Revision as of 08:08, 5 April 2010

User Friendly Categories: Organizing The Entire Wiki

So, everyone's probably noticed by now that while there's a ton of stuff in the Wiki categories, there are also a massive superton of pages that don't connect anywhere else, or only connect back to the main pages of the person who made them. This is sad, because it makes many ideas difficult (or VERY difficult) to find.

Therefore, I propose a solution - category-sorting by those of us who would like to see newcomers able to just browse the wiki by concept.

Before everyone's jaw hits the floor, I will make a couple notes about what the goal of this project isn't:

• It is not about imposing format standards on existing or new pages - that's pretty much a lost cause, and anyway no one could really agree on what those standards would be.
• It's not about minor edits or top-level pages. Again, not much can be done about those issues.

It is entirely about making sure that everything that gets made is linked to next to its related material, instead of being hidden in the middle of nowhere.

My proposal is as follows: Everyone who is interested, claim a letter of the alphabet. Go through the list of users with that name, and check their material against link locations; add material to locations as makes sense. When you finish a letter completely, grab another one and move on. Use this page to keep track of who is working on what; if every letter gets finished, we can go in and work backwards to help people who haven't finished a letter yet.

While doing this, we can try encouraging new posters to do the same, and possibly keep an eye on new posting in letters that are 'finished' in order to keep this going as long as possible.

The UserPages section lists 681 users with material online. With that in mind, 25 people each doing one user a day could finish this in a month. One person (ie me) doing one user a day could finish in two years.

Difficult? Yes. Nearly-impossible? Maybe. Quixotic? Definately a possibility. But I'd love to give it a try, because I have a lot of trouble finding things sometimes, and I would like to make life easier for people.

Who's with me?

List Of Organizers, And Letters They Are Working On

FrivYeti - A C N O P Q R T

Wordman - B D F G I J K L M S U V X Y Z

TheHoverpope - E H

Heru - S

EwindaleMoss - W

How YOU can help

You can help this effort in two ways:

1. Check the Orphans page for your own work. If you see some of your pages there, start from WikiContent and find a page where your page could be appropriately linked.
2. For the brave, grab a letter and add yourself to the list of organizers, above.

Lessons Learned

• There are a lot more orphans than expected. I guess after four years, this shouldn't be surprising.
• Many people don't know about redirects. I'd guess about a third to a half of the orphans were actually just "renamed" pages, where the old name is just abandoned, rather than redirected to the new name. I'm attempting to rectify this by throwing links to HowDoIDoThat all over the place, but that probably won't have much effect.
• Many people don't know how to make links that are named something other than the page, yet went through extraordinary lengths to find out how to build these bizarre external linking mechanisms. Most of these links also used the old URL, so no longer point to valid web pages.
• A large number of the people who use the wiki to store their characters don't use the Characters pages. Fixing these orphaned characters (and there are a lot of them, I'd say at least a fifth of the orphans) has been the most tedious aspect of this process.
• There doesn't seem to be a "general approach" for the basic layout of WikiContent pages, and consequently, most of the contents pages just do their own thing, even pages i the same "family". I have recommended a slightly more coherent approach, based on how many of the pages already work.

Commentary

I'm confused -- what about this idea wasn't already addressed with WikiContent? Also, I know you said you're not talking about format standards, but FormatStandards requests users to link to the relevant WikiContent page as well as to their UserPage.
~ Shataina

Ah, possibly unclear by me. Yes, linking is requested. But a lot of people don't, either because they forget, or get absent-minded, or so on. Basically, this is a project to go into WikiContent and compare it to UserPages, then fill in every spot that people haven't put there. That way, it will actually do what it should do, for the people who aren't doing it themselves. - FrivYeti

Sounds good. I tend to think that people who don't organize their own content aren't worth my effort, but I guess you're a better person than me :grin:. Kudos, and godspeed.
~ Shataina

Can you give us an example of what you are talking about? I think I see what you are getting at, but am not sure. - Wordman

Ok, I was just talking to Friv, and what he wrote is perhaps unclear on what he meant. What we are interested in is not a wholesale reorganisation scheme or anything like it - just sorting through the userpages, and adding links to the content indexes where there isn't one. Because as is, TheHoverpopeUserFriendlyCategories/Lightninginthemountains style is not listed on MartialArts, and so cannot be found by anyone who doesn't know to look through my charms, and the same is true for a large portion of the content on the wiki - regrettably, I am not the only one who hasn't been assiduously adding links to the content indexes. That's the goal of this project - adding material that is only on individual UserPages to WikiContent. - TheHoverpope

I may be able to accelerate this process. As I mentioned in my Wordman/CommissionEngine, I have ASCII versions of all these pages that I update daily. I also have some Perl that can do various things with it. It should be possible to build a link network and figure out which pages are only pointed to by a single page (or only by user pages, or something similar). - Wordman

That would make things much faster, in the sense that they might actually be accomplishable. Thanks! - FrivYeti

I might sporadically participate but I have not yet eliminated all vestiges of a social life, so no promises. And needless to say quality is not a sure thing when I'm involved. - Han'ya

I have added a page called Orphans, which will eventually hold the automatically generated list I mentioned above. I think the rules on that page are sufficient to identify orphans, but am not sure. Also, don't hold your breath waiting for me. Please proceed, as I'm not sure when this will be done. - Wordman

OK. A first pass at finding Orphans is done as best as I (and the Graph module from CPAN) could manage. I can't post the full results, because they are longer than the wiki can handle in a single page. So, at the moment I only go to letter C. I also found a bunch of MissingPages (also can't fit the whole list), so have added a page to hold those, too, with links to the pages that link to them. If you try to "un-orphan" a page and find yourself thinking "wait, there are already correct links to this page", it is probable that there are bugs in my software. Please mention them here. Also, if you are interested in the output of my code, I made a web page that lists the "shortest path" found from WikiContent to all other pages (if such a path exists). Looking at this page may reveal other bugs as well. Note also that just because a page isn't on the orphans page, that doesn't mean it has correct links leading to it. Some of the paths on that web page are a bit bizarre. I will try to update the orphans page once a week while there is interest in it. One last note: in some cases you'll find user pages in the list of orphans. If so, it probably means that the user isn't listed in UserPages. Please add them there and the next run of my code will eliminate them as orphans. - Wordman

I've started on some of the B's. It appears that you can use a soft touch in many cases to clear a bunch of orphans at once. For example, the first twenty or so B links are all cleared up by adding a single line to BerserkSeraphUserFriendlyCategories/AbilityLunars to the main Lunars page. Some others need a bit more individual attention. Some are also just crap, and I'm not sure what to do with them (as an example: Bat-Shit Insane As The Vice-President's Wife). Also, found and fixed some more bugs in my code. - Wordman

Very nice - but my god, are there a lot of missing links. We're going to beef up the WikiContent significantly by the time this is through. ;) I'm currently hard at work on the As, and will start up the Cs when (if) those get finished. - FrivYeti

Friv seems to be cutting the A list down quite a bit. I'm slowly clunking down the B list. As these lists get shorter, Orphans can probably start to hold an additional letter. Which new letter should be added? I started just in alphabetical order, but are there any letters that people would like to see before others? Oh, also... my code has a bug I haven't been able to track down yet: links with spaces or ' marks seem to always get marked as orphans, even when they are not. Working on it. - Wordman

I think we should probably just push ahead alphabetically; it's mostly just you and I working on this, near as I can tell. You could toss up the Es, and we can make Hoverpope put his money where his mouth is. ;) I was wondering about that spacing thing; I thought the mistake was on my end. Now I know better (half the battle, etc.) - FrivYeti

• whew* Alright, I have finished the As, and am going to start in on the Cs at some point in the near future. Good times, good times. ;) - FrivYeti, who is totally kicking himself for starting this project but is bloody well going to finish it.

And I, with no sound of trumpets, enter the fray. Thirty-something Es done; a mere scratch, I'm afraid. But a good way to spend a class. - TheHoverpope

Woo hoo! One note: if you find duplicate pages, or pages that obviously are supposed to redirect to another page, please change them to use the real #REDIRECT mechanism. Looks like a lot of authors don't know about this, so redirect in all sorts of squirrelly ways. My code will now pick up redirects properly. Redirect pages are automatically not considered orphans, even if no one links to them. This should cut down on the number of pages that need to be listed in the IntentionalOrphans page. Oh, also, I think I solved the problem with links with spaces. - Wordman

With Friv's reworking of the Artifacts pages, a bunch of orphans appeared with "path to long" problems. Looking at these, I realized there was a conceptual problem in my code. I wanted to allow five links before declaring a page an orphan, but I was really only allowing four. This is now fixed. - Wordman

Possible bug in the code - do you know why those links are showing up in the Orphans page that don't actually connect to existing pages? I can't figure out what to do with them. - FrivYeti

The ones I have seen boil down to case problems. I run my code on a file system (HFS+) that is case-preserving, but -insensitive, so the files "DooM.txt" and "DoOM.txt" are the same file, even though DoOM is a link and DooM isn't. I have to hand fix these in my code. If the link is non-existent, there will still be a link with slightly different case that is an orphan. Usually you have to use searching to find these (which is case-insensitive). Or, just ignore them and I'll take care of them. - Wordman

In connecting many of these orphans, I've noticed that stranded pages from the "A Dance of Angels" campaign keep coming up over and over. It appears that this campaign was moved off site, deliberately detached from pages and intended to be deleted; however, this wasn't really completed and there seems to be some confusion over what it's state should really be. There are a lot of these pages. I suggest that one of the following be done:

1. Embrace the "delete them all" idea and replace the contents of all of these stranded pages with a redirect to the main page for the campaign, which would contain instructions to where the campaign is now hosted, and an explanation of what was done to the wiki version. This is sort of a "scorched earth" idea, but would settle the question.
2. Edit the base campaign page to contain all these stranded links in a big list.

I'm willing to do the work for either, and don't care which. If someone involved in the campaign could offer an opinion, I'd appreciate it.

- Wordman

Delete them. DELETE THEM ALL! *Mwahahahahas*

And thanks. We had no idea it'd cause problems when we moved it. ~ GoldenCat

Hmm. Looking at the off-site page, some of it contains links that point back into this wiki. Thus, the "delete them all" isn't right, because some of the pages still exist here. Is there some kind of way to tell which is a page that can really be deleted vs. one that really can't? If not, I suggest option #2. - Wordman

I think the Es are done, so I've claimed the Hs, although I can't guarantee speed in the upcoming few bits. - TheHoverpope.

Added a "How YOU can help" section above. If people cleaned up their own stuff, that would be a big help. - Wordman

I want to make note that I check on the Orphan page relatively frequently to ever see if my name is on them. When it is I then take the time to make sure the page is indexed appropriately. That said, while I don't have the time to actually work on this project I do want to say that I veyr much appreciate what you are doing and thing that is an awesome thing. So yeah, :) - Heru

Ran into a weird problem - the page seems to have stopped removing redirects from the Orphans; a number of redirected Cs are still showing up as orphaned pages. Is this a code bug, or a redirect bug? - FrivYeti

Examples? - Wordman

Everything starting at CaptainPenguinUserFriendlyCategories/TheSevenUnspeakableRonin and leading down to Charms/HarenAbyssalSurvivial were completed a few weeks back, and haven't registered as gone yet. - FrivYeti

I'm still here too. Just stuck in the middle of a hellish week of moving and work. I expect I'll be back on the H trail early next week for the most part. - Thehoverpope

I think I fixed the redirect problem. The issue occurred with pages that had the same "leaf" name (e.g. CharmsUserFriendlyCategories/AbbyssalCraft and AbyssalCraft), where my code thought that such pages were redirecting to themselves and so ignoring them. - Wordman

I've run into a number of pages where the user page is entirely a campaign page. This kind of hoses the "orphan detection", even though there really isn't anything wrong with using your home page like this. The pages like this that I have seen have not been updated since 2005 or so. While I'm tempted just to build campaign pages in such cases, this is a bit presumptuous. What I'm thinking may be a better idea is to link to such pages in the Campaigns page, just like any other, and then tell my code that any user pages with links on the Campaigns page should be considered valid in a path from WikiContent. The problem is that I can't tell which of these pages work like this vs. other stray user page links, which is bad. So, I may end up just hard-coding some stuff into my code that exempts certain user pages from being ignored. This is less satisfying, because then anyone else who wants to do this has to let me know what user to exempt. Anyone have any better ideas? - Wordman

Be presumptuous. If it bothers someone and they actually log on and find out, they can change it however they want. - Capric

I dunno. Violates the Confucian golden rule ("don't to to others what you wouldn't want done to you"), so seems a little smarmy to me. - Wordman

I guess that's more what I meant--if you came in and dicked with my user page to make it fit in with an organizational scheme for the entire wiki, it would bother me not at all ~Capric

Ever found something you were better off not knowing? I found an evil bug in my code that caused pages that were only linked to by a user page and contained only links to other user pages to be ignored. So, for example Kahavi/Csilla was listed as an orphan (because it links to VirtueFlawsUserFriendlyCategories/Conviction), but Kahavi/Adaeze wasn't, even though they were both only linked to by the author's page. This bug turned out to have an easy fix, but results in many more orphans being uncovered. For example, 64 more links show up in the "A" section alone. Part of me wants to just forget I ever found this bug. Part of me wants to do the honest thing, even though it is sort of demoralizing. (Conviction check.) What do people think? - Wordman

OK. Since no one else cares, I'm biting the bullet and posted all the new orphans exposed by this bug fix. After looking at the "new" orphans, I realized that a decent percentage of them were actually way more interesting than the other orphans I've seen so far (which, lets face it, are all duplicates, characters or campaign pages). - Wordman

So one could say it turned out reasonably well. That's nice. - Han'ya

Also, I'm going to do some work on this, but it will be very minor stuff and very slow going, so I won't claim a specific letter (If I did I'd ensure that nothing will ever get done on it.) - Han'ya

One of the biggest pains in the ass in doing this is the large number of orphaned character pages. It occurred to me that it should be possible to run through the list of orphans with code and make a pretty good guess if the page is character and, if so, what caste and edition it is. I have done this, and updated the Characters pages with the results. There are still about two dozen or so that my code identified as characters, but couldn't figure out the type, edition or caste. - Wordman

Wow, kickass. I don't know how you program things, but I'm impressed. I am still working on this, incidentally, just very slowly; my free time dropped precipitously this season. ;) - FrivYeti

The character detection code is actually kind of primitive. I look for a bunch of keywords in the text of a page, and if they are all found, I assume it is a character. In this case, any page with "Willpower", "Essence", and all four virtues is considered a character (this naturally generates false positives). I assume a character page is second edition, unless it contains "Endurance", "Brawl", "Initiative" or "Base init". There is similar brute force stuff to figure out out the character type and caste, which I had to fine tune. The trick was really trying to make sure that false positives or ambiguous results got discovered and reported as such. For example, if a page contained "Serentity" and "Fire", and lacked some basic syntax structures (e.g. "Aspect: "), the code can't figure the character types and punts. I managed to get things to where there were only a couple dozen of these ambiguous cases. - Wordman

As I said this is a brilliant project and I have decided to participate. I will be working on the Letter S. Question, what do I do with pages whose authors say it should be deleted. And for the record, all I have to do is look at a page and then place a link to that page in one of the main WikiContent sections like characters, campaigns or charms? I just want to make sure that I am doing this right. -Heru

In almost every case I've seen a page where the author says "this page should be deleted", what was really happening is that they were redirecting the page, but didn't know how redirecting worked. So, when you see pages like this, see if you can figure out where they moved it and add the #REDIRECT. If this isn't the case, then add a link to the page to DeletedPages. As for "doing it right", it seems like you are doing the right thing. Also, don't be afraid to put the same link on more than one page if it seems useful. - Wordman

Thanks for the response, its appreciated. Here is another question, do you have to reload the orphan page personally or does it do it ons its own. If it does it automatically after a certain period of time could you increase the rate. If you do it manually, never the mind, :). -Heru

I do it manually. - Wordman

Sorry for the disappearance but life just got a whole lot difficult for me, with a reduction of time being the result. So until things change I won't be able to work on this project. Sorry.-Heru

Sorry to hear it, but these things happen. I can't remember the last time TheHoverpope worked on the project, and I'm sort of doing it intermittently. - FrivYeti

Also, welcome aboard, EwindaleMoss! Glad to see you joining our mildly quixotic quest. ;) And now, to meander around the Ts for a bit while I try to work on my upcoming game. - FrivYeti

Not a problem. Glad to offer my support. I thought I may as well (for obsessive compulsive and other disorder sake) help with the organization attempts. It's fairly zen compared to my ongoing beta test of a game. Eventually its rules will all be posted and 1E will have been written as if only Core had ever existed (long drawn out explanation behind that)... So compared to that monstrous beast of a project, this is a refreshing frolic in flowers. :D - EwindaleMoss

So, we have theoretically run the alphabet, so I'm looking for what to do next. I guess since Heru bowed out, I should probably tackle the S's, yes? -- Wordman

Probably, yeah. I'm going to pick up the Hs from TheHoverpope once I get through the Ts. - FrivYeti

Any other letters or part lists or whatnot I can take to help out? Don't want to step on anyone's toes in particular. - EwindaleMoss

How about you work with the As through Gs that have popped up while we've been working on the late letters? Many thanks! - FrivYeti

So, started on December 13, 2007, completed September 18, 2008. - Wordman

Woot! Wordman, you deserve major props for practically carrying this project. :D Many thanks, man. - FrivYeti

I was going to say the same about you. I would have totally dropped the ball if I hadn't seen you handling more entries several times. - Wordman

Major 'props' to you guys who did this for the ExWikiCommunity. Sorry I was a hassle at the last few moments and I'll try to be a bit more bold about posting my content. Hats off to you ladies and gentlemen for your brave works here.