Defining quality

Starting point

Woodwalker considered how quality might be defined or measured. The starting point was his essay " On Quality" covering assessment of quality, edits, and editors, and the concept of "content erosion":

The Dutch Writing Contest criteria list seven factors for assessing quality (link): Lede ("lead"), article structure, page layout, article content, style, verifiability, findability.

Woodwalker proposes a more general set of factors:

1. Content requirements

   • encyclopedicity, verifiability, neutrality, balance

2. Reader requirements

   • interesting, completeness, depth/specialism level/reader level, relevance of content to subject matter

3. Requirements of form

   • correct (or good) language, encyclopaedic tone, text style and readability, clear and easy article structure, clear and easy layout,

4. Broader project requirements

   • findability of topics, and coverage , balance, consistency, and completeness of topics in general (across the project).

He also divides edits and editors by their impact on quality:

• Edits that only add quality, those that only remove it, those that are neutral (eg US English to UK English), and those that add quality in some ways but remove it in others.

• Editors who add quality by adding new content; editors who add quality by maintaining (preventing degradation, and making minor improvements to existing content - a continuum exists), editors who remove quality, and "problematic users" who add quality some ways or work hard generally, but overall come to be seen as harmful or negative overall in quality terms.

He then describes the (important) concept of content erosion:

"If we assume the rate of change and the percentages of all four edit types to stay constant, the rates of quality increase and quality decrease are constant too. This means any page in the project is subject to a slow decay in quality, which I call wiki-erosion. Quality is guarded by the community though. The ability to revert destructive edits of all kinds is thus related to the amount of knowledge in the community. This means destruction can only go as far as a certain quality level. If the community is larger, that level will be higher, if it is smaller, the level will be lower. Thus, in the long term, quality of any sort will stand a larger risk of being destroyed at small projects, even though the wiki-erosion rate is much smaller."

Woodwalker feels current analysis of quality is lacking, but perhaps by analyzing these components, the statistical team can find better ways to assess quality. It may also lead to practical ways to improve specific quality factors rather than unhelpful generalized suggestions, that projects can tailor to their culture.

General discussion of quality measurement

Piotrus argued metrics are overrated. Virtually all scholarly publications agree quality is high and rising, and internal categorizations (Featured/good/assessed/stub etc) are adequate for the rest, for now. He drew attention to improvements of this scheme, including articles outside WikiProjects and hence not rated, and low activity Wikiprojects without adequate discussions or members to support rating. He sees more (and more active) Wikiprojects as a core quality tool.

FT2 states three kinds of metrics are useful and attainable: 1/ crude metrics such as computer assessments based on tagging, cite to word ratios etc (and some calculation based on these) which can be used to crudely identify major issues and assess articles up to a simple baseline; 2/ article progression and conversion metrics based upon article standing (new -> baseline quality -> good -> featured) and time taken between these, and article stability; 3/ assessments based on user and reader feedback ("rate this article").

Baseline quality and the "low fruit"

FT2 observed that "the 'low fruit' is appealing [to focus upon] -- metrics relating to substandard articles that don't meet a agreed baseline for quality, or measuring how long they take -- because there's lots of them, they make a big impression, they are easy to identify and quantify the issues, and they are easy to fix. Maybe for now, we should recommend focusing on that."

Incentivizing and promoting quality

FT2 explored this topic in several thread posts. Comments included:

• Adding a baseline quality and crude automated ratings, would "capture basic issues that are a concern and could flag them to the author and the community. If we take care of the worst articles then over time the average will improve. Nobody is more motivated to work on an article than those who have already edited it, so they may be interested in a simple "score" plus information why it's low."

• Giving an editor even a crude rating on an article ("This article is rated as 5.4, click here to see what's needed to improve it") will incentivize and stretch users ("We need quality things to be pushed, incentivized, fun, enjoyable, and desirable to go for.... Suggesting incremental ways to do better... [even] a crude automated evaluation of an article's weaknesses [can help]"). A suggested wizard/popup was proposed, to embody this approach [1].

• Other major organizations (Macdonalds, Coke, Nike, etc) promote by "mak[ing] it simple, easy, intuitive -- and plaster things (tastefully) wherever they can that channel people towards the ways that help that organization. We're no different in a way. We want readers to be nudged to check out possible corrections and facts to cite, and we want to make that really easy and obvious... we want editors who write an article to have it made really simple and attractive to revisit it to get it one more notch up a crude quality number... and so on."

• AndyZ's assessment tool, one of several quality rating tools identified by Piotrus, "has real potential if it could prioritize the key issues and suggest them, and if it was made simple with an integrated interface thing that was "once click away" on each page. Every last article that's not GA/FA could have a little tasteful slow-blink icon saying "Improvements we want on this article", listing 2 or 3 selected improvements the article needed and a "Let's fix it!" button... That would get the wider public's involvement."

Piotrus felt this sounded good, advocated simplicity, and asked whether Andy's tool (or one like it) should be recommended for future development.

File uploads

There was a short subthread about file uploads. Bhneihouse commented "the fact that anyone on this team said they "hope xxx works" is a huge statement about reliability factors on Wikipedia. That is a quality statement right there".

Branding

Bhneihouse discussed the idea of "branding" in the broad sense of purpose, mission or "being-ness" (as opposed to just "visual identity"), and how the brand (roughly, what Wikipedia is) drives what Wikipedia does:

"What we are talking about here is really about what Wikipedia is, and thus how it does what it does... we cannot have a conversation about quality without starting the conversation with brand... Brand is intangible but is expressed through that which is tangible, whether it be a mark/logo, or the way customer service responds to a customer or the way that a user experiences Wikipedia".

She also added that "a consistent framework would serve Wikipedia's goals" and remove guesswork. She stated

"[W]henever Wikipedia allows that which is not consistent with its brand to exist as Wikipedia, it dilutes the brand... Wikipedia is about accurate knowledge. Standards in keeping with Wikipedia's core ideals and values keep Wikipedia being Wikipedia".

User feedback mechanisms

Woodwalker asked about obtaining reader or user feedback, and suggested neutral, balanced, complete, well cited as the four axes. FT2 noted Flagged revisions has such a tool already, and suggested balanced, sources, coverage (completeness), up to date" as axes, adding that capturing the reader's knowledge level on the topic (casual editor | knowledgable | very knowledgable | formally qualified) would be extremely valuable (shows the rating that different levels of reader give the article, and reader existing knowledge levels).

There was some agreement (FT2, Woodwalker) that "rate this" popups would be seen as "spammy" compared to a toolbar, and agreement about the value of obtaining the reader's knowledge level in terms of "actual audience".

Article rating

Woodwalker states (later) that assessment may not help smaller projects lacking the skills to rate quality, and that rating systems vary between projects, stating in summary, "having more editors is important, but let's be fair: for quality we especially love to have more 'quality' editors". Bhneihouse stated that she felt the quality framework needed consistency and buy-in across projects, not a "pick and choose" structure. Woodwalker felt WMF's capabilities to force change on all projects are very limited (eg BLP) (see below), and that offering options if they wanted to improve this or that aspect was more respectful and likely to obtain higher uptake. He noted that "all projects should in the end have the same quality goals, but they may be in different project phases and therefore need different approaches".

WMF's abilities to force change and working in the "real world"

This was a significant point of taskforce philosophy.

Woodwalker felt WMF has only a very limited ability to force change on communities (see: BLP). FT2 agreed, expressing concern that some ideas would be "dead in the water" in any practical sense, and only certain things can be effectively shaped or altered.

Woodwalker commented he was "philosophizing for perfect world", but that it wasn't our problem if projects did (or were able to do in practice) with it "isn't up to us".

FT2 argued the taskforce's aim was to maximize effect, which meant designing for the real world of the projects, not a perfect world ("the criteria is 'what's most likely to deliver given these things' "). Anticipating or considering possible issues and practicalities that could affect likely productivity was part of the job:

"[I]t has to be a path that's got a good chance of people following it, otherwise it's pointless. So our optimum result might be categorized somewhat openly as the best path that has a good chance of enough people following it to make the necessary difference. Human nature, variety of views, and inertia, will ultimately limit what we can achieve in any given "bite" at the quality cherry. Best to respect there are limits on the achievable (although not giving in lightly), and see what's the most we can progress quality for this time."

Decision making

Woodwalker stated that paralysis in decision making within projects meant that new ways needed to be found, rather than just trying to modify old ways, but felt that ultimately frustration with 'red tape' would eventually force progress.

Sjc commented that "It's not just the amount and complexity of 'red tape', it is also the fact that 'red tape' is a tool which can be bent to purpose by all and sundry. In fact, a creative editor can make the red tape mean exactly what he or she wants it to mean. Red tape is in this respect more of a liability than an asset, where edit wars can be won by the editor most adept at bending the laws of reality to their own intent than others". He (Sjc) advocated removal of policy ("appears to be a fundamental enemy of content and quality of content") in favour of unhindered good faith editing.

(Sjc's essay on this and related points)

Summing up

Woodwalker closed the thread to date, commenting that:

1. We've found that quality isn't easy to measure by simple metrics, perhaps it's impossible unless we would have some form of feedback from the reader.

   Piotrus states this isn't impossible. Rather, there are several different metrics to doing that, and what may be impossible is selecting the "best one".

   Woodwalker disagreed, noting that while subjective feedback helps, there is no clear way to quantize key goals like "completeness", "balance" or "structure"; even reverting can be for both good and bad reasons.

2. We've come across some nice ideas how this feedback can be obtained. Important also is to know who gives the feedback (expert/interested person/school kid?).
3. Piotrus suggests Wikiprojects can play a part and we need more; Bhneihouse suggests Wikipedia can only become a quality brand when there is a consistent basic level of quality across Wikimedia projects (I assume this is not just about Wikipedia); FT2 thinks our recommendations should be in the form of realistic suggestions likely to make the biggest positive effect on quality as they perpetuate, allowing for where the communities are today [as amended]. I suggest this feedback thing could become our second practical recommendation (after 1. creating more manuals/wizards).

I'll follow Piotrus' advice and start a new topic for this. We are (I pressume) still in the warming-up phase of this task force, so I don't mind having conversations to get to know each other and do some general brainstorming. We've come to some interesting problems and even discussed potential solutions. Yet I think we have to get to answering the questions Philippe gave us sooner or later, and imho that is not well possible without first defining what quality is. I have begun to write an essay on that here (I thought I'd better not do it here to prevent confusion and to be able to think clearly). It's not finished yet but I'd like to share it with you and see your comments and suggestions.

To summarize it, I split the rather vague word 'quality' into 17 (!) things which I call 'quality factors'. All of these things have been used in the past to define quality. Perhaps there are more. It is obvious that all users have their own perception of which factors are the most important (I have my own preferences too). Projects make different choices between these factors too, judging from differences in guidelines. I began writing this essay with two immediate goals:

1. I want to show that current statistical research is lacking! Quality isn't as easily measured as quantity, because quality is less easy to define. By splitting it into 17 quantities I try to show what should be measured, if the statistical team can find ways.
2. It can give our task force a better 'grip' on the problem. The fact that different projects have different preferences can be seen as an enrichment, and imho Wikimedia shouldn't force them to make solid choices (even if they could). If we can advise on how to increase independent 'quality factors' instead of giving vague general advises on how to raise quality, our advice will be more practical. What's more, every project can pick the thing they like in the relative dose they like.

I think there are two main kinds of metrics that would be useful to editors and to improving content, one being crude and probably machine-manageable.

On a crude front, we can attach logic to the key tags:

(Article NPOV = X. Inline cite missing = Y. several inline cites missing = Z. Word to reference ratio over 20 = P. Word to reference ratio over 100 = Q. Current edit warring in the past week = R. Plus evaluating a single rating from these.)

It wouldn't capture "high quality", but we'd capture basis issues that are a concern and could flag them to the author and the community. If we take care of the worst articles then over time the average will improve. Nobody is more motivated to work on an article than those who have already edited it, so they may be interested in a simple "score" plus information why it's low.

More sophisticated metrics are hard. I'd be looking for projects to define a few standards on articles (newly created - baseline acceptable - GA - FA), and then measure quality in terms of average time taken for new and existing articles to reach the next quality level and conversion rates ("what proportion and how long it takes"), user feedback, and stability.

That said, the "low fruit" is appealing -- metrics relating to substandard articles that don't meet a agreed baseline for quality, or measuring how long they take -- because there's lots of them, they make a big impression, they are easy to identify and quantify the issues, and they are easy to fix. Maybe for now, we should recommend focusing on that.

Defining quality is important, but I'll be the devil's advocate here and argue why it is not that important:

• virtually all scholarly publications at WP:ACST, no matter how do they define quality, agree that Wikipedia's quality is high and raising
• WP:ASSESSMENT has a widely accepted metric for determining quality (FA/A/GA/ down to STUB). It seems to be working well; I'd suggest focusing on recommenations to improve it which are as follows:
  • some articles are outside the scope of any existing WikiProject, and as such are not rated
  • most projects are not active enough to support discussions/votes for A-class assessment (not to mention B-class), WP:MILHIST is a notable exceptions

As such, my solution to determining quality is simple: we need more wikiprojects, and we need the existing ones to be more active. For that, we need more editors - which takes us back to the problem of their declining numbers (and if I sound like a broken record by stressing this over and over, please let me know :D). --Piotrus 22:11, 26 November 2009 (UTC)

Agree defining quality isn't important (for editors). It is useful at a WMF level, in promotion and when the media ask, to be able to point to specific evidenced data. That's important at that level.

What does help is to be able to give an editor a crude rating on an article they wrote or are reading, and say something like "This article is rated as 5.4, click here to see what's needed to improve it". Those kind of metrics are there to incentivize, and stretch users, to make them think "hey, I can fix that" or "yeah, I want my article to be a bit better and I wonder what the computer says is holding it down". I want to consider that maybe a user should see a box that says:

Congratulations! The article you just wrote is rated at 4.1. The major issues identified at present are 1) it's missing citations for a large block of text starting "Lorem Ipsum...", and 2) it's had its neutrality questioned by a user who has explained their concerns here. Fixing these would raise its quality to 6.2. Reaching 9 or more would allow you to seek peer review by other users, and official "Good Article" recognition!

We need quality things to be pushed, incentivized, fun, enjoyable, and desirable to go for. Not just "if you ever heard of it, it kinda... exists." Suggesting incremental ways to do better is Good, and in that sense a crude automated evaluation of an article's weaknesses is also Good.

I think we have tools to do both. For readers, it is the WikiProject assessment rating. There's a lovely new gadget in preferences: "Display an assessment of an article's quality as part of the page header for each article" (more info). It would be easy to enable this by default (it is currently turned off). For editors, unautomated feedback on article is hard, even with WikiProject specific new article annoucements (courtesy of Alex's bot), simply because we don't have enough active people reading the bot generated lists (in WP:POLAND I am the only one doing that). But there is a useful automated tool that could be developed to provide a feedback for editors (AndyZ automated peer reviewer). I find it quite useful; unfortunately it has been abandoned for a year or so. A recommendation for it to be adopted by developers, turned on for new editors (who could opt out if they think its too annoying) could be a good idea (or even better: the scripts adds an option for automated review as a tab to each article edited - we could just implement it by default, and stress its functionality in tutorials for new editors). --Piotrus 22:46, 26 November 2009 (UTC)

You know how Macdonalds, Coke, Nike, and so on, do promotion? They make it simple, easy, intuitive -- and plaster things (tastefully) wherever they can that channel people towards the ways that help that organization.

We're no different in a way. We want readers to be nudged to check out possible corrections and facts to cite, and we want to make that really easy and obvious... we want editors who write an article to have it made really simple and attractive to revisit it to get it one more notch up a crude quality number... and so on. The latter tool you link (AndyZ's) has real potential if it could prioritize the key issues and suggest them, and if it was made simple with an integrated interface thing that was "once click away" on each page. Every last article that's not GA/FA could have a little tasteful slow-blink icon saying "Improvements we want on this article", listing 2 or 3 selected improvements the article needed and a "Let's fix it!" button.

Do it that way, not as an editor gadget. That would get the wider public's involvement!

Sounds good - shall we endorse AndyZ tool as one of our recommendation for future development, per your description above? And KISS all the way, of course (incidentally, this is why I like Wizard 1.0 interface better compared to information overload in Wizard 2.0). --Piotrus 03:29, 27 November 2009 (UTC)

In some ways it seems that you are playing off of "branding," which is kind of where I have been going with my comments about framework and consistency. I am going to upload a branding document that may help us with consistency. It harkens back to "what is Wikipedia growing up to be?" When someone understands the brand, all else falls into place.

Ok, Help! How do I post a graphic here?

Upload a file, then include as usual ([[media:Filename.jpg|right|thumb|200px|Description]] should do it). if you can't upload here then ask for help from whoever runs this wiki, or upload it to commons and hope through-links work :)

thanks but that assumes I have uploaded files in the past and know how to do it. I have never uploaded images to Wikipedia.

the fact that anyone on this team said they "hope xxx works" is a huge statement about reliability factors on Wikipedia. that is a quality statement right there.

thanks FT2.

this interface should allow for us to easily share information, as noted below in the PAGE thread, this isnt working for me.

Sorry, figured anyone here would probably be a long term user and likely have that experience. Quick guide (it's easy enough):

1. There's an option, "Upload file", on the sidebar to the left. Click it.
2. Choose the file to upload, and the name to give it (often the same, but you can change it). Give a brief description -- on the major reference wiki's there's a lot to go here, but for this one a brief note will do.
3. Click the button to upload it.
4. The image will appear once uploaded, on a "File:" page, eg "File:Mypicture.jpg".
5. Include it as follows (simple usage):

• To include full size: [[File:Mypicture.jpg]] (As simple as that)
• To inclue it as a small "thumbnail" with a caption, that can be clicked for the full image: [[File:Mypicture.jpg|thumb|position|size|This is my picture.]]
  where position is left/center/right, size is usually in pixels (enter as 200px, 400px, etc) and the rest is a caption.

Hope that helps.

Branding, yes. If you think about the essence of (one aspect of) branding, it's to make life simple for users, who see conceptually similar thuings that look similar, are familiar and "known", etc. A number of branding truisms also apply to the work we do, in the sense we want to reach out and attract people and then guide them into best ways (if they choose). So yes, a lot in common. Not via "visual image" or graphical design, but philosophically.

Branding actually goes a lot further than that. While branding may simplify the user experience as far as choices, branding is actually how a thing exists in the mind of the user. Brand is NOT a physical image. The graphic I wished to upload that I sent you actually explains the complexity of brand. What we are talking about here is really about what Wikipedia is, and thus how it does what it does, because how Wikipedia does what it does is a reflection of the Wikipedia brand.

In truth, we cannot have a conversation about quality without starting the conversation with brand.

Because "Branding" is a word with strong connotations/meanings, it can confuse a discussion. Would you be okay using terms such as expectation and perception, which I think cover what you're talking about, rather than branding which tends to imply the visual design identity.

Actually I would prefer other people understand what the word "brand" really means rather than use expectation/perception because brand is not encompassed by either/both concepts. I am not talking about expectation and perception, I am talking about what something IS and how the IS drives the what something DOES. In this case, Wikipedia is the something. An IS is not necessarily visual. Brand is intangible but is expressed through that which is tangible, whether it be a mark/logo, or the way customer service responds to a customer or the way that a user experiences Wikipedia (not necessarily tangible but could be.) I have uploaded the graphic, if you need me to explain it, please ask.

for some reason my message asking for help uploading a graphic didn't show up. Help. :)

It would be a fork with a similar concept. Used differently, coded differently, adaptable to different projects, highlighting and prioritizing and scoring issues, not just noting them, and feeding other processes that would live feed, notify them to users, allow editors to be notified when filtered types of edits were of interest on a page they viewed, etc. I wouldn't make this alone "a recommendation". I'd bundle the concept it's part of all together as one concept package (when we finalize our ideas) and say "this package of stuff that works together is recommendation #1".

But our final conclusions will probably be reshaped quite a lot between now and then.

But this is constructive! I get the feeling we are really getting somewhere. How about presenting the reader with an optional little multiple choice-question list, which can be toggled by clicking a new, extra tab? My quality factors can be used to make statements:

• The content of this article is neutral
• The content of this article is balanced
• The content of this article is complete
• The sources cited in this article are good
• etc

The reader can then choose between good/average/insufficient/bad/I don't know for every statement.

Flagged Revisions has such options already in its code. Possibly not too hard to extract the relevant code and use it "stand alone" for rating purposes only if desired (or by having the flagged rev's functions disabled) on wikis that wanted it. But yes.

One box to add: "I am <a casual editor | knowledgable | very knowledgable | formally qualified> on this topic", then we can see how it's rated based on respondent's self-claimed knowledge, and what its audience is like.

Also note an interface point: to most users, "rate this item" is so familiar from other sites, that a popup is likely to be ignored as spam and may even be seen as annoying. I'd do it how FR does it -- in a small bar on the interface itself:

I totally agree about the interface. No pop-ups please, just a small tab that opens this rating plus the mini-questionnaire will do.

Also agree with the box to add about the expertise of the reader. The 'quality of demand factors' are also great to have feedback on, for example:

• This article contained everything I expected to find, judging from its subject.

That's "coverage", isn't it?

Yes, I think "coverage" is synonym with "completeness" (no 2.2). Woodwalker 17:38, 27 November 2009 (UTC)

@Piotrus:

• We're trying to raise quality, looking back and applauding ourselves is nice but gets us nowhere.
• I wrote somewhere that the equivalent of WP:ASSESSMENT was deleted at wp-nl, much to my displeasure. We're just talking about the big projects then. I don't object to that but we should keep in mind that most projects are not that good in rating quality yet. As for the big projects, the rating systems between wp-en and wp-de are rather different, with different results too.

Having more editors is important, but let's be fair: for quality we especially love to have more 'quality editors'. We like to have more 'maintenance' and 'good' editors too, but these are second on our wish-list.

The only exception I have to your post, Woodwalker, is that I believe that the quality framework should (yes normative) be consistent across the various communities. I think if we create a framework that communities pick and choose from that the end result will be sorely lacking. In this quest for a broad reaching yet specific framework, we really need to take into account the needs of all communities. This way, the guidelines we craft will speak to all of the communities and get their buy-in.

@Bhneihouse: actually, I'm not against Wikimedia forcing projects to do this or that, if it increases quality. I just think Wikimedia's abilities are very limited. Some time ago now, Wikimedia pushed for a strict guideline about biographies for living people on all projects. I guess it hasn't been put into place yet in most projects. This illustrates how little real power Wikimedia seems to have.

What I meant was to tell projects: 'Look, if you want to increase your project completeness (one of the 17 quality factors I found), do this. If you rather want to increase your verifiability, you should instead do that.As is obvious from this example, larger projects have the luxury to go for depth, whereas small projects will probably like to focus on quantity. Size is just one reason why projects may have different priorities. Cultural, personal and other differences also play a role.

Some implementations may help increase a certain factor, yet be destructive for another factor. If, for example, the amount of non-encyclopaedic content is a priority, a project could be advised to use flagged revisions. This reduces the amount of IP-users.^* Less IP-users means less non-encyclopaedic content is added^* and the hardcore group of editors will have more time to focus on removing non-encyclopaedic content. On the other hand, another project may find that their priority is to raise their project balance, because they have little coverage of certain fields. They want to encourage more new users and can be advised not to use flagged revisions because it decreases the influx of newbies.^*

I agree that all projects should in the end have the same quality goals, but they may be in different project phases and therefore need different approaches.

^* I'm not sure this is true, it's just an example.

I do agree on Wikimedia (WMF)'s power to obtain change. What it can do is change the river-bed, so the water will "most likely" move in a "somewhat predictably" different manner ( :) ) I also agree on how projects differ, etc.

Where I don't think it works is that you're perhaps assuming decision making finesse beyond most project's capabilities in your main example, I fear. (I'm trying to imagine an enwiki debate where user consensus was sought on which position the project was in and which guidance (or any) to follow. "Herding cats" doesn't come into it!) Nice in theory, but sadly, dead in the water in any practical sense. Might work for a few projects, at most?

I do take the idea it's an example. But I think we can shape and effectively alter certain things only.

True, I know I'm philosophizing for a perfect world. However, the imperfectness of decision-making isn't really our problem, is it? We can give advice, what projects do with it (or rather: what they are practically able to do with it) isn't up to us.

Off-topic: the fact that large communities tend to be slow in decision-making surprises me. From a logical POV, only so many things can be said in a certain discussion. After that, there's either consensus or not. If there's no consensus a poll is held, if that's indecisive, a vote is held. In larger projects, there are more users able to accommodate/coordinate a poll or vote when the discussion gets to the point where people are repeating themselves, so decision-making should be easier, but it doesn't seem to work that way.

It's not our problem, but that's the same as the law of gravity not being our problem when designing a plane.

We're making recommendations; the recommendations are to maximize specific desirable effects; we need to make the recommendations based on a best analysis and understanding of the way reality will work on any given suggestion. Some will work better than a "perfect world" case, some worse. We're not designing for a perfect world, we're designing for the communities, human nature, encyclopedic issues, and public, that we're used to.

The criteria isn't "what might work best if the world wasn't like this", it's "what's most likely to deliver given these things". In that sense anticipating what can cause an idea to crash and burn, and what's workable or likely to be productive, is precisely what we should be considering.

good points, but I really do think that a consistent framework would serve Wikipedia's goals. That would take the guesswork out of "what stage" a project is in, how much content is up, etc. I do not agree with going for filling in the spaces and letting quality lag in the process. I believe that a standard should be consistent at all times. Otherwise, if we start a new language Wiki for example, for the Abkajeckfa language, there might be a lot of new pages with very little consistency and quality that do not mirror the overall brand of Wikipedia. People reading the pages may not understand that low quality is NOT Wikipedia. In truth, whenever Wikipedia allows that which is not consistent with its brand to exist as Wikipedia, it dilutes the brand. Kind of like IBM and laptops. There is a reason Lenovo is now not IBM. IBM truly isn't about laptops any more than Dell is really about consumer products. Wikipedia is about accurate knowledge. Standards in keeping with Wikipedia's core ideals and values keep Wikipedia being Wikipedia, no matter how it expands.

They're consistent. A standard framework can exist, that allows for human nature and cultural or wiki-community variations.

But so long as a wiki (eg Abkajeckfa) is poor quality, then for people who read that and inconveniently don't read enwiki or de/frwiki, that is Wikipedia, and we are poor quality. That said, inevitably the wider world will judge us by our bigger wikis alone.

this is the thing with brand, as you guys will see when someone posts my graphic (LOL) or not see, or as you may already know from your own experiences with various brands -- ONE customer experience can and will define the entire brand for any given customer/user. ONE bad wiki -- one incomplete, poorly written, unvetted wiki -- can and will define the Wikipedia brand for millions of users. As any brand specialist will tell you -- it is common knowledge that getting a customer back is SO MUCH HARDER that just keeping him or her in the first place.

that said, consistency is key. users need to know at all times what state a wiki is in, and they need to understand what the ultimate goal is. With the current structure for wikis, that is just not possible. Currently, wikis are sometimes really bad or incomplete or... yet they still carry the Wikipedia brand.

Originally, creating Wikipedia out of nothing took a lot of daring just to make it exist. But the same conditions of introducing a brand no longer exist. Now Wikipedia is an established brand and users see it as a valid information source. Going to the next level may mean changing the way the infrastructure works in order to stay TRUE to the Wikipedia brand. That may mean that inaccurate or misleading information may never be tolerated as a "public" or "finished" wiki. That may mean that we have to create an interim wiki for discussion and work (FT2's idea off list that we are still hashing out a bit in small group -- email me if you want to be included in the email) that eventually gets posted with a "done" (or close to done) status... there are many possibilities, but in the end, the brand that users experience as Wikipedia matters.

@FT2 (10:37, 27 November 2009): seems true, but that could bring us to an advice of the following sort:

'Well, actually we know how you can increase quality standards dramatically, but because of the way you're cooperating right now we assume you're not able to, so instead we'll tell you something that has little effect but certainly works'.

I think we're allowed to think big (I mean also: larger time-scale) and assume some problems in decision-making can be overcome in the future. Additionally, not all communities have become paralysed to the same degree as the en-wp community.

Because of the paralysis in decission-making, progress can only be made by creating new constructive ways (like our wizards), not by removing or changing old ways. This cannot continue eternally, people can only work with a limited quantity of 'red tape'. That's why I'm still optimistic that something will be done to make decission-making more effective in the future.

It's not just the amount and complexity of 'red tape', it is also the fact that 'red tape' is a tool which can be bent to purpose by all and sundry. In fact, a creative editor can make the red tape mean exactly what he or she wants it to mean. Red tape is in this respect more of a liability than an asset, where edit wars can be won by the editor most adept at bending the laws of reality to their own intent than others. I already have a fairly radical proposal up to the effect that we move away from what we now seem to have somehow ended up with, policy driven content, and back to what we once had and which made Wikipedia such an amenable and fascinating environment in which to work, one of content drive content. The paper which underpins the thinking for this is to be found here: Task force/Enhance community health and culture task force/Making Wikipedia a Happier Community. Policy appears to be a fundamental enemy of content and quality of content. Good faith, well researched edits by editors who care about their subjects results in ineffably better articles than anything cobbled together from a bunch of dubious web pages annotated with copious citations until your eyes blee with an overarching eye to policy compliance.

I like this part of the observation: that it's easier to suggest a new place to go or a new tool, than to suggest a change to existing wiki-cultural aproaches.

But I think the communities are looking for significant changes or findings here, so we're free to say "if you aren't working this way, try it, that's our conclusion" and provided it's viable they might.

However (and I think this is your point)... the more you tell people "kill that old approach and do this" instead of the softer and easier "start moving this direction", the more you risk an outright rejection by significant numbers.

In other words, it has to be a path that's got a good chance of people following it, otherwise it's pointless. So our optimum result might be categorized somewhat openly as the best path that has a good chance of enough people following it to make the necessary difference.

Human nature, variety of views, and inertia, will ultimately limit what we can achieve in any given "bite" at the quality cherry. Best to respect there are limits on the achievable (although not giving in lightly), and see what's the most we can progress quality for this time.

I'll try to summarize this thread so far:

1. We've found that quality isn't easy to measure by simple metrics, perhaps it's impossible unless we would have some form of feedback from the reader.
2. We've come across some nice ideas how this feedback can be obtained. Important also is to know who gives the feedback (expert/interested person/school kid?).

Besides, Piotrus suggested Wikiprojects can play a part and we need more; Bhneihouse suggested Wikipedia can only become a quality brand when there is a consistent basic level of quality across Wikimedia projects (I assume this is not just about Wikipedia); FT2 thinks our recommendations should be in a realistic form that has a chance of being accepted by communities (given the indecisive nature of discussions in the larger communities).

I suggest this feedback thing could become our second practical recommendation (after 1. creating more manuals/wizards).

...FT2 thinks our recommendations should be in the form of realistic suggestions likely to make the biggest positive effect on quality as they perpetuate, allowing for where the communities are today.

Regarding 1) I disagree with that measuring quality is impossible; rather, there are several different metrics to doing that, and what may be impossible is selecting the "best one".

It depends. I think having clear, subjective feedback potentially really helps, but this exact type of feedback hasn't been used in statistics or project models yet. In such a situation there is no way to measure quality factors like "article completeness", "balance", "structure", etc. The only way in which the statistics currently are able to measure quality is by looking if an edit got reverted and comparison with the editor's other contributions. This tells us very little since the revert can (as far as I see) have 17 valid direct reasons, be neutral to quality, or/and even have a negative direct effect on quality (in 17 possible ways).

Per Woodwalker. Metrics we could develop include:

1. Crudely measuring things like user tags, word to cite ratios, stability, and the like.
2. Reporting user feedback.
3. Setting standards for articles (newly created | baseline quality | good | featured) and measuring time taken and progression rates between these.

These aren't bad by themselves. But I'm not aware of any way to calculate useful metrics for genuine quality aside from those things.

I just ran a reader survey on five articles that I wrote or had a hand in editing. I found the results to be extremely enlightening and helpful in my work as an editor, and also as an important input to policy disputes in the project I am working on. You can see the results of the survey here.

I performed the survey by creating the survey form at www.surveymonkey.com, and attaching a link at the top of each article (see, for example, this revision of one of the articles.

As an editor, I would love a tool that I could use to develop a survey with article-specific questions, attach it to the end of an article, and analyze the responses.

In numerous other forums, I have pointed out that Wikipedia is an editor-centric, rather than reader-centric, institution. All the mechanisms and rules of behavior are designed to create cooperation of a community of editors. In this dynamic, the reader is most often shunted aside; to the extent that, when I proposed my survey, there were editors who clearly didn't want to know what their readers were thinking.

This is something that has to change if Wikipedia is to move forward, and that change will occur only when features of the editing environment support the change. That is why I think a tool like this would be invaluable, not only to me but to the entire Wikipedia weltschaum. --Ravpapa 15:09, 8 March 2010 (UTC)