Communal enabling of quality

Valid points, but are we sure that the community decision processes are having trouble? I do see problems with the community, but in other areas. --Piotrus 03:52, 24 November 2009 (UTC)

Piotrus03:52, 24 November 2009

Piotrus, where do you see problems in the community, in what areas, and what would be your suggestions to address these concerns?

Bhneihouse14:46, 24 November 2009

Keeping it short, see my mini-essay here. Bottom line: I've seen many good, productive editors - including several academics - leave the project. Most of them were relatively experienced Wikipedians, who got fed up with various personal attacks and otherwise stressful incidents. Hence, a major problem we are facing can be summarize as follows: 1) quality is proportional to the number of editors 2) we have mostly exhausted the low-hanging fruits (techy geeks who find editing encyclopedia fun) - as in they have already became Wikipedians 3) we are having trouble attracting other groups (but that is what the usability and other task forces are taking care of, partially) 4) we are running into a problem that we are burning through our existing contributors (they are getting fed up with the project and leaving for less stressful and more rewarding tasks) 5) at this point ([1]) we have reached a stage where we are loosing contributors faster than gaining new ones 6) thus, quality may be and is likely to suffer due to a diminishing number of dedicated editors 7) my conclusion: we need to understand why editors are leaving, address the issues causing them to leave, reach out to those who left and try to bring them back, and support plans to tap new pools of editors. --Piotrus 03:59, 25 November 2009 (UTC)

Piotrus03:59, 25 November 2009

Just checking in from the community health task force... Piotrus is onto something. Ortega actually said that quality is highly correlated with the number of editors working on an article, and that the loss of editors risks making even basic maintenance an issue. At our task force, we are trying very hard to figure out what is causing editor burnout and how to stop it.

A theme that might be relevant to your task force are disputes and hostility. I'm not sure how much of it is just a change in culture, as Wikipedia probably attracts different personalities in 2009 than it did in 2001. But I think there's a real systemic problem with how disputes are settled. At least, I hope it's systemic, because that's the only kind of problem we can have control over. We can't change peoples' personalities, but we CAN change the system so that contrary and disruptive personalities aren't empowered.

Which brings it back to the concern about consensus-building. Ortega suggests that there is a problem with consensus-building falling off, and more people pushing their personal opinion. I suspect that consensus building is falling off simply because of scale, if nothing else. It's easy for 2 or 3 people to agree about how to change a couple of sentences on an article. But it's hard for 50 or so people to come together and figure out the best way to cover an entire topic area. And yes, those two person debates escalate into WikiProject discussions or RFCs or policy discussions, which attract entire cartels of opinion pushers. And sadly, many of those opinion pushers are not interested in consensus building. They've actually realized that they can accomplish more by filibustering debate than by compromising. So you keep on having the same debate over and over, hoping to make progress, only to see it fall apart because the different extremes unified in their desire to keep the debate going forever.

So not only does quality never get a decisive resolution... you actually lose good contributors who just got sick of trying to mediate between the different fanatics.

Just my perspective from the community health task force. Don't let me interfere with your work... only trying to help.

Randomran04:20, 25 November 2009

Thanks Piotrus, that was insightful. I already had your 'morsels of wisdom' among my bookmarks and read them some time ago (I think they're really good). And Randomran thanks too, I've put your project on my watchlist and will follow your discussions.

I'd like to point out that:

  • We still don't have a sound definition of quality to start with. Ortega uses the FA status as objective measure and then combines this with the number of people editing an article. However, in my experience this can only have real value at the English Wikipedia and some other really large projects. (At smaller projects the number of people editing an article is not always a positive factor when an article reaches FA status. There are not enough quality contributors at these projects to make teams. At smaller projects, one quality contributor can write/translate an FA article while ten unknowing but well meaning others can destroy it);
  • We are now implicitly assuming that the number of total editors is equal to the growth/decline in quality at a project. I think it is not that simple. There are 'quality editors', and their number does matter. There are other editors who mean well but don't improve quality a lot, their number doesn't matter that much. There are 'vandals' (including other users that decrease quality of content such as POV-pushers), their number does matter since they destroy quality. There are editors mainly involved in maintenance (including bots), they decrease the amount of damage done to quality by vandalism and thus matter too.
  • The risk that a certain editor leaves the project exists, the average user account has a certain life time.
  • Let's face it, the average Wikimedia contributor is an individualistic, technically well developed person, often with a liberal attitude about social interactions. He (I dare guess most Wikimedia users are males) knows about computers, but is not so good at communicating with others. This bad effect is strengthened by the choice/option to stay anonymous at Wikimedia projects (Citizendium tries to exploit this weak point of Wikipedia).
  • The number of quality editors and 'maintenance editors' can grow when the help functions of the project are improved. We've explored this in our discussion above (practical example: idea of wizards).
  • I assume most users that grow into 'maintenance users' more or less fit the profile I sketched above. I dare assume that many of the best 'quality users' don't fit this profile though. They know less about computers (they know about the areas where they are adding quality instead), are generally more civilized/communicative and thus have a bigger risk of burning out and leaving. Even though we need both types of user for quality, the balance will slowly develop in favour of the 'maintenance users' (because they have a higher 'life time' expectancy).
  • We end up with the situation that most of the users at a project are primarily involved in maintenance. They hold the best positions to have their ways too, because doing maintenance is a requirement for being an admin/bureacrat/etc (this does matter to have your way, see Piotrus' essays on wikipolitics and why good users leave).
  • The process thus builds environments in which it is very easy to do maintenance and the most active maintenance users are highly respected. Adding quality isn't easy though, because there is simply little thought/political influence left for quality users. The type of user adding quality is confronted with bad behaviour often and often doesn't have the final say about content either. No wonder they leave sooner...
  • The influx of new users of all types is probably related to the size and thus the growth of a project, but let's assume it is constant for now. At larger projects like wp-en, the influx of new quality users is at a constant high level, at smaller projects, it is at a constant low level. This means that quality users are often found in isolation at small projects, surrounded by maintenance users and vandals. When they leave, the quality they added is at risk of being destroyed again.
  • This is imho one of the biggest problems our task group has to find solutions for. The questions are: 1) How can we ensure quality users get more respect and/or influence in communities (and in that way prolong their life time)? 2) How can we get quality users at local projects out of their isolation? Besides, we may want to explore the questions how to decrease the amount of vandal users and increase the amount of maintenance users too, since this increases quality as well.
Woodwalker06:20, 25 November 2009

I see I made a mistake about Ortega's comments on FA status. The statistics do show that the number of editors increases the chance an article becomes FA. I think it is an objective measure, yet not representative. It only shows featured articles have been edited by many users, not that content with high quality was generally edited by many users.

Woodwalker17:06, 25 November 2009

Yeah, it's a correlation and he's careful to note that it requires further investigation.

Randomran18:20, 25 November 2009

Having written ~20 FAs - and this is my subjective experience - in vast majority of cases I was doing the work alone. Other editors helped a little, particularly with improving by non-native English, and of course there were hundreds of tiny additions/improvements over time. Did the article become better with more contributors editing it? Yes. But if I were to quantify it, I'd say that if on average my contribution was 100, than the contribution of 100 other editors was 0.1 each (*100 meant they improved the article from 100 to 110). In other words, in my experience as a FA (and GA, and DYK) creator, one user does most of the work, others do only a little bit that even if added together does not exceed the contributions of a single, main creator (that said, I am familiar with exceptions to this rule - I've seen FA/GA/DYKs which were truly collaborative - but they are, I again stress this, exceptions to what I've usually observed).

Piotrus19:38, 25 November 2009

My experience with FA's is that actually after the certain level of quality has been achieved (for instance the article has been selected as FA) the majority of subsequent edits are actually detrimental and substandard - they only make the article worse and lower the quality. In my opinion, protection of the quality id one of the major issues we need to discuss at some point.

Yaroslav Blanter14:42, 8 December 2009
 

I'd say the same thing. Usually I'm working on a FA by myself, or at most in a "tag team". Maybe one or two peer reviewers will contribute a pile of copy edits. The correlation is likely attributed to the fact that FAs have usually existed for a much longer time, and something that's been around that long is bound to have more editors. Also, a FA is likely to be a high-attention topic, so it wouldn't be surprising to see more readers *and* more contributors.

But I think the broader point linking community health to quality is valid.

Ortega notes that "core" editors are definitely the main contributors to featured articles. And I think there is widespread agreement (although no conclusive evidence) that good editors are burning out. The closest thing I've found to definitive evidence: Ortega studied all the different language Wikipedias and found that "core" editors slowly fall out of the core, and fade away with time. But on the English Wikipedia, "core" editors go from being highly active to suddenly disappearing. I think that's the difference between someone who left Wikipedia because they were just running out of time and interest, versus someone who left Wikipedia because they had suddenly gotten fed up.

Randomran21:20, 25 November 2009

I understand the reason for sudden disappearances quite well. Fading users may be losing interest or get busy with RL and so on - I expect that's normal, and I am sure we have our share of those on en Wikipedia as well; vanishing ones are those who get sufficiently antagonized by negative reinforcement (read: PAs, harassment, etc.) that they say "that's it, I am outta here". --Piotrus 19:56, 26 November 2009 (UTC)

Piotrus19:56, 26 November 2009

Just a personal perspective about online discord, I hate to say how long I have been online, (LOL) and I have literally left a ton of communities because of the discord factor, because things like my clients or my business or my daughter are just more important than watching people arguing or having them diss my opinion, whether valid or invalid. And I think it likely that is a common thread for the exact people we want to attract and retain as contributors and editors. So the whole supermajority/consensus issue is probably more important than I originally thought, having had more time to think about it. I believe we need subject matter experts who are dedicated to different sections who can help vet and edit. Those subject matter experts are likely to have the same kind of zero tolerance I have for unnecessary conflict. Just mho.

Bhneihouse03:36, 27 November 2009

Agree 100%, this is likely a very major cause.

Important observation #1: This isn't just experts, it's most mature and capable users.

Important observation #2: "They are likely to have zero tolerance for unnecessary conflict"... yes, but also may not be entirely happy that others will have zero tolerance for theirs too. A lot of people expect their standards and others' to differ (usually leniently to themselves!). Same thing said elsewhere, even experts need induction/newcomer handling, "This is how we work, these are the expectations..."

FT2 (Talk | email)03:54, 27 November 2009

some of this is, of course, going to be cultural. Across the en-wiki, we have how many countries/cultures speaking English? and even in the US, we have so many different ways that people interact, specific to geographic locations.

I think these factors, especially when dealing with a wiki that encompasses a large culturally diverse population, can be mitigated by the kinds of policies we have been discussing. let people know up front what the expectations are, let them know that Wikipedia takes protecting their rights and contributions seriously and we will attract and keep the mature and capable users. (I wasnt just meaning experts, btw.) I think the ideas we have been tossing around of creating a kind of incubator for a wiki prior to stamping it "launched" may also attract the kind of contributors we are talking about.

Bhneihouse20:23, 28 November 2009
 
 
 
 
 
 

Valid points. Few comments:

  • if we want to define/operationalize quality, I suggest we do it in a new section (how about the one I started few days ago, which raises the same issues in Q1-Q2 below?)
  • there is certainly a major difference between highly productive editors and less productive editors; as some past studies have shown, 10% of Wikipedia editors produce 90% of content (more or less). Overall, the number of editors does matter, but the number of high end contributors of course matters even more.
  • I don't think that editors "grow" into maintenance editors (gnomes). I think some chose to become them early from the start. In any case, maintenance editors are crucial for stability, content creators are crucial for growth. Both groups must be nurtured, but I would agree that content creators are at a higher risk of burning out, as creating content is probably more controversial than maintaining it (albeit I can be wrong, as I can think of some examples of large wiki conflicts about maintenance issues - but they could be exception to the rule).
  • Your two questions:
    • 1) How can we ensure quality users get more respect and/or influence in communities (and in that way prolong their life time)?
        • My answer: tough question :) One solution: encourage and give special powers to non-anonymous users (as described here). This would also address a major quality issue: anonymous experts getting overruled by anonymous (but more active, or experienced) POV pushers. This is indeed a major strength of Citizendium. Other: careful analysis of this. One one hand, content creators should not be given a carte blanche to be disruptive. On the other hand, I've seen examples where editors who have created vast amount of content were harassed, successfully, by editors who have done very few edits, yet admins and arbcom in their investigations did not seem to care about that difference. Solution 3: positive reinforcement. When faced with bad atmosphere and harassment, it is important to show editors that we still respect them and care - yet wikilove is much rarer than wikistress (for example, arbcoms commonly warn/admonish/sanction editors; very rarely they also recognize that those editors do something right).
    • 2) How can we get quality users at local projects out of their isolation?
        • I am not sure I understand your question. Editors edit what they want to do, I know some editors whom I'd like to see more active in the community, but they don't care about anything but their little field. In the end, as long as they are happy in it, is this such a problem?
    • 3) Besides, we may want to explore the questions how to decrease the amount of vandal users and increase the amount of maintenance users too, since this increases quality as well.
        • Regular vandals are a non-issue. The problem are those editors: POV pushers, or however we want to call them. They are harder to identify as disruptive, yet they are the ones primarily responsible for creating bad atmosphere and driving others out of this project. In my experience, it is those type of editors that are primarily responsible for driving good editors away, and I am not seeing any signs the Wikipedia system is able to deal with this issue. In fact, the danger to quality in the future is not only the loss of contributors - it is the possibly shifting proportion between good content editors and POV pushers. --Piotrus 19:54, 25 November 2009 (UTC)
Piotrus19:54, 25 November 2009

@Piotrus (point for point):

  • I started a new section (see above). I think the topic you started is the right way to proceed, but I think we should separate the definition of quality from Philippe's questions.
  • I agree.
  • Initially, most constructive users are prepared to do maintenance, though there is a group which is only interested in writing too. Actually, I didn't mean only gnomes, the 'classical' gnome type is supposed to be low-profile and uninterested in wikipolitics. There are a lot of users who do a lot of maintenance, don't add content, yet work themselves up to admins and have high social statuses. The thing is, just like it is easier to start a new article than it is to revise an existing article, it is easier to 200 times revert a vandal than it is to remove POV from one long article. Many users that start doing some maintenance will do more and more, for this reason. I'm not saying that doing maintenance is a bad thing, at the contrary. I just think the difference in the amount of 'award' the system gives for maintenance (enough) and editing content (too little) is a problem if we want to have projects growing faster.
  • 1) I agree rewarding or at least protecting non-anonymous users is a very good idea, as long as constructive critique remains possible. This could at least increase the amount of quality content added by this group. Anonymous users can be 'quality users' too though.
  • 2) I'm sorry, I forgot to give my arguments. I have two: my theory of wiki-erosion and my assumption that quality users are motivated by constructive cooperation.
    • Wiki-erosion is the reduction of quality by maintenance edits (or the absence of maintenance reverts), because the community's maintenance users don't have enough specialist knowledge. As an FA writer (we all have more or less compatible wikiprofiles, that's perhaps why we're in this task force together :) ) I know this natural erosion rate from watching my own FA being victims to erosion every so often. The 'natural' rate of wiki-erosion is related to the community size: the smaller the community, the smaller the chance specialist knowledge to identify erroneous edits exists. Thus, specialist/quality content at small projects has a higher risk of being 'eroded' than at larger projects. Wiki-erosion decreases the quality of the content to a level related with the size of the community. We can prevent this by identifying isolated quality users and reaching out to them, to help them move quality content to places where the risk of erosion is smaller. From my own experience, I know the example of Pelex, who mainly edits the Slovakian Wikipedia. He understands other languages but finds writing in English more difficult. I encouraged him to translate some of his articles to wp-en and helped to correct grammatical errors. There must be many users of this type (I only understand so many languages and my specialist knowledge is limited to geology so I can't identify most of them). At the moment, the Slovakian Wikipedia contains some nice material about geology not available in other languages. The language barrier keeps this quality content in isolation. Unlike its big English brother, the Slovakian Wikipedia doesn't get translated often (furthermore, translation of quality content requires a translator familiar with the subject) so the chance that this content will stay isolated is higher too. Imagine the specialist user stops editing after some time, the content is then in the hands of the maintenance users of the Slovakian Wikipedia. Due to the natural rate of wiki-erosion, errors will slip into it.
    • There is one more reason why we should get isolated quality users out of their isolation: I think it prolongs their life time. Let me cite Ortega again: he expresses some surprise at the fact that departure rates aren't related to policy and hints that it is probably rather related to the general style of social interaction. The fact doesn't surprise me at all. Quality users don't contribute to wiki-projects because they want to change policy, but because they want to show the world their knowledge. What all wiki-users crave for is some sort of 'applause' (all users are, in a way, like trolls, they seek some form of attention). If you wouldn't have written an extensive answer, I wouldn't have given this reply or at least I would have become less interested. The process of adding content to projects works the same way. What all wiki-users hate is the opposite of constructive cooperation: non-constructive cooperation, obstructive or rude behaviour and fights - according to your own model these things make users radicalise or leave.
  • Third and last point ( :-) ). I don't agree vandals are a non-issue. Regular vandals are because they form a small group and will easily get caught. Being non-vandalistic is the easiest quality requirement to recognize and projects have been successful in battling vandalism (though I occasionally see something slipping through the net). Looking at the 'content-related factors of quality' I've identified, I'd say other forms of non-encyclopaedicity are the next easiest to identify (for example, a list of unknown family members is added to a biography). NPOV and verifiability are yet harder to identify, yet the most difficult is balance. This is maybe the reason that no project has a guideline about balance alone. The type of user you refer to ('true believers') mainly reduces NPOV, verifiability and balance. I agree with you: these are bigger problems than vandalism and we should focus on them.
Woodwalker08:26, 26 November 2009

Randomran said it for me - "there's a real systemic problem with how disputes are settled... Ortega suggests that there is a problem with consensus-building falling off, and more people pushing their personal opinion. I suspect that consensus building is falling off simply because of scale, if nothing else. It's easy for 2 or 3 people to agree about how to change a couple of sentences on an article. But it's hard for 50 or so people to come together and figure out the best way to cover an entire topic area. And yes, those two person debates escalate into WikiProject discussions or RFCs or policy discussions, which attract entire cartels of opinion pushers. And sadly, many of those opinion pushers are not interested in consensus building. They've actually realized that they can accomplish more by filibustering..."

Woodwalker also - "This is imho one of the biggest problems our task group has to find solutions for... 1) How can we ensure quality users get more respect and/or influence in communities (and in that way prolong their life time)? 2) How can we get quality users at local projects out of their isolation?"

I'd add a third aspect: big problems often need outside help -- and on enwiki we have problems so difficult, so extended in some areas, that by the time you exclude involved users, people who will be accused of bias, users unwilling to endure attacks, and only users with very high experience and capability, there's nobody (including its Arbcom) able, capable and willing to address these.

As our mandate is to produce a few recommendations, we need to think hard, what one change here is most likely to succeed and make the biggest incremental difference. I think having a way to formally recognize "trusted content editors" (noted before) is actually the one biggest change that could help. I post below, my thoughts why.

FT2 (Talk | email)19:15, 26 November 2009
 

I would not disagree with saying that content creation is not valued high enough, but as a major content creator, I may be somewhat biased here :>

Anonymous users certainly can be high quality contributors. The problem is two fold: 1) when anonymous users quarrel with/harass non-anonymous experts 2) when anonymous experts lack incentive to become non-anonymous, while the fact that we have non-anonymous, verifiable experts would boost Wikipedia standing in the eyes of its readers.

Wiki-erosion occurs everywhere; you make an interesting point that it would be larger in smaller communities than in the larger ones - I am not necessarily convinced by it. In smaller communities, there is also less editor who can contribute to erosion. You are assuming that as community grows, the ratio of experts to non-experts improves - I am not necessarily convinced it is the case. That said, I do think we should be encouraging content translation and editor migration into English Wikipedia (the biggest, the best...). Invinting quality editors from lesser project will improve our quality, but in the end, its a zero sum game (they write for us - they don't write for their projects).

"Quality users don't contribute to wiki-projects because they want to change policy, but because they want to show the world their knowledge. What all wiki-users crave for is some sort of 'applause'". Certainly. This is why positive reinforcement is important, and negative reinforcement is so bad (IMHO, the major cause for vanishing of editors).

If we classify POV-pushers as vandals, than I agree, vandals are a problem. I meant the the "wheely on wheels" or "YOU SUCK" type of vandals are a non-issue; I think we agree on that :) Discussing the type of advanced vandals, I think there are two groups, both dangerous to quality, but one of them much more than the others. The first group are POV pushers who don't engage in harassment of their opponents, the other group does. The first group can be controlled by the community, as other than being slightly annoying they don't cause other editors to leave. It is the other group - the "true believers" who POV push and harass their opponents, often resulting in their burning out and leaving - that is a serious problems (as they cause quality contributors to disappear). --Piotrus 20:08, 26 November 2009 (UTC)

Piotrus20:08, 26 November 2009

@Piotrus: about wiki-erosion: it is indeed also dependent on the amount of 'bad edits', on the project size. However, this only determines the erosion rate. So, at the English Wikipedia the erosion rate is higher than on the Greek Wikipedia, because there are more edits (on average of course, having patrolled both projects for vandalism I can assure you there's more vandalism at wp-el than at wp-en at some specific times of the day ;-) ). Sadly, this fact doesn't help. I assume Wiki-erosion is a hyperbolic function with time. It levels the quality of content to a certain level. That level is (in my assumption) directly related to the amount of specialist knowledge in a community. Therefore, even though it will take (on average) much more time to erode quality at wp-sk, the erosion will in the end be way more destructive than at wp-en.

Woodwalker10:10, 27 November 2009
 

Concerning the translations: I once made a suggestion on Meta a couple of years ago, but as with all Meta suggestions it did not go anywhere. I think translations of good and featured articles should be encouraged by all means. The problem is mostly to monitor the quality (not all projects, including en.wp, maintain high standards for FA and GA, and not all really good articles get selected since the authors do not bother about the status). We may think about bot monitoring the size of the articles, and if an article has been greatly improved and not reversed (to exclude vandalism) it gets checked manually and makes it to the list of articles for translation. This could be a Meta-wide list with individual projects encouraging the translation. Usually there are manu users willing to translate articles, they are just not sure where there is quality stuff.

Yaroslav Blanter14:49, 8 December 2009