Benefits of having "trusted / high quality" user recognition

Jump to navigation Jump to search

Grrrr: LT has eaten my post twice, and apparently I didn't learn the first time, to compose in OO :-(

So: suffice to say, very quickly .... based on my own experiences as a new editor, I can tell you I would have found trust-labelling-of-editors very very helpful. So yes, I can imagine it being useful for non-new-editors, and it would also be useful for new people. It's a good idea.

I would however make a plea for trustworthiness being automatically derived out of on-wiki actions, rather than being manually assessed on a case-by-case basis. 1) It scales better, 2) it eliminates popularity contests, and 3) it prevents people needing to waste a bunch of time debating particularly controversial / edge cases. I know there are lots of challenges inherent in an automated approach, but I think the benefits would outweigh the drawbacks.

22:28, 26 November 2009

I want to think carefully on this. You wouldn't add a post like this lightly, so I'm going to think on it.

The first thought is roughly, keep it simple, and using automation to cut down the number of cases that need more than cursory human review (80/20 rule).

However, the benefits of something like this flow from the idea that some users can generally be trusted to do right, in article and article discussions, of any kind. Those are the users we want in this pool, because once identified, they provide a large population of experienced content writers not needing much guidance or checking, and capable of being given heavy disputes to put into good editorial order.

I don't see a way to judge it on any automated basis though. The benefits here flow from their ackowledged trust to do right (broadly speaking) on any content matter, up to and including self-management of bias, interaction style, and the like.

That should be what we're guiding people towards. That's actually what we want and need -- and it's not hard to do either. It'll become a norm that people'll want to get it. Make it valuable, and people will value it. But I don't think it's open to easy automation. It would probably be easier to design a system with a simpler human element instead. I agree the intensity, drama, and diversion of communal resources seen at RFA isn't where something like this or on this scale would end up. Probably very different.

I'll think about it harder, but that's my initial thoughts why I've positioned it as I have.

FT2 (Talk | email)

23:58, 26 November 2009

I agree about not automating this process. I also agree about popularity contests, thanks Piotrus. So the admins surveying these users themselves have to be pretty much beyond reproach. Perhaps a pivotal point in quality control is how Wikipedia "approves" and trusts editors? Perhaps another pivotal point is the actual "structure" of this process? I apologize if what I am saying is obvious and derived from all of your statements -- I am trying to get us to bullet points that become more cohesive as we work.

03:57, 27 November 2009
 

Hi FT2.

I think I may have overstated the strength of my position in my earlier post; I was having some LT problems, which made me, in the end, probably overly succinct :-)

So let me make a couple of longer comments now:

It sounds to me like what you want to create is a marker/label for people who are particularly trusted to have good judgment. Probably these would be people who've been around for a while, and understand the policies well, who are reasonable and thoughtful.

I think that's a great idea. I think new editors would really appreciate being able to tell at-a-glance if an editor they didn't know was someone they should trust and listen to. I think also that one of the big points of pain for new people is that when their edits are reverted, they automatically assume "the Wikimedia community" has rejected their edit --- they don't understand that it's the act of an individual, and not necessarily a good or wise decision. If some editors were labelled as particularly trusted then A) new people might not be so quick to assume that everyone speaks with consensus authority, and B) they might actually be motivated to seek out advice and counsel from the ones who are specifically labelled as known to have good judgment.

I can also see how labelling-of-particularly-trusted-editors would be helpful for other experienced editors -- for the ArbCom, for people who do OTRS work, and so on. I don't have that kind of experience myself, but I can imagine how this would be useful for the people who do.

(((Basically, we currently all have our own informal mechanisms for assessing people's reputations, and learning who to trust. I trust people I know personally, like Philippe. And I trust people who my trusted people trust -- like, I first began trusting you, because Jimmy told me I could :-) But that's limited: it doesn't scale very well, and it takes a long time for those networks to develop, which makes things especially hard for new people.)))

So I think your idea makes a lot of sense: it would enable trust to scale better than it currently does.

So, after thinking it through some more, I have two comments/suggestions for you:

1) I worry about the word “trusted.” If some editors are labelled as trustworthy, by implication other editors will be seen, or will feel as though they're seen, as not trustworthy. Which I think would make lots of people feel bad, and could make the “trusted” people targets for envy and anger.

In my old world (journalism) we used the designation "senior editor" for exactly this kind of person, and I'd recommend you think about using it for this. A senior editor in a newsroom generally does the same work as other editors, but the designation is an acknowledgement, and a signal to others, that they are especially seasoned and credible and wise. It's a label that new people would understand. And it wouldn't alienate other people – they can still be good, constructive, useful editors, and they can aspire to earn senior status without feeling diminished by not having it.

2) I really worry about the system for attaining this status being gameable. I don't think you could afford to have an open voting component, because there will always be a few trolls and cranks who are super-motivated to game the system, and practically anybody can rustle up a few dozen friends to help them do that. I worry that if there's a voting component, it'd be a magnet for posturing and rabble-rousing and drama, which would end up wasting tons of good people's time.

I also think voting might make it impossible for anyone who edits controversial topics to gain this status. That would be bad, because as I understand it, some of our best and wisest editors focus on trying to bring neutrality to difficult topics. In a voting-based system they would get voted down, I think, by power blocs of editors with strong POVs.

So I don't think voting would work. You want these decisions to be made thoughtfully, by experienced people. So I would suggest this instead: I think you could have an (automated, low) bar that people need to reach before they're considered for this status. Like, maybe one year of editing experience, and a minimum of 200 edits. That would screen out people who simply haven't been around long enough, or edited enough, to have developed a good understanding of the policies. And then I think you would need a trusted team of people, who would investigate people's edit histories on a more qualitative basis, and seek out people who are particularly thoughtful and wise and constructive.

Who could make those decisions? I don't know the internal workings of the projects well enough to say – but I would guess the ArbCom could, or the Arb Com could nominate a group to do it, or the community could nominate a group that the Arb Com could then approve. Or maybe the ArbCom could nominate the first dozen senior editors, who could then set up a system for expanding their own ranks. It seems to me that the kind of people who create Featured Articles would be a good starting point – but like I said, I don't know the internal workings well enough to really say.

Sorry this is so long, but I hope it's helpful. It's a good idea: I would really like to see it work :-)

05:24, 27 November 2009

Agreed on the controversial topics issue. I've seen reasonable people caught in the middle, and ending up with nothing but harassment (from both sides) to show for it. The more we can rely on other measures the better. The human component can be there as a screen, as a veto, but people should really be judged by accomplishments that the widest number of Wikipedians cannot deny.

05:45, 27 November 2009
 

Framing is important, so I support the "senior editor" name.

ArbCom doesn't have time to deal with that stuff; instead, how about we make all editors who have written 2+ FAs "senior"?

05:48, 27 November 2009

I think that's a good place to start. Simple, objective, hard to game, but achievable within 6 months to a year if you want it.

05:51, 27 November 2009

Urgh. No.

Good content writers may or may not be good at interacting and working with other users, may not be good in other topic areas, and so on. It has its fair proportion of users who couldn't meet the kind of role we're talking about, though.

FT2 (Talk | email)

05:56, 27 November 2009
 

I'd suggest if anything this:

  • Users will be assessed on trust in their editorship. They must submit a portfolio of significant experiences and skills covering:

  • Basic and peer reviewed article writing - typically at least 10 non-stub articles and 1 GA/FA.
  • Specific editor skills - responded on at least 120 noticeboard issues spread across the major noticeboards including FRINGE, RS, NPOV, BLP/N; COI/N; EDITWAR/WQA, content RFC/3O, xFD (including at least one "rescued" AFD), plus basic template skills.
  • Peer review skills - typically at least 5 GA and 2 FA reviews
  • Collaboration skills - significant active involvement in a Wikiproject for at least 2 months, or equivalent.
  • Editor dispute skills - addressing 10 or more disputes with a mix of amicable and hostile/improper editorship.
  • Your own showcase - at least 3 items (not otherwise used here) that showcase your interests and abilities in any wiki area. These could be unusual or interesting content or editorial matters, media work, admin or patrolling, or any other area - your choice!
  • Users wishing to object must show diffs that clearly evidence any of the following in the last 9 months:

  • Two or more instances of clear poor judgement (not just legitimate disagreement) related to NPOV, OR, CITE, V, RS, or COPYRIGHT.
  • Two or more clear demonstrations of personal attack, attacking a person not the evidence, a thread in which the user fillibusted "gamed" edit warred or obstructed consensus in an unreasonable manner, or threats.
  • A history of poor xFD or other content process contribution
  • A pattern of undue serious incivility covering at least 5 instances.
  • Evidence that the portfolio grossly misrepresents their content editorship.
  • Any blocks or other formal warnings or sanctions by an administrator.
  • Gross bad faith, breach of trust, deception, or any access removal related to poor conduct (including but not limited to puppetry, faked content or citations, concealed POV warring, conspiracy to disrupt the wiki, and the like) - without time restriction.
  • Any diffs should be self evident, with minimal context or explanation, and clearly show the behavior concerned.

  • Claims (portfolio or concern) not evidenced as above are disqualified except in exceptional circumstances.

That allows users a rough criterion for "evidence that should be publicly shown".

It's not even that demanding - 10 basic articles, one decent peer-reviewed article, half a dozen peer reviews, a couple of days work on noticeboards (to show specific areas), and some dispute resolution and collaboration.

I don't like being prescriptive, I think it's gameable, but it's still better than 1 - 2 FAs. It'd probably work.

FT2 (Talk | email)

06:09, 27 November 2009

I'm not against formalizing the status of quality users (the German wiki has already experimented with something like that in the form of 'quality revisions', revisions that can be flagged by experts only). However a user that is an expert in Indian cooking isn't an expert in quantum physics, for example. The most it would give us is a very general indication if we can assume good faith in a certain contributor.

05:53, 29 November 2009

The essence of wiki working is, that certain approaches are key. Approaches (or their lack) aren't the same as expertize (or its lack).

If given the choice of an expert who could not show good editorial approaches, or a good editor with the right approaches who lacked specific topic expertize, then for this project, choose the latter, not the expert. Why? Many reasons:

  • The expert (in such a scenario) may have his own foibles and bias, or unwillingness to hear others, or non-neutral stance
  • The expert doesn't know how to collaborate, or work with others in a mass edited project. As a liability he may drive others off and absorb immense time and harm the communal fabric. We get one perfect article (only we don't know if it's biased because every community dialog about it descends into argument and name calling) and not a lot more.
  • The non-expert with good approaches will listen to others, consider the views, research them and check the details. They may not know, but they know how to examine others' work and check facts. They foster others to work with them and as a community the work gets done to a high standard even so.

Part of Wikipedia is that although we want high quality, we aren't a cutting edge academic source. We'd like to have some of that, but it's not (as I understand it) our actual core goal.

If we're assessing what kind of editor can be broadly trusted to work on all kinds of difficult articles unsupervized and do so in a proper way (as this thread considers), then the qualities I've outlined will get you that person, and fairly high quality (though not cutting edge) writing. A PhD won't.

$0.02 :)

FT2 (Talk | email)

06:18, 29 November 2009

I agree, but let's not call them 'experts' when in fact they're rather 'good editors'. I like the idea to create some kind of special user status whenever an editor reaches the requirements you mentioned (06:09, 27 November 2009). What do you think about quality revisions though? It's also a way of showing the reader how the best editors rate the quality of an article.

07:42, 29 November 2009
 
 

I don't necessarily see it as better, but I am not opposing making this more complex (still, I like KISS...). Anyway, regarding opposing, I would strongly suggest taking editors activity into consideration. In other words: editors who are very active and have edited for a long, long time (and logically, would be likely to be good or trusted editors) also are more likely to have enemies, or at least, more "exception-to-the-rule" dirt that can be brought up against them (lookie, warning from 2006, PA from 2007, ArbCom from 2008...etc.). As such I'd suggest that any assuming the editor applying has been reasonably active in the past 6 months, examples (diffs) of poor judgment should be not older than that period. --Piotrus 05:22, 30 November 2009 (UTC)

05:22, 30 November 2009
  1. My own view is that most "professions" have some kind of "continuing professional development" post initial qualification. While wiki isn't a "qualification", we could well ask "what are we offering to users to stretch their skills and as a means of self evaluation and development as editors". Something like this, a "recognized wiki-editor qualification", would be good for the ethos that way too.
  2. Piotrus - the "reasons to object" were crafted as requiring both specific types of bad activity or judgment to be shown via evidence alone, within a time limit of the last 9 months, as drafted. With luck that solves your concern?

FT2 (Talk | email)

11:02, 30 November 2009
 
 
 
 

Long thread, quick replies:

  1. Per Piotrus: Never underestimate the importance of choosing terminology well. "Senior editor" works just fine for me. We can debate terminology if this goes ahead, but the basic point's good.
  2. A panel's fine, but you hit the old problem then: this is a panel that indirectly controls who's designated as a "senior editor". So that becomes a focus of allegations, games, and so on, as Arbcom can be. We know where that path goes, and if avoidable, let's avoid it.
  3. This area's easier than Arbcom because editorial behaviors are almost all public record (even alliances emailed in private become obvious on wiki a lot of the time), so bad conduct's visible. The community was founded on open decision making, and for all other senior roles, it works just fine. Admins, arbs, both done by the open community. let's see if we can avoid losing or diminishing that. It's part of the "trust" model to trust the wider community where we can (with suitable precautions).
  4. Instead of a panel, or trying to be super-ideal on selection, we can have a "pretty good" selection, if we also have ways to effectively catch the exceptions. Don't let the minority fringe case distort what's fine and simple process for the majority of cases. So we might back a 95% valid (but slightly able to be gamed if determined) nomination system, by also having:
    • A clear and standardized removal process
    • Perhaps some kind of scrutineers panel who can review cases claimed to be grossly affected by bias and canvassing, or where the results don't reflect appropriately on the user.
  5. Arbcom's definitely the wrong ones for this.

FT2 (Talk | email)

05:51, 27 November 2009

Agree with FT2's no 3, I think a pannel isn't necessary. The arbcom isn't supposed to be involved in matters of content anyway (at least, the Dutch one isn't, I'm not totally sure about other arbcoms). Let's trust that, after a careful analysis of the portfolio and hearing the opinion of at least, say, three other 'senior editors', the community is able to choose a new 'senior'. Having the community choose them is more in line with the spirit of Wikimedia projects than having an elite pannel.

07:51, 29 November 2009
 
 
 

"it eliminates popularity contests" - very, very important, per my comments above.

03:32, 27 November 2009