Finding and interviewing ex-editors

I guess to answer question 2 we need to find and interview people that used to contribute. One suggestion for this that springs to mind is to ask the question on any other forums we're members of. For example, I contribute to a fairly busy forum related to a national newspaper. I could just throw the question out ("has anyone here edited a Wikimedia project and stopped? If so, why did you leave?") and see if anyone bites.

Another approach is to go to the projects themselves and ask "are you thinking of leaving? If so, why?"

These are good ideas. We could definitely use some more informal feedback. I'll try to think of something more scientific that we can do relatively quickly, so we have a more representative sample to work from. Randomran 21:02, 29 October 2009 (UTC)

Sounds good, Random. I might be wrong but I dimly recall there *was* a survey done that got input from leavers or those thinking of leaving. But I can't remember where I may have seen it. I have had some feedback from the newspaper forum I mentioned. I'll post it all when I have all the feedback. But so far it does seem to chime with what others have said and reported on here; that people were put off by fighting editing battles and being reverted.

That's interesting and pretty helpful. I guess it's the only thing that would make intuitive sense. People either leave on good terms because they have some exciting opportunities, or they leave on bad terms probably because of editing battles.

Then again, and I'm only half-joking, I think some people get addicted *because* of the battles :o)

Hehe. I know you're joking. But I think some amount of debate is healthy, and most people who love knowledge will love a good debate. The problem is when we create an environment where it's fun to compete for influence, and rally some troops, and try to get their "enemies" in trouble. Then we have a battleground, and it's less attractive to jump in, and burnout is more common.

Is there really a 'universal' link between loving debate and loving knowledge or is it culturally defined? Henna 00:03, 16 November 2009 (UTC)

Also a good point: not everyone likes to have a debate, and Wikipedia's discussion style may repel certain segments of the population. All the more reason to find ways to make discussions smoother, and cultivate a sense of teamwork (without cultivating a sense of "my team versus your team").

I really like this informal approach of leveraging everyone's contacts--seems like a great way to pull valuable insights/perspectives quickly.

Yeah, I agree this is helpful. Just that we may need to take them with a grain of salt, since they may not represent the population at large. We should use them to brainstorm ideas and insights, but should avoid relying on them for firm conclusions.

I agree. Seeking out informal views is good as a way to get broader input about a matter. But we need to consider the views as a snapshot of a subgroup's thinking about the situation. Taken in that context, the insights offered can enhance our knowledge since they might fill in gaps or at least give us a sketchy idea of missing perspectives. By considering it this way we don't give too much weight to what might turn out to be a tiny minority opinion. If over time the opinions are repeated, then we'll know that we are on to something significant and we (or someone else) can more fully document the views in a more formal way. Our end product is part of a bigger process. There will be opportunities for others to take what we find and build on it if it is something interesting that we don't fully flesh out. FloNight 10:51, 31 October 2009 (UTC)

Totally agree, FloNight.

Why not ask some retired or inactive users directly? There must be loads of former users who activated email on their accounts and still have those email addresses.

Howie Fung, who's doing some work for the Foundation, is interested in doing an "exit" survey of contributors who leave. This is a good opportunity to combine interests and actually get this information. I'm pointing Howie to this thread so he can contribute himself.

In the meantime, I'd like to encourage someone to summarize this thread and to start collecting ideas on whom to survey and what questions to ask.

This is great news! I'd like to work with him on the survey. I think this has the potential to be our most useful source of data. But we want to get a good mix of questions, without overwhelming people to the point that they don't feel like answering.

We also need a different survey for different stages of leaving. Users we survey on the first, second or third anniversary of their last edit are likely to have acquired some perspective on their experience; whilst users who just posted "retired" on their users page more often than not will be back in a day or two.

Do you think we'd ask them different questions? Or would we just get different answers? If the latter, then we can probably just ask a few questions about timing up front... and then see if there are any trends based on that.

I've started summarising this at Proposal:Survey Former users

Thanks to Eekim for introducing me to this thread. I think it’s great that there are folks in the community who are keen on the idea of getting a better understanding of why contributors leave. I think the people on this thread have a good understanding of the pitfalls of doing research like this (e.g., representativeness of sample), so I’m not overly worried about us misinterpreting the information that comes back. I do think a meaningful distinction to make is very active contributor vs. someone who might have made a handful of edits and then left. My guess is that the reasons for leaving would be different for these two groups so it’s probably worth treating them separately.

I think the idea of contacting these users directly (vs. say an online survey) is a good one. This allows us to keep the questions open ended and not presume answers. We may not be able to contact as many users, but I think we'll get more textured information. I do think we should ask a set of questions which are consistent across each group. This will help us calibrate the data. But we should also stay flexible. I’m sure these conversations will lead us down unexpected paths, which is a very good thing. We can also do an online survey, but I think the objectives would be different.

Does this group have a good sense of why people are leaving? In reviewing all of the available research, two themes popped out really clearly to me: polices are a big problem, and people just aren't nice to one another! (You can find some of this research at Participation.)

I wonder if we could brainstorm potential solutions (via the Template:Recommendations posted up, for example) before looking for additional granularity via a new survey. My sense is that this group has good networks within the community, and you all also have a good sense of why people are leaving or not contributing more than they would otherwise.

It also strikes me that we're focusing heavily on ex-editors, though this task force is about community health broadly. Issues we might want to consider more generally include how we might make Wikipedia a friendlier place overall.

I think you raise a good point about making Wikipedia a healthier (friendlier, more productive) place for all editors. I think that reducing the turnover of ex-editors (both veteran editors and newbies who don't like their first few weeks) is important, but hopefully we can find some recommendations that will also make Wikipedia better for the people who have stuck around as is.

Not sure if this is a already known but: Ubuntu is a linux distribution, started afaik partially because of the harsh nature of the Debian project. One of the tools they use is: http://www.ubuntu.com/community/conduct

Maybe we can look at what other (open source) projects did as well who where dealing with these issues?

That's a nice statement of ideals...

That's very interesting! I would like to learn more about the people who founded that spin-off. What prompted them to start Ubuntu, and whether it's improved things. Is there anyone influential in that community that we can interview?

Benjamin Mako Hill is a Wikimedia advisor and is a long-time Debian contributor. He also helped Ubuntu design its community process. Bridgespan interviewed him, but I don't see his notes under Interviews. John, can you provide some insight?

I think I now some slightly less influential people, I'll ask around Henna 00:07, 17 November 2009 (UTC)

Sounds good! If you're looking for ideas for questions, check out here. These are kind of Wikimedia-centric, but you might be able to tailor them to get great insights from the Linux community.

Unfortunately, Benjamin Mako Hill hasn't gotten back to us just yet with approval to post up his notes publicly! But I'll start reaching out to Ubunut to set up time to chat. Do we have any key questions that we would like to hit on?

Does it work? If it works, what do people see as the reason it works?

I guess I'd like to know why they did it in the first place. What problems they were noticing. And yes, we definitely want to know if it worked. It might also be helpful to have some numbers about the size of the different communities. (Even better would be the growth curves of each community! We obviously know that a young community will grow quickly. But did one do a better job of resisting a stagnation in growth?)

John, I'll ping Mako too. Sometimes, he just needs to be prodded. :-)

Thanks, Eekim!

It is a vital question, and IMHO a survey of editors who left is likely to be one of the most informative and important initiatives that the task forces here could achieve. --Piotrus 20:10, 25 November 2009 (UTC)

Can I ask if you folks are moving forward with a survey?

I raise it because on the weekend, I happened to be reading the history of the enWP 2008 Mumbai attacks article, and I was startled to find that three editors who had made substantial, excellent contributions to that article have since "retired." These were great editors, and I would really like to know why they left, and what might have persuaded them to stay. I'm not suggesting their stories would necessarily be representative of other people's --- to me, it just reinforced the need for a survey, in general.

I also think JohnF is right, and both current and former editors should be surveyed.

+1 :)

Eekim is working closely with Howie on that.  :)

We'll be putting up a survey over the next few days. We've taken the existing proposal (at Proposal:Survey_Former_users) and will be condensing it so that it's a little easier to complete. I'll update the team once it is up and running.

Hey Howie, could we get a closer look at it before you throw it up?

I promise not to raise so much of a stink that it becomes too long. But just want to be sure that the most important questions are there, and take one last look at the phrasing. Some of the stuff we had was sloppy.

Let us know :)

Sure! It would be great to get feedback. I'll be moving the draft survey to a wiki page, but in the meantime, folks can access the draft as a Google Doc here. The doc is open for editing, so please feel free to make changes directly on the doc.

I've posted the most recent version of the survey on this page: Former Contributors Survey. It's mostly a condensed version of the previous proposal with a few other changes. The page also includes a link to the draft online survey so people can take a look at what our users will be experiencing. Have a look!

Hey Howie, I have a few quick comments:

• There's going to be a temptation to say "all of the above" for the questions that have a "check all that apply". Is there any way we can get them to rank it, or at least ask them to choose their top 4? I think that would be much more informative.

• Re: Giving them a list of things they might like to see changed... it's a bad idea. In my experience in business, the customer does not have very good ideas. If you ask a patient "what's wrong?", they'll tell you their SYMPTOMS. They won't tell you the DISEASE. Much more useful is to ask a customer "what was your best experience with X"? You'll learn much more as they describe that situation. I would strongly recommend replacing this question with "What was your best experience at Wikipedia?"

• By the same token, asking them if there was a specific incidence that caused them to leave should be replaced with "what was your worst experience at Wikipedia". It's always more complicated than pointing to one specific incidence, and it's usually a culmination of factors. Let's dig deeper.

Those are the big things. But otherwise, I think the format of the survey is good, and the number of options are basically good. I'd just like to make some wording changes if that's alright with you. Would you be okay with that?

Randomran,

Thanks for your comments. In response:

• I think limiting selections to the top 4 is a great idea. There may be an issue with the Limesurvey though. For multiple choice answers where multiple answers are allowed, the default text is "Check all that apply" and I don't think there's a way to change that. Let me look into that and get back to everyone. (Note: Limesurvey doesn't have a "select all" option, so a user would actually have to click on all of the boxes to select all. But regardless, I think the idea is good).
• I actually disagree on giving users a list of things they want changed. In my experience, if you ask users an open ended question about what specifically they'd want to change, they generally won't give very insightful/specific responses. Yet if you give them a list of options, they usually don't have problems picking the ones that resonate with them. I do like the open ended question of "What was your best experience at Wikipedia?" But while that question will shed light on the positive things we can do to help users stay (e.g., make Wikipedia more friendly), it doesn't do as good of a job shedding light on the negative things we might fix (e.g., remove destructive editors.). Maybe we can ask both sets of questions?
• Agreed on changing the wording on #4 to "What was your worst experience at Wikipedia." This question will work nicely with "What was your best experience at Wikipedia"

Also, please feel free to make any wording changes on the wiki. And thanks again for your feedback!

Finally found some time to take a shot at it. Not sure if I went overboard. But let's talk it out. I'm sure we can find a way to cover the things that we both think need to be covered.

Thanks for the edits. This is very helpful.

Do you have a moment to chat about this in realtime, either via Google chat or IRC?

That's a great idea! I know the holidays can be too busy to make time. But let me know what your schedule is. I'm free except on the 24th and 25th. Maybe tonight or tomorrow, if you get this in time? Let me know.

IRC would be good.

Sorry, I just this now. Are you available sometime later today? I will be in the strategy IRC channel to chat. This shouldn't take more than 10 minutes.

Thanks!

hey, need to do some christmas shopping after work. but I'm popping in now, if you're around.

Here is a draft of the online survey. There are a number of kinks to be worked out, but this should give folks a feel for what the survey will look like.

Please let me know if you have any final suggestions. We're hoping to send the first set of emails by end of Tuesday.

Three form issues:

• Question 3: The number of edits (at the very beginning) -- 100 to 999 is pretty broad. We could afford to break this up into two ranges, if not three. (e.g.: 100 to 199, 200 to 499, 500 to 999)
• Question 5, 8, 8a: "Check any that apply". It's really important that we get people to pick their top three, instead of just "any". I know some people will click more, but we want to avoid people clicking everything. There's a good chance that they all apply, to some degree.
• Question 5 AND 8a: We need an "other", just in case.
• ... could we throw in an optional "tell us your username" thing at the end? Could be useful for follow up.

A few substantive issues:

• Question 6: This question about "was there anything that needed correction" is too obvious. Almost everyone will say YES. It would be more useful to phrase it in a way that emphasizes that this was how much they personally wanted to contribute. Of course anyone could find something else to do. But was it something that they thought was important? Something they thought that they had a special set of skills that made them really suitable to fix it?
• Question 8: Re: complexity, we should flat out test a null hypothesis: writing a good article is hard work, and maybe people leave when they realize it's more than they can handle. This is distinct from maintaining articles. "Yes, writing an encyclopedic article is difficult and time consuming."
• Question 9: on the agree/disagree questions... I think the very first one is already addressed in the earlier question about why they left, and later in the agree/disagree form when we ask them if they would come back when their personal life permits it. I'd just drop it and shorten it. But if there's room, I'd also like to know if people would agree/disagree that "the community is too lenient on disruptive editors". Just a thought.

You also have some numbering issues. But I used the numbers I saw in the current survey.

All in all it's very good though. We did a great job, IMO.

So there are a few issues that I haven't gotten around to with Limesurvey (our survey tool):

• Numbering -- I just need to get around to this.
• "Check all that apply" -- I don't remember if I've mentioned this before, but this is the default text that Limesurvey uses for multiple choice options is "Check all that apply." I haven't found an easy way to suppress this, but I will do a little more digging tonight.
• Regarding the "Other" option -- yes, I'll need to make sure we have these in the final version where it's appropriate. There is another issue, however, with Limewire which affects the questions with multiple "yes/no" answers. Limesurvey defaults "other + <please specify free text>" as the last option and I don't think there is a way to change this. So for Yes/No questions, the answer choices may be "Yes 1, Yes 2, Yes 3. . .No, Other (please specify). . ." where what we really want is "Yes 1, Yes 2, Yes 3, Yes, Other (please specify), No." I'm not sure there's way around this with Limewire. The only way (I think) we can have "other" grouped with the "yes" answers is if the "other" choice doesn't have the "please specify" field. (Sorry if this wasn't the best explanation).

In response to your other points:

• Q3: I know the response buckets are a bit broad, but we're trying to keep this consistent with Ed Chi's (researcher from PARC) buckets.
• Q6: This question has been a tricky one to get right. I see your point -- the question is phrased as a yes/no. What if we changed the phrasing to ". . .how much missing or incorrect content. . ."? This doesn't help us unearth the personal motivation aspect, but Q5 (Why did you stop contributing to Wikipedia?) should help us with that.
• Q8: Agreed. I think the phrasing you used above is really good -- eliminates possibility of a user selecting this choice because they had difficulties with the interface. I may be splitting hairs, but the subtle change is a good one.

Thanks again for the thoughts. The changes should be up on the wiki page in an hour or so.

Howie

Hey Howie,

I tweaked some wording for clarity. I also added an answer about feedback and appreciation (focusing specifically on the "reward" hypothesis) as a criticism of the community.

I also took a crack at question four, so it wasn't such a no-brainer. Everyone will say that Wikipedia has a lot of missing and incorrect content. Maybe if we focus on "important" content we'll get a more interesting answer. (I'm still not fully happy with this question. I'd really like to find out if we lost an editor who could have been really helpful, or an editor that had more or less hit their peak. Not sure how to phrase it.)

Everything else looks pretty much solid though.

Getting the "check the top three answers" is key, though. Very important. If we have to slip "select three" into the actual question phrasing, then do it, even if it looks sloppy.