Survey INTERIM results
Howie, Philippe, and I chatted about the survey results earlier today, and I wanted to briefly report on what we discussed.
First, as FloNight points out, a significant number of people say they have not stopped contributing. That could be selection bias (e.g. prompt responders), or it could be that our definition of ex-contributors is too rigid. People who are relatively inactive contributors may not edit for three months, but they don't necessarily feel like they've actively left in those cases.
I think it would be useful to ask this question (e.g. "have you stopped contributing?") up front in the survey so that we can segment the responses accordingly.
Second, Howie did a comparison between these results and the results of people who said they edited 10 or more times a month. There were some significant differences. Specifically, people who edited 10 or more times a month were more likely to complain about conflicts within the community. They also were more likely to say that they still had something to contribute to the project. Most of the research tends to lump Wikimedians into contributors and active contributors. This segmentation suggests that we really need to differentiate between active contributors and very active contributors as well.
One of the challenges with these numbers is that they're self-reported. It makes sense to split out this data based on actual behavior.
Next steps:
- Howie and Philippe are going to work on a second draft of the survey that will ask up-front whether or not the respondent has stopped contributing. We will also batch the surveys according to number of monthly contributions (1-10, 10-99, >99).
- An amazing number (over 30%) of respondents said that they were willing to be contacted. We should contact them. Would someone here like to coordinate this? I think we should come up with a basic script of questions to ask, and then we should get volunteers to contact a set of respondents and talk to them.
I also want to point out two other possibilites for the "not stopped contributing" - a) I set up another account and am contributing there (ie, i hated my username) or b) I'm contributing without being logged in. Anecdotal evidence suggests there's some of both happening...
As for the "contacting" , we specifically asked about online chat, so this would not necessarily necessitate telephone contact.
I kind of had a feeling that this would happen. But this stuff is hard to do, and it's good that we learn from it and refine it. I agree -- we need to ask better questions up front to segment people into those who left versus those who just scaled back, and those who were active versus those who simply signed up and left. I think you guys have it under control, but I wouldn't mind keeping an eye on it before you finalize the new design.
I would also be very glad to help out with follow up questions. It might be good to make it transparent (without necessarily asking people to give up their anonymity), just so the paranoid types don't accuse us of fudging the results.
I am 100% unsurprised that more active contributors were more likely to cite problems with the community.
We'll post a draft here for feedback before sending out a second batch.
It would be fantastic if you would help with follow-up questions. Would you be up to coordinating the volunteers to help with this effort? Count me in as a volunteer.
Sure. I know a thing or two about primary research / surveys. But more than anything, I'd need to know what Howie and Philippe are trying to achieve.
Great! Just to clarify: by follow-up questions, I meant asking respondents follow-up questions. About 30% said they were willing to talk to someone about their experiences. I think it would be helpful if we followed up with as many people as possible. As Howie has said, it will add texture to the results, and it will give all of us a chance to get some one-on-one time with ex-editors. Who knows? Maybe it will help convince some ex-contributors to come back?
We would need two things: a "script" of questions to ask the various respondents, and volunteers to talk to 3-10 people per volunteer.
How does this sound? Others interested in helping?
Designing adequate follow up questions is tricky business. As you saw with the survey, you learn more as you go along that helps you come up with better questions.
We haven't leveraged the feedback from the open-ended questions yet. I think that would give us the best direction for asking follow up questions. I know we don't want to start invading privacy or anything. But certainly, we could aggregate some of the open-ended responses, and look for trends... that would tell us where to dig in our follow-up interviews.
There might be multiple good ways to aggregate them, BTW. One is by the number of edits. But another is by the type of complaint -- some who said "complexity was/wasn't a reason", some who said "community was/wasn't a reason". The more we can parse and re-parse the data, the better.