Finding and interviewing ex-editors

Eekim is working closely with Howie on that.  :)

~Philippe (WMF)18:46, 7 December 2009

We'll be putting up a survey over the next few days. We've taken the existing proposal (at Proposal:Survey_Former_users) and will be condensing it so that it's a little easier to complete. I'll update the team once it is up and running.

Howief00:53, 9 December 2009

Hey Howie, could we get a closer look at it before you throw it up?

I promise not to raise so much of a stink that it becomes too long. But just want to be sure that the most important questions are there, and take one last look at the phrasing. Some of the stuff we had was sloppy.

Let us know :)

Randomran01:44, 9 December 2009

Sure! It would be great to get feedback. I'll be moving the draft survey to a wiki page, but in the meantime, folks can access the draft as a Google Doc here. The doc is open for editing, so please feel free to make changes directly on the doc.

Howief02:36, 9 December 2009
 

I've posted the most recent version of the survey on this page: Former Contributors Survey. It's mostly a condensed version of the previous proposal with a few other changes. The page also includes a link to the draft online survey so people can take a look at what our users will be experiencing. Have a look!

Howief01:07, 10 December 2009
 

Hey Howie, I have a few quick comments:

  • There's going to be a temptation to say "all of the above" for the questions that have a "check all that apply". Is there any way we can get them to rank it, or at least ask them to choose their top 4? I think that would be much more informative.
  • Re: Giving them a list of things they might like to see changed... it's a bad idea. In my experience in business, the customer does not have very good ideas. If you ask a patient "what's wrong?", they'll tell you their SYMPTOMS. They won't tell you the DISEASE. Much more useful is to ask a customer "what was your best experience with X"? You'll learn much more as they describe that situation. I would strongly recommend replacing this question with "What was your best experience at Wikipedia?"
  • By the same token, asking them if there was a specific incidence that caused them to leave should be replaced with "what was your worst experience at Wikipedia". It's always more complicated than pointing to one specific incidence, and it's usually a culmination of factors. Let's dig deeper.

Those are the big things. But otherwise, I think the format of the survey is good, and the number of options are basically good. I'd just like to make some wording changes if that's alright with you. Would you be okay with that?

Randomran04:03, 10 December 2009
 

Randomran,

Thanks for your comments. In response:

  • I think limiting selections to the top 4 is a great idea. There may be an issue with the Limesurvey though. For multiple choice answers where multiple answers are allowed, the default text is "Check all that apply" and I don't think there's a way to change that. Let me look into that and get back to everyone. (Note: Limesurvey doesn't have a "select all" option, so a user would actually have to click on all of the boxes to select all. But regardless, I think the idea is good).
  • I actually disagree on giving users a list of things they want changed. In my experience, if you ask users an open ended question about what specifically they'd want to change, they generally won't give very insightful/specific responses. Yet if you give them a list of options, they usually don't have problems picking the ones that resonate with them. I do like the open ended question of "What was your best experience at Wikipedia?" But while that question will shed light on the positive things we can do to help users stay (e.g., make Wikipedia more friendly), it doesn't do as good of a job shedding light on the negative things we might fix (e.g., remove destructive editors.). Maybe we can ask both sets of questions?
  • Agreed on changing the wording on #4 to "What was your worst experience at Wikipedia." This question will work nicely with "What was your best experience at Wikipedia"

Also, please feel free to make any wording changes on the wiki. And thanks again for your feedback!

Howief23:46, 15 December 2009
 

Finally found some time to take a shot at it. Not sure if I went overboard. But let's talk it out. I'm sure we can find a way to cover the things that we both think need to be covered.

Randomran19:06, 20 December 2009
 

Thanks for the edits. This is very helpful.

Do you have a moment to chat about this in realtime, either via Google chat or IRC?

Howief21:05, 21 December 2009
 

That's a great idea! I know the holidays can be too busy to make time. But let me know what your schedule is. I'm free except on the 24th and 25th. Maybe tonight or tomorrow, if you get this in time? Let me know.

IRC would be good.

Randomran23:25, 22 December 2009

Sorry, I just this now. Are you available sometime later today? I will be in the strategy IRC channel to chat. This shouldn't take more than 10 minutes.

Thanks!

Howief21:41, 23 December 2009
 

hey, need to do some christmas shopping after work. but I'm popping in now, if you're around.

Randomran21:58, 23 December 2009
 

Here is a draft of the online survey. There are a number of kinks to be worked out, but this should give folks a feel for what the survey will look like.

Please let me know if you have any final suggestions. We're hoping to send the first set of emails by end of Tuesday.

Howief19:32, 28 December 2009
 

Three form issues:

  • Question 3: The number of edits (at the very beginning) -- 100 to 999 is pretty broad. We could afford to break this up into two ranges, if not three. (e.g.: 100 to 199, 200 to 499, 500 to 999)
  • Question 5, 8, 8a: "Check any that apply". It's really important that we get people to pick their top three, instead of just "any". I know some people will click more, but we want to avoid people clicking everything. There's a good chance that they all apply, to some degree.
  • Question 5 AND 8a: We need an "other", just in case.
  • ... could we throw in an optional "tell us your username" thing at the end? Could be useful for follow up.

A few substantive issues:

  • Question 6: This question about "was there anything that needed correction" is too obvious. Almost everyone will say YES. It would be more useful to phrase it in a way that emphasizes that this was how much they personally wanted to contribute. Of course anyone could find something else to do. But was it something that they thought was important? Something they thought that they had a special set of skills that made them really suitable to fix it?
  • Question 8: Re: complexity, we should flat out test a null hypothesis: writing a good article is hard work, and maybe people leave when they realize it's more than they can handle. This is distinct from maintaining articles. "Yes, writing an encyclopedic article is difficult and time consuming."
  • Question 9: on the agree/disagree questions... I think the very first one is already addressed in the earlier question about why they left, and later in the agree/disagree form when we ask them if they would come back when their personal life permits it. I'd just drop it and shorten it. But if there's room, I'd also like to know if people would agree/disagree that "the community is too lenient on disruptive editors". Just a thought.

You also have some numbering issues. But I used the numbers I saw in the current survey.

All in all it's very good though. We did a great job, IMO.

Randomran04:02, 29 December 2009
 

So there are a few issues that I haven't gotten around to with Limesurvey (our survey tool):

  • Numbering -- I just need to get around to this.
  • "Check all that apply" -- I don't remember if I've mentioned this before, but this is the default text that Limesurvey uses for multiple choice options is "Check all that apply." I haven't found an easy way to suppress this, but I will do a little more digging tonight.
  • Regarding the "Other" option -- yes, I'll need to make sure we have these in the final version where it's appropriate. There is another issue, however, with Limewire which affects the questions with multiple "yes/no" answers. Limesurvey defaults "other + <please specify free text>" as the last option and I don't think there is a way to change this. So for Yes/No questions, the answer choices may be "Yes 1, Yes 2, Yes 3. . .No, Other (please specify). . ." where what we really want is "Yes 1, Yes 2, Yes 3, Yes, Other (please specify), No." I'm not sure there's way around this with Limewire. The only way (I think) we can have "other" grouped with the "yes" answers is if the "other" choice doesn't have the "please specify" field. (Sorry if this wasn't the best explanation).

In response to your other points:

  • Q3: I know the response buckets are a bit broad, but we're trying to keep this consistent with Ed Chi's (researcher from PARC) buckets.
  • Q6: This question has been a tricky one to get right. I see your point -- the question is phrased as a yes/no. What if we changed the phrasing to ". . .how much missing or incorrect content. . ."? This doesn't help us unearth the personal motivation aspect, but Q5 (Why did you stop contributing to Wikipedia?) should help us with that.
  • Q8: Agreed. I think the phrasing you used above is really good -- eliminates possibility of a user selecting this choice because they had difficulties with the interface. I may be splitting hairs, but the subtle change is a good one.

Thanks again for the thoughts. The changes should be up on the wiki page in an hour or so.

Howie

Howief02:36, 30 December 2009
 

Hey Howie,

I tweaked some wording for clarity. I also added an answer about feedback and appreciation (focusing specifically on the "reward" hypothesis) as a criticism of the community.

I also took a crack at question four, so it wasn't such a no-brainer. Everyone will say that Wikipedia has a lot of missing and incorrect content. Maybe if we focus on "important" content we'll get a more interesting answer. (I'm still not fully happy with this question. I'd really like to find out if we lost an editor who could have been really helpful, or an editor that had more or less hit their peak. Not sure how to phrase it.)

Everything else looks pretty much solid though.

Getting the "check the top three answers" is key, though. Very important. If we have to slip "select three" into the actual question phrasing, then do it, even if it looks sloppy.

Randomran03:51, 30 December 2009