Jump to content

Proposal:Discussion Interface Study

From Strategic Planning
Status (see valid statuses)

The status of this proposal is:
Request for Discussion / Sign-Ups

This proposal is associated with the bolded strategic priorities below.


  1. Achieve continued growth in readership
  2. Focus on quality content
  3. Increase Participation.
  4. Ensure that the underlying project infrastructure is secure, stable, and sufficient to guarantee the permanence of the projects and support ongoing growth.
  5. Encourage Innovation.


Summary

Run a study to gather scientific data about the affordances of different talk page interfaces. If the study is conducted properly, Wikimedia will gain valuable insight into the direction the MediaWiki platform should take for improving the constructiveness and accessibility of discussions. This proposal is being put forward by human computer interface researchers at the University of Washington, who are capable of developing the technical infrastructure of the study, as well as designing and conducting the study.

Motivation

A great deal of time, effort, and emotional energy in Wikimedia communities takes place on talk pages. Consequently the interface used shapes the experiences that many people have, everyday, in their hard work.

By default, talk pages are standard wiki pages. This interface has some excellent affordances, such as enabling anyone to refactor a conversation, but it does raise the barrier to entry. The LiquidThreads extension introduces structured discussions to the MediaWiki platform, potentially making it easier to contribute to a discussion.

While it is easy to speculate on the tradeoffs of different interfaces, it is difficult to make good decisions without proof. The purpose of the proposed study is to generate solid empirical data that can inform how the discussion interface of MediaWiki is developed in the future. It can also serve to justify decisions that Wikimedia makes with respect to the discussion interface.

Proposal

The study is aimed at testing hypotheses about how different discussion interfaces impact large discussions. In particular, we propose to conduct two separate, simultaneous discussions on a single topic of interest, over a period of two weeks. Each of the discussions will use a different interface. We will then be able to evaluate the differences between the two conditions.

What follows is an outline of a study plan, with some details omitted.


Proposed Discussion Topic

We propose to base both discussions around the following question, which concerns the role of social media features (e.g avatars, userpages/userboxes, a "thanks" button on posts) in the MediaWiki software platform:

What social features should be incorporated into Mediawiki to help foster an effective and engaging work environment for volunteers, without distracting from the core mission?


Study Phase One (Weeks 1-2)

  1. Conditions. There are two interfaces we are proposing to test:
    1. MediaWiki+LiquidThreads: The basic installation of LiquidThreads
    2. MediaWiki+LiquidThreads+Reflect: Reflect is a tweak to online forums. It helps close the feedback loop between speakers and listeners by introducing a space for bulleted summaries next to every comment in a thread. Reflect can help speakers figure out if they are being heard and understood, and is also an intermediate step toward discussion summarization. It is available as a MediaWiki extension, with a dependency on LiquidThreads (see Figure 1). Disclosure: the university researchers designed Reflect, and want to evaluate whether it helps discussions.
      Figure 1: LiquidThreads + Reflect integration. User is mousing over a summary bullet point. Click to see fullsize.
  2. Discussion topic. An issue of current concern for the Wikimedia community is chosen. Something that people care about. It could be something specific to a single community, or of broader concern to the whole Wikimedia ecosystem (e.g. a specific proposal that addresses retention of contributors). Regardless, the topic must interest enough people to participate. On the order of 15-20 people per condition.
  3. Recruiting. We recruit people through a number channels (e.g. Village Pump, mailing lists, wikisignpost, Facebook groups). We describe that the results of the study will inform the development of the MediaWiki discussion interface. We also let people know that the discussion will be available after the study is complete for anyone to see. We will want to try to recruit a mix of hardcore and light Wikimedia participants.
  4. Assignment. After someone volunteers to take part, they are assigned to one of the two discussion conditions. We will not force participants to stick within their condition, but we will ask them to do so. They are invited to contribute as much as they want for the duration of the study. Participants in a given condition will also be given a special URL that they are invited to share with other people who might want to take part.


Study Phase Two (Week 3-?)

This phase is intended to allow broader participation in the discussion of social media features in Mediawiki. We will remove all of the study components (except for the in-situ surveys) and open the discussion up to anyone who wishes to participate, with no sign-up required. The discussion interface for this phase will be LiquidThreads+Reflect. For this phase, we could either a)start the conversation fresh on a new talk page, with links to the two previous, closed discussions, or b)include all threads from the two existing discussions and allow new participants the choice of either contributing to those threads or start new ones.

Hypotheses

We will need to develop a set of focused hypotheses that answer important questions. The university researchers have spent the most time considering hypotheses around Reflect's impact on discourse, so it will be worthwhile for interested people to help develop hypotheses about baseline MediaWiki and LiquidThreads. To give a sample of the possible hypotheses:

  • LiquidThreads is easier to learn, and less burdensome over the whole discussion, because participants do not need to wade through much WikiText.
  • LiquidThreads reduces potential barriers to contribution by a wider range of editors by providing interactional cues that are similar to other threaded comment boards.
  • Reflect improves the ability of participants find important points in a large discussion by scanning the summary bullet points that others have written.
  • Reflect improves the probability that a speaker feels that they are being heard and understood by enabling other people to provide evidence of listening (summaries) that may reveal flaws in their understanding of the original statement. One consequence may be a reduction in redundant posts or flaming.

Methods

We can use a variety of methods to provide evidence for proving or disproving our hypotheses.

  1. Instrumentation. We can collect behavioral data like mouse clicks and hovers. This can help us understand where user attention is. We can also log how often people return to the discussion.
  2. In-situ Surveys. We can ask participants periodic short survey questions about something that has just happened in the discussion (for instance, why they just posted a comment, or what their reaction is to a reply that someone has made to their comment). Participants can be automatically prompted to answer these questions on a random basis, or every n times they perform a particular action (such as leaving a comment).
  3. Extended Surveys+interviews. During the course of the study, we may ask participants short questions about their experience (e.g. what motivated them to leave or respond to certain comments or contributors?). After the study, we can also interview a sample of participants from each condition.
  4. Conversation analysis. We can analyze the content and structure of the threaded comments in order to understand how people are using LiquidThreads and Reflect as well as certain characteristics of the discussions (such as how responsive, civil and interactive people are). This may help us understand whether the affordances of the different technologies affect the content and tone of people's comments. For example, we can measure the (hopefully low!) prevalence of personal attacks across the different discussions, as well as the degree to which people are responding to one another's comments (rather than simply expressing their own opinion) across the base MediaWiki/LiquidThreads/LiquidThreads+Reflect conditions.

This is a simple sketch of a study. There are lots of details to work out. But the researchers (and yes, I'm speaking in the third person) are very motivated to carry this project forward.

Potential Costs

The majority of the proposed work will be conducted by University of Washington researchers. The researchers are funded by a National Science Foundation grant. The MediaWiki installations would be hosted on a Wikimedia domain, which should take a minimal amount of work to set up and administer.

However, recruitment through multiple channels will likely take a bit of work. Wikimedians who are interested in helping out with recruitment should contact Travis Kriplean or Jonathan Morgan to find out how they can contribute, or to share possible recruitment strategies.



Community Discussion

Do you have a thought about this proposal? A suggestion? Discuss this proposal by going to Proposal talk:Discussion Interface Study.

Want to work on this proposal?

  1. User:tkriplean
  2. Jtmorgan 18:35, 3 July 2010 (UTC)
  3. Eekim 05:30, 8 July 2010 (UTC)
Retrieved from "https://strategy.wikimedia.org/w/index.php?title=Proposal:Discussion_Interface_Study&oldid=73849"