Proposal:Distributed Infrastructure
Every proposal should be tied to one of the strategic priorities below.
Edit this page to help identify the priorities related to this proposal!
- Achieve continued growth in readership
- Focus on quality content
- Increase Participation
- Stabilize and improve the infrastructure
- Encourage Innovation
|
Share this:
|
|
This is a featured proposal. This template automatically categorizes into Category:Featured proposals. All proposals marked by this template should be on this list too. |
 |
It has been suggested that this page be merged with Proposal:Distributed backup of Wikimedia content. (Discuss) |
|
|
It has been suggested that this page be merged with Proposal:Distributed Wikipedia. (Discuss) |
Summary
The Wikimedia Foundation should sponsor software for 'Wikimedia At Home,' public service grid-computing software similar to Folding At Home,[1] but supporting Wikimedia.
Proposal
It should be possible to improve search, multiply storage and bandwidth by permitting individuals or organizations to host a standard software package to carry out the goals of the Wikimedia Foundation.
The techniques have been developed in isolation: Folding at Home proves that heterogeneous computing service is possible and valuable. Google has developed distributed search and storage. BitTorrent distributes bandwidth serving. Inferno is a small, portable OS that provides a general-purpose distributed computing infrastructure designed for easy, secure, central management. Wikimedia has proven that individuals will produce valuable media for public uses.
Motivation
It improves service
Quality and speed of search could be improved. A Google-style massively parallel search would be affordable, including full text searches, contextual searches, perhaps even spelling checks.
Speed of download could be improved. The most idle, best connected, etc. volunteer servers could deliver content in a massively parallel fashion. The result would be a dramatic improvement in bandwidth, and a dramatic widening in the types of media that would be affordable.
Uptime might increase, as well, if operations and storage can be sufficiently decentralized.
It organizes a new form of community service
It provides another directly helpful way that individuals and organizations can help the Wikimedia Foundation achieve its public service goals. This is, of course, a main thrust of Wikimedia: organizing voluntary public service.
Properly designed, it would enable organizations to directly donate idle net resources, for substantial tax advantages, while retaining use of their servers for their own peak bandwidth needs.
It distributes and leverages costs
A major cost of the Wikimedia Foundation is the maintenance of central servers, storage, backup and bandwidth. While still necessary, they could be demoted to providing a central, secured backup, rather than the first line of service. The rate of growth should decrease dramatically.
It also distributes and leverages technical vulnerabilities. For example, the foundation could focus resources on providing services rather than updating infrastructure.
Key Questions
Managing a Heterogenous environment
System management in particular must be extremely automated in order to be workable at all. There is, however, a compact, effective method: Inferno is a small (<1meg), public domain operating system designed to provide homogeneous system resources via heterogenous networks of heterogenous computers. It is fully open-sourced. It has an efficient, portable, public-domain computer language, standard APIs, and preexisting powerful methods for centralized administration and distributed storage systems, with automatic versioning. It's already hosted on PCs, under Linux, Windows, and others, as well as stand-alone on a variety of CPUs. It also has several large sponsors and users, including AT&T, IBM, a number of national labs, and a specialized consulting company that sells binaries, training, porting services and development.
Some in-flight entertainment systems provide versioned roll-out of code and media updates. Download is distributed over time, and switch-over occurs at a certain time. If Wikimedia used this model, improved code could flow from the central servers out to heterogenous servers without interrupting service. At a particular time, dual operation could begin, with servers providing both versions of a service. Shortly after, the old service could be retired. With a few controls, this provides coordinated roll-out and roll-back of new distributed services.
Algorithms
Grid computing algorithms can be difficult.
One solution is to develop and deploy a preexisting, thoroughly distributed infrastructure with centralized secure management (e.g. Inferno), then use it as a platform to host new services incrementally.
For example, first provide a grid platform, with a prototype of a crucial service, like a somewhat improved search service. Then incrementally roll out new services to the platform (e.g. distributed storage or download) as they become available and make sense.
At every stage, give resource owners control of their donation.
Applicability
BitTorrent is best suited to serving widely desired files. It's a question whether something like Wikipedia, with large amounts of slow-moving text, could be effectively hosted by a decentralized voluntary system of servers. However, I think that Google's decentralized search method indicates a -yes-.
Centralization and merging of edits is another challenge. However, several version control systems (CVS and SVN) can automatically merge lines. A merge of sectional edits seems possible. Flagging contradictory merges for editorial attention also seems possible.
Possibly Better Availability
Distribution of media could be peer-to-peer. A central coordination facility (BitTorrent calls it a "tracker") can probably help a great deal with balancing loads and storage among a heterogenous organization of servers.
Potential Costs
The main cost is developing the software. I think Wikimedia might be able to reduce continuing costs a lot with volunteers, interns and possibly consultants.
Also, the foundation must invest in a few technology experts. The best way is probably to take trusted, existing technical employees, several of them, and train them in any new technologies.
References
- Folding At Home- Describes an example of a grid software application for public service
- A possible contractor for Inferno Development, including developer training
- Wikipedia.en's Inferno article
- Google's Search Tech. Note that Wikimedia content is much more standardized, and inherently easier to search than the web.
- BitTorrent FAQ
- BitTorrent's Web Site
- Transdigital Communications Corporation in Brea, CA, Thales entertainment,
and Panasonic In-flight entertainment all have versioned roll out in some form.
Community Discussion
Do you have a thought about this proposal? A suggestion? Discuss this proposal by going to Proposal talk:Distributed Infrastructure.
Want to work on this proposal?
- .. Sign your name here!