A wiki is a web site that allows any visitor to edit or extend its content; the term "wiki" (from a Hawaiian word meaning "quick" or "super-fast") is also used to refer to the software that enables such editing. Wikis were invented in 1995, but their popularity has really started to take off since 2000 or 2001, boosted partly by the success of Wikipiedia (http://www.wikipedia.org/), a wiki-based free-content encyclopedia. Think of a wiki as falling somewhere between IRC and web pages: wikis don't happen in real time, so people get a chance to ponder and polish their contributions, but they are also very easy to add to, involving less interface overhead than editing a regular web page.
Wikis are not yet standard equipment for open source projects, but they probably will be soon. As they are relatively new technology, and people are still experimenting with different ways of using them, I will just offer a few words of caution here—at this stage, it's easier to analyze misuses of wikis than to analyze their successes.
If you decide to run a wiki, put a lot of effort into having a clear page organization and pleasing visual layout, so that visitors (i.e., potential editors) will instinctively know how to fit in their contributions. Equally important, post those standards on the wiki itself, so people have somewhere to go for guidance. Too often, wiki administrators fall victim to the fantasy that because hordes of visitors are individually adding high quality content to the site, the sum of all these contributions must therefore also be of high quality. That's not how web sites work. Each individual page or paragraph may be good when considered by itself, but it will not be good if embedded in a disorganized or confusing whole. Too often, wikis suffer from:
A well-organized web site makes visitors feel like they know where they are at any time. For example, if the pages are well-designed, people can intuitively tell the difference between a "table of contents" region and a "content" region. Contributors to a wiki will respect such differences too, but only if the differences are present to begin with.
Wikis frequently end up with different pages saying similar things, because the individual contributors did not notice the duplications. This can be partly a consequence of the lack of navigational principles noted above, in that people may not find the duplicate content if it is not where they expect it to be.
To some degree this problem is inevitable when there are so many authors, but it can be lessened if there are written guidelines about how to create new content. It also helps to aggressively edit new contributions at the beginning, as an example, so that the standards start to sink in.
The common solution to all these problems is the same: have editorial standards, and demonstrate them not only by posting them, but by editing pages to adhere to them. In general, wikis will amplify any failings in their original material, since contributors imitate whatever patterns they see in front of them. Don't just set up the wiki and hope everything falls into place. You must also prime it with well-written content, so people have a template to follow.
The shining example of a well-run wiki is Wikipedia, though this may be partly because the content (encyclopedia entries) is naturally well suited to the wiki format. But if you examine Wikipedia closely, you'll see that its administrators laid a very thorough foundation for cooperation. There is extensive documentation on how to write new entries, how to maintain an appropriate point of view, what sorts of edits to make, what edits to avoid, a dispute resolution process for contested edits (involving several stages, including eventual arbitration), and so forth. They also have authorization controls, so that if a page is the target of repeated inappropriate edits, they can lock it down until the problem is resolved. In other words, they didn't just throw some templates onto a web site and hope for the best. Wikipedia works because its founders thought carefully about how to get thousands of strangers to tailor their writing to a common vision. While you may not need the same level of preparedness to run a wiki for a free software project, the spirit is worth emulating.
For more information about wikis, see http://en.wikipedia.org/wiki/Wiki. Also, the first wiki remains alive and well, and contains a lot of discussion about running wikis: see http://www.c2.com/cgi/wiki?WelcomeVisitors, http://www.c2.com/cgi/wiki?WhyWikiWorks, and http://www.c2.com/cgi/wiki?WhyWikiDoesntWork for various points of view.