Putting "Putting Wikipedia to the test: a case study" to the test

Posted on

A fairly small study on the quality of Wikipedia called Putting Wikipedia to the test: a case study seems to have been electronically published from Brisbane, Australia in 2008.

The study looked on just three medical topics:

and did a blinded comparison with three other online services

For UpToDate the researchers aggregated separated articles into one. For Wikipedia non-scientific sections were omitted, e.g., history and movie characters with multiple sclerosis.

Three medical academics evaluated each four of the 12 articles. They were blinded and ranked the content for accuracy, coverage, concision, currency and overall suitability for medical students. Medical librarians (unblinded presumably) assessed accessibility and usability (cost and login requirement) “ease of finding and navigating the information” and “the quality of presentation”.

Results are summarized in their Table 3 with AccessMedicine generally scoring the best (39 points) followed with some distance by UpToDate (32 points), Wikipedia (30 points) and eMedicine (29).

My critique is:

  1. Overall well-carried out study with blinding.
  2. Very small study with only three topics evaluated. This makes the study one of the smallest I have ever seen, and questions how much we can generalize from the study.
  3. The researchers summarize their results on Wikipedia with “relatively concise and current, but failed to cover key aspects of two topics and contained some factual errors. Each reviewer deemed it unsuitable as a learning resource for medical students”. It should be noted that for the Otitis media article both the eMedicine and the AccessMedicine article were deemed unsuitable. Also the eMedicine’s ‘Multiple sclerosis’ article were deemed unsuitable. UpToDate’s ‘Otitis Media’ was only “generally suitable but with some limitations.” This leaves the medical student with no really good online resource for ‘Otitis Media’.
  4. Factual errors were reported for 5 of the 12 articles (these numbers include 2 of the 3 Wikipedia articles). It is not reported what precisely these factual errors were.
  5. I find it interesting that they find “navigating around the information was simpler in UpToDate and eMedicine than in Wikipedia”. I wonder what that is due to. This is not explained.

The study points to two other recent Wikipedia quality studies Scope,
completeness, and accuracy of drug information in Wikipedia
and Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference.

(Correction 16:48, 2011-01-17, 18 & 19 other minor corrections)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s