Thursday, December 30, 2004

MetaCritic 0-100 Rating Scale

Just discovered this explanation of the MetaCritic.com scoring system which uses a 0-100 rating scale. Of particular relevance to this blog is the "incredibly detailed version" of the explanation, detailing how a variety of alternative rating/review systems can be mapped on to their 0-100 rating scale.

A symmetric rating system has been adopted by MetaCritic.com, so 50 represents an average rating/review as indicated below:

81-100 Universal Acclaim
61-80 Generally Favorable Reviews
40-60 Mixed or Average Reviews
20-39 Generally Unfavorable Reviews
0-19 Overwhelming Dislike

This is similar to the system proposed by this blog:

81-100 Excellent
61-80 Good
41-60 Middling
21-40 Poor
1-20 Terrible

...with a couple of exceptions.

Firstly, it has previously been suggested that a score of 0 (zero) be used to represent Unrated/Not Rated; this is similar to MetaCritic's "Score Pending" or "XX". From their website:

"To help ensure that METASCORES accurately reflect the reviews given by critics for any particular movie, game, or album, we no longer display a METASCORE for those items that do not have a minimum number of reviews. Once the minimum number of reviews is reached, the METASCORE will display.

Items that do not have a METASCORE are marked either "Score Pending" or "XX" depending on the space available. These scores (or non-scores) are given a neutral color to further distinguish them, so they appear like this: xx ."

By using a score of 0 (zero) it would be possible to represent Unrated, Not Rated, Score Pending or XX in RSS feeds with a simple numerical value. I am not sure if MetaCritic can or do represent this in RSS feeds at the moment.

Also, a score of -1 could be used to represent that something was wrong with providing a rating for a particular aspect of a resource. This could be for a variety of reasons. For example, if a rater was providing a rating for a piece of music and one of the categories under which it was being rated was genre, but the rater considered the genre to be inappropriate, a rating value of -1 could signify this.

So, to expand a little, the piece of music may be good or excellent in all respects, but may have simply been incorrectly categorised. Providing a poor/terrible rating for the song under the genre category would, in this case, be inappropriate as it results from incorrect metadata and not the resource itself. It would distort subsequent analysis of user ratings, and would fail to signal that something may be wrong with the rating category. By enabling users to select a value outside normal rating ranges (-1 as a negative integer is not common in rating options) via a "wrong" option, rating systems could be alerted that something should be looked at/corrected.

So in summary, MetaCritic have adopted a 0-100 rating scale which is similar to that proposed by this blog. The 0 (zero) value could be used to signify “Score Pending” or “XX” in the MetaCritic system, and -1 could signify that for some reason something was wrong with providing a rating which could then be looked into/corrected.

This page is powered by Blogger. Isn't yours?