Review Rating System

[slaveofconvention]

There have been some discussions among the staff in recent months about our current "out of 5" rating system. Some want to leave it the way it is because, frankly, it seems to work, and some want to move to an "out of 10" system to give a wider range of scores for products.

One example where this might have been useful was in our recent review of the InWin Buc (http://www.thebestcasescenario.com/frontpage/?q=in-win-BUC-review)case - we awarded the case a solid 4/5 based on our initial review, but have since had some additional information which casts a slightly better light on the case. It isn't enough to push it all the way to a perfect 5/5 score but if we'd been working on an out of 10 system, it MIGHT have pushed it from an 8 to a 9....

We're running a facebook poll on our new fan page so please, head on over and give us your vote (and click on that "like" button if you didn't already - you DO like us, right??)....

http://www.facebook.com/pages/TheBestCaseScenariocom/144072398999501

[Ichbin]

You've nailed the head on rating.

I would keep the X/5, only because you'll find that you'll argue yourself in some instance on the exact score. And next thing you'll find yourself wondering if going to a metacritic scoring system is better (x/100)

If you want more "Detail" with the rating system, split up the general category. For instance...

General : x/5
Design : x/5
Construction : x/5
Modability : x/5

(Or something along those lines)

If it were me personally, I'd just say "This has or doesn't have the TBCS stamp of approval!"

But that's just me :)

[SXRguyinMA]

I says the out of 10 would work better, as I had some issues with that BUC. It is a VERY well rounded case with lots of nice features, but there's only a couple minor things that keep it from getting a full 5/5. If we were on a 10 scale it'd definitely get a 9/10 or possibly even a 9.5/10.

[Fuganater]

Switch to 10. You can be much more critical than you can with only 5 points.

[AmEv]

And add points.

Like 4.5, 3.7.....

maybe not......

[slaveofconvention]

ok so we're talking about going to out of 10 - and SXR starts talking about 9.5 which REALLY is a score out of 20 - then AmEv is talking about 1 decimal place, which effectively makes it a score out of 100!!!

5 or 10 guys lol 5 or 10 :P

[SXRguyinMA]

well if we do categories like stated above:

Aesthetics - 8/10
Ease of use - 9/10
Cable management - 7/10
Features - 10/10

Total - 8.5/10 :D

[slaveofconvention]

And who do you think gets to re-do all the rating logos with 20 new ones to cover 0-10 with 0.5 increments? lol

[SXRguyinMA]

better get crackin there then kiddo :)

[OvRiDe]

The only problem I have with Category scores is that not all items have the same criteria/categories to compare. In a lot of ways I am with Ichbin.. Its good or its bad, but I think people and the manufacturers would like to see an actual rating. It looks like there is a legitimate need to go to a 10 point rating system, but I think segmenting it beyond that is overkill.

Colin .. no worries on redoing them.. I have them in a template already. :)

[Oneslowz28]

I like the 1-10 as well. I went with the 1-5 in the beginning because it was sort of already in place from previous editors. But as SXR showed, there is a real need for us to bump it up a little. There are many products I would have rated a 5 but should have been a 9 to reflect some issues I noticed.

[RogueOpportunist]

Personally I don't even look at ratings, I find the whole idea behind reviewing something is to peel back the layers, show people the nitty gritty, try to expose any unseen flaws that couldn't be exposed from looking at a product brochure and once thoroughly fleshed out the reviewer then gives their overall opinion and the readers then decide for themselves whether the product is good or bad.

I understand the ratings systems aren't going anywhere but by that same token I really don't think it matters in the slightest whether it is out of 5 stars or 10, the target audience that is going to pay attention to things like that aren't likely to care if your rating is accurate down to the Nth degree and if anything the people that pay attention to things like star ratings are doing so because they want a "simple" summary of all that was said.

Hate - Dislike - Like - Like Very Much - Love

When it comes to rating systems that is the standard everyone recognizes to some extent or other and while I can definitely sympathize with the reviewers who desire more range to express themselves in their ratings it should also be considered that the reviews aren't written for the reviewers, they are written for the reader and as such the logical course of action would be to maintain whatever standard the reader is most familiar with.

This is all just my opinion though, take it with a grain of salt.



Just as an added thought the "flaw" in 5 grade ranking systems comes about when people impose perfection on a 5 star rating, just because an item has 1 or 2 things wrong with it doesn't mean it isn't deserving of a 5 star rating, nothing is perfect and no product is ever regarded the same by all people who buy it so having your 5th star reserved for perfection or even near perfection leads to a flawed scale where any and all results are going to be skewed to the lower end by having a higher end that is almost unreachable.

For the sake of accuracy IF your top end is indeed going to be reserved for near perfection then the skewing effect that has would be at least somewhat mitigated by a larger scale and instead of only 1 in 100 reviews getting 5 stars you could have 30 in 100 at least getting 9 stars so while 10 stars would still be nearly unobtainable the bell curve of your average ratings would at least span a range close to that maximum as opposed to dropping off a cliff after 4.

[AmEv]

Or was I talking about 50???????


Seriously. 10-scale sounds good.

[SXRguyinMA]

Rogue - the manufacturer's like to see it. I for one don't pay much attention to the ratings either, but if you look at Manufacturer X's individual product page that lists their reviews, 99% of them have Review Sit X's rating image there somewhere. It's more important to them than to Joe Consumer :D

[Kayin]

I like to 10 for leeway purposes, but honestly, if they want "better scores", give us better stuff. I want boards like the Shuttle AN35N Ultra. 35 bucks and 250 FSB 1:1 with a simple volt mod. Serious engineering without a serious price tag. I finally moved to server hardware to get some real stability. Tired of things flaking out or catching fire. Or exploding, like my last Asus.

When I started, there were a lot fewer choices, but they were more solidly put together. You want a case built to put a few fans and hard drives in? Chieftec Dragon, or a Lian-Li. Maybe an early CoolerMaster. You want to do something spectacular? Dremel it. Now we have all these plasticky premods out there and it's more to choose from, sure, but not as much innovation in the average day-to-day build. Some people choose to do more, but increasingly it's either just bolt on parts to it (throw money at it) or simply start from scratch. The last category is where the real innovation is coming in, but even so someone tries to rush a product to market similar and stuffs it up almost every time.

I purchased a rather expensive reservoir a while back (Typhoon III.) It had a huge parting seam and horribly sloppy weld line all around the back and sides that was visible from all angles. It also leaked. This was a well-known problem with the reservoir, and after a few revisions it was finally scrapped. Its replacement, the Monsoosn, is everything the original set out to be, but how long did it take to finally get there? Years. Why did we have to suffer for that long? Manufacturers cheaping out on materials and process, not poor design-I know Geno, and he has great ideas and he fixed his own flaws every time he had the chance. It took changing companies (go go Danger Den) to get him somewhere. It's getting to where everything is multiple revisions to get it right (I'm looking at you, P67) and even 1-10 isn't gonna help that.

Release less, get it right, and we'll be more willing to buy. Or at least I will.

I'm getting more vitriolic in my old age. But that's one I've sat on for a while.

[slaveofconvention]

So do I - I just wanted to moan a bit :P (have templates that is heh)

[Ichbin]

The only problem I have with Category scores is that not all items have the same criteria/categories to compare. In a lot of ways I am with Ichbin.. Its good or its bad, but I think people and the manufacturers would like to see an actual rating. It looks like there is a legitimate need to go to a 10 point rating system, but I think segmenting it beyond that is overkill.

Colin .. no worries on redoing them.. I have them in a template already. :)


When I did reviews for my (Now dead) business on games, I found myself at the same crossroads.

X/10 works fine. I used X/10. Something that needs to be taken into consideration is the writers ability to bring more detail to the table because of it. Explaining why you chose 4/5 is easier than explaining the difference of 8 and a 9 (In abstract terms, Immah pop you in the mouth if you say "EASY, ITS OFF BY A DIGIT!" derp derp derp:hurt:)


Just playing Devils Advocate :D

[RogueOpportunist]

Rogue - the manufacturer's like to see it. I for one don't pay much attention to the ratings either, but if you look at Manufacturer X's individual product page that lists their reviews, 99% of them have Review Sit X's rating image there somewhere. It's more important to them than to Joe Consumer :D

Yeah but that's kinda my point, for what the rating systems are used for 5 point accuracy versus 10 point accuracy really doesn't matter in the long run, either way I think moving from 5 to 10 wouldn't really "displease" anyone but seems as though it would definitely please some people writing the reviews so I guess there's nothing to lose and something to gain from having a larger scale.