Why Wine Ratings are Badly Flawed

[Red Bigot]

Article in the Wall Street Journal, including some properly documented research on show judging and correlation of medals won by the same wines across various shows.
http://online.wsj.com/article/SB1000142 ... 53628.html

There has been some similar research on published show results in Australia, with I believe comparable findings, but they have not been published.

[daver6]

Thanks...very interesting read. A couple of things that i thought of were:

* With regards to different judges using different descriptors for the same wine, my understanding is really a descriptor, eg leather is just one persons association of a certain aroma with a word. So while two different people might associate different descriptors with the same aroma, it would be interesting to see if the same judge used to same descriptor for the same aroma in another wine.

* The variation of scores between the same wine being judged 3 times at different time. Surely the variation could also be put down to how long the wine has had to breathe?

Still, my belief is a rating is just one persons opinion and should be used as just that, not gospell.

david

[Rawshack]

I deplore the 100 point system and it saddens me greatly that we all in this country have come to rely upon it so much.

It beggars belief that we can attempt to be so exact with something so subjective, so passionate, as good wine.

I'm sure many people use a sliding scale with the basic formula of low price + high points = purchase

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

[Michael McNally]

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

G'day Rawshack

I would recommend that you read the note and get an appreciation for the wine and whether or not you might like it. Then, ignore completely (or take a spoonful of salt with) the score. Put your thumb over it if it distracts you.

Shame to throw the baby out with the bathwater. Even if the bathwater scores 96.

Cheers

Michael

[orpheus]

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

G'day Rawshack

I would recommend that you read the note and get an appreciation for the wine and whether or not you might like it. Then, ignore completely (or take a spoonful of salt with) the score. Put your thumb over it if it distracts you.

Shame to throw the baby out with the bathwater. Even if the bathwater scores 96.

Cheers

Michael

Michael, the spoonful of salt may be detrimental to your palate.

Of course, the article calls into question more than just scores. Descriptions also seem to be very variable.

It may be a problem with the wine show setting; too many wines, hence jaded palate.

Another clearly observable phenomenon is that each wine critic has their own biases in terms of descriptors.

Descriptors run the risk of being boring and unhelpful. If someone lists, in a tasting note, a kaleidoscope of flavours which immediately brings to my mind my kitchen pantry, I really don't have too much of a sense of what the wine tastes like. I am likely to describe it differently.

At the same time, I enjoy reading Halliday's notes, and tend to find him reasonably consistent. I can usually tell if I will like a wine from his notes. (I will value some qualities more highly than he, and some less, but I know where I stand.)

[jeremy]

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

G'day Rawshack

I would recommend that you read the note and get an appreciation for the wine and whether or not you might like it. Then, ignore completely (or take a spoonful of salt with) the score. Put your thumb over it if it distracts you.

Shame to throw the baby out with the bathwater. Even if the bathwater scores 96.

Cheers

Michael

Michael, the spoonful of salt may be detrimental to your palate.

Of course, the article calls into question more than just scores. Descriptions also seem to be very variable.

It may be a problem with the wine show setting; too many wines, hence jaded palate.

Another clearly observable phenomenon is that each wine critic has their own biases in terms of descriptors.

Descriptors run the risk of being boring and unhelpful. If someone lists, in a tasting note, a kaleidoscope of flavours which immediately brings to my mind my kitchen pantry, I really don't have too much of a sense of what the wine tastes like. I am likely to describe it differently.

At the same time, I enjoy reading Halliday's notes, and tend to find him reasonably consistent. I can usually tell if I will like a wine from his notes. (I will value some qualities more highly than he, and some less, but I know where I stand.)

Halliday is excellent, but his tasting notes often leave much to be desired as far as I'm concerned. The lurking variable is that most of us are very familiar with JH, thus we know "how" to read him. This lets him write far less descriptive notes whilst retaining a degree of usefulness. But that's not what's it is all about. And that is why I think you should famialirise yourself with other writers. I'd hate to rely on JH when it comes to descriptions of Sangiovese or Nebbiolo. He just doesn't seem interested enough to put in the effort. Hardly a floor when you consider all he has done. But pertinent none the less.

[Rawshack]

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

G'day Rawshack

I would recommend that you read the note and get an appreciation for the wine and whether or not you might like it. Then, ignore completely (or take a spoonful of salt with) the score. Put your thumb over it if it distracts you.

Shame to throw the baby out with the bathwater. Even if the bathwater scores 96.

Cheers

Michael

Well Michael, thank you for you recommendation, it's certainly something that I have never considered before. Still, many hands make light work. The problem for me, Michael, is that perhaps the words are only the Watson to the scores Holmes (I hope you appreciate the literary reference at this time of the morning).

Still, this debate could go on and on and frankly, we've probably both got better things to do

[Mandingo]

Perhaps simplifying the descriptors will provide some assistance for the punters.

The questions I usually ask are: (for a cheaper wine) Were the grapes ripe when they were picked (rather than over ripe or under ripe), what does it feel like in the mouth, how does it finish and will I be wanting to reach for for another glass. (for a more expensive bottle) the same questions with the addition of do I want to keep my nose in the glass all night, are the tannins ripe and the acid in balance and does it have that X factor that sticks in the memory beyond the life of the drunk bottle.

One American winemaker in his book describes a monthly lunch organised for winemakers - nobody waxes lyrical about the wines, but when one of them stands up and states "that is great wine, I wish I'd made it!" the others take note. A little more of this pragmatic approach could be helpful with some wine writers - a bit of interest in style is also welcome (thanks CM and GW).

Like those before, I would not want to throw the baby out with the bathwater as there is too little time and money to be trying all of the wines being released. I will keep reading the reviews, trying the wines and making up my own mine.

Cheers
John

[Tristram Shandy]

The love using numbers in a rating system derives from the advent of modern technology through the use of the scientific method. This was fairly straightforward concept when applied to most scientific and enginneering pursuits. It's relatively simple to measure right and wrong in a mathematics exam or standards for an engineering project, but this becomes more problematic when judging an English essay or a musical performance or a work of art. But because we've all been so impressed by the advance of technology, we have tried to borrow some of the means used to achieve these results. 100 points sounds so precise and scientific, but as this discussion has indicated, it is very difficult to communicate the subjective experiences of smell and taste. In fact, philosophers have been arguing for decades about the nature of this physicality and whether or not it can be said to have any objective validity, or more importantly, any real discursive value.

I think that taste and smell descriptors can be useful pointers. The points system, however, is a waste of time - a delusion at best. Simple verbal categories such as "not recommended", "recommended", "highly recommended", "excellent" and "outstanding" would probably carry sufficient meaning for most of us: a more serviceable and honest assessment, with descriptors playing a supporting role. While each category need not be defined, a reasonable agreement for what "recommended" might mean could probably be reached.

After that, it's really up to the individual judge or critic.

The only other factor that might be of interest is QPR. We are all attracted to good value for money.

[orpheus]

Perhaps simplifying the descriptors will provide some assistance for the punters.

The questions I usually ask are: (for a cheaper wine) Were the grapes ripe when they were picked (rather than over ripe or under ripe), what does it feel like in the mouth, how does it finish and will I be wanting to reach for for another glass. (for a more expensive bottle) the same questions with the addition of do I want to keep my nose in the glass all night, are the tannins ripe and the acid in balance and does it have that X factor that sticks in the memory beyond the life of the drunk bottle.

One American winemaker in his book describes a monthly lunch organised for winemakers - nobody waxes lyrical about the wines, but when one of them stands up and states "that is great wine, I wish I'd made it!" the others take note. A little more of this pragmatic approach could be helpful with some wine writers - a bit of interest in style is also welcome (thanks CM and GW).

Like those before, I would not want to throw the baby out with the bathwater as there is too little time and money to be trying all of the wines being released. I will keep reading the reviews, trying the wines and making up my own mine.

Cheers
John

John, this approach is an excellent one, I think.

[Craig(NZ)]

Jeez, ive never read a thread on this topic before!

pass me today's news

[Red]

Highlights to me the validity of the approach that critics like Mattinson and Walsh take, where they taste a wine over the course of a couple of days. While still imperfect (doesn't overcome some of the other factors the article highlights), its more likely that you'll at some point taste the wine at its best, rather than being influenced by tasting the wine at one point in time when any number of factors may affect your taste buds.

Red

[David]

I deplore the 100 point system and it saddens me greatly that we all in this country have come to rely upon it so much.

It beggars belief that we can attempt to be so exact with something so subjective, so passionate, as good wine.

I'm sure many people use a sliding scale with the basic formula of low price + high points = purchase

Frankly, it depresses the hell out of me. Hard to find any reviews now of wine that don't involve the demonic 100 points. Perhaps I should stop reading reviews full stop.

Couldn't have said it better myself. I am a great fan how the late Len Evans wrote reviews. It has occurred to me that the 100 point system and complicated tasting notes are a bit of an ego trip. Of course, I do not refer to any contributers here. I think I know a good wine when I see it but as to saying its 93 point and not 94 or 92 im buggered if i could tell.

[odyssey]

With the ratings system being so clinical and all, I think reviews should be more subjective rather than less subjective. Surely wine writers can't really be everything for everybody, after all many wines have bipolar responses from the community (eg. recently 2007 Wynns BL). If all wine writers love it but I dislike it, then that's personal preference. I'm not going to buy it just because someone else tells me that I am supposed to like it, when I don't. More power to subjectivity.

Good on Halliday for admitting his dislike for Nebs, for example. All the better that he admits it, then at least you know what you are getting. I mean how could you trust a wine writer that doesn't admit to not truly appreciating the finer points of a style they dislike? If I loved Nebs and wanted a review I'd simply ask a different person.

Noone expects movie reviewers, for example, to eliminate subjectivity in their movie reviews. When you watch "At The Movies", each reviewer has their own preference of movie styles and each rate differently, and then you take it or leave it. Disagreement is actively encouraged - you go spend 2 hours and $18 watching a movie and then decide whether you dislike it, like it, love it, or you want the 2 hours of your life and $18 back. Why not also in the wine world?

Anyway, some like Burgundy more than Bordeaux, Red more than White, or Chardonnay more than Sauv Blanc. Good on em. Wine writers are human and have preferences too.

[Polymer]

I don't know if I'd call ratings flawed...If you're expecting them to make wine objective then yes it is....

I personally look at scores as one person's opinion about a specific wine at that point in time. Based on what wine I've had before that was also reviewed by that person, it gives me a decent idea of what I can expect from wine I've never had the pleasure of trying yet myself. When I taste a wine, I could care less what it scored, what other people thought of it or their description of it...It doesn't matter. None of those things should ever matter to anyone but those things helped me at least narrow down my buying decision. Does the good rating/good review of a new wine I've never had attract my attention? Sure..especially if it's a reviewer that I tend to agree with. Why not take their experience and use it to help me increase the number of good "buys" that I have vs. the number of bad ones? I don't like buying junky wine more than anyone else and if using these reviews helps me avoid some of them, it's a good thing.

This does NOT mean I will always like a wine that someone rated highly but I definitely feel it helps me find good wine more often than not...

[orpheus]

Jeez, ive never read a thread on this topic before!

pass me today's news

Craig, this criticism could be leveled against most threads on a wine forum (or just about any other enthusiast's forum).

It is not compulsory for you to read it

[Hindmarsh]

[quote="Polymer"]I don't know if I'd call ratings flawed...If you're expecting them to make wine objective then yes it is....

I personally look at scores as one person's opinion about a specific wine at that point in time. Based on what wine I've had before that was also reviewed by that person, it gives me a decent idea of what I can expect from wine I've never had the pleasure of trying yet myself. When I taste a wine, I could care less what it scored, what other people thought of it or their description of it...It doesn't matter. None of those things should ever matter to anyone but those things helped me at least narrow down my buying decision. Does the good rating/good review of a new wine I've never had attract my attention? Sure..especially if it's a reviewer that I tend to agree with. Why not take their experience and use it to help me increase the number of good "buys" that I have vs. the number of bad ones? I don't like buying junky wine more than anyone else and if using these reviews helps me avoid some of them, it's a good thing.

This does NOT mean I will always like a wine that someone rated highly but I definitely feel it helps me find good wine more often than not...[/quote]


Very well said. I agree with lots of that.

While I am not a fan of tasting notes that have 17 descriptors, half of which are flavours / scents about .085% of the population will ever be exposed to (or be able to discern), a highly rated wine with effective tasting notes can be a decent guide to find good wines - especially if there is a modicum of consensus amongst wine critics, and most importantly if you like it!!

You can get a sense of which critic likes what wine style (eg the mention of Hallidays seeming aversion to Nebiolo), and the differences in the ratings for certain critics. It is not a precise science and the article from WSJ (which is excellent) really proves how subjective and inconsistent wine reviews can be (which one of the beauties of wine as opposed to other alcoholic drinks!)

[Red Bigot]

Can you believe Ringbolt Margaret River Cabernet 2007 is the best Cabernet 2007 and older in Australia? Well it was judged best of all those entered in the prestigious Cabernet Trophy class at the National Wine Show. Most of the red trophies went to Western Australian wines, with Langi Shiraz 2004 taking the best of the Museum classes.
A clear signal to South Australia? Or just too warm during the judging for SA reds to show their best (NWS venue only has evaporative cooling). Randomness in action? Or...?
List of trophies here: http://www.ozwinereview.com/

[Craig(NZ)]

Craig, this criticism could be leveled against most threads on a wine forum (or just about any other enthusiast's forum).

It is not compulsory for you to read it

no, you miss the point i was making. surely we all know by now that rating /100 and any objective attempt to rank wines in any forum by any method it flawed to buggary. This is wine 101, accept it and move on.

surely we also know that although imperfect it does have its uses (baby with bathwater used above is perfect). accept that and move on too

it is the universe we are in. Maybe when we all get to heaven, the dear lord will have the perfect system, all our palates will be the same, there will be no bottle variation or any other bias. we can all buy a book on heavens 100 greatest wines and never need to argue its contents. We will be all able to count the number of top 10 wines we own and puff our chest out when everyone murmours about how clever we were to obtain them. But on this mortal coil its never gonna be that way.

[Rawshack]

Craig, this criticism could be leveled against most threads on a wine forum (or just about any other enthusiast's forum).

It is not compulsory for you to read it

no, you miss the point i was making. surely we all know by now that rating /100 and any objective attempt to rank wines in any forum by any method it flawed to buggary. This is wine 101, accept it and move on.

surely we also know that although imperfect it does have its uses (baby with bathwater used above is perfect). accept that and move on too

it is the universe we are in. Maybe when we all get to heaven, the dear lord will have the perfect system, all our palates will be the same, there will be no bottle variation or any other bias. we can all buy a book on heavens 100 greatest wines and never need to argue its contents. We will be all able to count the number of top 10 wines we own and puff our chest out when everyone murmours about how clever we were to obtain them. But on this mortal coil its never gonna be that way.

Don't agree, if people are passionate about an issue, it's going to get raised again and again, whether like it or not. The 100 system is obviously something that generates debate. Besides, I've declared war on them (the 100 pt system) and won't rest until it's rotting corpse has been thrown on a mangy old tip, along with Parker to keep it company

My main gripe; and someone else said this far more eloquently than I ever can, is that good wine is a magical juxtaposition of a myriad of factors. I know that sounds a bit wank, but it's Friday, and I haven't had nearly enough coffee yet. With the 100 pt system we're applying a rigid, scientific approach to something that clearly isn't and never should be. As a society we want nice, digestible bite size pieces that we can regurgitate in an instant. It seems that with everything, current affairs, news, sport, entertainment, we want to know a bit of everything, rather than actually understand something and the same is happening with wine reviews.

I don't care whether I repeat myself on this - or any other subject - or not. I came to the realisation a long time ago that I am as dull as Sauvignon Blanc (only kidding) and have a habit of carping on about the same old shit.

[Craig(NZ)]

Besides, I've declared war on them (the 100 pt system) and won't rest until it's rotting corpse has been thrown on a mangy old tip, along with Parker to keep it company

I wish you all good fortune in your quest. Then the 109 point system will indeed rule the world

[Gavin Trott]

Then the 109 point system will indeed rule the world

>>Jeez, ive never read a thread on this topic before!

pass me today's news>>



Craig, couldn't resist!

.

[Rawshack]

Besides, I've declared war on them (the 100 pt system) and won't rest until it's rotting corpse has been thrown on a mangy old tip, along with Parker to keep it company

Then the 109 point system will indeed rule the world

Or a 112 if you're Halliday

[RogerPike]

Besides, I've declared war on them (the 100 pt system) and won't rest until it's rotting corpse has been thrown on a mangy old tip, along with Parker to keep it company

Then the 109 point system will indeed rule the world

Or a 112 if you're Halliday

Rawshack,

Just a cheap shot. When you have 10% of the experience of Halliday or when your promotion and support of the Australian wine industry reaches 10% of that achieved by Halliday then you will have a smidgeon of credibility.

Roger

[GravyMaker]

it is the universe we are in. Maybe when we all get to heaven, the dear lord will have the perfect system, all our palates will be the same, there will be no bottle variation or any other bias. we can all buy a book on heavens 100 greatest wines and never need to argue its contents. We will be all able to count the number of top 10 wines we own and puff our chest out when everyone murmours about how clever we were to obtain them. But on this mortal coil its never gonna be that way.

Sounds extraordinarily boring doesn't it...

[Rawshack]

Or a 112 if you're Halliday

Rawshack,

Just a cheap shot. When you have 10% of the experience of Halliday or when your promotion and support of the Australian wine industry reaches 10% of that achieved by Halliday then you will have a smidgeon of credibility.

Roger[/quote]

Go to http://www.dictionary.com and look up the word 'Humour' old chap. However, you must understand it as your post certainly made me smile

EDIT - Before anyone else says it, I know full well I am not, nor ever have been, funny once in my life. Someone once said that I was once in 1998, but they unfortunately mistook me for someone else.

[RogerPike]

Or a 112 if you're Halliday

Rawshack,

Just a cheap shot. When you have 10% of the experience of Halliday or when your promotion and support of the Australian wine industry reaches 10% of that achieved by Halliday then you will have a smidgeon of credibility.

Roger

Go to http://www.dictionary.com and look up the word 'Humour' old chap. However, you must understand it as your post certainly made me smile

EDIT - Before anyone else says it, I know full well I am not, nor ever have been, funny once in my life. Someone once said that I was once in 1998, but they unfortunately mistook me for someone else.

Just checked the dictionary Rawshack. Couldn’t find any reference to young pretenders taking a poke at respected leaders who have earned that respect with a lifetime of hard work.

Roger

[Rawshack]

Or a 112 if you're Halliday

Rawshack,

Just a cheap shot. When you have 10% of the experience of Halliday or when your promotion and support of the Australian wine industry reaches 10% of that achieved by Halliday then you will have a smidgeon of credibility.

Roger

Go to http://www.dictionary.com and look up the word 'Humour' old chap. However, you must understand it as your post certainly made me smile

EDIT - Before anyone else says it, I know full well I am not, nor ever have been, funny once in my life. Someone once said that I was once in 1998, but they unfortunately mistook me for someone else.

Just checked the dictionary Rawshack. Couldn’t find any reference to young pretenders taking a poke at respected leaders who have earned that respect with a lifetime of hard work.

Roger

Roger, I think perhaps you're taking this a little seriously. Now, if I apologise, can we have a nice big hug and make friends? Please, this negativity is bringing me down.

[RogerPike]

Any apology is not due to me.

Roger

[Rawshack]

Any apology is not due to me.

Roger

Have a drink Roger, please. On me.