A few years ago a team of New Zealand wine scientists made news when they declared that their country’s famous Sauvignon Blanc tastes like pee. (Cat pee, to be exact). To be fair, that wasn’t all the $12 million chemical analysis concluded; they also compared the taste of the wine to other scents, like that of “sweaty” passion fruit.
I belong to a wine group myself, but we have no funding agency, perhaps because our tastes are less discerning – when we conclude that a wine tastes like pee, we rarely specify the species, and most of us wouldn’t know a sweaty passion fruit from one that had just showered. As summer approaches, a prime time for wine drinking – and the time of the most prestigious wine tasting in North America, the California State Fair wine competition (June 10-12) – it is a good time to consider the accuracy of all the analyses, and – especially – the numerical ratings upon which many of us base our purchase decisions.
Numerical wine ratings are a subjective measurement of quality. Like that of any measurement, the usefulness of the ratings is dependent upon the degree to which they vary. If a set of fifteen critics all agree that a wine is worth a 90 rating, that is one thing. But if a set of critics rate it 80, 81, 82, 87, 89, 89, 90, 90, 90, 91, 91, 94, 97, 99, and 100, that sends another message. Both sets of data have the same mean, but they don’t mean the same thing.
- CNBC Slideshow: Best Premium Liquor Brands
The first set of ratings, all 90s, instills confidence that the wine has some definable quality that you might agree with, making it a good guide. Scientists would characterize the second set by saying its standard deviation is 6, meaning that as a rule of thumb all you can really say is that it the wine’s rating is probably in the range 84 – 96, which is not a useful guide. Which type set do actual wine ratings most resemble?
Imagine a blind wine tasting competition in which 90% of the judges give identical samples poured from the same bottle, significantly different scores. Or to first reject a vintage as sub-par, and then award it a double gold medal. Might sound like your monthly wine group onto its tenth 4-ounce pour, but according to a recent academic study these facts apply to the judgments of the California State Fair competition.
Disagreement across different judging bodies can be even more extreme. A few years ago The Penguin Good Australian Wine Guide and On Wine’s Australian Wine Annual both reviewed the 1999 Vintage of the Mitchelton Blackwood Park Riesling, and the Penguin guide named it Penguin Best Wine of the Year, whereas On Wine rated it Worst Vintage of the Decade.
Numerous laboratory studies support the conclusion that expert pronouncements on wine taste and quality are less consistent, hence less meaningful, than people commonly believe. For instance a group or researchers asked an ensemble of experts to analyze twelve characteristics – such as tannins, sweetness, and fruitiness – of a set of wines. The experts disagreed significantly on nine of the twelve. In another study wine experts were presented with three unlabeled wines, two of which were identical. In one third of these taste challenges the gurus couldn’t distinguish the odd wine. And when wine experts who were asked to taste white wines aside the same whites which had secretly been tinted to look like rosés, they rated the fake rosés as sweeter than the whites.
Though many in the industry seem to realize that wine ratings are unreliable, they like the system because it makes wine purchase decisions easier and less intimidating. And numerical data implies a certain gravitas which, even if unwarranted, has been a useful tool in generating excitement about certain wines and vintages. One retailer told the New York Times that if a certain cabernet had been awarded an 89 instead of the 90 it received, “we would have sold a tiny fraction of what we’ll end up moving.”
Those in the industry might be on to something: people seem to enjoy a wine with a high rating, and the price it commands, even if that rating is meaningless. That effect was confirmed in a recent study by economists in which a group of volunteers was asked to rate five wines labeled by price. One was a $90 bottle that appeared in the line-up twice, labeled the second time as costing $10. The volunteers rated the bottle labeled $90 higher than the one rated $10. Moreover, the test was conducted while the volunteers were having their brains imaged in an MRI machine. The scans showed that the area of the brain thought to encode pleasure was truly more active when the subjects drank the wine they believed was more expensive. So if you want to experience true pleasure at a discount price, save those empties with the fancy prices and labels, and employ them as decanters, thus saving on the cost of both the wine and the glassware.
