Appreciate the feedback, thanks, and glad to see the discussion.
Some good points raised about the analysis too. It was just a bit of fun - not a scientifically rigorous study - so there are obviously plenty of holes in it. For a start, there's the issue that the number of wines reviewed per year (and published) are quite different, which brings the whole statistical significance of the analysis into question.
The comparison of 2005 and 2010 was chosen for the 5 years separation and the (larger) quantity of data available for each year. Dates quoted in the figures are tasting dates, not vintage dates, and multiple taster scores are included (for The Wine Front).
The figure only suggests score creep between 2005 and 2010 - and, I agree, that's not enough to confirm a 'creep' trend. I'm not actually so sure the trend would hold for other tasting/publication years. It's potentially confounded by the fact that there's no account made for wines being of different vintages (as others have noted) - and the average vintage between different critics might be different for the same publication/tasting year (i.e., most of the wines scored in 2009 by critic A could be 2007 wines while most of those scored by critic B might be 2008).
Comparing similar vintages across different tasting dates would be a good way to go. Originally I thought comparing one mutiple year range to another mutiple year range (like 2000-2005 versus 2006-2010) would be a good way to go (especially if the number of wines reviewed in each of those two datasets could be similar), but the available databases were too small for that... so it would be back to data collection/collation... given some time and being so inclined, it might be interesting to look at other variables too...
Just to answer some questions:
pstarr wrote:When did Gary Walsh shut down Winorama and move across to Winefront? Are you comparing Campbell Mattinson's scoring in the 2005 sample to a 2009 sample with different reviewers?
The Wine Front scores are for all reviewers (perhaps not stated clearly enough in small text in original post).
Duncan Disorderly wrote:Noting the evenness of The Wine Front's curve would it be fair to say that they simply reviewed more wines in 2009 than 2005, noting that the high point of the graph is at 90 which is not a particularly high score
In the database used it's the other way around: there're about 3300 scores for 2005 and about 2200 for 2009. Shape of the curve is another story. The fact that the Halliday scores aren't normally distributed (and especially around scores of 91 to 93) is IMO... strange ("manipulated"?), irrespective the issues raised about the analysis. The distribution's like that for all scores from 2005-2010, BTW.
rossmckay wrote:The quality of vintages over the sample time has been raised. Another might be the number of reviews that are published versus not published?
Sure, but then, it is about score creep in published scores anyway.
rossmckay wrote:Has a similar comparison been made about wine show results?
That would be interesting. I've seen it for some small-scale show results (mainly in the USA) but I've never seen any similar analysis done on the scores of critics. If anyone here has, I'd be interested to hear.
Polymer wrote:If the 2005 publication is EVERYTHING published by them up to that point..and 2009 is EVERYTHING published by them at that point (which also includes all the data from 2005) then the data is even more damning and really
The figures are for scores from wines tasted in 2005 only, and scores from wines tasted in 2009 only. Perhaps not a very fair comparison, as previously mentioned (need more data).