Beware of Sound-Bite Logic – It’s Compelling but Misleading

Thursday, January 21, 2010

01-21 542439_59528342-200“Question everything!” Excellent advice, especially for claims based on simplified statistics. What is amazing – read, disconcerting – is how pervasive improper statistical analysis is. So, we need to ensure pronouncements and conclusions we deem important are accurate. Here is a non-investment example…

The New York Times had an entertainment news story: “Oscar Race Begins at Golden Globes” (January 18, page C-1):

Because the Globes are awarded while Academy Awards voters still have their nominating ballots in hand, they exert considerable gravitational pull over the Oscar process. The best picture Oscar has mirrored the association’s choice for best drama or best comedy-musical in 15 of the last 22 years.

Two things to note:

  • The conclusion is absolute – “they exert….”
  • The data appears supportive – 15 of 22

Now let’s discuss what’s wrong with the reporter’s conclusion.

First, the hypothesis. Report says that timing creates influence. That only covers the “How.” Missing is the “Why?” – cause and effect. Why should the Golden Globe’s 90 foreign press member votes influence the Academy Award’s 6,000 Hollywood members?

Second, the data. Report used 22-year period: 1987/1988 to 2008/2009 (first number is film year, second is awards year). Academy Awards started in 1928/1929, and Golden Globes, in its current form, began in 1963/1964. Using the latter date, there are 46 years of combined data, meaning the reporter left out 24 years without explanation.

Third, the methodology. Reporter calculated number of years best film choices were the same, implying it was the significant datum. But is it? In the field of picking best picture, what would we expect? There are common criteria used to judge a movie, so many groups can end up selecting the same best movie. This happened with Slumdog Millionaire last year. In that context 15 of 22 (68%) is less significant.

Fourth, adjustment factors. Studies often need special adjustments, and this study certainly requires some. For example, a measure of film popularity that could influence both awards. Also, the Golden Globes have two best picture awards: drama and comedy/musical. Clearly, two choices raise the probability of the Academy matching one.

The last piece of analysis is a bite-sized examination of the data, looking for anything that might be lurking. With so few data to examine here, we can look at the individual years. Here a simple graph showing the pattern over the 46 years:

01-21 Statistics-yearly

Note that in last five years, only the last was a match. In the previous five, all were matches. So, the percent match dropped from 100% to 20%. Why? Who knows. But what it means is that one-half of the reporter’s data is so volatile that the 22-year results are questionable. Here is what the entire period looks like, with rolling 5-year periods:

01-21 Statistics-rolling 5

Had enough statistics for today? Me, too.

But there is one last question I want to address: Why are statistical analyses so often wrong?

Three main reasons:

  1. Lack of training or education. This is especially prevalent in the media, where sound-bite logic is prized:  Simplistic conclusions that sound plausible, backed by simplistic data (like the example above)
  2. Improper use by those that should know better. Dishearting, I know, but far too common – particularly in the investment world. The misuse is especially high nowadays because of the ease with which statistical analysis can be done.
  3. Deception. Particularly prevalent in the sale of products and services. Whenever someone’s livelihood is on the line, they will work the numbers to find enticing support. This is the root of the harsh truth: Figures don’t lie, but liars can figure.

So, question everything.

Tags: ,

Leave a Reply

You must be logged in to post a comment.


John Tobey on Seeking Alpha

Seeking Alpha Certified

January 2010