Tha Gallup/USAToday poll, taken 6/1-3/07, produced a surprising lede:
Hillary Rodham Clinton and Barack Obama are essentially tied for the Democratic presidential nomination, according to a USA TODAY/Gallup Poll, the first time that the New York senator hasn't clearly led the field. (Link to full story.)To the credit of reporter Susan Page and the editors at USA Today, the story also notes that other polls do not show such a sharp tightening of the race:
No other major national poll has shown the Democratic race so close. An ABC News/Washington Post poll taken last Tuesday through Friday gave Clinton a 12-point lead.(A strong pet peeve of mine is stories that refuse to note comparisons with other polling, especially when results are surprising. I wish other news outlets would emulate the openness we see here.)
This result has produced a lot of excitement and over-interpretation on both sides of the Clinton-Obama camps. Clinton's advisers found reason to dispute and discredit the poll while Obama supporters preferred to embrace the result as proof of Clinton's vulnerability.
Let's take a closer look.
The chart above shows the Gallup poll results since November 2006 in bold red and blue for Clinton and Obama respectively. The 95% confidence intervals (more commonly called the "margin of error") are shown by the vertical lines around each Gallup poll data point. All other polls over this time are shown in light red or light blue so you can see how Gallup compares with everyone else. And the trend estimates I calculate based on all polls (including Gallup) are shown as the red and blue trend lines. (These are my standard, more conservative, trends, also known as "old blue". This trend doesn't respond as much to individual poll bumps and wiggles, but is a little slow to respond to real changes of trend. The results here don't depend on which estimator I use, but I've included the chart for the "ready red" sensitive estimator in the plot at the bottom for the interest of skeptics. I've written at length on differences between the two trend estimators. See those posts if you wish here and here.)
The most important result is that the new Gallup poll does look quite a bit different from most, but not all, of their previous polls since November. For 17 of the 22 Gallup readings (11 polls x 2 candidates) the confidence interval overlaps my trend estimate. The new poll is unique in that both Clinton's and Obama's confidence intervals do not include the current trend estimate. That means either the trend has changed and my estimator hasn't detected it, or that the new Gallup result is further away from the trend than we would expect 95% of the time. (My sensitive estimator, shown at the bottom, thinks that Clinton's trend is a bit more down, and Obama's a little more up, but the confidence intervals don't overlap the trend in that plot either.)
So based on this, the new Gallup result does not fit what we would expect based on sampling theory and the overall trend estimates. In that sense, the poll is a statistical outlier, and the critics are right to discount the result and any claim of a "statistical tie".
We've seen some earlier Gallup results in which the confidence interval fails to overlap the trend for one, but not both, candidates. Twice Clinton has fallen below trend and outside the CI (one of those was just barely outside the CI). And once Obama was below trend and outside the CI.
But here is the important point: For those polls that were farther from trend than expected, both the previous AND the subsequent polls bounced back to within the margin of error of the trend, and in fact happened to fall above the trend in all three cases.
And that is the really important thing: If these large deviations from trend were telling us something "real" about the trend, we'd expect to see the subsequent poll ALSO reflect that. Instead, these polls act very much like random variables are supposed to do: they may deviate in a single reading, but they bounce back to near their mean ("expected value" is the technical term here) as soon as we take a new sample. NOTE that the trend changes over time, so I'm NOT saying this is all random noise. When we look over all the polls we can detect real shifts in Clinton and Obama support, shown by the red and blue trend lines that are our best estimate of true changes in the candidate's fortunes. The mistake, which is very common, is to think that each single poll's movements are meaningful and MUST be explained by some political event or "short term" trend.
Frank Newport, Gallup's editor in chief, has a discussion of the new poll in his Gallup Guru blog at USAToday here. He stresses that the change from May to June for each candidate is within sampling error and so can the interpreted as "not statistically significant". Obama gained four points, from 26% in May to 30%. The confidence interval for that change is 5.7%, so not statistically significant. Clinton dropped from 35% to 29%, a 6 point drop. The confidence interval for that is 5.9%. We could quibble whether to round the CI to 6% or not, but certainly that change is at most just at the margin of statistical significance, and not "strong evidence" of a change in fortune for either candidate.
The one quibble I have with Newport's comments is that even as he (correctly) cautions about sampling error and non-significance of the changes, he can't quite resist the possibility of real change:
So we don’t know exactly what to make out of the change in the current poll. The relative stability of the structure of the Democratic race certainly suggests that this latest bump up by Obama could be temporary, as was the last such change in April.It is the "latest bump up by Obama could be temporary" that jerks my chain. If the best explanation is random noise, which the non-significance of changes suggests, then we shouldn't turn around and consider that the "bump" is in any way meaningful. The "last such change in April", for example was a non-significant point above the trend at 26 which followed the low outlier for Obama at 19 earlier in the month. That "7 point gain" was mostly random noise from a lower than expected reading to a higher than expected one, but neither signaled a meaningful shift in trend, which was relatively flat during that time.
The worst implication of taking the current poll at face value is that when the next one comes out and shows Clinton back at around 34% and Obama back around 23% (my current trend estimates) there will be a powerful urge to write that Clinton has rebounded and Obama has fallen off. Neither is likely to be true. Rather, random variation has worked its invisible hand and brought the polls closer to the actual trend in support. A substantive reading of that future poll will be quite as misleading as reading the current one as showing a sharp change in fortune.
(Interests Disclosed: I was interviewed by reporter Susan Page for the USAToday article and briefly quoted, though not on this specific point.)
The plot below shows the trend estimate based on "ready red", the more sensitive measure of trend, but one that is also more likely to mistake noise for changes in trend. While the trend estimate here is a little different, the polls that fall outside the confidence interval are the same as in the chart above using my more stable estimate of trend.
