Campaign Fairy Tales

This election has been all about polls.  Far from being barometers of public opinion, they have dictated opinion so thoroughly that they have eliminated candidates and narrowed the range of voter options.

This election has been all about polls because of the unspeakable Ford years.  The majority of Torontonians are so traumatized that they’ve been talking about strategic voting since the start of this campaign.

Strategic voters depend on polling results to tell them when it’s time to switch from their first choice to the lesser of two evils.  In this election, that occurred in July when it became apparent that Ford support was growing instead of shrinking.

With the goal of electing anyone but Ford, it was up to the pollsters and the media to inform voters who they should support to defeat him.  Left to chance, the voters might otherwise get it wrong again.

Chow had led since January, but her support had been in decline for the first three weeks of July.  Tory’s support had been rising gradually over the same period.  As of July 21st, they were in a dead heat: Chow, 29%; Ford 27%, and Tory, 28%.

Panic set in.  The pollsters couldn’t point to a leader, so the media didn’t know who to favour.  Strategic voters were without a strategy.

It took seven weeks, six polls, and four polling companies (Forum, Maple Leaf, Main Street, and Nanos), to solve the crisis.  By August 6th, John Tory had leapt to a commanding lead, while Olivia Chow had slipped to third place.  Those were in fact the verbs used in the morning’s headlines; “leap” and “slip.” The choice was clear, if undeclared:  It was time to back Tory if you don’t want “Ford more years.”

This narrative prevailed for another month, apparently evidenced by the polls.  By September 9th, Tory reached 45% in a Main Street poll that totalled 107% according to the Star, instead of the usual 100%.  Tory was pulling away, the headlines assured us.  Six days later, a Nanos poll gave him 49% in a poll that totalled 106% (seriously, do the math yourself here).  “One out of every two Toronto voters” was ready to cast a ballot for Tory, trumpeted the morning radio show hosts.

For anyone who liked having at least one candidate on the ballot who is not a white, male, suburban, Conservative, millionaire, this was a problem.  In what the media declared to be a two horse race, Chow was the odd woman out.  Poll driven headlines, and headline driven polls, cemented her in third place while real contenders slugged it out.

The certitude of this prophecy didn’t last long.  Rob Ford revealed a potentially fatal illness, Doug Ford wrote himself onto the ballot, and suddenly any criticism of the outrageous Ford record was off limits.  No one was willing to trash a dying man’s legacy, and his brother of course denied complicity in the incumbent Mayor’s many, “private” misdeeds.  Having effectively disqualified Chow, the pollsters and the press watched helplessly as Doug Ford started putting pressure on their designated champion, Tory.

Flash forward to the present day, and the result of all this manipulative polling and reporting is a city thrown back into panic.  Tory has a meagre two percent lead over Ford with only a few weeks to go, if you believe the latest reports on the latest poll.  Now any dilution of Tory support by Chow seems unthinkable in such a tight race.  Murmurs about her withdrawal have already started.

This angers me.  A month or two ago, when it might have made a difference, the media could have told a very different story than the tale they crafted about the leaping Tory, the slipping Chow, and the menacing, but impotent, Ford.  If they had only moderated their rhetoric about the polls, other outcomes might still be possible now.

While it is true that Chow support declined while Tory’s increased, with fluctuations on both sides, the trend wasn’t clear enough to anoint Tory when the media chose its anti-Ford champion back in late summer.  As I’ve pointed out elsewhere, the trend line established by Forum Research over the first six months of the campaign was completely corrupted by the inclusion of polls by different companies, using different techniques, in July, August, and September.

Here’s a screen grab from the Star’s poll tracker site, mixing polls by Forum, Maple Leaf, Main Street, Nanos, and Ipsos Reid. If you can discern any trend in this mashup of sampling and weighting methods, you’re either a genius or you belong on the editorial board of the Sun, Star, Globe, or Post. You can’t be both.


As you can see, the trend line moved more or less plausibly until July, because all of those polls were conducted by a single company (Forum) using a consistent method.  By July, the chart became an incoherent mess, and as the frequency of polls, by different companies, went up in September, reporters were hanging headlines on peaks in popularity that reflected changes in polling companies, not changes in voter intent.

To make my point, I’ve separated the Forum results from those of all the other companies.  In the following chart, you can see the same general trends that the media reported, but without the exaggerated effects of different polling methods to fuel the Tory hype in July and August.  In general, the range of values is much more modest than the numbers that made headlines over the past few months.



At no point do the undecided respondents exceed 7% of the total, for example, and as might be expected, appear to be shrinking in number as the campaign goes on.  Tory has peaked, and is in decline, while Ford is ascending.  Chow appears to have bottomed out and may be starting to recover.

Viewed in this context, the meagre 39%-37% Tory lead over Ford in the latest Forum poll, isn’t as momentous as the pundits make out.  If you eliminate the polls that don’t belong in this trend line, you don’t have to explain how the gap between Tory and Ford just narrowed by nearly 20%, because Tory didn’t fall that far and Ford didn’t gain that much.  Their latest results are only moderately different than the results of the previous Forum poll.

Inconsistent with the media narrative, there appears to be an inverse relationship between Chow and Ford’s numbers after positions changed in July.  Although the media story was that this had become a two-horse race, note how Chow’s numbers dip after each Ford uptick, and vice versa.  Rather than Ford and Tory fighting for support, mano a mano, it was Chow who suffered every time Ford put a scare into voters, and who recovered every time Ford support declined.  Were it not for this media construct of a two horse race, might Chow have recovered and retained more support, and might she have forced Ford to fight on a second front, instead of ignoring her and concentrating on Tory?

No such hypothesis is possible in the mess created by plotting the work of other pollsters onto the Forum trend line in the chart below.  Instead there are huge, inexplicable variations appearing in July that become magnified as the frequency of polling increased in August, September, and October.  The result is statistically absurd.

By “absurd,” I mean that the results of all the polls in the mixed sample defy interpretation.  I’ve experimented with polynomial trend lines on both charts with completely different results.  The Forum data anchors a forecast that falls within a plausible range of values, whether or not it’s predictively accurate.  But when that Forum data is mixed with the data from other pollsters, the resulting forecasts shoot off into the realm of mathematical impossibility – less than 0% or more than 100%.  There is simply no integrity to trend lines based on inconsistently gathered and weighted data.

mixed year

This isn’t a technical problem, it’s a problem of journalistic ethics and accountability.  Exaggerated and misleading headlines were generated from these polls.  They created the impression that Tory can win and that Chow cannot, and there’s no way to recover from the damage this has done to her campaign.  Journalists should report the news, not manufacture it.

Nor is this a question of defective polling.  No matter what you think about the relative merits of pollsters’ methods, it should be obvious that trends become meaningless when the methods of data collection and analysis are arbitrarily changed midway through a campaign.  Forum, for example, asked about support for the candidates without prompting respondents to say that they were undecided.  Nanos, in contrast, presented the “undecided” option as if “undecided” was the fourth candidate.  The media companies that commission or buy these polls are fully cognizant of their differences, yet they reported on them as if each new poll related in a coherent way to the previous one as part of an established trend.

Finally, it is not a question of competence or integrity.   With 10% discrepancies in the undecided vote between polls, and 6-7% over-reporting of total respondents on occasion, the media’s premature acclamation of Tory was based on a shaky premise.  Add up the mathematical error and the discrepancy about undecided voters in a couple of those polls, and you have to wonder who was really in first place, and by how much.  Simple logic and arithmetic should have made reporters and editors pause before composing fantastic headlines about Tory the Invincible, Ford the Terrible, and Chow the hopeless Shield-bearer.

If I was a Chow supporter, you could dismiss this complaint as sour grapes, but my preferred candidate was knocked out by polling headlines long ago.  The disservice done to Chow is simply a better illustration of the same effect.

Let me provide another demonstration of the lazy, irresponsible reporting that has suffocated Chow’s campaign.  Look at the chart of undecided voters, representing the 12 polls conducted since Labour Day.  The Forum trend is clear and plausible – a steady, moderate decline in the number of undecided voters is occurring as the campaign progresses, ending, presumably at or near zero on voting day.


The alternative, provided by the mix of other pollsters, is obviously absurd.  It shows the undecided vote swinging wildly in a 13% range, only a few percent lower than it was a month ago, as if no one has learned anything about the candidates over that time.  There is no explanation of changes in voter intention that fit this broken line, however each dramatic variation does happen to correspond with a change in pollster.

That’s a simplified version of the chart below, which shows the Forum results in a bold solid line, and the mixed results from other companies in a broken line.  As with the undecided voters, this chart shows that the mixed polling lines depict inexplicable variances in the candidates’ support, whereas the Forum results fall within a plausible range that makes sense of the latest numbers.


In fact the reason the media was startled by the closeness of the October 7th poll numbers is because the blended results obscured what is revealed by isolating a single pollster’s data (in this case, Forum’s).  The differences between the dotted and solid lines suggests that Tory support has been frequently overstated since Labour Day, while Ford support has been understated.  All that’s required to see this clearly is the elimination of sampling and weighting inconsistencies from the sample.  Suddenly, the latest poll makes sense in the context of the established trend line.  Nevertheless, the media reacted hysterically, and are now waiting, tight-lipped and sweaty-palmed, for clarity from the next poll.

As the chart also shows, Chow is left struggling under the weight of the media’s story line, despite the fact that no coherent trend line supports this narrative.  For as long as Tory is held out as the antidote to Ford, her numbers won’t improve.

Am I still putting too fine a point on this?  I want to be utterly clear.  Knowing the source and quality of the polling data they were buying, the media should have known better than to write Chow off the ballot.

When Tory registered 49% support in a Nanos poll, September 15th, the headline should have been:


Or, when Nanos reported that 17% of voters were undecided, compared with 7% in the previous poll by Forum, the headline should have been:


But we never saw those headlines.  Instead, the media has engaged in dramatized recitations of the polls to fill cheap column inches and air time, while selling more advertising.

The worst consequence of this, in my view, is the narrowing of voter choices.  People don’t have choices if they don’t believe in them.  If a free press is a pillar of democracy, the media should not be creating statistical fictions to enhance the prospects of one candidate to the extent that it eliminates another.  As Stintz and Soknacki can attest, the polls can kill your campaign, whether or not you hang in there like Chow.

This is especially damaging in an election where the motive for strategic voting is so strong.  Journalists should know that polling relies on expressions of intent by committed voters.  They have no predictive value otherwise.  Strategic voters are, by definition, fickle, so in this Ford-fearing election, polls are even less reliable than usual.  If they’re newsworthy at all, they need strong caveats and careful interpretation.

Comments are closed.