'No, I'm not going to stop back when you have an opinion.'

You’re on some web site looking at a screaming headline about the latest political poll. It might be trumpeting a double digit lead, or a narrow margin, or even a shift—all for or away from your preferred candidate. You get a bit tense. You want your guy* to win, of course. And you’d like the comfort of knowing right now that he will. You click on the link and are bombarded with numbers. Before you give up, and before you get buried, here’s what you need to know to understand such polls.

First, skip the headlines, the tables, and graphs. Start by looking for a text description of how the poll was taken. This is usually buried in the very last paragraph or even the footnotes. Sometimes, when a web site is summarizing someone elses poll, it isn’t even in the article and you have to follow a link to the original poll report. It will tell you how many people were surveyed, how the survey was conducted (online, cell phone, land line phone, in person), and what level of responder was included (likely voter, registered voter, adult)

Take, for example, the punchline from this NBC poll. The footnote reads:

The NBC News|SurveyMonkey Weekly Election Tracking poll was conducted online May 30 through June 5, 2016 among a national sample of 10,520 adults aged 18 and over, including 9,240 who say they are registered to vote. Respondents for this non-probability survey were selected from the nearly three million people who take surveys on the SurveyMonkey platform each day. Results have an error estimate of plus or minus 1.4 percentage points. For full results and methodology for this weekly tracking poll, please click here

So this is 10000+ people (a big sample) of adults, who are mostly registered to vote. So what’s with that last bit?

When a survey like this is taken, the first few questions go like this:
How old are you?
Are you registered to vote?
Did you vote in the last election?
How likely are you to vote in the next election?

To be a ‘likely’ voter you have to answer yes to question 3 and give one of the more positive answers to question 4 (i.e very or somewhat likely). When a poll is of registered voters, they answered yes to question 2, but not 3 and 4. When a person is just ‘Adult’, they answered no to question 2 and gave an age old enough to vote.

Now, why would you believe that the opinion of someone who is not going to vote is an accurate reflection of those that are? They aren’t, and polls of just registered voters are notoriously inaccurate. They will tell you they have a margin of error of some small percent. This is based on the number of people polled and reflects the accuracy for comparing to an equivalent sample (i.e. people who likely won’t vote). The polling agency is happy for you to misunderstand the margin of error as meaning its relation to how the election will turn out, but the two are unrelated. My rule of thumb: for polls of registered voters, take the margin of error and multiply by 5. That’s how much the result could really swing by.

Even worse are those who classify as ‘Adults’. So they aren’t registered and didn’t vote last time. They may run out and get registered…or may not. These tend to be younger, less interested, and more liberal people, so their presence usually skews a poll towards whatever liberal candidate or issue is being considered

Now take a look at this poll. The headline screams that Hillary is up 7. The final paragraph reads:

The Fox News Poll is conducted under the joint direction of Anderson Robbins Research (D) and Shaw & Company Research (R). The poll was conducted July 9-12, 2016, by telephone (landline and cellphone) with live interviewers among a sample of 601 Virginia registered voters selected from a statewide voter file and has a margin of sampling error of plus or minus four percentage points for the total sample.

But this item is in paragraph 3:

The good news for Trump is that among just those voters “extremely” interested, he’s up by three (45-42 percent). That’s driven by the fact that more Republicans (43 percent) than Democrats (38 percent) say they are extremely interested in the presidential election.

Okay, got that? Headline insists Hillary is up by 7, but for folks really double dog dare ya likely to vote, trump is up 3. But that can’t be the headline because it’s a subset of an already small sample size (survey was 600 people, remember). Good thing they didn’t ask more people.

As I was writing this, a battleground poll came out showing Trump up in Florida. It was a poll of likely voters and gave the first indication of what numbers are like when the simple deceits are not used. But there is still a nasty trick polling companies can use to skew the results of a poll of likely voters. I’ll use some silly numbers to make the point clearly:

suppose the poll included 100 likely voters, 20 of whom are democrats, 10 independents and 70 republicans. Their preferences go like this
Dems go 19 to 1 for Hillary
Independents go 7 to 3 for Hillary
Republicans got 60 to 10 for Trump

So if you put all those together, Trump is ahead 68 to 32. But of course that’s not how the election will turn out because those arent the demographics of the entire electorate or, more accurately, of the actual voters on that single election day. To fix the, the Polling agency has to predict what the composition of the election day turnout will be.

If they select 40%D, 20% I, 40% R, then scale the answers from those groups, the new poll result is Hillary 58 (38+14+6) and Trump 42 (2+6+34). Or they could wiggle their selection to anything they want and can make sound reasonable until they answers say what they (or their employers) think they should say.

In essence, the result of such a poll can still be faked, though the level of fakery is limited compared to other polls, assuming the fakery is to go unnoticed. You may, over time, find some polls that don’t do much fakery of this sort and trust them outright, but you must always consider each individually

So how do you know what the poll results really say? It’s actually quite simple. The candidates pay for unpublished polls that actually tell them the truth. The candidate who looks tense, angry and defensive, or who just changed their campaign strategy? They’re behind in the polls.

*using ‘guy’ for its familiar tone and simplicity, replace with gender-specific term of your choice, if you need to

Advertisements