Separatists, Unite!

Let’s analyze the latest poll by the Western Standard, which is hosted HERE and HERE by Nealenews, shall we?

The poll found that the most pro-separatist age category was the 18-29 age bracket, which I fall into. I can see the reasons why younger people would advocate such politics, but I’m not ready to jump on the bandwagon just yet. Albertans my age look at how Quebec has used separation as a credible threat (as we economists call it) in order to maintain their preferred outcome - more cash from Ottawa. Then we look at how much cash supposedly flows out of Alberta into those Federal coffers to begin with, and we wonder.

“35% of Western Canadians in favour of separation” is just another way of saying “65% of Western Canadians are in favour of a united Canada”.

Spin, pure and simple.

The methodology used by the pollster seems okay, and you can get it HERE.

But I have to point something out: an issue of what we economists call self-selection. The methodological section does not state how many people refused to answer the survey. All it states is:

Data were collected between June 29 and July 5 2005 by Dr. Faron Ellis.

A total of 1,448 adult residents of western Canada were interviewed by telephone. Respondents were selected randomly. Where necessary, the sample has been statistically weighted to more accurately represent the demographic distribution of the population. The margin of error for the weighed sample is + 2.6 per cent, 19 times out of 20. The margin of error increases when analyzing sub-samples of the total.

Notice the sample period. Right around Canada day, when the study assumes that support for federalism should be at an all-time high. This is an assertion, as the Shotgun does not cite any other study which would indicate that this is the case. It’s also a time when people are busy making holiday plans, getting drunk or trying to get away from annoying telemarketers/pollsters.

And over at Nealenews it says:

The poll sampled 1,448 adults in British Columbia, Alberta, Saskatchewan and Manitoba and had a margin of error of plus or minus 2.6 per cent, 19 times out of 20.

I can agree with the randomness of using a random phone-dialer, but I have to point something out. Usually polls cite the number or percentage of people who declined to comment on the survey, and they incorporate this into the construction of the error term (the plus or minus percentage).

One could just argue that the people who excluded themselves from the survey are no different from the people who were included in the survey, but since we don’t know what these people answered, that’s a big assumption to make.

With self-selection, it is possible that the people who declined to comment were self-selecting themselves out of the poll. This could be due to time pressures, but it could be due to political belief. What if 50% of the people surveyed opted out of the survey, and what if these people all disagreed with separatism? This is highly improbable, just as it is improbable that all favoured separatism, but the point of the matter is that the construction of the confidence interval is affected. What the above line from Nealenews is saying is that 19 times out of 20, if we picked 100 of these respondents, we will get a percentage of 35% in favour of separation, plus or minus 2.6 percent.

This doesn’t mean much, because we don’t know how many opted out and hence how they feel about separatism in the West.

To illustrate, consider the following experiment.

There were 1,448 respondents. For illustrative purposes, assume there were 1,448 opt-outs, meaning 50% of the people dialed opted in and 50% opted out.

Case 1: All of the opt-outs favoured separatism.

35% of 1,448 is 507 pro-separatists, plus 1,448 opt-out separatists = 1,956/2896, which is 68% FOR, 32% AGAINST

Case 2: All of the opt-outs were against separatism.

65% of 1,448 is 941 non-separatists, plus 1,448 opt-out non-separatist = 2,389/2896, which is 18% FOR, 82% AGAINST.

So, there is a possibility that the support for separatism could be as little as 18% or as high as 68% (in this experiment) and these represent the extreme bounds of possibility. The greater the number of opt-outs, the wider these bounds are, and vice versa. All this shows is that the exclusion of a part of the sample does affect the confidence intervals. That 2.6% number means absolutely nothing to me without the number of opt-outs cited.

Finally, 1,448 respondents is a very, very small sample size relative to the population over which these results are extrapolated.

This almost looks like a push-poll to me. Certain parts of the methodology were left out so that critics can’t really assess what’s going on.

**UPDATE** CLICK HERE to read a letter to Faron Ellis which contains the exact same criticism I have about the methodology and more.

Furthermore, someone was nice enough to point out that the people who operate the polling firm are also co-founders of the Western Standard. I’ll re-post the comment in case it gets deleted.

May wrote:

Since this was done with such relish by conservo bloggers when polls came out backing the Liberals, allow me to conspiratorially find bias in the Western Standard’s poll.

Faron Ellis is the chief pollster for JMCK Communications, whose principals — Stephen Johnston, Sean McKinsley and Matthew Johnston — are founding managers of Ezra’s magazine. This fact is not in the story, nor is Ellis’ JMCK tag.

But let’s assume that there is no political bias in the pollster. Let’s look at the full poll Ezra laudibly provided to us.

Q1 In the last federal election campaign, Prime Minister Paul Martin dedicated his government to eliminating the democratic deficit by making the federal government more representative and responsible. Please indicate how effective you think the Martin government has been at eliminating the democratic deficit since being elected one year ago.

Q2 In the last federal election campaign, Prime Minister Paul Martin made it a priority to end western alienation. Please indicate how effective you think the Martin government has been at ending western alienation since being elected one year ago.

Q3 Please tell me how much you agree or disagree with the following statement. Western Canadians should begin to explore the idea of forming their own country.
——
Any political science or communications students out there? Ask people two questions about apparent failures of a government, then throw in your hardball question. Inflame, inflame, then reap the benefits.
Whoever said “push poll” in the last entry hit it.

In the story: Writer likens the numbers to Quebec’s. Do pollsters on PQ sovereignty ask, “Quebecers should begin to explore the idea of forming their own country. Agree?”
Shall i get into the issue that nowhere in it mentions that of the 65% of Westerners that are happy with Canada, more than three-quarters are in the “strongly agree” camp.
This doesn’t bolster the cause, but does indicate the degree of soft support for independence, perhaps bolstered by the two questions prior.

There’s my nightly conspiracy fer ya.