All Hell Will Break Loose
By Riaz Haq
http://riazhaq.blogspot.com

On the eve of the voting in Pakistan, someone responded on Facebook to
my post Pakistan Election Rigging 101 as follows: "All Hell Will Break
Loose" tomorrow. As much as I would like to disagree with this
comment, the history of elections in Pakistan makes me pause and
ponder on it. Regardless of whether the elections are free and fair,
there is usually trouble in the aftermath. The general elections of
1970, held under the military regime of General Yahya Khan, were
widely believed as free and fair. However, there was serious trouble
and Pakistan lost its eastern wing resulting in the creation of
Bangladesh. The elections organized by Zulfikar Ali Bhutto in 1977
were generally accepted as flawed and caused widespread rioting
leading to military takeover by General Ziaul Haq who executed
Zulfikar Ali Bhutto. So the lesson is that Pakistan has a checkered
history of general elections. And there is mounting concern, given the
opposition leaders' threats of mass protests if the results are not
favorable to them. Such protests invariably descend into major
rioting, widespread chaos and violence on the streets. Recently, I
wrote a blog post titled "What If Pakistan Opinion Polls Are Wrong?".
This post outlined the possibility that the opinion polls like the
IRI's may be wrong based on the limitations and the history of such
polls. But the key reason was to warn my readers and Pakistanis at
large to use caution in assuming that the results of the general
elections would just exactly reflect the findings of the latest IRI
poll. I felt it is particularly important to point to the flaws
because the potential for widespread violence and mass casualties in
the aftermath of the February 18 elections is so great that anything
anyone can do to minimize it is worthwhile. Since I wrote this piece,
Wall Street Journal has published the details of the recent flawed
polls in the US Presidential primaries. I reproduce this Wall Street
Journal article by Carl Bialik as follows:

In the month leading up to last week's delegate-rich California
primary, at least a dozen polling firms canvassed the state,
collectively calling tens of thousands of households.

Political junkies tracking television, newspaper and online coverage
of the voting also heard the names of two main providers of polling
data that didn't place a single call: Real Clear Politics and
Pollster.com. Both are mashing up surveys from various sources this
election year to produce composite numbers meant to smooth out
aberrant results. Their methods are criticized by statisticians, but
their numbers are embraced by news organizations eager for a way to
make sense of conflicting polls.

Numbers from Real Clear Politics, which has been averaging polls since
the 2002 congressional races, are used regularly on Fox News, MSNBC's
"Hardball," and the Web sites of CBS News and the Washington Post.
Pollster, which started combining polls in 2006 and attempts a more
complicated mix than a straight average, is featured on Slate and the
political Web site, Talking Points Memo.

Stirring disparate pollsters in one pot has its critics. "That's
dangerous." says Michael Traugott, professor at the University of
Michigan, and author of a recent guide to election polls. "I don't
believe in this technique."

Among the pitfalls: Polls have different sample sizes, yet in the
composite, those with more respondents are weighted the same. They are
fielded at different times, some before respondents have absorbed the
results from other states' primaries. They cover different
populations, especially during primaries when turnout is traditionally
lower. It's expensive to reach the target number of likely voters, so
some pollsters apply looser screens. Also, pollsters apply different
weights to adjust for voters they've missed. And wording of questions
can differ, which makes it especially tricky to count undecided
voters. Even identifying these differences isn't easy, as some of the
included polls aren't adequately footnoted.

Mark Blumenthal, a former Democratic pollster and co-founder of
Pollster.com, admits that combining different polls violates "a rule
we were taught at pollster school." The site attempts to address
concerns over using older polls by giving more weight to the latest
ones.

John McIntyre, co-founder of Real Clear Politics, says that averaging
polls is better than cherry-picking individual ones, which is what
campaigns might do to highlight numbers favoring their candidate, or
journalists might do to create the impression of a close race. He
requires at least three polls before producing an average. His site's
numbers, he says, provide "a clearer picture of where things truly
stand."

Combining polls isn't perfect, adds Mr. Blumenthal, but "let's hope
that by combining them we're getting some better version of the
truth."

A case in point is the Reuters/Zogby poll completed a day before last
week's California primary showing Sen. Barack Obama leading Sen.
Hillary Clinton in the state, 49% to 36%. That day, SurveyUSA had Sen.
Clinton ahead by 10 points -- her eventual winning margin. (Zogby
blames an underestimate of the Hispanic vote and an overestimate of
the African-American vote.) Still, the composite numbers produced by
Pollster and Real Clear Politics showed a close race where one didn't
exist.

But then, SurveyUSA had Sen. Clinton ahead by 11 points in Missouri, a
state Sen. Obama narrowly won. Zogby had Sen. Obama up by three
points. Noting that the Associated Press initially called Missouri for
Sen. Clinton as election results came in, SurveyUSA President Jay Leve
says, "Missouri is a tough state." The Pollster and Real Clear
Politics combined polls did correctly depict a close race.

Jeff Jones, managing editor of the Gallup Poll, complains that some of
his competitors use more-careful methodologies than others. "So, the
firms that attempt to do things the right way are treated as no better
than the ones that use less-accepted methods," he says.

The problem is identifying that best poll. Pollsters tend to do well
here, badly there. "There are 50 almost uniquely different state
systems," says Mack Shelley, an Iowa State University political
scientist. "You almost have to have a poll that's right for each
particular state or region."

"There's no single best pollster who always gets all the races right,"
says Charles Franklin, professor at the University of Wisconsin and
co-developer of Pollster.com. He says even the best poll occasionally
misses the result by a few points or more, thanks to statistical
sampling error. Pollster and Real Clear Politics could bolster their
case by comparing their numbers directly against those from individual
polling firms in terms of election accuracy, a step neither has taken.

Scott Keeter, director of survey research at the Pew Research Center,
embodies the conflict many pollsters feel about the averages. "The
average of good and bad polls could be worse than just looking at the
best available single poll," he says. However, Mr. Keeter adds, "I
confess to looking at the averages myself."

Mr. Blumenthal says visitors to his site don't necessarily share the
concerns of statisticians. "Asking political junkies to stop paying
attention to a horse race is like asking them to stop blinking," he
says.