In today's Opinion Journal, Steven Moore takes a hard look at the much discussed Lancet study that claimed astronomical numbers of civilian casualties in Iraq. As Moore says, it is not that anyone is defending any civilian deaths, just that wildly inflating the numbers is a dreadful way to try to influence public policy.
After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don't get me wrong, there have been far too many deaths in Iraq by anyone's measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5%–not 1200%.
The group–associated with the Johns Hopkins Bloomberg School of Public Health–employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in "clusters" within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.
However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.
Many people, myself included, have denounced this study as a politically motivated hack job. Moore provides a couple of bits of information that show that this "study" is hideously flawed, which only reinforces the political motivation of the study:
Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million.
When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.
….
Dr. Roberts said that his team's surveyors did not ask demographic questions. I was so surprised to hear this that I emailed him later in the day to ask a second time if his team asked demographic questions and compared the results to the 1997 Iraqi census. Dr. Roberts replied that he had not even looked at the Iraqi census.
And so, while the gender and the age of the deceased were recorded in the 2006 Johns Hopkins study, nobody, according to Dr. Roberts, recorded demographic information for the living survey respondents. This would be the first survey I have looked at in my 15 years of looking that did not ask demographic questions of its respondents. But don't take my word for it–try using Google to find a survey that does not ask demographic questions.
You have a complete invalid methodology here. As Moore points out, the lack of demographic data alone make the survey impossible to prove - or contradict. In other words the results cannot be replicated. That alone has always been enough to invalidate a study in the sciences. As I have said before, even opponents of the war should not want to promote lies to advance their agenda.
UPDATE: Others: Sundries Shack, Confederate Yankee, Penraker, Fausta, Kesher Talk, Volokh Conspiracy, STACLU, Cold Fury, Decision '08, Sister Toldjah,