The Wall Street Journal ran a headline a few days ago that was simply, “Britain’s Muslim Problem,” proving once and for all that the WSJ has be fully integrated into the Murdoch news machine, which is sad, but also a bit hilarious, as it clearly isn’t trying very hard to cover up its bias.
For example, the article starts by saying, “at least 800 British Muslims have left the country to wage jihad with Islamic State. Another 600 were caught trying to join the group.” Setting aside for a moment that it gives no citation for those numbers, that’s a total of 1400 people. The population of British Muslims is roughly 2.7M, as of the last census. That means that the WSJ is warning us that 0.052% of Muslims have been radicalized (which is approximately 0.013% of the total population of Britain). Even if they want to claim that this is an annual rate, that means that it would take 1,929 years for the entire population of Muslims to be radicalized, assuming the rate and population remain stable. Presumably by the time the entire Muslim population of Britain is radicalized, they’ll be ‘waging jihad’ over those shiny, aluminum jumpsuits we all wear in the future.
Unfortunately, startling statistics about 0.013% of the British population isn’t all they have. They’re also citing a survey conducted recently by ICM for a Channel 4 documentary, entitled “What Muslims really think,” to air later this month.
First, for the uninitiated – “documentary,” when attached to the words “Channel 4” doesn’t really mean what you’re thinking. Channel 4 is well-known for producing scandalizing documentaries, often exploiting minorities and vulnerable populations to do so, producing what often amount to modern-day freakshows. Perhaps the best illustration of this is the satirical version produced by That Mitchell and Webb Show a few years back, called “The Boy with an Arse for a Face” (trigger warning for language that would match something with that title). So that the survey is for Channel 4 is already not a vote in favor of its even-handedness. Moreover, the survey included only 1,047 respondents, and as several people have pointed out, focused on neighborhoods that had at least 20% Muslim population, which are also some of the poorest neighborhoods in Britain. The survey is corrected for socio-economic status, but only within the sample group, meaning they didn’t find additional respondents to balance out the difference, they just weighted the responses from the existing pool of middle and upper class participants more.
There’s a lot to talk about with the survey – most reporters have focused in on the result that half of the participants said that they did not support the legalization of homosexuality or of gay marriage, which I think I want to treat in a separate post – but I think one of the most overlooked aspects is the issue of sample size.
For example, the WJS notes that the survey found that “7% of respondents support the establishment of an Islamic state.” To start with, only 1% of respondents, or 12 people, said they would support “a fully-separate Islamic area in Britain, subject to Sharia Law and government.” Another 17% percent, or 187 people, supported integrating in some ways, but maintaining a Islamic lifestyle as much as possible. The overwhelming majority (49%, 532 people) said they would like to “fully integrate with non-Muslims in all aspects of life.” While that suggests that the majority support integration, we’re still only talking about the opinions of 1,047 people for a population of 2.7M (that’s 0.039%, for those playing along on our home game).
Survey sizes obviously have to be smaller than the full population, but when you get down to survey populations that small standing in for actual populations that large, it leads to the interjection of way too many alternative variables. For example, it’s entirely possible that all 12 of those people who claimed to support a Muslim state in Britain were far-right nutjobs pretending to be Muslims. You could easily convince 12 people to do that. Or they could all be from the same family, or have studied under the same Imam, or they could have all just made some weird pack to answer the survey in as extreme a manner as possible. When it’s only 12 people, there are just way too many other options for how those answers might have come about.
Plenty of other people have talked about the problem with humans’ difficulty in understanding the relationship between large numbers, but when it comes to population-wide observations, we really do need to train ourselves to see these things in context. A less than 0.1% sample size is just not significant, unless you can provide some really compelling evidence for how this cohort was assembled. Otherwise, it’s exactly the same as me just stopping people on the street, asking their opinion, and then claiming that that’s “what Americans think” or “what white people think.” It’s not just not informative, it’s actually mis-informative, because it’s presenting something essentially anecdotal as statistically significant.