Ben Goldacre
Saturday April 1, 2006
The Guardian
Nothing comes for free: if you can cope with 400 words on statistics, we can trash a front page news story together. “Cocaine floods the playground,” roared the front page of the Times last Friday. “Use of the addictive drug by children doubles in a year.”
Doubles? Now that was odd, because the press release for this government survey said it found “almost no change in patterns of drug use, drinking or smoking since 2000”. But the Telegraph ran with the story as well. So did the Mirror. Perhaps they had found the news themselves, buried in the report.
So I got the document. It’s a survey of 9,000 children, aged 11 to 15, in 305 schools. The three-page summary said, again, there was no change in prevalence of drug use. I found the data tables, and for the question about using cocaine in the past year, 1% said yes in 2004, and 2% said yes in 2005. Except almost all the figures were 1%, or 2%. They’d all been rounded off. By asking around, I found that the actual figures were 1.4% for 2004 and 1.9% for 2005, not 1% and 2%. So it hadn’t doubled. But if that alone was my story, this would be a pretty lame column, so read on.
What we now have is an increase of 0.5%: out of 9,000 kids, about 45 more kids saying “yes” to the question. Presented with a small increase like this, you have to think: is it statistically significant? Well, I did the maths, and the answer is yes, it is, in that you get a p-value of less than 0.05. What does that mean? Well, sometimes you might throw “heads” five times in a row, just by chance. Let’s imagine that there was definitely no difference in cocaine use, the odds were even, but you took the same survey 100 times: you might get a difference like we have seen here just by chance, but less than five times out of your 100 surveys.
But this is an isolated figure. To “data mine”, and take it out of its real world context, and say it is significant, is misleading. The statistical test for significance assumes that every data point, every child, is independent. But, of course, here the data is “clustered”. They are not data, they are real children, in 305 schools. They hang out together, they copy each other, they buy drugs off each other, there are crazes, epidemics, group interactions.
The increase of 45 kids taking cocaine could have been three major epidemics of cocaine use in three schools, or mini-epidemics in a handful of schools. This makes our result less significant. The small increase of 0.5% was only significant because it came from a large sample of 9,000 data points – like 9,000 tosses of a coin – but if they’re not independent data points, then you have to treat it, in some respects, like a smaller sample, and so the results become less significant. As statisticians would say, you must “correct for clustering”.
Then there is a final problem with the data. In the report, there are dozens of data points reported: on solvents, smoking, ketamine, cannabis, and so on. Standard practice in research is to say we only accept a finding as significant if it has a p-value of 0.05 or less. But like we said, a p-value of 0.05 means that for every 100 comparisons you do, five will be positive by chance alone. From this report you could have done dozens of comparisons, and some of them would indeed have been positive, but by chance alone, and the cocaine figure could be one of those. This is why statisticians do a “correction for multiple comparisons”, which is particularly brutal on the data, and often reduces the significance of findings dramatically, just like correcting for clustering can.
Mining is a dangerous profession – and data mining is just the same. The stats nerds who compiled this government report knew about clustering, and Bonferroni’s correction for multiple comparisons, and that, presumably, is why they said, quite clearly, that there was no change from 2004-2005. The journalists, apparently, did not want to believe this: they tried to re-interpret the data, and the story went from an increase of 0.5%, that might be a gradual trend, but could well be an entirely chance finding, to being a front page story in the Times about cocaine use doubling. You might not trust the press release, but if you don’t know your science, you take a big chance, when you delve under the bonnet of a study to find a story.
· Please send your bad science to bad.science@goldacre.net
Alan Harrison said,
April 4, 2006 at 8:27 pm
Well I’m late again, but I seem to have missed a stats practical (again) so no loss 😉
profnick said some years ago: “I would have replaced your instances of “0.5%†with “from 1.4% to 1.9% and omittted all the stuff about correcting for clustering””
Ben said, rightly, that would leave a weak premise for a column.
In general most commenters forget that Ben is a journalist and this is his weekly column in the Guardian. Yes he could write tons of clever stuff that would keep all you pedants happy but he’d lose his job. Ben’s challenge to you (i.e. what would you write?) is a good one. If you can’t take it then shut up.
I’m left with memories of 3 hour stats practicals in Sheffield Uni in 1986 which, after the first, I spent feeding pizza crusts to the ducks in the park behind the Arts tower. Happy days.
Why Dont You…Blog? » Bad Science - Bad Statistics… said,
April 4, 2006 at 8:39 pm
[…] This weeks article (at www.badscience.net/?p=230) is about the way news papers fight for headlines by really overdoing the actual data. The headline claims of the number of children using cocaine has “doubled” is based on an increase from 1.4% to 1.9%. Even my basic understanding of maths doesnt see that as a “doubling.” […]
Jason Ditton said,
April 5, 2006 at 6:24 pm
Sorry. But how do you control for cluster effects?
oharar said,
April 5, 2006 at 7:28 pm
“Sorry. But how do you control for cluster effects?”
As we’re already in the nerdiest thread, I don’t feel so bad about answering this. And the answer is…
It depends. In general, the effect of clusters is to increase variation in the data. The solution is to include this variation in the analysis, but how exactly that is done depends on the structure of the data and where the clustering is. You need some sort of replication (e.g. between schools, between years etc.), and then you model this variation. There’s a whole class of models called hierarchical models that deal with this, by estimating the variance at the different levels in the data.
Bob
Steve said,
April 5, 2006 at 8:28 pm
Yes. What hierarchical models do is allow you to understand how much of any variation is attributable to the properties of ‘clusters’ rather than the properties of individuals . In the example that talked about in the article there could be several types of clustering eg schools, classes within schools. It then becomes complicated as there is then the possibility of cross-classification ie clustering of classes within a year group in all schools (a cohort effect). Some researchers like to control for clustering but I (as a geographer) am interested in the properties of the clusters themselves ie a cllective school or classroom ‘effect’ on drug use..
Anyone interested in looking at this should check the pioneering work of Harvey Goldstein the education researcher who developed and popularised multi-level models.
Introductions to multi-level models have been written by Kelvyn Jones and others (within geography)
oharar said,
April 7, 2006 at 6:01 am
Yes! We’ve finally found the level of this community! Consecutive posts about hierarchical models and everyone shuts up.
Of course this also means that Steve and I are über-nerds (unter-nerds?), but I guess we both already knew that anyway.
Bob
Sockatume said,
April 7, 2006 at 8:17 am
I can wholeheartedly reccomend introducing statistics (particularly the Monty Hall Problem) into dinner conversation.
Aspiring Pedant said,
April 7, 2006 at 12:18 pm
Why has this message – ” You must bee logged in to post a comment” appeared? I really needed another password to forget.
As a humble engineer with a fairly basic understanding of statistics it seems to me that a common problem with statistics, as they appear in the media, is that journalists will often try to compare 2 data points and make out there is an underlying trend. The statistics Coracle referred to in comment 72 – news.bbc.co.uk/1/hi/uk/4871854.stm are another good example. The story says that teenage driver deaths are on the rise but quotes only figures from 2000 & 2004. Are there no figures available from other years? Or would those figures not create the right impression?
I suspect the same might be true for the drug use survey; were we to look at figures for the last 5 years we might find that the figures vary a little between 1 & 2% or maybe there is a genuine trend, but referring to only 2 sets of figures is never going to convince me that there is any kind of trend.
Alan Harrison – ” In general most commenters forget that Ben is a journalist ” – Ben is definitely not a jounalist; he works full time for the NHS and writes a weekly column for the Guardian. The fact that Ben is not a journalist, and doesn’t write like one, is what makes his column so refreshing. However, I agree that he has to write a column that can be understood by most Guardian readers and so too much detail on statistics is unlikely to be welcome. I have to say that I found his latest column a bit dull anyway – and I find statistics quite interesting.
stever said,
April 7, 2006 at 1:10 pm
AP – it was because the forum got hit with a wave of crappy gambling spam yesterday from automated SPAM BOTS! registering should stop it happening.
Fyse said,
April 7, 2006 at 8:21 pm
Sockatume – I’ve tried to explain the Monty Hall problem a number of times now, particularly to my Dad who just wont buy it. While I was initially persuaded by working through the maths, if your victim doesn’t have the necessary background it’s really difficult to form a convincing explanation. Any tips?!
By the way Ben, I was wondering whether the passwords we choose are visible in plain text in your SQL database? Obviously I trust you not to abuse the privilege, but I was curious for future reference when registering on other blogs.
briantist said,
April 10, 2006 at 9:49 pm
Ben, what Fyse means is that if you store the passwords as plain text in your database, it is easy for a hacker to view them. The best way to sort this is to use a function like this one in your PHP code to encode the password in a one-way process, so no-one can just read the password..
function crypt($strPassword)
{
$strmd5=md5(strtolower(trim($strPassword)));
$strcrc=dechex(crc32($strPassword));
$strV1=md5($strmd5 . $strcrc) .md5(strlen($strcrc) . $strcrc);
return (md5($strV1));
}
Ben Goldacre said,
April 10, 2006 at 9:53 pm
you’re clearly mistaking me for someone who programs websites.
briantist said,
April 10, 2006 at 9:58 pm
I should have guessed! A programmer wouldn’t leave “Fatal error: Call to undefined function: () in /home/users/web/b2624/pow.bengoldacre/htdocs/wp-content/plugins/stattraq.php on line 90” at the bottom of each page. 😀
briantist said,
April 10, 2006 at 10:08 pm
OK, back to statistics again then. Have had a look at Annex I of this document if you have a minute.
www.ofcom.org.uk/research/tv/reports/dtv/dtu_2005_q4/q4_2005.pdf
Is there any statical basis to the A1.14 adjustment that removes 4.3 million Freeview boxes from the figures? It represents 44% of the boxes sold, and its only purpose seems only to allow Sky to remain infront.
RS said,
April 11, 2006 at 7:55 am
briantist, they give their reasoning:
“Second set duplication
A1.11 Latest available data (from Q3 2005) suggest that around 30.2% of Freeview boxes
are being used on secondary sets by viewers who already have digital (either
Freeview or Sky or cable) on their main set (source: GfK). Ofcom estimates that this
equates to a total of 3.2 million DTT receivers on secondary sets.
Inactive boxes
A1.12 A number of DTT boxes are currently inactive, possibly because they were never
installed by consumers, have been replaced, or because of reception issues. The
latest estimate for this figure (from Q4 2005) is around 1,040,000 (source: GfK).
ITV Digital legacy boxes
A1.13 There are also around an estimated 250,000 ITV Digital legacy boxes remaining in
the market. The number of homes where the ITV Digital box is the only digital
platform is estimated at 130,000 homes (source: GfK).
Ofcom adjustment
A1.14 Ofcom has therefore deducted around 4.3 million from Freeview sales in order to
account for these adjustments. This means the number of Freeview-only homes is
therefore calculated as a little under 6.5 million.”
briantist said,
April 11, 2006 at 12:24 pm
Yeah, I read that. I just don’t think it’s valid.
It seems that anyone who has, say, a Freeview box and NTL digital cable gets discounted as a Freeview customer.
Likewise if you have Sky and Freeview the Freeview number gets reduced, but not the Sky number. This hands a double advantage to the Sky count!
To say that people who bought a DVB-T box with an “ondigital” label or an “itv digital” sticker are not watching Freeview with it is diengenious to say the least.
Also, people who buy a top-up TV badged box can watch all the Freeview services, but they don’t get counted either.
I just can’t see that for every twenty DVB-T boxes sold that NINE have been binned – if this was even remotely true then there would be no need for the BBC licence fee to go up to deal with free boxes for the underprivaldged, all that would be needed is a “put your unwanted Freeview box here” box in every high street (in the style of the cat-food collection boxes in my local Sainsbury’s).
hatter said,
April 11, 2006 at 3:38 pm
Cocaine flooding? Anybody drown?
Pendantica it is true that boys are likely to generally claim more sex than reality and girls will claim less.
With drug surveys things can go either way. Answering yes to appear cool is possible, but equally possible is the sneaking suspicion that drug warrior nut is going to use your answers to entrap you.
It is interesting to know the trends, drug fads come and go, but this needs to be set against overall usage. Substitution is typical.
Did this report state levels of usage? I mean actual rates of use rather than the standard loaded terms like heavy, casual, etc. While we might prefer teenagers to not be using recreational drugs, including ones like alcohol and tobacco, both at least as harmful as anything illegal, if they’re just taking them relatively infrequently, no more than a few times a month, and in relatively small doses, it is not a major concern.
I doubt dealers are generally as forward thinking as to consider things like setting up a long-term relationship. Of course if someone can find a reliable dealer who consistenly sells a quality product, then they should stick with them. Of course that’s rare, like reliable banks.
They do typically just exaggerate the value of intercepted drugs. It is particularly noticeable with drugs that are not or cannot be diluted such as cannabis and MDMA tablets. They also like to quote big numbers for the quantities involved. Of course if the authorities were to mention that what they catch is a small percentage of the total amount smuggled and that that their efforts have almost no impact on street level supply they wouldn’t look nearly so good.
briantist said,
April 20, 2006 at 1:10 pm
The Ofcom logic boggles me! It seems to be done like this:
How to count fruit, the Ofcom way.
– apples sold on Monday or Tuesday don’t count as apples
– any apple bought by an orange owner doesn’t count as an apple
– any apple bought by a banana owner doesn’t count as an apple
– if you buy an lemon, orange or a banana after you have bought an apple, the apple doesn’t count anymore
– lemons only count as lemons if they were origianally oranges, not if they are bought as lemons
vcnmwwgr - Google Search said,
December 28, 2006 at 3:38 pm
[…] […]
kyvpjrdr - Google Search said,
December 29, 2006 at 6:53 pm
[…] […]
mickjames said,
June 14, 2007 at 4:14 pm
Fyse: the best way to deal with Monty Hall refuseniks is to play the game for money.
ghghgh said,
April 19, 2010 at 3:33 am
asdfasd
Observasjon er vitenskapens mor, men klokkertro er vitenskapens morder | Hjernedvask said,
May 25, 2011 at 8:50 pm
[…] Goldacre om journalisters vidløftige tolkning av […]
Are there benefits to gaining a strong statistical background? | humblecontrarian said,
June 19, 2013 at 1:17 pm
[…] www.badscience.net/2006/03/cocaine-floods-the-playground/ […]