How policy works

April 12th, 2008 by Ben Goldacre in badscience, laws, nutritionists | 32 Comments »

Ben Goldacre
The Guardian
Saturday 12th April, 2008

If you put aside the fact that most of the people who campaign against food additives should be taken out and shot for crimes against the enlightenment, even a stopped clock shows the right time twice a day, and the evidence overall genuinely shows that some food additives probably aren’t too good for you. Perhaps there should be a special body for issuing warnings on the rare occasions when scares aren’t bogus.

The study that the stories were based on this week was, genuinely, a double blind randomised controlled trial in which a large number of children – around 300 – were given drinks containing colourings and a preservative, or not, and it was done properly. The children were a fairly representative sample of normal kids, people didn’t know which drink they were getting, and the exposure to additives was similar to what you’d get in the real world if you ate bad food, and the kids’ behaviour was measured properly, and there was a small difference between additives and placebo, and it was both statistically significant (which is to say, probably not just a chance finding) and also fairly clinically significant (which is to say, it wasn’t just a genuine but trivial finding).

That’s the research. Entirely separate is what you choose to do about it. Speaking as someone with no interest in making laws, I’ve always been amused by how completely arbitrary the whole process is. For example, the research itself didn’t come out this week, the Food Standards agency knew about the results in March 2007, and the research was fully published in the Lancet last September. The news story is the FSA saying the additives should be phased out, and this is getting more news coverage than the publication of the experimental data itself. We clearly set a lot of store by these decisions.

So how are the judgement calls made? Last September, after they’d already been thinking about it for 6 months, the FSA said something rather different (“avoid additives if your kids is hyperactive”). Now, after a bit more of a think, they call for a ban. Fair enough. A week ago the European Food Safety Authority gave their report, on the exact same evidence, and they say the evidence of harm is poor, and nothing should change.

After six hours of bleary eyed amazement poring through all the relevant documents I can only say that these decisions appear to be entirely random. The EFSA, for example, have produced three PDF reports full of technical material, including a massive statistical e-analysis of the original data (kindly provided by the original researchers) which they harp on about enormously, but which they conclude changes basically nothing. For a massive organisation representing the interests of 700 million people, well-funded by UKIP voters’ tax money, and with a massive advisory panel, they also make some slightly bizarre criticisms of the science.

For example they worry about the GHA, the outcome measure used in the study, because it’s “novel” with “unknown clinical relevance”, although the GHA is simply an aggregate score of a few other outcome measures which are well known and validated. They give dubious weight to a 25 year old meta-analysis with known flaws. It’s all a bit disagreeable, but it’s not quite enough to explain the disparity of view with the FSA. Meanwhile the FSA have produced a lengthy appendix (Annexe II of document FSA- 08/04/04 if you’re interested) detailing the disagreements of interpretation between the UK Committee on Toxicity and the EFSA on 14 technical points, but… there aren’t really any.

I guess hese decisions are a mixture of whim and politics. They’re vaguely informed by research, but only partly, and as it goes, this is entirely as it should be: because it’s a complicated world, and things have risks and benefits beyond the microcosm of a controlled study. You might really like bright green mushy peas, they might sparkle up your day like nothing else. And on the other side of things, additives aren’t just bad in a special, toxicological, narrow sense, but also because they are “facilitators”: they allow us to produce rubbish food, preserve it, and make it more attractive, ultimately producing fat people with quintessentially English teeth. In those terms additives were always part of a bigger problem of cruddy lifestyles, and this goes way beyond 7 special molecules, a few p-values, a headline, an easy scapegoat, and some ratings scales. What a gloriously complicated world.

Seriously, knock yourself out, read all these government reports, it’s good for you.

Here’s the FSA report including Annexe II where they compare the COT and the EFSA takes on the evidence

Here’s the original FSA take

EFSA presser

EFSA Opinion

Their stats reanalysis:

The COT analysis Sept 2007

And here’s the report itself:

If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

32 Responses

  1. Nellie Dean said,

    April 12, 2008 at 7:39 am

    I need a bit of extra pap. Do additives make adults hyperactive? If not, why not, and at what age do people (teenagers?) grow out of responding?

  2. muscleman said,

    April 12, 2008 at 9:29 am

    The comic misunderstandings of your first paragraph in the Gruaniad comments section is very funny. You really shouldn’t expect people to get irony using metaphorical language in a newspaper Ben, but points for trying.

    I suppose at least the EFSA and FSA both looked carefully and in depth at the evidence. A nice change from Brown’s ignoring the evidence on cannabis. It is the hypocrisy or sheer ‘I am right despite the evidence’ attitude over that one that really annoys. And they criticised the Tories…

  3. Tim Worstall said,

    April 12, 2008 at 9:35 am

    A quick question on a minor point. How do we get to 700 million people? That’s some 200 million more than the population of the EU isn’t it?

  4. muscleman said,

    April 12, 2008 at 9:43 am

    Hmmm, well I expect the EFSA rulings might operate in associate states like Norway, but last I looked Norway wasn’t 200million. Maybe their writ runs in Turkey too, does that make the numbers work? Morocco?

  5. dbhb said,

    April 12, 2008 at 10:43 am

    Love the column, Ben– always have– but what happened to the amusing titles that used to make us smile? Lately they’ve all been so dull and dry I can’t imagine any new readers being drawn to your wisdom, which would be a shame.

  6. evidencebasedeating said,

    April 12, 2008 at 10:55 am

    ‘Additives’ are an essential part of human diet – and need to be in order to ensure there’s enough safe food available to feed the urban population who don’t have a few thou acres to farm at their disposal.
    I agree that the E100s (food colours) contain these artificial dyes – but also included are E100(curcumin),E101(Vit B2), E140/1 (chlorophyll), E160a (beta-carotene), E163 (anthocyanins). They’re termed ‘nature identical’ or ‘natural’ additives because, well, they have the same structure as the original – and their effect in the body will be the same as from ‘wholesome’ fruit and veg- even if the carrier (gummy bears and percy pigs spring to mind) may not provide the ideal dietary product – especially in terms of sugar.

    The E200’s and E300’s are preservatives, including anti-oxidants, and the E400s and E500’s emulsifiers and stabilisers. All needed to keep food safe and edible.

  7. evidencebasedeating said,

    April 12, 2008 at 11:00 am

    Interesting, there’s no call to ban Brilliant Blue FCF (E133, aka F,D&C #1 in the USA). It used to be added to liquid feeds to check if critically ill patients were silently vomiting back into their lungs, given that the lungs don’t normally possess blue fluid.

    It proved to be a naff test, but of interest was the ability in septic patients for them to turn blue temporarily. If they were jaundiced, they turned an interesting shade of green. All due to the blue dye leaking into the body between the gut cells.

    An amusing aside, but because the dye had a side chain similar to ADP, it was found to block the formation of ATP energy in cell mitochondria (their power stations), causing cell, and then organ death.

    Absolutely no need to be concerned about it in health, but both the test and the dye are totally, totally banned on ICU’s now as a serious risk to seriously ill patients…..

  8. gimpyblog said,

    April 12, 2008 at 11:28 am

    evidencebasedeating, your knowledge and understanding is, as always, an example to us all.
    If I may add something though, if we didn’t use ‘additives’ we would have to preserve food using huge quantities of salt or sugar. Both of which are generally best avoided in large quantities.

  9. trickcyclist said,

    April 12, 2008 at 11:34 am

    I do have a slight issue with this paper being the sole evidence for banning these additives. True, it was well conducted, and the results were clinically significant, but it should be pointed out that the amounts of additives they fed these kids would give even the most ardent junk-food lover pause for thought –

    “Doses for mixes A and B for 3-year-old children were roughly the same as the amount of food colouring in two 56-g bags of sweets. For 8/9-year-old children, the dose for mix A was equal to about two bags of sweets a day and for mix B about four bags of sweets a day.”

    That’s a quarter of a pound of sweets for a 3 year old! Half a pound for an 8 year old! I loved sweets as much as the next sugar-addicted 8 year old, but even I would have had difficulties with half a pound of sweets a day!

    Now that they’ve shown effect at those doses, I would like to see a little more research with smaller quantities, different sub-sets of children etc before I’m entirely convinced that a glass of squash is poison.

  10. Ben Goldacre said,

    April 12, 2008 at 1:40 pm

    mm i wouldnt say it’s just that one study, it was the first well conducted rct in normal children, but there have been a fair few before that, with variable results, but in this meta-analysis it comes out as a significant effect;jsessionid=LQgVrYrnpW0Kp152Qy2F9Lv66kqglJvy0mp1FpCSJG6KZnzK24HV!509222201!181195629!8091!-1

    annoyingly i can’t get to read it on my athens login so haven’t bigged it up, therefore buyer beware, and send us a copy if you have access. it’s approvingly referenced in various sensible places.

    Do Artificial Food Colors Promote Hyperactivity in Children with Hyperactive Syndromes? A Meta-Analysis of Double-Blind Placebo-Controlled Trials.

    Journal of Developmental & Behavioral Pediatrics. 25(6):423-434, December 2004.
    SCHAB, DAVID W. M.D., M.P.H. 1; TRINH, NHI-HA T. M.D., M.P.H. 2

    Burgeoning estimates of the prevalence of childhood attention-deficit/hyperactivity disorder (ADHD) raise the possibility of a widespread risk factor. We seek to assess whether artificial food colorings (AFCs) contribute to the behavioral symptomatology of hyperactive syndromes. We searched ten electronic databases for double-blind placebo-controlled trials evaluating the effects of AFCs. Fifteen trials met the primary inclusion criteria. Meta-analytic modeling determined the overall effect size of AFCs on hyperactivity to be 0.283 (95% CI, 0.079 to 0.488), falling to 0.210 (95% CI, 0.007 to 0.414) when the smallest and lowest quality trials were excluded. Trials screening for responsiveness before enrollment demonstrated the greatest effects. Despite indications of publication bias and other limitations, this study is consistent with accumulating evidence that neurobehavioral toxicity may characterize a variety of widely distributed chemicals. Improvement in the identification of responders is required before strong clinical recommendations can be made.

  11. Ben Goldacre said,

    April 12, 2008 at 2:17 pm

    wow, thanks to my mystery benefactors, i should try this more often when i can’t get access to a paper!

  12. trickcyclist said,

    April 12, 2008 at 4:53 pm

    Thank you too mystery benefactor! Free, as all good information should be…
    So it looks from a skim-read that a statistically significant effect is seen only in children who’s parents already suspect a reaction to AFCs, or where a screening challenge has suggested a response, right?
    Anyway, I don’t know why I sweat it – those colours will be phased out, others will take their place, the turkey twizzlers will continue to be munched, and people will continue to demand nice simple answers to hang their children’s ‘difficult’ behaviour on…

  13. Robert Carnegie said,

    April 13, 2008 at 1:12 am

    Well, what does the paper actually say about bias issues? (I’m terribly afraid that if I try to read it myself it’ll be incomprehensible.)

    My own thought is that 300 isn’t a lot of recuits compared to medical trials that we hear about, but presumably the results are sufficiently clear to woork with nevertheless. There’s also a risk of missing effects in a small percentage of the population – not to set a hare running yet again, but the rate of autistic spectrum disorder is usually said to be down at that level or less.

  14. Jamie Horder said,

    April 13, 2008 at 9:27 am

    Well, the study was double blinded, randomized and placebo controlled, so any bias will have been equal across the drug and the placebo groups, and hence shouldn’t have impacted on the conclusions. That’s the strength of an RCT. The report also says that the additives seemed to have the same effect in children regardless of social class or their level of “hyperactivity” before the test.

    I general this looks like a well designed study and the researchers seem very competent. But the results were far from dramatic. The drug-placebo differences were only just statistically significant, and if you look at the breakdown of the results (table on page 8 of the report pdf), it’s clear that there was no really consistent effects across the two different additive mixtures they used (A and B), or the two age groups (3 year olds vs. 8-9 year olds) on any of the measures they used. Overall there was a significant difference, but it wasn’t especially robust, which is not necessarily a sign that the results are a chance finding, but it’s a bit worrying.

    So I’d want to see more data before I was convinced. The scientist’s favorite phrase – “More work is needed” – is actually appropriate for once. This is potentially really important stuff but I’d want to see if it stands up to replication.

  15. trickcyclist said,

    April 13, 2008 at 11:36 am

    Quite apart from the scientific points, Ben does raise an interesting question about the rather strained relationship between government policy and evidence.

    It reminds me rather of Gordon Brown’s current approach to cannabis: “I will ask the experts to look again at the classification of cannabis, and I will ask them to ‘look again’ repeatedly until they recommend it is re-classified to B, and if they don’t take the hint, I’ll have it re-classified anyway”.

  16. nickyb said,

    April 13, 2008 at 2:54 pm

    It must surely be clear to most people with half a brain, banshee, that the overwhelming majority of people are completely incapable of analysing statistical data, and, moreover, do not want to be able to.

    This is the society that has been created, whether deliberately or not I know not, by successive governments over the last umpteen years since universal education was introduced.

    It terrifies me that I, who gave up maths at 16 and came away with a mere B at GCSE, can generally analyse, evaluate and apply statistical data (and, on a related note, probability) infinitely more ably than the overwhelming majority of my fellow citizens.

    I would be interested in reading an analysis by Ben (cos I can’t be arsed and don’t have the time or the energy to do it myself) of NICE’s decision making, and the evidence basis thereof.

  17. danohuiginn said,

    April 13, 2008 at 7:56 pm

    “should be taken out and shot for crimes against the enlightenment”

    I believe a guillotine is traditional.

  18. muscleman said,

    April 14, 2008 at 11:01 am

    Indeed, danohuiginn Msr Guillotine designed his machine from sound enlightenment principles. It was a great advance on alternative methods, experiments with shouting at corpseless heads notwithstanding.

  19. emilypk said,

    April 14, 2008 at 2:18 pm

    I have to quibble with “statistically significant (which is to say, probably not just a chance finding)”

    As we all know statisticas never ever tell you *whether* something is caused by chance, only how often it would occur *if it was* caused by chance.

    Any conclusiosn beyond that fact (what was it in this case, 1 in 20… 1 in 100?) is the job of our brains not our statistics and depends on klnowing far more than the a-obtained.

  20. emilypk said,

    April 14, 2008 at 2:19 pm

    p.s. why is this blog taking ages to load now?

  21. Robert Carnegie said,

    April 17, 2008 at 1:24 am

    It can be difficult to control the quantity that a child eats of any particular food. Particularly if they turn fussy and only eat one thing.

  22. Ben Goldacre said,

    April 28, 2008 at 10:18 pm

    from the above post:

    They tested two groups of twenty people to see if they could discriminate between the drinks, and pooled those data; the data don’t quite cut statistical significance. So it is only just double blind. But hey; why test more people when you could ruin the blindness ?

    reader beware, as far as i can tell this is just gibberish.

  23. peroxisome said,

    April 29, 2008 at 10:57 am

    “reader beware, as far as i can tell this is just gibberish.”
    dear ben,
    why is this gibberish ?
    is it gibberish because
    1) you don’t like what it says ?
    2) you have a fact that contradicts what it says ?

    I am quite prepared to accept that what i said was approximate rather than precise. However, please feel free to share any facts that contradict what I said. If you can, I will apologise.

    if you cannot, perhaps you can explain why you labelled my comment as gibberish ?

  24. Squander Two said,

    April 29, 2008 at 1:54 pm


    > With regard to food colourings, there are no known benefits (except for marketing purposes)

    I think that’s a little glib. Fact is, the reason that certain colours of certain foods work well for marketing purposes is that lots of people like them. Now, I’m not one of them: I prefer my raspberry yoghurt not to be bright purple and am quite fond of food that looks sort of greyish as long as it tastes good — Patum Peparium, for instance. But that’s just me. I’ve met plenty of people — I’m sure we all have — for whom the colour of food is so important that they will actually refuse to eat stuff that looks wrong to them. I may think they’re all a bit barmy, but I’m not sure that’s a reason to ban stuff they like. And I’d hesitate to claim with such confidence that the only benefit of allowing fussy eaters access to food they’ll eat instead of food they won’t eat is “marketing”.

  25. peroxisome said,

    April 29, 2008 at 9:30 pm

    it’s interesting to read the reports. Firstly, the FSA report (08/04/04) paragraph 11 says “The EFSA Panel
    also agreed with the COT that it was not possible to attribute causality to the
    association nor could the observed effects be ascribed to any of the individual

    it is worth parsing that carefully. That is both scientific expert groups declined to accept from this study that the chemicals caused hyperactivity.

    by contrast, para 22 sets out the FSA reasoning:
    “The available
    evidence now leaves uncertainty as to whether that safety can be confidently asserted”

    so FSA is acting because it is uncertain as to whether the chemicals are safe. Now it strikes me you could have a good debate as to whether this, err, principle, is good or bad science.

    other interesting reading is the publically available report on this research ( Figure 5 shows that objective observers couldn’t detect any effect of gving additives. Disaggregated analysis of the GHA (annexes) shows that, oh, surprise ! it is only the parents that picked up any results.
    (and did you know that parents had to volunteer to take part in the study ?)

    There is also the issue that this is a double-blind study. It might have escaped some people’s notice that this is a test of four food colourings in a drink. So how do you fill a drink with food colouring, and still have a blind trial ? Enquiring minds might think that this is an important question…

    Para 10 of the COT opinion makes clear that two panels of 20 adults were tested, so 40 in total. Now as it happens, in the main study, you have a sample size of ~120, and the results are only borderline significant with this sample size (oh, and contradictory between different age groups). Do you really think you can safely conclude that the sample was blinded, when you only tested 40 adults, and the results were skewed ?

  26. Robert Carnegie said,

    May 3, 2008 at 4:17 pm

    Peroxisome: you seemed earlier to be confusing “blind” with statistical significance. I see what you mean now, but food colouring could be hidden from experimental subjects in various ways: blindfold, baby juice cup with non-see-through cover, add a dark brown food colouring as well. I’m assuming that different colourings were added to the same solution. I would expect a description to state what was actually done.

  27. peroxisome said,

    May 4, 2008 at 12:03 pm

    the drink was a mixture of fruit juices, and was given on a weekly basis to the parents. I think the idea is that the mixture of juices is meant to be the same colouring as juices with additive. Different proportions of juices were added to the various mixes. There was nothing to stop participants seeing the drink.

    Nonetheless, the blinding test is described in the Lancet paper, in the last paragraph of “Study design and challenge protocols”. Their X2 value for the blinding test is 4.4, and the 1 in 20 cut-off is 6. It is also a bit interesting to have done the blinding test in two tranches. If there is any batch-to-batch variation in the colour of the juices, this could change the outcome of the blinding trial.

    Just directly answering your point, this is a test of blinding. Statistical significance in the outcome would show that the participants were not blind. They didn’t test as big a group as they did in their study, and they didn’t test in children; children are sometimes able to discern tastes that adults cannot. You could certainly argue that they have not tested the blindness of the study as rigourously as they could (and should) have done.

  28. peroxisome said,

    May 4, 2008 at 12:43 pm

    robert, i should thank you for your interaction.

    I thought this was a site where science was discussed, but I have been disappointed by the lack of engagement with the facts, and the rather abusive nature of some of the comments.

    EFSA had a legitimate concern, which they state, that the manipulation of the data could affect the outcome of the statistical analysis. When they did the re-analysis, several of the statistical findings evaporated (page 3). It is really difficult to pick up this bit of science from the summary above.

    The only significant thing in the GHA is the parental response, as agreed by EFSA, the Stephenson paper, etc. Why does anyone take exception to the finding that parental findings that cannot be replicated by independent observers are of “unknown clinical relevance” ?

  29. wokao123 said,

    October 15, 2009 at 8:44 am

    i like this article Links of London Links of London Links of London Links of London Tiffany Tiffany Tiffany Tiffany ED hardy ED hardy ED hardy UGG BOOTS UGG BOOTS UGG BOOTS UGG BOOTS

  30. longyan said,

    October 28, 2009 at 12:36 pm

    The ugg knightsbridge world is a comedy to those who thinks, a tragedy to those ugg classic cardy who feels.Look not mournfully into the ugg boots past, it comes not back again. Wisely improve the present, it is ugg lo pro button thine. Go forth to meet the shadowy future, without fear, and with a manly ugg bailey button heart.

  31. longyan said,

    November 6, 2009 at 2:44 am

    It is no use doing what ugg bailey button you like ugg boots ; you have got to like ugg classic cardy what you do  My philosophy of ugg lo pro button life is work . When work is a pleasure , life is joy ! When work is duty ,ugg knightsbridge life is slavery .Work banishes those three great evils : boredom , vice, and poverty.

  32. laptopbatteriesshop said,

    April 3, 2010 at 4:49 am

    We don’t just offer the typical cheap laptop batteries that you may find from other e-retailers, we pride ourselves with providing our customers with the most cost effective solution towards laptop battery replacement without sacrificing quality. All laptop batteries and AC adapters will meet or exceed OEM (Original Equipment Manufacturer) specifications. Every that is available on our website is also guaranteed to look, fit, and perform just like your original laptop battery (and usually better). The only real difference is the price. You don’t have to pay a ridiculously high price just for a laptop battery. We produce over 95% of the laptop batteries, laptop chargers, laptop chargers and other products that we sell. There’s also no middle man for us to pay. Lower cost for us translates into lower prices for you.