Ben Goldacre, The Guardian, Saturday 20 November 2010
Obviously I like nerdy days out: like Kelvedon Hatch secret nuclear bunker, maybe, with its sign on the A128 saying “secret nuclear bunker this way”. Last month eight of us commissioned a boat to get onto a rotting man-made WW2 sea-fort in the middle of the ocean through Project Redsand (we genuinely thought we might die climbing the ladders), and a couple of weeks earlier, myself and Mrs Bad Science travelled to Dungeness, where a toytown narrow gauge railway takes you through amusement parks and back gardens, past Derek Jarman’s house, then into barren wasteland, before depositing you incongruously at the base of a magnificent, enormous, and terrifying nuclear power station.
I tell you this, because I should declare an interest: I quite like nuclear power stations, not just because they’re clever, or even because I regretfully concede they might be one of our least bad options for power. I secretly like nuclear power stations because they remind me, in the way nostalgia makes us pine for things we disliked at the time, of a childhood in the early 1980s when I knew that I would definitely die in a nuclear holocaust.
So. Last month energy company EDF conducted a poll on whether people near Hinkley Point nuclear power station would like it to be expanded. The BBC dutifully reported the results: “EDF survey shows support for Hinkley power station” said the headline. “Six in 10 people support a new power station at Hinkley”. Polls like this convince locals, and politicians.
But Leo Barasi at the blog ClimateSock has diligently obtained the original polling questions from ICM, and what he has found is a masterclass in how to manipulate answers to a single question.
First, respondents are set into the frame with a simple starter. “How favourable or unfavourable is your opinion of the nuclear energy industry?” Then things heat up. “To what extent do you agree or disagree with the following statement: nuclear energy has disadvantages but the country needs nuclear power as part of the energy balance with coal, gas and wind power.” As Leo says, this is structured in a way that makes it harder to disagree. “It appear reasoned: taking on board the downsides of nuclear before drawing a measured conclusion that it’s a necessary evil to produce a greater good.” As a result, only 13% disagree, but the whole audience is gently nudged.
Then locals are asked a whole series of branching questions, forcing them to weigh up the positive and negative impacts a new power station would have on the area. People who think it would be positive are asked to also weigh up the negative, and people who think it would be negative are asked to weigh up the positive factors, and everyone is asked to say why they think what they think.
Then, in a killer move, they’re asked: “How important, if at all, do you consider a new power station at Hinkley to each of the following? To the creation of local jobs? To the future of local businesses?” And take a moment to reinforce those concerns: “Why do you say that?”
Finally, after being led on this thoughtful journey, and immediately after mulling over the beneficial economic impact it would have in their community, the locals are asked if they’re in favour of a new nuclear power station. It’s the results of this, the final question, that are reported in the press release and headlines.
To me it seems clear that this long series of preceding questions will guide people down a path of thinking about a nuclear power station in a very different way to how they normally would. It’s a line of reasoning, and that might make sense if you were trying to advocate for a kind of structured decision making, but it’s very unlikely to produce results that reflect local views, partly because we’re all a bit thoughtless, in the real world, and follow our guts in odd ways; but partly because the penultimate question is “do you want your children to be unemployed?” rather than “are you all secretly terrified we might cock up and give you cancer?”.
So I still quite like nuclear power stations, but more than that, as ever, I salute the PR industry for finding new and elaborate ways to muddy the waters, and I salute the nerds who bust them for it.
JuJu said,
November 20, 2010 at 12:19 am
Glad to hear you spread the word about Kelvedon Hatch!
Onto the subject at hand, push polling was brilliantly defined on Yes Prime Minister
www.youtube.com/watch?v=2yhN1IDLQjo
James G said,
November 20, 2010 at 12:22 am
Ah, JuJu beat me to it. Finding the right clip was proving a little difficult from my mobile.
Guy Chapman said,
November 20, 2010 at 1:05 am
This reminds me of the late and much lamented David Silsoe QC, who was lead counsel for CEGB at Hinkley Point (and Sizewell and Drax). Ben, if you know Adam Hart-Davis do ask him about David some time. There are some delightful stories and he was very passionate about “his” power lines.
fnorman said,
November 20, 2010 at 5:42 am
I recall a survey of public opinion on animals in research that found the answer changed dramatically if respondents were given some information about the benefits to medicine.
www.ncbi.nlm.nih.gov/pmc/articles/PMC1115830/
At what point does helping people make an informed decision tip over into manipulation?
PsyPro said,
November 20, 2010 at 7:43 am
Ben opines: “So I still quite like nuclear power stations”. So do I. As a child of the 50s and 60s, nuclear power stations were the coolest things ever—the solution to all our problems, and amazing technology. I would take every tour I could to the experimental station in Manitoba (yes, Canada was and still is in the forefront of nuclear fission technology), proudly wearing my exposure badge, and envying the man with the Waldos manipulating some really cool and, obviously, dangerous shit. I suspect it was growing up with that (and reading all that R.A. Heinlein) that I really find it hard still to see nuclear fission technology as some extreme, bad thing. Dangerous? Well, duh! But bad? Not so obviously.
briantist said,
November 20, 2010 at 8:17 am
Reminds me of Yes, Prime Minister:
www.youtube.com/watch?v=2yhN1IDLQjo
“Mr Wooley, are you worried about the number of young people without jobs”…
Yogzotot said,
November 20, 2010 at 9:48 am
I have to chime in – though my first affective reaction to nuclear power is always a negative one, I accept it as, how Ben aptly puts it, “one of our least bad options for power”. I work in the power industry (not for EDF, and soon even specifically in renewables), and I think I have quite some understanding of what is possible, needed, and likely. For the foreseeable future there is no way around nuclear power if we want to cut greenhouse gases and still have a sustainable power supply. Give me a feasible large-scale electricity storage solution, and I change my mind.
On the topic, at least EDF’s polling firm seems not to have fudged it completely. One can question the affective ‘journey’ through the questionnaire, but obviously they were keen in providing information on both positive and negative effects.
fnorman is right: “On what point does helping make an informed decision tip over to manipulation?” – This is a very sensible and difficult matter. In always suggest two brilliant books on this: “Nudge” on “paternalistic libertarianism”, and Ropeik’s “How Risky Is It, Really?”.
www.amazon.co.uk/Nudge-Improving-Decisions-Health-Happiness/dp/0141040017/ref=sr_1_1?ie=UTF8&qid=1290246486&sr=8-1
www.amazon.co.uk/How-Risky-Really-Fears-Always/dp/0071629696/ref=sr_1_1?s=books&ie=UTF8&qid=1290246519&sr=1-1
Zod said,
November 20, 2010 at 10:12 am
The first person to come up with a set of Mr. Woolley questions that persuade the surveyee to agree that yes, medical testing on animals should be banned, wins eight experience points and a small heap of those glowing “health” thingies.
SteveGJ said,
November 20, 2010 at 10:36 am
I’m always suspicious of any survey purporting to show public attitudes as they are famously vulnerable to manipulation through wording. It’s also far to easy to present people with leading questions or allow for lazy moralising by opting for the nice sounding answer without thinking through consequences. This applies to a host of controversial issues – runway construction, wind turbines, nuclear power, testing on animals, public spending, the war in Afghanistan …
What the questions above appear to be is not so much a survey of public attitudes as directing people to ever so slightly more than the superfial questioning on a single dimensional aspect of policy. As limited as it is, it does invite at least some contemplation of consequences.
zaichishka said,
November 20, 2010 at 11:25 am
The Yes, Prime Minister scene sprang to my mind immediately as well. Here it is in written form, for those with slower computers or bandwidth restrictions:
paranoidangel42.livejournal.com/343220.html
TwentyMuleTeam said,
November 20, 2010 at 11:58 am
How ’bout this: “Do you prefer nuclear power plants over the alternative of people’s craniums exploding while shaving?”
asclegg said,
November 20, 2010 at 1:08 pm
I live near Hinkley Point. I used to take kids round it when I taught in Taunton. And I buy my electricity from EDF. But I work here in Namibia where EDF are currently turning over 40 square kilometres of the edge of the Namib, one of the planets oldest, most unique and fragile habitats, to a depth of 40m in order to extract uranium present there at less than 0.01%. They are using a technique which will leave the groundwater underneath permanently saline. If we are to benefit from Hinkley Point then surely we should be prepared to sacrifice our own Dartmoor which, unlike the Namib is not one of the planet’s biodiversity hotspots
eebkent said,
November 20, 2010 at 2:32 pm
Like several earlier contributors, the first thing which came to mind when reading the article was the marvellous and quite spot-on “Yes Minister” scene, which should be compulsory viewing for statistics students.
A couple of extra points. All surveys – and those on sensitive topics in particular – depend for the validity of their conclusions on how representative the respondents are. Reputable pollsters do make efforts to achieve this, but it’s only easy for demographic factors. What is pretty near impossible is to avoid bias through disproportionate inclusion of those with strong views, since they are naturally more likely to agree to take part.
Perhaps a future Bad Science column could be devoted to suggestions for a sequence of questions to be included in the General Household Survey to measure happiness. This loopy government proposal is even now the topic of choice during coffee breaks in ONS.
Marcus Hill said,
November 22, 2010 at 9:38 am
eebkent: It IS compulsory viewing for my statistics students!
SonofRojBlake said,
November 22, 2010 at 12:27 pm
@ Zod:
First, a League of Gentlemen reference: “Would you say you are very kind, kind, or not at all kind to animals?”
Now, the actual answer/questions:
Mr. Woolley, would you say you’re a compassionate person?
Are you worried that society is becoming more tolerant of the suffering of others?
Do you agree that doctors and scientists do some things that they cannot explain or justify to members of the public?
Would you support legislation that would reduce deliberately-inflicted pain and suffering?
Would you support a ban on all animal testing?
(Some implied questions are “Do you agree animals feel pain like we do?” and some degree of fudging would be necessary to equate cosmetic testing with medical testing. But you could do it, easily.)
schaff said,
November 22, 2010 at 1:01 pm
Of course you could make an argument that, when forced to think about the issues at stake rather than make knee-jerk reactions (the survey did solicit positive and negative factors), most people consider nuclear favourably – which is an interesting message.
Bruce44 said,
November 22, 2010 at 1:08 pm
“but it’s very unlikely to produce results that reflect local views, partly because we’re all a bit thoughtless, in the real world, and follow our guts in odd ways”
Sounds like an argument for being very wary of paying too much attention to local views to me.
warumich said,
November 22, 2010 at 2:04 pm
Hm, somewhat dodgy survey perhaps, but the result is not totally unexpected – (proper) social survey research has long known that people who live near existing nuclear power infrastructure are more positive about it than the population as a whole (look up the splendid work of Nick Pidgeon and colleagues at Cardiff). I would also add that this belongs to the cock-up rather than conspiracy pile. Big Energy, like Big Pharma, may not be the world’s cuddliest industry, but it’s definitely not in their interest to have a distorted idea of what the local public thinks about their project – EDF doesn’t want to spend £silly on projects that are cancelled at a late stage due to public opposition. The communications people at energy companies that I spoke to are _genuinely_ interested in public opinion and don’t see these surveys as merely a way to produce nice PR. Now what their press offices are doing with them may be a different thing.
bagpuss7 said,
November 22, 2010 at 4:26 pm
I once had a job being on call for earthquake monitors in Nuclear power stations so had to use and wear radiation badges…and pass the safety course on all stations.
Of the 13 stations I visited, only two ever gave a positive (and within limits) reading, Wlyfa and Sizewell “B”. Both of those readings were beaten by a reading I got working at Drax Coal fired station. Honest.
mrjohnc said,
November 22, 2010 at 5:21 pm
@bagpuss7
interesting
from Wikipedia – Fossil fuel power station
“As most ores in the Earth’s crust, coal also contains low levels of uranium, thorium, and other naturally occurring radioactive isotopes whose release into the environment leads to radioactive contamination. While these substances are present as very small trace impurities, enough coal is burned that significant amounts of these substances are released. A 1,000 MW coal-burning power plant could have an uncontrolled release of as much as 5.2 metric tons per year of uranium (containing 82 pounds (37 kg) of uranium-235) and 12.8 metric tons per year of thorium.[17] In comparison, a 1,000 MW nuclear plant will generate about 500 pounds of plutonium and 30 short tons of high-level radioactive controlled waste”
also
www.scientificamerican.com/article.cfm?id=coal-ash-is-more-radioactive-than-nuclear-waste
bagpuss7 said,
November 22, 2010 at 5:33 pm
…yep, that’s pretty much my point. I’d assumed radon at the time.
I did notice at the time that Nuclear stations had a far more diglent approach to safety in general as well, although to be fair, any kind of accident was bad publicity for them.
Some of the example accidents we were shown were in the “Darwin Award” category. Scary stuff.
RogerMexico said,
November 26, 2010 at 5:24 pm
Late to the party, but two points to pick from the original piece.
Firstly, as Leo Barasi says in his piece, it doesn’t take much investigative skill to get to the questions behind this sort of thing. If the polling company is a member of the the British Polling Council, current list here:
www.britishpollingcouncil.org/officers.html
they are obliged to publish all the details of the poll on their website within a day or two of publication. If they’re not a member, the poll will probably be at best amateurish and at worst dodgy.
Secondly even the most minor changes in wording can have a big impact on a poll’s results. A classic example of this happened last week concerning the “Twitter Joke Trial” and the fining of Paul Chambers. YouGov asked a question about it in their daily poll:
A man has been convicted of sending a menancing message after making a comment on the microblogging site Twitter and has been fined £385 plus costs. Paul Chambers’ comment said “Robin Hood airport is closed. You’ve got a week and a bit to get your s— together otherwise I’m blowing the airport sky high!!”
Mr Chambers said the comment was a joke reflecting his fustration at the airport closure, but the court ruled that an ordinary person would have been alarmed by the message and that Chambers should have been aware of
the possible consequences.
Do you think it was right or wrong to prosecute Mr Chambers for making the comment?
Of those polled 52% agreed with the prosecution against 36% opposed. However some commenters on the UKPR blog pointed out that the question wasn’t exactly right; the actual Tweet had started “Crap! Robin Hood airport is closed…”. The difference in tone might make some difference.
YouGov actually did this and, to their surprise, adding one word changed the result to 44% agree, 43% against (decent sample sizes, so way outside margin of error). Full story is here:
ukpollingreport.co.uk/blog/archives/2880
What a difference a Crap! makes.
(Of course you can argue that there are other problems with the question – and some of us did – but remember only change one variable at a time)
Martin B said,
November 29, 2010 at 7:21 am
Regarding what Bagpuss and mrjohnc said, when I worked in the nuclear industry the quantity of radiation emitted from Sellafield was reckoned to be about a quarter of that emitted from a paper-making factory down the coast. (IIRC, radiation was used to check the thickness of the paper.)
Staberinde said,
November 29, 2010 at 4:54 pm
I commission market research like this for corporate clients regularly.
As other contributors have pointed out, there’s no utility in companies paying for research which offers a distorted view of local opinion. The reason why they research local attitudes is to understand how best to persuade local people of the company’s point of view.
An important part of that communication is the research itself. Often, the street survey is the first locals will have heard of a new project or initiative – ‘launching’ it via a survey is a much cuddlier way of announcing these things than waiting for the local paper to run knocking copy. It feels consultative, rather than imposing.
It’s also pretty pointless asking people whether they want a potential Chernobyl near their kids’ school – the answer is generally ‘no thanks’. So if you were to conduct a survey of this kind, you’d be interested in considered (rather than knee-jerk) opinions. The local communications plan might focus on job creation, but there are other potential messages too: investment in the area, infrastructural improvements, safety, CSR etc. EDF can’t assume that the only thing it can or should say to local people is employment-related. But unless it pokes people and makes them think about these facets, it won’t know which route to go down.
If this all sounds smugly sinister, it really isn’t. EDF are in the legal business of making nuclear power stations. If you don’t like that, it’s a matter for Parliament, not for EDF. Since EDF are a big brand, they can be held accountable. Journalists can kick them and anyone who feels like it can do so too, online or locally. It’s much easier to do this with companies everyone’s heard of, which is why Greenpeace prefer to attack Nestlé rather than Sinar Mas (who?). And because EDF’s brand can be held accountable, it invests time and money in understanding the concerns of local people through market research, so that it can try to address those concerns where it can.
It makes for better business than waiting for people to go on a march or slap them around on a blog or newspaper front page.
Sometimes mundane things can make a big difference. For example, locals might worry about construction and road noise at night, while EDF are building the plant. It’s not out of the question for a company to build a new road, or schedule construction to minimise noise. But you won’t get that sort of insight from leading questions or assumption.
And finally, if a company can dig out some spurious stat and claim that local people are in favour of the new Whatever, yeah, they’ll probably use it. Because while it may not be independent, it’s a finding ‘valid’ enough to make the business case for potentially £hundreds of thousands of local engagement communications and activities. So if it’s valid enough for that, it’s valid enough to PR. What’s needed is for the PRs to check back with the research agency to ensure the proper caveats go out on the news release – which is where these things typically fail.
Joffan said,
December 2, 2010 at 9:49 am
Well, the level of pre-conditioning for the yay-or-nay question there pales into insignificance besides some of the tricks employed on the anti-nuclear side of the same coin.
I direct your attention to an after-the-fact assessment of one such: nuclearpoweryesplease.org/blog/2009/09/08/how-to-get-professionals-to-agree-with-your-opinion/
doc123 said,
January 21, 2011 at 2:20 am
Another example of a poorly thought through question.
In 2009, a proposal to change the law in New Zealand to make it illegal to smack kids lead to a back-lash of people wanting to be able to parent ‘as they see fit’. This ultimately ended in a ‘people-led’ referendum at a cost of hundreds of thousands of dollars to tax-payers. The question on the referendum…
“Should a smack as part of good parental correction be a criminal offence in New Zealand?”
And the result of the referendum? 87.6 percent of the 1,622,150 New Zealanders that voted voted ‘NO’. Quelle surprise!
Subsequently the NZ government passed the legislation in parliament with a minimum of fuss..????
mmonika2009 said,
February 12, 2011 at 11:21 am
Nuclear power stations are important. But everything should be with in the limit. if exceeds then it will create many problems.