Ben Goldacre, The Guardian, Saturday 26 February 2011
Science is about disproving hypotheses, and no matter what the armchair conspiracy theorists tell you, torpedoing cherished ideas is a very good way to make a name for yourself in academia. Here are two fun ones from the literature this month.
Firstly: are sniffer dogs for real? Animals respond to humans, after all, and especially domesticated animals: that’s the point of them. This is why the placebo effect is so wonderfully effective in animals, and of course in children.
Clever Hans was a horse who could read, spell, perform arithmetic, handle fractions, and differentiate musical tones. But in 1907 a psychologist Oskar Pfungst found the horse was actually responding to his owner, and his audience, who unwittingly gave involuntary cues to the animal as he got nearer to tapping out the right answer with his hoof.
3 academics have now replicated this phenomenon in sniffer dogs. They took 18 dog and handler teams, certified for detection of drugs or explosives, where the dog was trained to indicate it had found something by standing next to it. They ran a search for drugs or explosives in a four room building, in one of four conditions. Sometimes there was nothing to find. Sometimes there was an empty box with a sign on it, telling the handler this was the target. Sometimes there was a box of delicious decoy sausages. And sometimes there was a box of decoy sausages, with a sign on it, telling the handler this was the target.
The dogs were supposed to do the searching work, and the handlers were supposed to ignore the signs. In no case were there any drugs or explosives, although the handlers thought there were.
The results were a smuggler’s dream come true. The “dogs” kept confidently finding the empty boxes, when the humans could see the marker saying where they were. So in reality, although the human handlers cheerily thought the dogs were finding these boxes, in fact it was the humans themselves, just like with Clever Hans. The dogs also found the sausages a lot (though they were supposed to be finding drugs and explosives), but they were twice as likely to do so when the handler saw a sign misleading them into believing this was the target. Overall, the handlers beliefs had a greater influence over what was found than the dogs. If you’re a smuggler, get a haircut and buy a suit.
Meanwhile, you’ll have heard all about brain scanners being the next horizon in lie detectors. Neuroimaging lie detectors work by watching areas of the brain which are known to exhibit modest changes in blood flow when you’re shown a stimulus that has “salience”, like the box of poison you used to kill your wife, perhaps.
In a new study, the experimenters taught their subjects how to fake that signal. The participants were put in a scanner, shown a series of dates, and asked if each was their birthday. One date was. If they tried to deny their genuine birthday, the brain scan gave them away: they had concealed knowledge, they recognised their date of birth but denied it, and the machine caught them 100% of the time.
But then they were given a faking strategy: whenever you see an irrelevant date second in the sequence, you imperceptibly move your left toe before clicking “no”, and so on. This introduced salience to the other dates, as well as the birthday date. The recognition of – and deception about – their birthday no longer stood out in their brain bloodflow activity, and the computer could only spot the subjects’ deception about their birth date 33% of the time. As a lie detector this is basically useless.
Sometimes, when I’m in a fanciful mood, I enjoy devices like brain-scanning lie detectors, and hi-tech sniffer dogs, because their appeal speaks to our desire for simple mechanical explanations in a complex world, and for machines to aggrandise intuition, or make it more sciencey. But I enjoy them mostly because – like the ridiculous new porno-scanners in US airports, that give staff a view of your breasts and penis – they show how much of security is about theatre rather than reality.
Kimpatsu said,
February 26, 2011 at 2:14 am
It’s long been known that airport security is about theatre, and not reality. All we have ever needed is better-secured cockpits (done since 9/11), and a willingness among the passengers to fight back if need be. So the real question becomes, why are governments licking Michael Chertoff’s shoes?
briantist said,
February 26, 2011 at 3:41 am
It’s nice to know sniffer dogs work the same way as TV Detector Vans.
DancingFool said,
February 26, 2011 at 5:02 am
So what you’re saying is they studied what dogs do when there is nothing to find. Though this may explain some dogs (or is that handlers?) high false positive rate, it gives no idea at all whether you could get by the dogs if there was something to find. If I was a smuggler, I wouldn’t relax yet.
Yogzotot said,
February 26, 2011 at 6:18 am
DancingFool beat me, exactly my thought: I bet this study gets easily reported as sniffer dogs being useless, however, it has only established that there is a high false positive rate – which, to some degree makes sense: If there are no explosives, but the handler thinks a specific area or package is likely to contain one, the dog may go for those “obvious” signals of the handler, as there are no other, stronger cues to use.
This does not mean that in the case explosives *are* there, any cue/smell associated with the trained target would supersede the cues given by the handler. A follow-up story is in order to establish (with a comparable design) what the hit rate for “real” cues is.
Geeb said,
February 26, 2011 at 7:32 am
Probably breasts *or* penis for most people. 😉
Bill Bigge said,
February 26, 2011 at 9:18 am
“The dogs were supposed to do the searching work, and the handlers were supposed to ignore the signs. In no case were there any drugs or explosives, although the handlers thought there were.”
Isn’t that like claiming that a drug doesn’t work by ONLY giving patients a placebo (never the actual drug)
There is no ‘signal’ in this test, just noise, so it doesn’t test to see if the dogs can find a signal, it just reveals that they generate false positives when there are no real positives, and that it is largely directed by the handler.
Like Dancingfool says, If I were a smuggler I wouldn’t relax yet – but when visiting an airport dress smartly to avoid delays.
Matt J said,
February 26, 2011 at 10:12 am
I think the study on sniffer dogs is somewhat misleading. It doesn’t matter too much if a bomb is ‘detected’ when there is none. I think the detection powers of dogs when there is something to find is fairly well documented. Just because their owners (or food) influence the dogs doesn’t make them useless. The American TV show Mythbusters tested sniffer dog distraction techniques. They tried both food decoys and placing the ‘drugs’ in something disgusting (which would discourage the handler from wanting to look thoroughly) but in all cases the dog was successful.
adzcliff said,
February 26, 2011 at 10:21 am
I think I’m with other posters here. Surely all this shows is that sniffer dogs will find false positives (rather than true negatives), when no true positive is available?
Not sure?
Adzcliff
Valis said,
February 26, 2011 at 10:54 am
I know this is anecdotal, but I used to smuggle cannabis about ten years ago. Looking the way I do they used to pull me off every time going over the border. They had sniffer dogs all over the car every single time, not once did they react. I had about 20 kilos of weed in the car every time, sniffer dogs never reacted once. They were specifically looking for cannabis at the border crossing, never found my stuff.
On a side note; I can’t believe people are referencing Mythbusters? It’s a TV show, for Pete’s sake! I’ve seen the show once or twice myself, it’s definitely not science.
AgentR said,
February 26, 2011 at 11:06 am
The strategy for foxing the “lie-detector” made me think about The Stainless Steel Rat, a series of Sci Fi romps written by Harry Harrison, published donkey’s years ago. 1960s?
The hero is being interrogated by some heavy duty forces of the state with the aid of a lie detector that, in essence, will sense his anxiety levels. They ask him two or three ‘calibration’ questions to get his normal response level when telling the truth. They are simple innocent factual-type questions. He answers them honestly/accurately and simultaneously thinks about something frightening (“Oh God they’ve got me, I’m gonna die” – that sort of thing). So when they ask him the ‘biggie’ and he lies, there’s no difference in his response! Simples.
Just thought I’d let you know the idea is by no means new. Apologies that I can’t be more specific with the details. Can anyone help with that?
Valis said,
February 26, 2011 at 11:43 am
Ah, “Slippery” Jim DiGriz :p Loved those books!
Everything I’ve read about lie-detectors seems to show they are complete woo. The only thing they detect seems to be peoples’ gullibility.
Richard said,
February 26, 2011 at 1:21 pm
There is an interesting book by Amber Marks called “Headspace”, where she investigated the evidence around the use of sniffer dogs.
She came to the conclusion that there was a suprising paucity of evidence, and what limited evidence there was suggested the dogs had no benefit. Though, of course, it was a book rather than a systematic review of the evidence.
PeteR said,
February 26, 2011 at 1:23 pm
It’s curious to know how an experiment is designed.
The sniffer dogs study shows that handlers influence their dogs behaviour – in the absence of the target scent the dogs were trained to identify.
That’s an interesting result. But why, one asks, didn’t the experimenters include a category where real drugs and explosives were hidden? Wouldn’t that have made the test results even more interesting?
So maybe they didn’t have any real drugs or explosives? Well, actually they did. Their paper explains:
“…at the beginning of each testing day, the experimenter carried a metal box containing 12 half-ounce samples of marijuana triple bagged in sealed plastic bags, and a canvas bag containing 12 half-ounce samples of gunpowder triple bagged in sealed plastic bags.”
Maybe the experimenters just weren’t interested in dogs’ detection rates in their real world working conditions:
“Importantly, this study was not evaluating abilities of these detection dogs to detect their target scents. Because all dogs were certified, many with confirmed deployment finds their ability to correctly locate target scent was considered to be previously established.”
But hang on, isn’t this study investigating to what degree handlers influence their dogs’ ability to find targets? Doesn’t that specifically call the above statement into question?
I don’t want to criticise this piece of research. It’s come up with some interesting results. I just hope the team will now be spurred to get the drugs and explosives out of their briefcase and do further tests.
vickiw said,
February 26, 2011 at 2:54 pm
Thinking about it surely what the research needs is a scenario whereby a real target is placed in a room alongside an empty box which reads target in order to identify if the handlers influence leads the dog to choose the fake target over the real one?
David Cruise said,
February 26, 2011 at 6:36 pm
I’m no smuggler but I REALLY want Valis to tell us how he did it.
rikkus said,
February 26, 2011 at 8:02 pm
It may indeed be that those who use sniffer dogs are useless, rather than simply influenced by handlers as the study shows, but I’d imagine they have some value as a deterrent.
Joel said,
February 26, 2011 at 8:27 pm
@Valis
“On a side note; I can’t believe people are referencing Mythbusters? It’s a TV show, for Pete’s sake! I’ve seen the show once or twice myself, it’s definitely not science.”
Feynman would (might) disagree
xkcd.com/397/
asuffield said,
February 26, 2011 at 8:29 pm
If sniffer dogs generate false positives based on the expectations of the handler, this seems to have profound legal implications on their status in searches.
Currently, if police suspect there are drugs in a car but they have no legal reason to search it, then they will sometimes try using a dog; if the dog reacts to the car then this is considered sufficient grounds for a search. If it turns out that the expectation of the handler is the only trigger, then a whole lot of possession convictions may be overturned because the justification for the search has been invalidated.
I’m fascinated to see how this one will play out in the courts.
James said,
February 26, 2011 at 9:41 pm
A field study of drug dogs in New South Wales found their false positive rate to be 73%. This was from a big sample of public stop and searches. But obviously it does not tell us the false negative rate. However there was one dog at Strangeways prison a few yars ago with a 100% detection record. Some of the inmates took a contract out on it and it was withdrawn from service. All the prison service could say was that ‘it was not operating in line with th other dogs at the prison’. Training a dog relies on operant conditioning, and the learning has to be constantly reinforced. How well that is done varies, and it is an open question as to how many drugs a dog can be trained to detect at any one time. In the meantime a tchnology called Ion-Tracking appears more accurate. Also it tells you which drug has been detected. Something Fido can’t do.
omnis said,
February 27, 2011 at 9:31 am
Last time I went through an airport it was slightly amusing to see all the sniffer dogs converge on an 80 year old man in a wheelchair. The police apologised to his 80 year old wife/wheelchair pusher and then searched all his bags.
It didn’t take long to find the sausages in there.
Of course, they never found the semtex/heroin stashed under his seat!
tialaramex said,
February 27, 2011 at 11:06 am
How about if the trial (perhaps tacitly) didn’t test the dogs actual ability to detect scent because that’s something those who agreed to the test don’t want to hear about? A more complete test which everyone refused to participate in would be a non-story, since it would have no results at all.
When introducing monitoring of bus positions you pretty much have to reassure the drivers that this information won’t be used to penalise them, but only to provide information to the travelling public.
What happens to the operators of sniffer dogs if a study is released which says plainly “they’re mostly theatre” and it so happens at that moment there’s political pressure to cut the budget to the relevant organisation? Nobody wants to be out of work.
HelenC said,
February 28, 2011 at 2:40 pm
Maybe sniffer dogs are just confused by the reactions of handlers, so if they can’t smell anything but they’re getting human non-verbal cues that something “is” there then they give the signal anyway?
Richard Gadsden said,
February 28, 2011 at 3:26 pm
What I’ve read about lie detectors is that they’re pretty good at detecting lies when those lies are being made by basically honest people who feel nervous and uncomfortable about lying, but are otherwise in largely unstressed conditions when there is little at stake.
If there is a lot at stake, if there is a lot of stress, or is the person is a habitual liar (or politician) then they don’t work. The problem is that those are the situations where the results of the lie detector are useful – the person who rarely lies (or at any rate, normally only makes “white” lies) who is nervous and shaking at the prospect of lying is easy to spot without a lie detector anyway.
MrNick said,
February 28, 2011 at 3:27 pm
“The strategy for foxing the “lie-detector” made me think about The Stainless Steel Rat, a series of Sci Fi romps written by Harry Harrison, published donkey’s years ago. 1960s?”
The Stainless Steel Rat series covers the period 1961 to 2010:
1961 The Stainless Steel Rat
1970 The Stainless Steel Rat’s Revenge
1972 The Stainless Steel Rat Saves the World
1978 The Stainless Steel Rat Wants You
1982 The Stainless Steel Rat for President
1985 A Stainless Steel Rat is Born
1987 The Stainless Steel Rat Gets Drafted
1994 The Stainless Steel Rat Sings the Blues
1996 The Stainless Steel Rat Goes to Hell
1999 The Stainless Steel Rat Joins the Circus
2010 The Stainless Steel Rat Returns
I have the first seven and enjoyed every one.
Nick
richardelguru said,
February 28, 2011 at 4:55 pm
@ omnis “semtex/heroin stashed under his seat”
Wow you’d really get blasted with that mix!
anon3455 said,
March 2, 2011 at 6:33 am
“Probably breasts *or* penis for most peoples.”
FTFY
ChrisPartridge said,
March 5, 2011 at 8:01 am
What this study proves is that sniffer dogs are used not to detect things but to make random searches socially acceptable.
You can imagine the rumpus if officers just hauled every tenth person out of the airport terminal and searched them. Instead, they say ‘Sorry Sir, the dog says we must search you.”
mikewhit said,
March 5, 2011 at 12:08 pm
Yes, really I think it’s the human who needs more training.
I would say that dog/handler teams need regular exercises of the nature described, to ensure they keep doing their jobs properly.
I don’t think anyone doubts that dogs have a highly developed sense of smell, and if trained correctly could tell if an incompletely sealed, or contaminated ‘package’ was there.
But since the dogs operate on scent, wouldn’t a better method, or at least an alternative, be to stand the person and luggage in a cubicle out of view (and previously unseen) to the dog and handler, and just pipe a flow of clean air through the cubicle and past the dog, which could then react, or not.
Richardo said,
March 20, 2011 at 4:51 pm
As a youthful chap I did work with the filth.
They had german shepherd dogs.
With mine own eyes I did see’th criminals tracked down who had decampethed from the stolen car. Not once did this occureth but several times.
Dog arrived. sniffed grouund. sniffed ground and appeared to be heading in a direction with a purpouse. Eventually the dog either lost the scent – due to contamination of a busy footpath or the clever car thief being picked up in another car, OR the naughty one was caught. (this happened – like loads of times when I was actually there. I saw it wiff mee own eyes)
So there.
Courts of law accept “propensity of animals” evidence in this regard.
Dont get me wrong – I am with you on most of the mallarkey written about here – but in this instance I think there is another side to consider. Unlike all that Mckeith talked about Mckeith.
therubberbishop said,
April 25, 2011 at 11:27 pm
Anything which weakens the case with sniffer dogs is fine. The extremely questionable fishing expeditions at main London Railheads (and elsewhere no doubt)are not morally acceptable. It’s only because most fish in this barrel are small time personal users on a night out that nobody has attempted to fight. (Hint simple posession is usually a wrist slap but can be a criminal record if the boat is rocked)
Less ranty and more germane,Penn and Teller, in the episode of Bullshit about polygraphs, had an expert who explained a simple strategy. During the questioning slowly and discreetly clench and unclench ths anus.
www.youtube.com/watch?v=NsoI92BfmqY
parasura said,
April 26, 2011 at 5:47 pm
I’m guessing that a study involving both illicit drugs AND explosives would involve a whole different world of scrutiny from one only pretending to have such things. Personally, whilst I agree that false positives in this case don’t really clarify much of anything (let’s face it: in the real world you’re either in possession or not as the case may be) being stopped by plod because you’ve aroused a dogs interest is very irksome. Mind you, having been subject to security checks at a local airport whilst driving an actual emergency vehicle on an actual emergency and in possession of a) Class A drugs b) Flammable pressurised containers c) More sharp pointy pieces of metal than you can shake a stick at and d) Almost certainly excessive amounts of both salty and sugary water, (I don’t know what they were looking for but it was none of thye above) I can only assume the world finally went mental.
flies said,
May 11, 2011 at 9:25 pm
Clearly, without a false negative rate this study tells us little about whether these dogs make us safer.
On the other hand, asuffield points out that this study has very clear implications in matters of law as far as what constitutes a legal search. If the world were fair (that is, run by rational people with the public interest at heart), this finding would have an actual effect on criminal justice. Would that it were so.