Blind prejudice

September 4th, 2010 by Ben Goldacre in bad science, irrationality research | 36 Comments »

Ben Goldacre, The Guardian, Saturday 4 September 2010

Everyone likes to imagine they are rational, fair, and free from prejudice. But how easily are we misled by appearances? Noola Griffiths is an academic who studies the psychology of music, and she’s published a cracking paper on what women wear, and how that effects your judgement of their performance. The results are predictable; but the context is interesting.Four female musicians were filmed playing in three different outfits: a concert dress, jeans, and a nightclubbing dress. They were also all filmed as points of light, wearing a black tracksuit in the dark, so that the only thing to be seen – once the images had been treated – was the movement of some bright white tape attached to their major joints.

All these violinists were music students, from the top 10% of their year, and to say they were vetted to ensure comparability would be an understatement: they were all white european, size 10 dress, size 4 or 5 shoe, and between 20 and 22. They were even equivalently attractive, according to their score on the MBA California Facial Mask, which seems to be some kind of effort to derive a numerical hotness quotient from the best fit of a geometric mask over someone’s face. I’m not saying that’s not ridiculous: I’m just saying they tried.

In fact they did better. All the performances were also standardised at 104 beats per minute, so the audio tracks from each musician could be replaced with a recording of a single performance, recorded by someone who was never filmed, for each of the various pieces in the study. This meant there was no room for anyone to argue that the clothes made the musicians perform differently, and when the researchers checked in a pilot study, nobody watching the clips had spotted the switch.

Then they got 30 different musicians – a mixture of music students and members of the Sheffield Philharmonic – and sat each of them down to watch video clips with various different permutations of clothing, player and piece. All were invited to give each performance a score out of 6 for technical proficiency and musicality, and the results were inevitable.

For technical proficiency, performers in a concert dress were rated higher than if they were in jeans or a clubbing dress, even though the actual audio performance was exactly the same every time (and played by a separate musician who was never filmed). The results for musicality were similar: musicians in a clubbing dress were rated worst.

Experiments offer small constricted worlds, which we hope act as models for wider phenomena. How far can you apply this work to wider society? There’s little doubt that women are still discriminated against in the workplace, but each individual situation has so many variables that it can be difficult to assess clearly.

In the world of music, assessment of performance goals can be restricted to make individuals broadly comparable, and so there’s a reasonably long tradition of the field being used as a test tube for bigotry.  In the 1970s and 1980s, in an attempt to overcome biases in hiring, most orchestras changed their audition policy, and began using screens to conceal the identity of the candidate.

Female musicians in the top five US symphony orchestras gradually rose from 5% in the 1970s to around 25%. This, of course, could simply have been due to wider societal shifts, so Goldin and Rouse conducted a very elegant study (titled “Orchestrating Impartality”): they compared the number of women being hired at auditions with and without screens, and found that women were several times more likely to be hired when nobody could see that they were a woman.

What’s more, using data on the changing gender make up of orchestras over time, they were able to estimate that from the 1970s to 2000 – the era which shifted from casual racism and sexism in popular culture, to more covert forms – between 30% and 55% the trend towards greater equality was driven simply by selectors being forced not to see who they were selecting. I don’t know how you’d apply the same tools to every workplace. But I’d like to see someone try.

Bugger:

Sorry, slip of the brain, while cutting the piece down I erased the bit where I wrote the join from judging by appearance generally to judging by gender, that was slack of me. One of the interesting things about editing your own piece is remembering to forget everything you think you’ve said, so you can read it as if you’d never read any of it before. I guess this column now just reads like my chain of reasoning derails itself a bit halfway through, oh well.


++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks! ++++++++++++++++++++++++++++++++++++++++++

36 Responses



  1. eddwilson said,

    September 4, 2010 at 12:09 am

    Nice. Most occupations involve language and personal interaction, so hiding gender at interview would be a bit hard, but we all have our anecdotes.

  2. Rob_ said,

    September 4, 2010 at 1:40 am

    I find it surprising and interesting that not one of thirty musicians could spot that a different recording had been substituted. Even if all involved were keeping to a strict tempo any personality in their playing would show through in tell tale differences in vibrato and slight variations in timing.

    I suppose they might have got away with it if the clips didn’t show the hands in close up. Otherwise unless music schools are turning out robots rather than musicians perhaps people are so used to miming these days that they don’t even register the discrepancies anymore?

  3. Kris said,

    September 4, 2010 at 2:07 am

    I wonder how this applies to Nigel Kennedy, who purposely dresses down and uses a mockney accent, and is still generally considered an exceptional musician. But it does explain a lot of things about Hilary Hahn’s public image.

    The continued sexual discrimination by orchestras is interesting to me. I think I would personally biased towards female musicians rather than against them, perhaps in part because of their relative rarity. It’s especially interesting considering that female opera singers have traditionally been generally overweight to obese. I’d be very interested to understand that selection bias. My best guess is that it’s a matter of a perceived physiological advantage (when none exists in fact).

  4. physicsmum said,

    September 4, 2010 at 3:40 am

    “Everyone likes to imagine that they are rational, fair and free from prejudice.”
    But we all know that no one else is!
    Is it also not the case that anyone is taken more seriously when dressed “properly”?

  5. thinkerhead said,

    September 4, 2010 at 3:53 am

    Why, Ben, have you cast this as a sex discrimination issue?

    Apart from Noola Griffiths’ stated interest in ‘gender and musical performance’, and therefore her possible intent, it’s difficult to see from your post, what this has to do with sex discrimination.

    To recap; 1)no comparison between male and female performers is made here, 2)the asessment panel are not identified as all male or all female, 3)the performers were chosen by the experimenter to be all female but their presumed attractiveness was carefully matched and then disguised in various ways, 4)the only stated variable was their choice of dress [Perhaps we should use the term clothing in case dress is considered too sexist], 5)the sex of the person who played the sound track in all the trials is not even stated.

    The only valid conclusion from what is reported here is that the clothing people perform in affects the way their musical performance is received. It may be that ‘inappropriately’ clothed male performers would be received better (or worse) than ‘inappropriately’ clothed female performers but this report doesn’t even show that.

    I think it would indeed be bad science to reach a conclusion of sex discrimination from this data. Does the original paper, which I don’t have free access to, contain further details that would support it?

    If you wish to tell us that sex discrimination occurs in performance art, and it’s unfair, that’s fine, but please don’t lose your analytical skills or rationality.

  6. dreg1 said,

    September 4, 2010 at 7:04 am

    The study says not much about discriminating against women. I would expect similar biases with differently dressed male musicians. The gender bias could be addressed by mixing male and female players. Without that, this simply demonstrates a conventionality bias.

  7. Jon d said,

    September 4, 2010 at 7:05 am

    I can’t penetrate the paywall and the abstract is a bit thin.
    the abstract says

    (1) observers have a strong concept of appropriate dress by genre of music; (2) inappropriate and body-focused dress may have a detrimental effect on perceptions of performers’ musical abilities; and (3) performers’ body movement style may also affect perceptions of their abilities.

    just the three hypothesies being tested then?

    The word ‘may’ in the 2nd and 3rd bullet makes it sound as if they found an effect but the stats didn’t come in at p<0.05 or better.

    Off the top of my head I can't help thinking it would have been a lot more informative to repeat with male musicians*

    also I wonder if the performers actually did perform better when dressed 'appropriately' than when dressed inappropriately.

    Anyway wrt the second paper audition performance, or performance in any test might not be the only factor in deciding wether to employ someone – women may leave to have children, dropping their employer right in it if they have a crucial role.

    * (with equivalent male typical clothing before anyone pipes up)

  8. nigel said,

    September 4, 2010 at 7:11 am

    The NHS Jobs website can be set up to strip off significant amounts of extraneous demographics so that short listers are less prone to unwitting or deliberate bias. It would be interesting to see whether this made a difference in shortlisting behaviour – I suppose we can guess the result. Interestingly the removal of personal information was one of a number of objections to the selection process for junior doctors – the main reasons were much more legitimate.

  9. adzcliff said,

    September 4, 2010 at 8:05 am

    Hi Dreg1

    “The study says not much about discriminating against women. I would expect similar biases with differently dressed male musicians.”

    I’m not sure the main study discussed here was looking at discrimination ‘against’ women, more discrimination ‘between’ women/performers. Agreed though, you would’ve thought a male arm and mixed arm to the study would’ve thrown up some interesting and useful findings about gender bias?

    Adzcliff

  10. dreg1 said,

    September 4, 2010 at 8:15 am

    @adzcliff:

    I know that the study did not address the gender question but Ben’s article suggested this. He wrote: “How far can you apply this work to wider society? There’s little doubt that women are still discriminated against in the workplace”

    While the second sentence is correct, this discrimination in the wider society is not explained by the biases demonstrated in the reported study.

  11. adzcliff said,

    September 4, 2010 at 8:23 am

    Thanks for that Dreg1

    Right you are, I mis-spotted when the gender bias issue was introduced in the piece. I suppose the study only provides some basis for discrimination ‘between’ types of women in the workplace.

    Note taken.

    Adzcliff

  12. rikkus said,

    September 4, 2010 at 9:42 am

    “and how that effects your judgement of their performance”

    Should be ‘affects’.

  13. phayes said,

    September 4, 2010 at 10:46 am

    “In the 1970s and 1980s, in an attempt to overcome biases in hiring, most orchestras changed their audition policy, and began using screens to conceal the identity of the candidate.”

    All candidates? For all positions? I’d be surprised if they ever rigorously applied that policy. An orchestra is a concert ticket and CD selling business and if I was running one and hiring violinists and the musically second best candidate happened to be Hilary Hahn…

  14. bagpuss7 said,

    September 4, 2010 at 11:08 am

    It would have been instructive to repeat the experiment with men with the same controls and conditions.

    I’d suspect the results would be broadly similar, which would be that ‘appearances make a difference to the opinions of those judging you’. No shock there.

    Gender bias in the workplace is another matter altogether, although if all workplaces were meritocracies then it wouldn’t matter. I’m not holding my breath.

  15. Luke Jostins said,

    September 4, 2010 at 11:19 am

    @Jon d

    I think they are just hedging here – the p-values for the effect of dress on technicality and musicality are both p < 0.001.

  16. Bonsai said,

    September 4, 2010 at 11:25 am

    String players are often characterised very differently… or rather conform to different personality structures manifest in diverse proclivities. String players are notoriously competitive because of the ranking within the sections. But what about the outliers? The lower middle class or lower class – perhaps even beer-drinking – without the networks that proceed from coming from a professional musical family. What about the difference between wood-wind, brass, and string-players? Interested in your new bent.

  17. Ben Goldacre said,

    September 4, 2010 at 11:28 am

    Hi there

    Sorry, slip of the brain, while cutting the piece down I erased the bit where I made the leap from judging by appearance generally to judging by gender, that was slack of me. Oh well, sorry!

  18. Martin said,

    September 4, 2010 at 11:29 am

    ?! Ben that was all reading fine and interesting until you suddenly derailed to sexism. Where did that come from? As others have said there’s no comparison here between the sexes.

    There’s possibly some things about how dress influences our perceptions of competence, but is that particularly new? The interesting thing here was the care taken with the experiment.

  19. Martin said,

    September 4, 2010 at 11:30 am

    Ha ha if only I’d posted that two minutes earlier

  20. MagsLHalliday said,

    September 4, 2010 at 11:41 am

    @Jon D
    “women may leave to have children, dropping their employer right in it if they have a crucial role.”

    And if you chose a man over an equally qualified woman on those grounds you’d be breaching the Sexual Discrimination Act. Men may also leave to become the primary care giver to children, especially if the mother earns more than – or has a more reliable wage – the father. So it doesn’t matter what gender your first violinist is, there’s a risk they’d “drop you right in it”.

  21. PeteF said,

    September 4, 2010 at 11:52 am

    On the issue of sexism, this study may be looking at discrimination within a gender, but, although this isn’t examined in the piece, it is a fair bet that the tendencies discovered operate more strongly for people judging women than judging men. Obviously the information would need to be compared against a similar study of men but then it could actually demonstrate a presumably sexist tendency such as “men/women’s abilities are more likely to be judged according to their clothing choices.” Like I say, my hunch is that men would be less affected by this, but either way gender could be an influence.

    Additionally, talk of employment slightly skews matters because the people in the study were presumably trying to make a disinterested aesthetic or technical judgement and it seems their actual judgement of the piece was impared by the player’s dress. It seems to me that this is more analogous to failing to enjoy a concert played by a woman in ‘night club dress’ rather than being disinclined to hire her?

  22. Bonsai said,

    September 4, 2010 at 11:57 am

    But in practice (albeit covertly) this is still a major reason that certain age-groups i.e. child-bearing (esp. 35-41), women are still discriminated against in almost every sector. Many ways to hide this in the evaluation of interview performance… etc.

  23. kristenSF said,

    September 4, 2010 at 12:30 pm

    @MagsLHalliday
    “And if you chose a man over an equally qualified woman on those grounds you’d be breaching the Sexual Discrimination Act”
    And that simply means they will find something else, however tenuous, to cite.

  24. Dan Kimberg said,

    September 4, 2010 at 1:06 pm

    This seems like a fairly mundane and predictable kind of study, the sort that has only one possible outcome, but that someone managed to turn into a line item on their CV. Many similar studies on biases in judgment are published every month, and as these studies go, this one seems harmless enough, even though its scientific impact will probably be negligible. But I missed what it’s doing here. Where’s the bad science?

  25. Bonsai said,

    September 4, 2010 at 2:00 pm

    I guess that depends on whether you consider epistemology to also apply to science… but then you’d be picking at the classical foundations underpinning western civilisation.

  26. 400guy said,

    September 4, 2010 at 7:12 pm

    Let me offer just an anecdote about sex-related differences in musical
    performance.

    In one rather dull evening back in the seventies, a few of us students
    at a music school abandoned our practising for a bull session.
    Seeking some light instead of the heat generated by our discussion of
    the those famous differences, we set out on a tour of the practice
    areas, up one corridor and down the next. The common arrangement of
    imperfect soundproofing and small windows in practice studios made it
    easy to listen and guess the sex of the inhabitant before peeking in.
    The result: none of our guesses was very confident, and they were not
    notably right either.

    Just FWIW, if anything.

  27. clt47 said,

    September 4, 2010 at 10:30 pm

    The conductor Jane Glover commented some years ago (on Woman’s Hour, IIRC) that reviewers often wrote as many words about her shoes and clothing as they did about her conducting, and that she had been described as ‘over-dressed’ and ‘dowdy’ (again, if memory serves, as this was some years ago and possibly pre-internet) by different reviewers of the same concert. I hope this is no longer the case.

    Confession: I always hope any female soloists in a formal concert will be wearing something beautiful, striking or both. This would not cancel out a substandard performance however, it just adds to the pleasure of a good one. I have been stunned by a performance of unacommpanied Bach by a violinist wearing jeans and teeshirt.

    It’s a pity the chaps are largely stuck with penguin suits.

  28. snoozeofreason said,

    September 5, 2010 at 12:07 pm

    “What’s more, using data on the changing gender make up of orchestras over time, they were able to estimate that from the 1970s to 2000 .. between 30% and 55% the trend towards greater equality was driven simply by selectors being forced not to see who they were selecting”

    That sounds like a very difficult thing to estimate if you are relying on observational data.

    The link to the Goldin and Rouse paper is broken, and my Googling has so far only found me a rather uninformative abstract. So it could be that some very clever trick was used. But in the absence of that kind of magic, how do you get round the problem that orchestras who use screens are probably more committed to eliminating bias than ones that aren’t, and would as like as not end up hiring more women even if they had not used the screens.

  29. ChrisPartridge said,

    September 6, 2010 at 7:56 am

    A fascinating study that underlines lots of research showing the overwhelming effect of the visual over any other sense when making judgements. Wine, for example – even trained tasters can be fooled into judging white wine as red simply by adding a little food dye.

  30. anon3455 said,

    September 6, 2010 at 12:49 pm

    @rikkus, conserning effects vs. affects.
    How would you know?

  31. elvisionary said,

    September 7, 2010 at 12:50 pm

    I would love to see the following hypotheses tested (I’m sure most of them have been):

    1. Both men and women are routinely judged (i.e. discriminated for or against in selection processes) on the basis of appearance.
    2. They are judged both for their dress and their underlying attractiveness (if the two can be distinguished)
    3. Women are judged in this way to a greater extent than men.
    4. Women are every bit as guilty as men of discriminating in this way (in judging both men and women).
    5. But women may use different discriminatory criteria from men.

    In other words, we’re all shallow and guilty of judging books by their covers, but women get the rough end of the deal, at the hands both of men and of other women. Anyone got evidence to support this?

  32. TheSacredMongoose said,

    September 7, 2010 at 8:58 pm

    As a musician and a busker I find this most interesting, as obviously what you wear often determines what money you get. For some reason low cut tops do quite well…..

  33. WCHB said,

    September 11, 2010 at 9:10 pm

    A forerunner to this is shown in Malcolm Gladwell’s book “Blink”, where he shows examples of auditioning musicians playing behind a screen.
    As a high school teacher (now retired) I wondered about the effect of grading of Music students if they played behind screens. No cash involved, but a difference between an A/B grade at an early stage could affect entry qualifications and selection to further education.
    I do believe that examiners are proud of their unbiased professionalism – but are they?
    No evidence, sorry.
    (And I’m not a failed music student!)

  34. timbod said,

    September 15, 2010 at 10:41 pm

    I’m dissapointed that all the raters were classical musicians, as I would expect that almost all of them would accept the norm of putting on very formal clothes for concerts (even for university student concerts). Preconceptions might not be as great for those who go to a wide variety of concerts with only one classical concert a year (or so). The worry that such people would be worse at discerning good classical violinists from weak ones is controlled for because the music sound track is the same in all cases.

  35. Mike said,

    September 20, 2010 at 4:41 pm

    I’ve just got round to reading this. The paper has a great design but statistically it is hardly ‘cracking’. Only p-values are reported and so we have no idea of the size of the effect(with a big enough sample almost anything is significant). Interestingly there appears to be more variation in both the proficiency and musicality outcomes between performers than between clothes styles, and for musicality a significant ‘interaction’ i.e. the results were not consistent from one performer to the next. This makes it very difficult to extrapolate.

  36. Felicia said,

    December 19, 2010 at 10:12 pm

    As a female musician I found this article fascinating regardless of any flaws. I always give serious thought to what I wear when performing, though it might not seem so to those outside of the folk world. I also try to “reinvent” my wardrobe on a regular basis. Generally, my husband/partner does not go to great lengths over his performance clothing though it’s always clean and neat in appearance and he does ask my opinion on specific items. Hm. Good food for thought. Thanks for posting!

You must be logged in to post a comment.