Blind listening tests are strongly critiqued by Robert Harley in this editorial, published in The Absolute Sound. Harley seems to think the tests prove nothing.

Every few years, the results of some blind listening test are announced that purportedly “prove” an absurd conclusion. These tests, ironically, say more about the flaws inherent in blind listening tests than about the phenomena in question.

The article goes on to describe research done by the Swedish Radio. Apparently an artifact inherent to a codec that hadn't been noticed in 20,000 blind listening evaluations, was noticed almost instantly in non-blind circumstances. This gives Harley grounds to claim the following:

The answer is that blind listening tests fundamentally distort the listening process and are worthless in determining the audibility of a certain phenomenon.

Is this true? Do blind listening tests significantly inhibit the listeners' ability to notice certain audible phenomena?

This article was mentioned in a comment thread by Christopher Galpin. Oddthinking suggested making a question about it. Since nothing has happened in almost a year and the question is interesting, I'm bringing it up.

  • Is this a fun question? Blind tests are made to exclude bias. How would you compare a blind test to a non blind test, to prove an advantage of the non blind test, to show it is superior, if you can't prove it? Of course you could test audio cables, by measuring the output. Such an instrument for measuring the output of an audio equipment exists: It is called microphone. If you can't measure a difference between cheap cable and expensive cables, I guess there is another excuse: Only humans can detect the difference - but only if not tested blind. Poor marketing. Mar 4, 2012 at 3:17
  • 1
    The claim feels ridiculous to me, too. If it were true, similar questions about double blind methodology in general might also be relevant. I can't think of a plausible explanation for Harley's claim (but I don't know that anyone has debunked it either). Still, I tried to frame the question in an "objective" way as showing bias in the question doesn't help anyone. Mar 4, 2012 at 9:12
  • 2
    I'm having trouble seeing how this question could possibly be answered, but I am keen to see someone smarter than me try. After all, I believe blinded-tests are better than non-blinded, and as a skeptic I should have evidence for my beliefs...
    – Oddthinking
    Mar 4, 2012 at 14:34
  • 1
    @Oddthinking I wonder whether a sophisticated range of tests that involve a mix including deliberate misleading in some non-blind experiments would tease out the issues. Maybe we need a statistician.
    – matt_black
    Mar 4, 2012 at 20:28
  • 2
    @matt_black: deliberately misleading? You mean like a placebo? Isn't that what a good blind test does? (Also, not sure that a statistician is the right specialist here.)
    – Oddthinking
    Mar 5, 2012 at 0:19


You must log in to answer this question.

Browse other questions tagged .