The Problem with A-B’ing and Why Neil Young is Right about Sound Quality

Allen Farmelo writes: With news of Neil Young pushing to get studio quality audio out to the consumer, it occurred to me that it would be good to get some perspective on how we go about comparing audio quality.  One of the most problematic issues surrounding audio quality and file resolution these days has been that, more and more, we hear about people not being able to tell the difference between compressed audio and full-resolution audio, and by extension some people are claiming that efforts like Neil Young's are out of touch and irrelevant.  I couldn't disagree more.

In my opinion, one of the biggest pitfalls in this arena is the dreaded A-B test.  In our seemingly infinite desire to manipulate the response of our fellow humans, we tend to stage A-B tests in two ways:

1) Almost-Barely: A-B because you can barely tell, and thus proving that the two items are close enough to be interchangable.

2) Absolutely-Boldy: A-B because you want to show how different two things are (and typically the superiority of one of them, which the test-designer is likely selling).

In my opinion, both types of A-B tests won't tells us what we want to know, but it's the first type, the Almost-Barely tests, that I take issue with here.  (And I'm also inadvertently and respectfully drawing into question what, Ethan Winer, (who I adore) has said about "scientific" A-B testing in Tape-Op #88, p.66.)

We've all done it at some point.  A.  B.  A.  B.  Flip.  Flip.  Flip, flip, flip.   This is B?  Ok, A.  B.  Ok, is that B?  Ok, do A?  Gosh, they're close!

The Almost-Barely tests seem so objective.  Here's A, and here's B.  See, soooo close!  The conclusion: because most .....