ABX Testing and Distortion


ABX testing is a way to compare two similar versions of something, in this case sound. The user is presented with an audio stream called the A stream, and another slightly different audio stream called the B stream (or vice versa). The user can freely toggle between the A and B streams listening for differences. Once the inferior stream has been picked out, the user can then audition another stream, called the X stream, and then decide if the X stream is the same as the A or B stream.

The benefit of ABX testing is that it’s immediate and doesn’t rely on memory.

The QA_ABX software below is unique from other ABX software in that it takes a single source file in the WAV format, loads the file into RAM, and then generates distortions selected by the user on the fly during playback. The switchover between distorted and undistorted is seamless and gapless. To the DAC, it appears as an uninterrupted stream. The DAC may be an external USB DAC or in the internal Audio DAC on your PC. The DAC you wish to use must be set as your primary driver in the Windows Control Panel. Currently, there is no option in the QA_ABX software to select playback devices.


The QA Audio Analyzers can measure vanishing low levels of distortion. But we still have people wondering if it can measure “low enough” and what precisely is “low enough” to matter. The QA_ABX program attempts to help a user answer these questions. The interface is shown below and is fairly simple. First, you load a WAV file. The file is pre-buffered into RAM and will begin playing. The user can then select the A, B or X buttons to listen to different streams and then vote on whether the X stream is actually the A or the B stream. The program will report the correctness of the vote in the status window. The streams will be randomized again, you can go directly to comparing again and voting again.

Statistical significance will occur after 5-10 trials of success. Normally, if you just toss a coin in your voting you’ll achieve a 50% success rate. So 60 or 70% barely demonstrates skill. For blind testing, 90 to 95% success is preferred, which suggests 10 to 20 trials overall. But QA_ABX doesn’t require a set number of trials. Just try as much as you like.

After completing several trials, the user can then sign the result. This will combine the output of the test along with a secret key and generate an MD5 hash of the test summary which can then be verified by others as being authentic or not. This will allow someone with “golden ears” and/or phenomenal equipment to in fact prove to others that they are able to hear differences with increasing degrees of statistical validity. It’s not completely fool proof. And we’ll work to make it more so in the future. But for now, it is a big step towards allowing others to verify and in fact understand how much better changes have made your equipment.

QA_ABX Distortion Generation

Existing ABX tests today generally use two different audio files as their sources. It’s a bit cumbersome, however, for normal use. People of course have a very good quality version of a performance in WAV someplace in their collection, but generating the degraded version is somewhat more difficult.

The QA_ABX app takes a single WAV file. When the wave file is loaded into the app, it’s converted to floats, and then a distorted stream is created in real time based on the type of distortion that the user has selected. In this first release, there’s 3 three types of distortion available. For the plots below, we’ll look at just the crossover distortion.

The cross over distortion is generated by slightly modifying the sample just before a zero crossing occurs. How much they are modified is a user setting. Below you can see the audio output of a PC without any changes to the signal. This shows a THD of roughly –92 dB, which is about what you’d expect for PC audio. An external DAC would hopefully be better.

And the plot below shows the same wave file with a very slight change made to the sample right before a zero crossing. In this case, the location of the sample just before the transition was multiplied by 0.99 (this figure is the setting in the Distortion Setting dialog). And from this very small change, a big change in the output spectrum was observed, with the THD dropped from about –92 dB down to –72 dB. That’s a 20 dB change just from nudging a sample by 1%. It’s likely very few would be able to discern this in listening.

If we instead multiply that sample just before the zero crossing by 0.96, or reduce it by 4% of it’s original value, then we see the THD drop from –72 to –60. And at this point, it’s very easy for even unskilled listeners to hear the difference in an ABX test. Music played with this level of distortion has a buzzy quality to it.

For completeness, a few cycles of the “heavily” distorted sine wave is shown here. Visually, this waveform appears perfect. Even if you zoom in you cannot visually see that a point just prior to the zero crossing was moved by 4%.

Installing the Application

The application comes in a zip file, and the download location is linked at the bottom of this post. The file you download is a zip. Once downloaded onto your machine, simple unzip the file into the directory of your choice. Inside the newly created directory you’ll see the QA_ABX app and a DLL that contains the audio library (NAudio).

Run the QA_ABX application

Selecting the WAV File

When the application opens, you’ll see something similar to the screen below. You need to first specify the WAV file you are interested in testing. Remember, for the QA_ABX app, you only specify a single WAV file. The file is loaded into RAM, and the user-selected distortions are applied on-the-fly. Your WAV file is not altered in any way.

After you’ve selected the WAV file, you will see the file load (which can take several seconds) and you will have the opportunity to select your distortion.

Selecting the Distortion

After specifying the wav file to load, the app will take a few seconds to cache the file. The app will then present a dialog allowing you to select the type of distortion you’d like for the degraded stream. In the screen shot below, there are 3 types of distortion:

Crossover distortion will slightly modify a sample just before a zero crossing. The adjustment allows you select a range from 0.94 (extreme distortion) to 0.99 (slight distortion). This number is the amount that the sample just before a zero crossing will be multiplied by.

Bit Depth allows you to simulate a reduced bit depth in playback. Your source material should be encoded at a bit depth much higher than your selection—it doesn’t make sense to take a 16-bit file ripped from a CD and set it to an 18-bit bit depth. But if you have true 24-bit source material, this will allow you set the bit depth all way down to 8 bits to hear what that material would sounds like on a lesser DAC. And be aware too that your selection should be less than your DAC capability.

Amplitude isn’t really distortion per se, but this selection will let you determine the minimum amount of gain imbalance you can detect. 0.5 dB is very hard to detect, while 3 dB is very easy to detect.


Understanding the Log

The log summarizes the use of the application. The log contents can be copied and shared with other users. When you share the log contents, you need to copy all of the log contents, from the beginning to the end. This is so that others can verify your results if they wish.

Verifying Results

Results of other users can be verified in the application by selecting the “Verify” menu option. To verify a result, copy the text from a log, paste it into the textbox that opens after you select “Verify” and then press the Verify button. If the log is authentic and unaltered, a message will appear indicating so.

What Is Next?

There are other types of distortion we’ll add to the QA_ABX tester. Clipping distortion is likely next. We’re also anxious to see users demonstrate their listening skills so that some knowledge can be gained about what levels of distortion can nominally be heard. The more you know about your personal listening capabilities, the more informed you can be about what matters in your equipment.

Download the latest version of QA_ABX at the downloads page located here.