Throw away your APx555s!

Discussion in 'Measurement Techniques Discussion' started by purr1n, Oct 16, 2022.

  1. purr1n

    purr1n Burned out

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    82,449
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Throw away your APx555s! Just kidding. Actually, there is some truth to this, particularly if you interest is SINAD and audibility from a scientific perspective. Considering that most of our listening with with CD-quality (or worse), which has a limit to 16-bits. 16-bits is 96db of dynamic range. So really, we don't need measurement gear that does better than that. Even moreso when we consider that the signal to noise ratio of microphoned instruments or electronic instruments plugged into mixers is probably no better than 55-85db.

    Which makes us wonder if we should not just cut off the lower part of the graph below the -100db line. Think of it like tolerances for automobile body panels. What's considered acceptable tolerance for the gap? 3mm, 1mm, 0.5mm? I'm sure Toyota's spec is probably much tighter than Ford's or Chrysler's (Tesla probably doesn't even have one, their spec manual likely being "use eyeball" or "whatever Elon says is OK for the day"). However the fact is, after a certain tolerance spec, it's not going to matter to the human eye. I'm not going to care if the gap between the hood and front quarter panel is 0.01mm off from spec. The reason is because I cannot see this.

    This is why taking measurements where we care about -120db spuraie is utterly ridiculous. While I do think maybe there is some information that can be gleaned indirect from such a spectrum analysis as a whole; anyone who tells you that the aggregate of the harmonics and "grass" below -100db (or -80db even), aka SINAD, can be used as a measure of audio quality, is either lying to your, has an ulterior motive, or has illusions of grandeur. Most likely the latter. There were some of these people in the past: Doug Self, NWAVGUY, etc., and there will be some in the present, and in the future.

    The fact is, modern A-D chips do a damn fine job. A good enough job - to take simple SINAD type measurements. Case in point, the MOTU Ultralink mk5. Let's start with the APx555 taking a steady-state sine measurement of the Schiit Modius ES DAC. I will include measurements from the APx555 with the the High Performance Sine Analyzer turned on and off. The HPSA on the high-end APs is kind of an ingenious cheat that gets us better results when measuring a single sine signal. The HPSA comes in very handy if one really wants to mastubate over SINAD type readings.

    Schiit Modius ES DAC
    1kHz steady state tone at 0dbFS (red)
    Y-axis not normalized to anything in particular
    32k FFT, 10 average
    APx555 (High Precision Sine Analyser On)
    upload_2022-10-16_19-15-25.png

    Schiit Modius ES DAC
    1kHz steady state tone at 0dbFS (red)
    Y-axis not normalized to anything in particular
    32k FFT, 10 average
    APx555 (High Precision Sine Analyser OFF)
    upload_2022-10-16_19-17-40.png

    Now here's the MOTU Ultralink mk6, into the front panel XLRs with the pad turned off (it must be for hot XLR line signals). Note that the APx555 is $35k these days. The MOTU is $600-$700.

    Schiit Modius ES DAC
    1kHz steady state tone at 0dbFS (green)
    Y-axis not normalized to anything in particular
    32k FFT, 10 average
    MOTU Ultralink mk6 with ARTA software
    upload_2022-10-16_19-33-4.png


    The APx555 has less random grass / spurs. However I'm getting a lower noise floor (and lower 3rd harmonic) from the MOTU. Surely this cannot be considering both are using a 32k FFT size? Actually, it's probably because I'm using a different algorithm Kaiser 7 for calculating the FFT in ARTA. I have the AP set to use their secret proprietary algorithm (which is super fast). Let me use a higher FFT size to get the same "noise floor": a 96k window size.

    Yes, setting different parameters with how the FFT is calculated will give us different results! This is one thing which is very important to note! (And also one way to give reader misleading results if they are sloppy or don't fully document the parameters of how they arrived at the measurements).

    Schiit Modius ES DAC
    1kHz steady state tone at 0dbFS (red)
    Y-axis not normalized to anything in particular
    96k FFT, 10 average
    APx555 (High Precision Sine Analyser OFF)
    upload_2022-10-16_19-30-40.png

    So here we go, the APx555 using the AP algorithm with a 96k FFT window size gets us about the same noise floor as the ARTA with the 32 FFT size using Kaiser 7. And not just that, the results between APx555 (in its non-cheat mode) and the MOTU Ultralink Mk6's front panel inputs with the ARTA software are very similar to each other, with neither one truly besting the other.
     
    • Like Like x 12
    • Epic Epic x 3
    • Agreed, ditto, +1 Agreed, ditto, +1 x 2
    • List
    Last edited: Oct 16, 2022
  2. atomicbob

    atomicbob dScope Yoda

    Friend BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    16,780
    Trophy Points:
    113
    Location:
    On planet
    Some very competitive results to APx555 may be obtained with RME ADI-2 Pro and HpW Works analyzer. For a LOT less money. However sweeps aren't nearly as easy to perform with the lower cost option.
     
  3. purr1n

    purr1n Burned out

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    82,449
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    The sweeps, automation (programmable testing), open loop measurement capability, the FFT speed especially at large window sizes, the bandwidth, the software, is what really sets the APs apart. BTW, one can get the AP software for the RME (I believe RME is one of the suggested hardware providers.) The AP software is extraordinarily expensive though.

    If one wanted to concentrate on SINAD, then the AP is a waste. The AverLAB was plenty good. The problem is that many people were fooled by the AP Schlonging, starting with Jude, and then Amir. The fact that more hobbyists own APx555 is a good thing. It keeps the schlongers in check, knowing they can be peer reviewed at any time. I would have much preferred an APx525 (doesn't have HPSA cheat mode nor the bandwidth). Unfortunately for the worse, an APx555 is almost necessity for a manufacturer if they wanted some insurance from being smeared by a self-anointed Internet expert peeking in at stuff that didn't matter.

    However, my point was that there is plenty of good affordable instrumentation out there if SINAD is your thing. I would encourage more hobbyists to take this route. You don't need an APx555.

    That being said, the below are more reasonable (they are still overkill) visualizations for the above measurements if one listens to anime soundtracks, CD quality music, full analog recordings from a bygone era, or anything that uses a microphone:

    upload_2022-10-16_19-51-34.png

    upload_2022-10-16_19-58-32.png
     
  4. atomicbob

    atomicbob dScope Yoda

    Friend BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    16,780
    Trophy Points:
    113
    Location:
    On planet
    I looked at the AP software pricing and it is so high that one might as well buy the APx555. At this point I could do that, but the fan noise is a huge problem for me. dScope has a much lower fan noise level and I've become so familiar with it I am not motivated to go the APx555 route. Other than a few internut personalities promoting the idea that only APx555 measurements are worthy. As you are so adeptly showing, even ARTA and probably a Focusrite Scarlett 2i2 is more than enough to produce very good quality measurements.
     
  5. Vtory

    Vtory Audiophileâ„¢

    Friend MZR
    Joined:
    Sep 12, 2016
    Likes Received:
    9,938
    Trophy Points:
    113
    Location:
    East Coast


    Major points agreed. I think this YT video (wow it was in 2014 lol) is relevant to what you examined. Tom Kite in the video basically mentions their "high performance" is a combo of more notch filters and extra AD to de-couple AD harmonics. I couldn't find this claim/approach be criticall wrong. I'd not call this 'cheat', either. It's just a matter of "I do vs don't need that level of accuracy" or "I do vs don't agree with AP's assumptions". We just have to be wary of possible wrong interpretations.

    The fault is usually on AP users or readers FWIW (rather than on AP).

    If some dudes intentionally or unintentionally turning it on and off to make certain products look betters over others.. then I agree that it's cheat and falsification. Cross-validation from multiple measuring folks can be a remedy though.

    There is also a big problem of inappropriate claims created by readers. IDK. We may just need more educations. I think posts like this or what atomicbob or GoldenOne posts could be really helpful.

    On the specific topic of HPSA front, my takes below
    1. It doesn't matter as long as HPSA on or off is consistent across measurements.
    2. I do think any AP measurements (either done by MFR, researchers, or hobbyists) should be released with the full report being downloadable, which clearly let readers know the detailed configurations. Below is copied from Schiit's one downloaded from the product page.

    upload_2022-10-16_21-17-16.png
     
  6. purr1n

    purr1n Burned out

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    82,449
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    The full report is only available in Sequence Mode (programmed sets of measurements). There's nothing wrong with using the AP in Benchmark mode and taking screenshots.

    Even in the report, not all the parameters may be indicated, e.g FFT algorithm, window size, averaging. There are also potentially crucial parameters like settling time, settling method, of which are a many combinations, which can lead to very different results when looking a stuff -100db down.

    None of the above really matter if we look at measurements holistically rather than treating them as a true indicator of audio quality depending upon how the leptons and quarks line up that day to how the algorithms are processing them.

    As reminder:

    [​IMG]

    [​IMG]
     
  7. Armaegis

    Armaegis Friend

    Friend BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    5,934
    Trophy Points:
    113
    Location:
    Winnipeg
    I'm surprised there isn't a business that builds ultra high end power supplies for measurement equipment for audio measurebators to improve the performance of the machines.
     
    • Agreed, ditto, +1 Agreed, ditto, +1 x 2
    • Like Like x 1
    • List
  8. LetMeBeFrank

    LetMeBeFrank Won't tell anyone my name is actually Francis

    Friend
    Joined:
    Aug 4, 2017
    Likes Received:
    3,547
    Trophy Points:
    93
    Location:
    Jackson, Mi
    My good friend (Chrysler engineer working in quality) tells me that on paper there are actually very strict specs and tolerances for every gap, however they were rarely enforced in practice lol
     
  9. purr1n

    purr1n Burned out

    Staff Member Friend BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    82,449
    Trophy Points:
    113
    Location:
    Padre Island CC TX
  10. Armaegis

    Armaegis Friend

    Friend BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    5,934
    Trophy Points:
    113
    Location:
    Winnipeg
  11. atomicbob

    atomicbob dScope Yoda

    Friend BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    16,780
    Trophy Points:
    113
    Location:
    On planet
    A hallmark of good science is when sufficient details are documented to allow measurement replication and independent verification of results obtained. @purr1n has done this with the APx555 vs ARTA + MOTU Ultralink measurements of Modius E (ES9028) which allowed me to replicate and verify his observations.

    My data presented were collected as follows:
    1. PrismSound dScope III
    2. ARTA + MOTU Ultralite mk4 audio interface
    3. Balanced XLR cables Belden 1800F with Neutrik 110R AES connectors
    4. Analyzer sample rate 48 KHz unless otherwise noted
    5. -4 dBu level used for testing unless otherwise noted
    6. Modius AES or spdif input, Bal output unless otherwise noted
    7. Audioquest Forest and Schiit Pyst USB cables used with measurement equipment
    8. Vaunix Lab Brick USB hub for measurement equipment
    9. Shielded 14AWG and 16AWG power cables
    10. ESD, EMI/RFI controlled lab bench and workspace

    Measurements are made in accordance with AES17:2015

    I am employing a Modius OG (AK4493) DAC and MOTU Ultralite mk4 which will result in slight variance from the OP, but close enough to make useful observations.

    Schiit Modius OG (AK4493) DAC
    1kHz steady state tone at 0dbFS (yel)
    Y-axis max 0 dBFS
    32k FFT, 10 average
    dScope analyzer FFT
    01 20221017 Distortion modius ak4493 -4dBFS FFT avg 10 AES Bal R ch dBFS y-axis.png

    Schiit Modius OG (AK4493) DAC
    1kHz steady state tone at 0dbFS (grn)
    Y-axis max 0 dBFS
    32k FFT, 10 average
    ARTA + MOTU Ultralite mk4 analyzer FFT
    02 20221017 Distortion modius ak4493 -4dBFS FFT avg 10 AES Bal R ch ARTA.png
    Results are substantially similar to those in the original post, including the differences caused by different windowing functions, proprietary Prism-7 for dScope and Kaiser 7 for ARTA similar to APx555 proprietary vs ARTA Kaiser 7.

    The MOTU Ultralite Audio Interfaces are approximately $600US as of this post.

    Now consider the following two graphs. A single measurement has been scaled two ways:
    1) 0 dBFS maximum y-axis
    2) 120 dB SPL maximum y-axis
    If the audio reproduction system is gain staged such that 0 dBFS = 120 dBSPL we can clearly observe the lower limit below which no one should obsess.

    dBFS y-axis
    03 20221017 Distortion modius ak4493 FFT avg 10 AES Bal R ch dBFS y-axis annotated.png

    dBSPL y-axis
    04 20221017 Distortion modius ak4493 FFT avg 10 AES Bal R ch dBSPL y-axis annotated.png
    Below 0 dBSPL is inaudible, period. For most listening environments any harmonics below 30 dBSPL will be masked by the ambient.

    So with the above consideration, the following two graphs are scaled 0 to -120 dBFS

    dScope
    05 20221017 Distortion modius ak4493 -4dBFS FFT avg 10 AES Bal R ch -120 dBFS.png

    ARTA+MOTU Ultralite
    06 20221017 Distortion modius ak4493 -4dBFS FFT avg 10 AES Bal R ch -120 dBFS ARTA.png
    The second harmonic in this case is barely visible above -120 dBFS. Again @purr1n results and conclusions are corroborated.

    ARTA + MOTU audio interfaces allow access to non-trivial measurement capability for a much larger portion of the audio obsessed population.

    *edit* 20221018 corrected typos
     
    • Like Like x 5
    • Epic Epic x 5
    • List
    Last edited: Oct 18, 2022

Share This Page