Audio Science Review Review

It's the same story every time.

If you do blind testing, and the results are statistically significant enough to show you could pick out presumably inaudibly different components, then you did something wrong.

Oh, you only tested 10 times, but got 10/10 correct? You have to do 100 for it to be statistically significant. Oh, you did 100? Well, you need to do 1000. You did 1000? Oh, well, you need to have more participants.

Oh, you level matched to under 1mv? Can you level match to under 0.01mv? 0.001mv?

Oh, you only did A/B testing? Well, you have to throw in some cases where nothing actually changes to rule out bias. And remove all signs of relay interruption and the like.

Oh, you accounted for all this? Well, your (X, Y, and/or Z) measures like this or that, which might be audible, and/or it might have electrical characteristics that influence your (A, B, and/or C) upstream gear, so of course it's audible. It's not an audibly transparent setup!

This crowd is always nitpicking (or straight up moving goal posts) about protocol, yet will blindly follow someone that regularly makes sloppy, protocol mistakes with their AP analyzer measurements.

Truth is, it's no different than a religious person talking religion, philosophy, and science to an atheist, or vice versa. They might be cordial enough, lay out solid arguments, and put on a semblance of understanding or partial agreement, but the fact is their mind put up a firewall as soon as that conversation started.

All these protocols are just firewalls. As important as this sort of testing can be, among other objective procedures, they're too often used more as an insurmountable barrier meant to keep you from even trying. "It's easier to accept my reality than it is bothering to try to prove me wrong via my own framework. And if you do, I won't believe you anyway."

Yeah, correct blind testing is hard. But when you use the fact that it's hard as a form of discouragement, that's the wrong approach. When you've already made up your mind and find yourself on the defensive, that's not the right place to be.

Fact of the matter is that we'd need multiple prevalent, widespread, peer-reviewed studies, with suggestions taken into account from both "camps", to possibly move the needle in either direction. A few individuals or small groups aren't going to influence anyone.

Now, what might actually work is finding new measurement methodologies. You'd still have to get buy-in from the analyzer powers that be, but measurements are what seem to talk today.

As such, I'd suggest that the blind testing part, while still important, is not going to be a fruitful approach. Better to spend time finding new ways to measure all the things.

 

Ding! Exactly, the "DBT" is merely a device. Note how my triple-blind volume matched test of 3+ vs Heresy is simply ignored, as if it never happened.

 

@ushanka While I don't have empirical evidence to support this claim (more just that it seems obvious), I believe the bulk of individuals that are willing to do some research in pursuit of higher end audio gear are the exact same sort that use any sort of seemingly authoritative site to help them pick TVs, monitors, GPUs, CPUs, keyboards, mice, and so on.

I've said it before, but I came into this hobby with a fairly objective slant. I grew up building gaming PCs and the like. What else could there be besides hard numbers? Something was either better, or it wasn't, based on tests.

And I think that's exactly the same place the bulk of the audiophile crowd has come from, whether budding or not, in the last couple decades. Not that this sort of objectivism is new to the 2000s, but that the ease of access and publication of such material via the internet has accelerated its place in an exponential manner.

Now, what I think the vast majority of these consumers, and even an unfortunate number of "authoritative sources", don't understand is that benchmarks are limited in scope, accuracy, precision, and real-world reflection to some varying extent.

Back in the day, all we got for GPU benchmarks was basic FPS average charts, power consumption, heat, and maybe overclocking results.

Now we more commonly see not just FPS, but 1% and 0.1% lows, frame time charts (arguably more important than FPS), benchmarks across a wide variety of games, resolutions, and settings, plus more, all to help us better understand a GPUs inherent and/or use-case specific performance.

Still, any reviewer worth their weight will at least demonstrate or accept the following:

1. At least imply on a regular basis that benchmarks don't always reflect real-world scenarios. There are too many variables. We can only try to get better.

2. They keep an eye on what their counterparts in the industry are doing. They collectively, whether intentionally or not, try to stay in sync with robust testing suites and new testing methodologies to remain competitive.

3, They are open to feedback. They will look into community suggestions, even if coming from a small minority, and even if they personally think there's nothing to their feedback or claims. They will present this data and adapt if they find something or show evidence contrary to community claims...though, even then, often in such a way to emphasize that while they have doubts, they can almost never rule anything out.

On the other hand, I suppose this same enthusiast field (PC gaming) had some parallels to our current scenario back in the mid-2000s and through the early 2010s. Can't see above 60FPS, frame times don't matter, you can't see micro stuttering, you can't detect the difference between 1ms and 30ms on a peripheral, blah blah blah.

Well, now it seems that realm is mostly convinced we can detect well above 240hz, albeit with minimal return margins as you go up, can detect polling rates over 1000hz (depends on refresh rate), that frame times and stability are more important than average FPS, and that everything should be covered in rainbow LEDs.

I dunno, maybe this industry is just really fuckin' bad at selling us higher sampling rates, DSD, and MQA. Certainly seems hard enough selling the crowd on R2R...

Another thought, e-sports are one driving factor of all this ridiculous (and cool) PC gaming equipment. Maybe we need to establish audiophile e-sports to really get the innovative and imaginary juices flowing? Maybe hire Razer to make some gear and Nvidia's PR to spin everything. I'm sure we can sway opinion however which way we like from then on.

 

@ushanka:

I don't think it's that hard. We can always do similar as Olive and take a poll. For example, 78% of members on SBAF either somewhat or strongly disagree with the Harmon Target. https://www.superbestaudiofriends.org/index.php?threads/harman-curve-poll-public.10130/ One can point out all the flaws, but this poll does mean something.

I bet we can take polls on specific pieces of gear. I bet the "good" sounding gear will not only score higher, but also have lower standard deviation. In fact, we did something like that at Changstar with some interesting results.

I also bet we can take a poll of whether Magnius how much different or the same as THX789: slightly different, quite a bit different, no difference, etc. Even if no blind tests are done, and we see a pattern with a large samples, it does mean something. The people to believe are the ones who actually had access to both pieces of gear and aren't talking armchair audio.

These approaches are still scientific. Like all other aspects of science, do have their limitations though.

Now in the case of THX789 vs Magnius: of course people will NOT hear a difference if people do not try or even have the two pieces of gear in their hands for more than long enough to just take measurements on their AP. The only "dissonance" is with people who only talk the talk but do not do, or people who truly bad ears. I truly believe that Amir cannot hear a difference. It's unlikely because his judgement is impaired by confirmation bias (from measurements and his SINAD believe system) and also because of his lack of time. Amir cranks out a measurement a day. Even if I dedicated my life to this, I would never be able to be as prolific as he. It takes me hours if not days of back and forth to do a suitable comparative subjective write up for gear. With measurements, same difference. @atomicbob can attest how difficult it is to take honest measurements.

As far as "authorities" or whatever, no one on SBAF cares. If you like SBAF, cool. If you like ASR, cool. I think RTings does a huge disservice because their TV ratings are biased against IPS panels and doesn't take into account viewer preferences, ambient light in the room (down to color of walls), and size to viewing distance. In the end, it all sorts out when there is dissonance. Look how crinacle and Oratory have moved away from Olive's target and suggested their own which are closer to SBAF preferences.

Finally, human sensitivity is not equal. Some people can jump higher than others. Some people can do more difficult math problems than others.

 

Mids are treble are close, but Harmon is still brighter. I don't know why he would say they are the same in the mids and treble when they are clearly not. A few decibels does matter. Details details. Look how much more Harmon suggests in the treble. Opt HiFi also suggests less 3kHz than Harmon. I would burn the HD600 if it were adjusted to Harmon.





Harmon has been tweaking their curves for years now with the latest 2018 not so offensive in the mids and highs. The lows are offensive though - and in most cases downright bad because EQing headphones to have flat bass when they cannot reproduce flat bass only results in massive distortion, which affects not only the lows, but also the mids and the highs.

The point is that Sean Olive should not be peddling his shit if it's still work in progress.

 

1. Depends which iteration of the PMx2. This headphone went though continuous development and later models actually sounded darker, scooped in the upper mids.
2. Another definition of neutrality: frequency response smoothness. PMx2 is unrivaled here.
3. I still feel the Oratory Opt HiFi target is just a tad too much for upper mids and highs. LIke 0.5-1db - which is probably more preference because I do prefer a darker slope.

 

CASE 3: Flawed listening tests / impressions not supported by measurements at the transducer or deeper analysis of amp measurements

Imma just gonna keep reposting the links from the first post.

 

There is no guess work. The digital reconstruction will be off, but we know how much off it will be. And it usually is below the noise floor. Unless the system is poorly engineered.

EDIT: BTW. When you mean analog, what do you mean? Vinyl and magnetic? Because in "digital" there is a lot of analog. The sources of error in vinyl and magnetic are higher though. The awesomeness in the analog format is in the classic level of effort placed in recording in such poor formats, which in many cases result in a significantly better recording. But if an equivalent level of effort was placed in "digital" recordings, such recordings would murder "analog".