Audio Science Review Review

• Write Autohotkey macro / scripts to switch outputs randomly. Put attentuator between DAC and amp to level match.
• Have someone else in the house to switch cables. Use same cable source and occlude sight of cable source to make the test triple-blind.

 

I know @purr1n is pretty vocal about their issues with RTINGS, and rightfully so, but I do appreciate the following:

1. They will listen to user feedback and try to incorporate that into their test methodologies.

2. They'll respond to Q/As and other comments in reviews, often going back and testing certain aspects or quirks in specific models.

3. They seem to try to test in a somewhat holistic nature. Lots and lots of little tests, less of a slant where one test is primarily used to rank/chart a product's performance. (Yes, even contrast ratio does not have quite the sway that SINAD does on ASR.)

4. They seem to have measurements, for headphones at least, that no one else does. More seemingly experimental sort of stuff, regardless of how official they suggest it to be in their methodology descriptions and write ups. Whether these are of any value is questionable (and thus possibly "dangerous"), but I'd rather see experimentation than stubborn simplicity and complacency.

5. While the way in which they numerically rate test and overall categories is problematic, as I see it, I do appreciate that they have prominent ratings for specific use cases. Flawed, yes, but at least there's a semblance of recommendations based on use case rather than "one rting to rule them all."

6. They seem more consistent in the precision of their testing and how they subjectively rate or comment on things, compared to the relatively sloppy ASR approach. (There are some exceptions with audio measurement accuracy, I think, especially with sound bars.)

Again, this is not to say RTINGS doesn't have flaws. There's plenty to complain about. But there are a lot of core, foundational elements I appreciate, the kind of behaviors that can lead to something really good as things progress. Not guaranteed, but a possibility. It's some of the same behaviors I've seen in the better computer hardware review sites, the ones that raised flags on frame times back in the day.

Whether or not I agree with everything, even some of the larger aspects, it's a hell of a lot better approach than ASR. More potential for good. Or it may just be that, in contrast to ASR, I'm viewing RTINGS too positively because of the juxtaposition.

 

On blind testing setups...

I'm not usually one to stress things like cables, jacks, and so on. I tend to err on the side of caution and go with properly constructed and shielded cables, with good materials, at a reasonable price. I tend to go for things like BNC jacks with SPDIF for engineering peace of mind, even if I don't think I've ever really heard a difference between it and an RCA SPDIF jack.

That said, I've used alligator clips to swap between potentiometers, and I was surprised at the differences I thought I heard. I might argue they can be just as or even more influential than caps and tubes.

As such, I think there's some amount of uncertainty around if A/B switchers can impact sound quality at all. Could the switch itself affect the sound? Does the brief relay interruption f**k with our perception? What about the grounding scheme and how that might interact with the multiple pieces of gear in your setup? It's kind of hard to subjectively test those to ensure transparency, you know?

Now, through a pot/attentuator in the mix to level match, that might be yet another variable. Same for adjusting the volume on DACs themselves, if they have that capability (of which there are many approaches).

Even if we strip all that out and had perfectly identical, dual setups, with only one component differing, and have to plug the headphone into each rig manually for testing, well, there's still that brief interruption that can mess with you.

I'd even go so far as to factor in digital sources. A subpar digital source, i.e. USB straight out of a PC, might muck up the sound too much to differentiate DACs. (But then you might have to ask, how much of this is the USB input on the DAC as it relates to my source?)

Similarly, are your amps and headphones good enough to show differences? Some amps, like the Magnius, are going to impart their will on anything and everything.

Do you have enough experience and training to pick differences out? (Or maybe, do you have enough of an imagination? Depends on who you ask.)

As @atomicbob mentioned, sometimes you have to listen for long periods of time before switching. I've found sometimes I do best with short bursts. It varies.


OK, so, that's an awful lot of nervosa, right? Do you really need to worry about all that? Not totally sure! I like to rule out variables where I can out of precaution, even if I mostly doubt it matters enough.

If you don't account for these variables and hear differences, well, then you hear differences. But if you don't hear differences, are the variables enough to cast doubt on there being no differences?

I think it's worth keeping in mind and poking around with, but nothing to stress over too much.

Part of the problem is people assume blind testing has to be stressful. Really, who cares? Your approach will never be totally perfect. You will never be totally perfect. That's not to say you should be sloppy, nor should you lose hair over minutiae, but find a balance between consistency and precision, accuracy, experimentation, and having some fun with it.

 

Their measurement methodology seems solid and their huge variety of different measurements is to be applauded. With RTings, there's always something there that those who know what they are looking for will find useful.

However, their numeric ratings, say 7.5 for Movies and 8.4 for HDR are butt.

There are also no measurements (at least seemingly) on why "@Hands (and now Merv) prefer IPS panels because their colors pop more than VA panels". And also how a weakness of IPS panels, is realistically not a weakness, and only serves to kill IPS panel's numeric ratings. The fact that their numeric ratings are translative (Movies, HDR, Gaming, etc.) only makes things more hokey.

Bottom line is that I think these guys are deaf (look at their suggestions to correct the HD650) and also possibly blind. Numbers are fine. But they need to mean something.

As for their receptiveness, you have to remember that one RTings guy came here to ask us about the Olive curve. We told him the Olive target was butt - in quite strong terms. He never came back.

I plan on write a doodle on this.

 

As far as RTINGS:

https://www.rtings.com/headphones/reviews/beyerdynamic/dt-990-pro

"The Beyerdynamic DT 990 Pro are excellent neutral listing headphones"
"They may sound a bit sharp at times, as the treble range is slightly too emphasized, but bass, instruments, and vocals are well-balanced and reproduced with high-fidelity"

Are you shitting me?! These cans have the freaking Matterhorn peak for treble.

Comfort: 7.5 vs HD600 with a 7.0.

This is bullshit.

I think it was because of RTINS or whatever it's predecessor was, that I bought these POS cans. The DT 990 Pros have a peak monstrosity in the treble region that actually hurts. Their measurements make it seem as if it was all positional error, or averaging issues. It is not. They are also way more uncomfortable than the classic Sennheiser headphones.

I'm not picking on RTINS cuz they also measure cans. Again, if I remember correctly I bought my old DT 990 Pros because I got bamboozled by their measurements, and found them horridly bright after the fact. This before I did any measurements or gave them to Merv to test.

 

LOL, well it's only a "slight" variance from Olive target:



The problem with RTings is lack of weighting, poor weighting, or poor algorithms. The horrible effect of that high amplitude 8kHz peak is likely minimized according to RTings automagically numeric scorer because the peak is narrow, and thus the "error" is minimal. The problem is that any errors, or peaks around this region, no matter how narrow, fricking hurt - a lot.

Anything with peak like this, on top of the Olive target which is hot in the treble, should be an automatic fail - a showstopper.

This is what happens when people decide to rely too much on graphs, Sean OIive's fake science, and not enough on actual experience.