DAC Blind Test: Modi2 vs ODAC

Discussion in 'Blind Testing and Psychoacoustics' started by purr1n, Nov 17, 2015.

  1. Xen

    Xen Friend

    Pyrate
    Joined:
    Sep 30, 2015
    Likes Received:
    207
    Trophy Points:
    43
    Location:
    Houston, TX
    Thanks for undergoing this ABX test, Marv! I don't see any issues with the test setup.

    I will have to disagree with those who think that the "Abort" choice is a good . If the "Abort" was a non-biased choice, then yes, it is a useful choice, especially for problems with the test setup. However, if the tester only chooses ABORT when they cannot tell the difference between the 2 selections, then "Abort" creates a severe bias away from the null hypothesis. As I read the progress of Marv going through the trouble of this test, I think only #10 is a neutral use of the "Abort" choice. #10 had issues with the test setup and so the abort was warranted. In test #3, 7, and 11, Marv aborts when he can't tell a difference, which would definitely bias the test against the null hypothesis.
     
  2. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,129
    Trophy Points:
    113
    Location:
    India
    And, more importantly, lack of proper level adjustment (or did I miss that?) but it all still comes under the heading of a good experiment.

    Err... when I say good experiment, I mean a fun thing, not good experiment in the scientific sense.
     
  3. mkozlows

    mkozlows Friend

    Pyrate
    Joined:
    Oct 6, 2015
    Likes Received:
    512
    Trophy Points:
    93
    Agree, although I understand why you'd do it.

    Still, even if we assume those are all fails, there's a 71% chance he's hearing a real difference. If you assume he'd have coin flipped those, which is probably the better way to think of it from a null testing sense, that rises to something like 90%.

    So I'm comfortable saying that real differences are very likely. At the same time, on 5 of 13 tries, he couldn't reliably hear a difference, so we're definitely not in clear cut obvious judgment territory.

    I'll reiterate my desire to see a test between super unambiguously different DACs, to see if the difference ever does rise to that level of obviousness.
     
  4. SKiring

    SKiring Friend

    Pyrate
    Joined:
    Oct 3, 2015
    Likes Received:
    203
    Trophy Points:
    43
    Location:
    Rotterdam
    This is true and to be fair I always enjoy both approaches at the same time. I can't stand either extremist, the objectivists seem too busy with debunking instead of musical joy and subjectivists are only busy with praising with bias without any form of scientific approach. I prefer it this way, like purr1n is going, just see how far you get and simply enjoy the experience.
     
  5. lm4der

    lm4der A very good sport - Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,461
    Trophy Points:
    113
    Location:
    Seattle, WA
    This is remarkably impressive. Sure, I was hoping that Marv would have zero difficultues. But this result is stunning considering that he picked 2 low end dacs that are so similar. This can't be over emphasized. Being able to blind a/b these two is astonishing, and I expected a couple errors.

    Jeebus, a/b'ing modi. vs. odac. Why do you guys that are good at this insist on tackling the most difficult? But I'm glad you did.

    It would be fun if Marv would do one with Yggdrasil vs odac just to prove that it is a walk in the park for him.
     
  6. Hands

    Hands Overzealous Auto Flusher - Measurbator

    Staff Member Pyrate MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    12,285
    Trophy Points:
    113
    Location:
    Colorado
    Home Page:
    Abortion is cheating at life! I will mark those as a total loss and failure. I see no reason to believe you're not just guessing left and right and trying to get out of things to make yourself look and feel better! Point proven: every DAC sounds the exact same and we're all just crazy. I am going back to the cheapest DAC and amp combo I can find with measurably inaudible imperfections (or audible perfections?) and upgrading to whatever the best headphone is. Probably the LCD-XC or Beyer T1v2.

    ;)
     
  7. lm4der

    lm4der A very good sport - Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,461
    Trophy Points:
    113
    Location:
    Seattle, WA
    [​IMG]
     
  8. Hands

    Hands Overzealous Auto Flusher - Measurbator

    Staff Member Pyrate MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    12,285
    Trophy Points:
    113
    Location:
    Colorado
    Home Page:
    Just trying to keep everyone's crazy placebo BS in check.

    Nothing matters, all we're doing is guessing, and everything actually sounds the same in reality.

    (People actually like that must live miserable, or at least horribly bland and boring, lives.)
     
  9. Chris F

    Chris F Boyz 4 Now Fanatic - Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    805
    Trophy Points:
    93
    Location:
    Ottawa, Canada
    I haven't done a formal test like Marv but when I informally did Yggdrasil vs low end DAC (MBP or DJ controller) the difference was immediate; the soundstage and tonality are completely different.
     
  10. TMoney

    TMoney Shits on SBAF over at Head-Case to be cool

    Joined:
    Sep 27, 2015
    Likes Received:
    760
    Trophy Points:
    93
    Location:
    Sausalito, CA
    Nice job on the first round of tests, Marv. Even when you ace the test as you did blind testing can remind you of how big (or small) the audible differences in gear really are.

    Moffat is right that getting laid is better than A/B'ing, but his dislike for blind testing in general doesn't mean that much to me.

    What Moffat wants to get out of the hobby isn't the same as what we want. To put it crudely, our goal as enthusiasts is to find the best sound at the best value. Moffat's goal is to make products that sell.

    Obviously Moffat's goal is closely aligned to ours as a great sounding good value product should sell well, but the two goals are not identical.
     
  11. OJneg

    OJneg The Most Insufferable

    Pyrate BWC
    Joined:
    Sep 25, 2015
    Likes Received:
    3,923
    Trophy Points:
    113
    Location:
    Grand Rapids, MI
    How are you coming up with any of this?

    Yes he's in the business of designing audio gear, but I don't think that statement sums up the man's priorities accurately or completely. And that's certainly not the impression that I've got from talking to the man.
     
  12. Tuco1965

    Tuco1965 Suffring from early onset Alzheimer's - Friend

    Pyrate
    Joined:
    Oct 4, 2015
    Likes Received:
    151
    Trophy Points:
    43
    Location:
    Just East Of West
    If Mike's only goal was to make products that sell, he could have stopped before the Yggdrasil. I'm sure that making the best sounding products that he can is a huge part. The kind of products that disappear, leaving only the mood music for getting laid, or fist pumping or toe tapping. :D
     
  13. Psalmanazar

    Psalmanazar Most improved member; A+

    Pyrate Slaytanic Cliff Clavin
    Joined:
    Sep 27, 2015
    Likes Received:
    5,345
    Trophy Points:
    113
    I can ABX WASAPI Event vs WASAPI Push vs DirectSound on my system with the ODAC+O2. Of course that's as WASAPI event clips and DirectSound sounds awful smearing guitar distortion and channel separation around somehow.
     
  14. paranoidroid

    paranoidroid Friend

    Pyrate
    Joined:
    Sep 30, 2015
    Likes Received:
    180
    Trophy Points:
    43
    Location:
    San Francisco
    Nice job @purrin! The Abort is an interesting concept - I'm not sure what it means statistically but I would have liked that option when doing my own tests. It does take the stress off of picking one if you just can't perceive a difference. I bet taking that pressure off gets you better results - something I'll try next time.

    Are you turning down the volume then up after every attempt? It wasn't clear since after trial #3 you didn't mention it. I think that's a good way to do it instead of trying to volume match perfectly. I did get a 10/10 hit on Bifrost Multibit / ODAC until I re volume matched, then got my guesses go in the other direction.

    Would definitely love to see Yggdrasil / ODAC to see if it confirms your feelings that it'd be cake.
     
  15. BioniclePhile

    BioniclePhile The Terminal Man - Friend

    Pyrate
    Joined:
    Oct 25, 2015
    Likes Received:
    386
    Trophy Points:
    63
    Location:
    Admudnson-Scott Antarctic Facility (Seattle WA)
    Home Page:
    Well, nothing's stopping you from ODAC and Yggdrasil, so..
    And we hope you got laid after all those tests, unless you're at that age when 'going to sleep' actually means going to sleep.
    And just wait 'till the Modibit that Schiit has under wraps. Compare that to the ODAC and we'll find who truly wins the budget war!
    Also, Modibit would be feckin' awesome, but it'd definitely probably be impossible due to lack of space. But who knows, we'll have the technology someday, even if it only runs the UltrafarterMuchachosEnchelada filter at 16/44.1 and has a total of 7 bits.
     
  16. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,771
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    1. Yes, I'm turning down the volume on every F11 press and DAC switch. Made it a habit. Gets annoying after a while. I think the test would have been easier with volume matching, but hard to know without doing it. The ODAC output is a nice 2.04 dbFS louder.
    2. I found that I couldn't tell or got things wrong when I changed the recording or used recordings I hadn't heard for years.
    3. If I used the same recording over and over again, I would have had an easier time. There are certain things which are easier to key on this way. I actually made it harder for myself by changing the music around.
    4. Tests are tests. They can be designed to make people fail or make people succeed.
    5. I originally thought the difference between these DACs were larger. After these tests, I've learned a lot more about the subtleties of these DACs and how they present themselves with different kinds of music. I learned about more about the sound of these DACs with wrong answers.
    6. I am certain a person can learn how to take this test and be more successful at it with practice. Keep in mind that this test was practiced with some of the same recordings on the night of the 16th before the formal test began.*
    7. The abort thing was something I learned from earlier tests: too much going back and forth equated to fatigue. The points made against it are valid. Score these results with 0.5 / 1 as an alternative tally.
    This begs the question of people who claim to do blind tests and find gear (say for example ODAC and Benchmark DAC1) to sound the same: how much practice or preparation did you do? I came into this test knowing what to listen for and what recordings to use that would best differentiate the DACs. Does preparing for the test invalidate the results?
     
    Last edited: Nov 18, 2015
  17. BioniclePhile

    BioniclePhile The Terminal Man - Friend

    Pyrate
    Joined:
    Oct 25, 2015
    Likes Received:
    386
    Trophy Points:
    63
    Location:
    Admudnson-Scott Antarctic Facility (Seattle WA)
    Home Page:
    Pretty sure I remember Jason saying that he was doing Schiit because it was fun and had no interest in growth unless it helped further the fun aspect. Maybe Mike is in it for bitches, but he's got to have some fun during the profiteering.
    Maybe one of them can chime in, but I remember the goal was to make good audio products that you can pass down to your children and so forth from their site. I was under the impression that if Schiit wasn't a commercial success, they'd still've had fun with it.
     
  18. Psalmanazar

    Psalmanazar Most improved member; A+

    Pyrate Slaytanic Cliff Clavin
    Joined:
    Sep 27, 2015
    Likes Received:
    5,345
    Trophy Points:
    113
    @purrin So they differ more in functionality rather than in sound quality.
     
  19. SSL

    SSL Friend

    Pyrate
    Joined:
    Nov 12, 2015
    Likes Received:
    1,239
    Trophy Points:
    93
    Probably none, since most people just want to listen to music. You said yourself that this kind of thing is boring and fatiguing.

    I think both perspectives are useful - the trained ear demonstrates that there is indeed a difference, while the untrained ear indicates the difference that most people are going to perceive. Schiit says the Magni/Modi might be the only amp/DAC you ever need for a reason.
     
  20. lm4der

    lm4der A very good sport - Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,461
    Trophy Points:
    113
    Location:
    Seattle, WA
    I can't blind a/b most dacs, but I know I still perceive a difference. Ie, I don't think the inability to a/b a difference is the same thing as inability to perceive a difference in normal listening. Doing the a/b type of test takes skill and practice. My own experience with going from Modi 2 to Bifrost Multibit taught me that the difference is significant in normal listening, but I still can't a/b it easily. The type of difference between modi 2 and Bifrost Multibit is, imho, at once both extremely subtle and yet profound.
     

Share This Page