Blind Test Part Deux: Schiit Heretic vs Magni+ (with statistical analysis)

Discussion in 'Blind Testing and Psychoacoustics' started by purr1n, Feb 16, 2023.

  1. ergopower

    ergopower Friend

    Pyrate
    Joined:
    Mar 21, 2018
    Likes Received:
    815
    Trophy Points:
    93
    Location:
    South Central PA
    I'm pretty sure a simple CATV splitter will do what you want. It will keep the 2 outputs at 75 ohm impedance. They use F connectors, but F to RCA adaptors are cheap.
    I have been looking at this myself, as I'm considering bypassing my AVR for my 2 channel rig.
    Or F to BNC adaptor
     
    Last edited: Feb 18, 2023
  2. Yethal

    Yethal Facebook Friend

    Joined:
    Sep 23, 2018
    Likes Received:
    230
    Trophy Points:
    43
    Location:
    Poland
    Pi2AES should be able to output to two dacs over coax and bnc (with an adapter)
     
  3. Armaegis

    Armaegis Friend

    Pyrate BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    7,467
    Trophy Points:
    113
    Location:
    Winnipeg
    For the validity of the test, the source connector should be the same for both.
     
  4. Michael Kelly

    Michael Kelly MOT: Pi 2 Design

    Pyrate
    Joined:
    Apr 22, 2016
    Likes Received:
    4,112
    Trophy Points:
    113
    Location:
    Rhode Island, USA
    Home Page:
    not really. Assuming the DAC’s process the signal the same. Mercury and Pi2AES drive the BNC and RCA from the same buffer chip.
    The only difference is the voltage level. Which for cables less than 1 meter (3ft) is totally irrelevant.
     
  5. luckybaer

    luckybaer Friend

    Pyrate Contributor
    Joined:
    Mar 19, 2018
    Likes Received:
    2,191
    Trophy Points:
    93
    Location:
    Missouri
    I can do it going from my PC via USB to Singxer SU-6 - and then taking coax or optical, etc. to the DACs.
     
  6. Armaegis

    Armaegis Friend

    Pyrate BWC
    Joined:
    Sep 27, 2015
    Likes Received:
    7,467
    Trophy Points:
    113
    Location:
    Winnipeg
    Oh I absolutely believe you... but you just know someone is going to cry foul over the test results if different cables/connectors are used.
     
  7. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,778
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    Just ordered the Van Alstine ABX comparator. It's time to put an end to the AmirNAD dogma nonsense and show that SINAD doesn't make much difference past ~55 to 75db.

    I expect them to keep moving the goalposts but I really don't care. They more they do, the more crazy they will appear, like the flat-earthers or the Church in the middle ages who thought the sun circled around the earth. Personally, I think blind tests (informal, formal, for fun, or according to rigorous statistical analysis, etc.) are fun. I'm looking forward to bringing these to meets, large and small. I'm hoping I will start a trend, that more people do these kinds of tests too, and maybe take the effort to make a two-hour YouTube video from beginning to end.

    Randy Cheap Audio Guy in coming into town at the end of next month and staying near me. We could do something fun.
     
    • Like Like x 9
    • Epic Epic x 9
    • Agreed, ditto, +1 Agreed, ditto, +1 x 2
    • List
  8. Skyline

    Skyline Double-blindly done with this hobby

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    3,426
    Trophy Points:
    113
    Excellent. As you said, Amir will keep making excuses, but having credible witnesses is probably the most valuable tool you have to make their excuses appear more and more asinine.
     
    • Like Like x 2
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  9. JK47

    JK47 Friend

    Pyrate
    Joined:
    Oct 31, 2022
    Likes Received:
    2,033
    Trophy Points:
    93
    Location:
    Surf City USA
    YESSSSS!! As I told Tyll when I met him way back, I would love to see blind test competitions for fun, or $$$. I want to see people put their money where their ears are. I would fail and finish last or maybe second last, but would love to see others put in their place!
     
  10. Vtory

    Vtory Audiophile™

    Pyrate MZR
    Joined:
    Sep 12, 2016
    Likes Received:
    10,831
    Trophy Points:
    113
    Location:
    East Coast
    I almost laughed. This A-blahblah guy obviously doesn't know what the garbage he is talking about. A few interesting points:
    • "The trick is to get > 95% confidence that there was no guess" --> This sentence hints he has zero background in stat analyses. Those who understand the nature of TWO types of errors (type-1 and type-2 errors) won't say like this.
    • "For 95% confidence, they need to get 9 answers right" --> The most obviously wrong part in his claim. Critical number of success with 0.05 alpha is, assuming one tail test (because we are not interested with the other end), 4 out of 5, 8 out of 10. Maybe he intended two-tail test but why (no rationale provided)??? Let me elaborate the process a bit:
      • Because he mentioned confidence, I'm guessing he meant type-1 error. This means everything is conditioned on random guesses. Type-1 error is, roughly speaking, the probability of getting more extreme outcomes conditioned on the null hypothesis.
      • Binomial distribution parameters with p=0.5 (random guess), n=10 (total trials), and x (critical number of successes)
      • We want to find the minimal x such that the cumulative sum of Binom(n,p,z) between z=0 and z=x is equal to or greater than 0.95 (=1-alpha=1-0.05) --> x=8
    • "They must run the test at least 10 times" --> In general, determining the required test size needs to consider the TWO errors together. Often more consideration toward the type-2 error (related to so-called "Statistical power"), requiring proper formulations of alternative hypotheses. This also introduces a number of relevant questions including "to what resolution we want to distinguish measures". Any co-statisticians or myself never talk solely about type-1 error when it comes to necessary sample sizes.
    • "No blind test is valid without statistical analysis" --> This sounds a bit funny. Of course statistical analysis is an umbrella term. So, the argument per se isn't too wrong. But I doubt if he really meant just two numbers lol.
      • For the reference,knowing the sample size and first moment (mean)--"A successes out of B trials" -- is enough from informational perspectives in the context of binomial distributions. The way Marv presented his results this time (full result sequence) is even dominantly better because of greater informational quantity (e.g., order of outcomes).
      • Amir seems to think statistical analysis is a big deal, but it's not. It's mostly about reducing dimensions and complexities. For simple scenario like binomial trials (a sequence of respectively independent Bernoulli trials), his wording sounds pretty awkward.
      • The only good point he made was Marv didn't report the results specifically in the Magni 3-heresy thread. But 10/10 is reported this time. lol
     
    • Epic Epic x 7
    • Like Like x 4
    • List
  11. Skyline

    Skyline Double-blindly done with this hobby

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    3,426
    Trophy Points:
    113
    @Vtory , I would recommend two-tailed in this instance which is why I went with a 95% confidence interval in my earlier post.

    If someone gets 0-1 correct, it is statistically significant and there's a reasonable chance that they were probably NOT guessing. They are just confused about the properties of the amps...they have associated the wrong amp name with a particular sound.

    Or, if the test is flawed, there is something else that is tipping them off. Either way, 0-1 correct guesses is just as significant as 9-10 correct guesses if all we are trying to determine is whether they are guessing randomly.
     
    • Like Like x 1
    • Agreed, ditto, +1 Agreed, ditto, +1 x 1
    • List
  12. Vtory

    Vtory Audiophile™

    Pyrate MZR
    Joined:
    Sep 12, 2016
    Likes Received:
    10,831
    Trophy Points:
    113
    Location:
    East Coast
    I see. That makes sense in the test you assumed, where inverse prediction is possible. My basic blind testing method is 3-point abx. Each Bernoulli trial is defined as hearing A and B then matching randomly selected X to either A or B. And in this test, I'd avoid two tail because of power loss other things being equal.

    PS. I think it's typo.. but isn't it that 1-10 correct guesses (1 correct out of 10 trials) is equivalent to 9-10 correct guesses (9 corrects out of 10 trials)? 0-1 is totally different story (i.e., different distribution at all). Type-1 error is drastically different.

    Pr (one incorrect out of 1 trial | random guess) = Pr (one correct out of 1 trial | random guess) = 0.5
    Pr (nine corrects out of 10 trials | random guess) = 10 * 0.5^10 = 0.0098


    Disregard above. I might misunderstand. You're just saying the range instead of outcome-total combination.

    PS2. Just for fun.. Heretic test's type-1 error..
    Pr (10 corrects out of 10 trials | random guess) = 0.5^10 = 0.00098
     
    Last edited: Feb 21, 2023
  13. Skyline

    Skyline Double-blindly done with this hobby

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    3,426
    Trophy Points:
    113
    If you're worried about power, just raise the sample size. As I tell my students, whenever you don't know the answer just say that you should raise n and you'll be right 90% of the time. :p
     
  14. Vtory

    Vtory Audiophile™

    Pyrate MZR
    Joined:
    Sep 12, 2016
    Likes Received:
    10,831
    Trophy Points:
    113
    Location:
    East Coast
    I was schooled that way decades ago, too. But the realty is trickier because we have to think about the costs. That's why I emphasized "Ceteris paribus". Increasing sample size is not a free lunch, particularly in human experiments. Even for my hobby experiments, considering each trial needs its respective wash-out/rest time, reducing trial size would be way more favorable -- even bigger benefits when we involve our friends or families. In other words, I'd prefer to maximize power given allocatable resources. I also prefer to design experiments to systematically avoid inverse-predictability when I'm allowed to design structure.

    Aside: I've been conducting some ABXs for myself (for curiosity). At the end of the day, I really always wanted to reduce as many # trials as possible...

    Anyway, back to the topic, I really don't think Amir meant two-tail. Look at his statement. "they need to get 9 answers right". He should've added "... or they must get 1 or less answers right".

    My original point is that two result numbers will be enough to report. Choice between one vs two tails in testing might be slightly subjective and/or context-based, I think.
     
    Last edited: Feb 21, 2023
  15. Skyline

    Skyline Double-blindly done with this hobby

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    3,426
    Trophy Points:
    113
    I don't think Amir had any clue what he was talking about, so I don't take his intent into consideration :p

    Sorry for the nerdy diversion, everyone. Back to your regularly scheduled (super cool and non-nerdy) programming. |\/|
     
  16. yotacowboy

    yotacowboy McRibs Kind of Guy

    Pyrate Contributor
    Joined:
    Feb 23, 2016
    Likes Received:
    10,696
    Trophy Points:
    113
    Location:
    NOVA
    Home Page:
    I very much appreciate the stat nerd discussion!!! I think that's part of the allure of SBAF. Simple comments like, "that's not exactly right," that come from folks that don't necessarily tout their academic/experiential accomplishments but often hint at a deeper understanding of the nuances involved in applied mathematics and the practice of science show the general restraint I've come to recognize as hidden gold on this forum.

    Hints, nuance, selective word choice, etc., are indicative of deeper understanding gained through something other than a textbook, imho.
     
  17. caute

    caute Lana Del Gayer than you

    Pyrate Contributor
    Joined:
    Jul 12, 2022
    Likes Received:
    1,990
    Trophy Points:
    93
    Location:
    The Deep South
    This is also why I love SBAF, when I saw what some of you do via the Jobs IRL thread, I wasn't surprised one bit—that we have scientists, PhDs, grad students, and other occupations of the intelligentsia.

    So true, the extreme restraint in selecting just the right words for something, an idea, concept, even impression, is usu indicative of careful, very subtly nuanced, but ultimately rigorous thought.
     
    Last edited: Feb 21, 2023
  18. msommers

    msommers High on Epipens

    Pyrate
    Joined:
    Nov 6, 2016
    Likes Received:
    2,750
    Trophy Points:
    113
    Location:
    Calgary, Alberta
    Home Page:
    Has anyone used any of the VAA products? Their power amp looks beefy and priced well considering.
     
  19. roshambo123

    roshambo123 Friend

    Pyrate Contributor
    Joined:
    May 26, 2018
    Likes Received:
    2,763
    Trophy Points:
    93
    Location:
    Los Angeles
    Stats people, correct me if any of this is wrong, but:

    I believe the intention of using a 95% confidence interval assumes the distribution of values is gaussian. In the event that the distribution isn't gaussian, (ie. it's bimodal or Mandelbrot-ian) then the stats may be misleading.

    An example of this might be that you have outliers that either are deaf or have golden ears. The average Joes would form a Gaussian distribution but the gifted or deaf listeners could create a skew. If you ignored the non-gaussian distribution and simply took the forced mean you might come to a wrong conclusion. The last time someone did this at scale it crashed the world economy, re: Black-Scholes Model

    Additionally, 95% confidence doesn't mean if the value isn't within 2 standard deviations it's a random result. My understanding is it means there's a 95% chance that the observed values fall within 2 standard deviations of a predicted gaussian distribution.

    As for getting 9 answers right, this is wrong. The hypothesis "People can't tell amps apart" doesn't require everyone is 90% accurate. If we have two comparable groups, where the control listens to two of the same amp (but is told they are different) and the test population listens to two different amps, and the test group identifies the different amp correctly and we use statistical analysis to be confident the test group result does not fall within the control group distribution (95% CI or 99% CI if you have a large sample) then the hypothesis is not supported.

    As for running 10 times, my understanding from college is a sample size of 30 is table stakes for statistical power at a 95% CI.
     
    Last edited: Feb 23, 2023
  20. purr1n

    purr1n Desire for betterer is endless.

    Staff Member Pyrate BWC
    Joined:
    Sep 24, 2015
    Likes Received:
    89,778
    Trophy Points:
    113
    Location:
    Padre Island CC TX
    I've probably had a bit too much to drink tonight (happens on business trips), but Amir is a turd.

    Look, I got a B- in my stats class. I'm sure @zerodeefex didn't much better when he followed me at the same uni years later. Not going to argue with you guys who obviously understand more. But f**k this Amir dude. The problem with him is trat he thinks he's smart about everything on the basis that he oversaw thousands of employees at Microsoft, of which in the consumer division he did jack shit and was obviously ejected from this position. (Look, Bill Gates does not tolerate failure).

    You guys wanna know my secret at my RL job? I assume I am a dumbshit and rely on the specific SMEs at work. I'd be so so fucked without them.
     
    • Agreed, ditto, +1 Agreed, ditto, +1 x 6
    • Like Like x 2
    • Epic Epic x 1
    • List

Share This Page