Expectation Bias

Discussion in 'Random Thoughts' started by Thad E Ginathom, Oct 20, 2015.

  1. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,134
    Trophy Points:
    113
    Location:
    India
    Well, maybe.

    And then again, maybe not. If I recall correctly, Ethan Winer's first anti-audiophool article was aimed at his fellow professionals. He re-wrote it for us lot, the audiophiles. You you think that audio professionals don't have fads, biases, and even an imagination, then you are wrong. Hey, maybe surgeons do to! And did you know that double-blind testing is very much a part of serious wine-waiter training? No bullshit: those people can tell one wine from another without the label.

    Of course a recording engineer uses his expertise to mould the sound. Of course he knows more about how to achieve that than I ever will. Does that mean he is not subect to bias? Of course not, Because everybody is.

    One of the best comparisons of audiophile and engineer I heard was: they both hear the same thing, but engineers know how to put numbers on it.

    You must also understand that blind testing means trusting your ears. It means finding out what your ears are really telling you, not what your eyesight is telling you, not what your expectations are telling you, etc etc etc.

    True technicians use the tools at their disposal. Whether a track sounds better with just a little more 600Hz is a matter of personal preference and experience. That is what engineers are using their ears, and their minds for. One can disagree with their decision, but it is still their decision. Whether a cable, a support, a bit depth, a sampling rate, or even a green marker pen on the CD makes a difference or not is a matter of fact, not preference --- excepting the aesthetics of the thing. cf JJ Johnston.

    There I go with another of those appeal-to-authority things, I suppose. But I prefer to look at it as an appeal to take on board what those gurus of the digital age actually have to say, and I think you'll find that they most certainly use their ears!
     
  2. Original Ken

    Original Ken Friend

    Pyrate
    Joined:
    Oct 18, 2015
    Likes Received:
    94
    Trophy Points:
    28
    Location:
    Central California
    Home Page:
    The problem with the expectation bias and similar bias arguments is that they are junk science.

    The idea that you can make your $1 earbuds sound as good as HD-800's, simply because you expect them to, is the same idea that if you get cancer and just think positive thoughts about the cancer cells, they will go away.

    Ultimately, the "expectation bias" argument is not falsifiable and thus not scientific. Double blind and other such methods are comparing subjective impressions. Music does not occur outside of a mind, so sound quality cannot ever be objectively determined - that is inherent in the word "quality" as opposed to "quantity".

    One telling aspect is that if you read all the anti-audiophile posts on the Internet, 99% of them have a $ somewhere in them, as in "$10,000 cable". It's always an emotional argument about money, even though it is framed in terms of "science". Science does not care whether one cable is $20 and the other is $10,000.
     
  3. lm4der

    lm4der A very good sport - Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,461
    Trophy Points:
    113
    Location:
    Seattle, WA
    These are really good points. But I think that it is too strong to say that expectation bias is not falsifiable. If one can reliably DBT a difference, then we know that expectation bias is not a factor. As for the legitimacy of expectation bias as a phenomenon, I have read stories from recording engineers that have talked about spending hours EQing a recording, and making what they believed were the improvements they wanted, only to realize that their EQ interface was not engaged, and they had changed *nothing*. Furthermore, the placebo effect is a documented medical phenomena. A certain percentage of people do get "cured" taking the sugar pill, who otherwise wouldn't have been. Why this happens is not well understood.
     
  4. OJneg

    OJneg The Most Insufferable

    Pyrate BWC
    Joined:
    Sep 25, 2015
    Likes Received:
    3,923
    Trophy Points:
    113
    Location:
    Grand Rapids, MI
    Pro audio guys (musicians, producers, recording, mixing, mastering) run just as wide of a gamet as audiophiles do. Some of them are like Ethan Winer and put complete faith in a dScope, while others will wax poetic about how little stabilizer tweaks on their mic pres improved sonics by 169%. It comes down to relying on a person's credibility and methodology.
     
  5. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,134
    Trophy Points:
    113
    Location:
    India
    Really? I don't rate your understanding very highly.

    What idea re $1 buds and HD-800s? I never heard anybody say anything like that. Is this what is called "a straw man?"

    You have not understood at all. Biases, expectation and all, are not a matter of will --- and that is the problem

    Hold that thought!

    If you put some of your thoughts together in a different order, then I think we may be getting somewhere.

    On the other hand, I remember having decided that I have had enough of these conversations. I've left forums because of them (and been made unwelcome too). That is not a cop out. If you would like me to answer any further points you have, I will.
     
  6. Original Ken

    Original Ken Friend

    Pyrate
    Joined:
    Oct 18, 2015
    Likes Received:
    94
    Trophy Points:
    28
    Location:
    Central California
    Home Page:
    From your signature, you have a bias. Since it is your signature, you consider it to be part of your personal identity. So, you are never going to actually ever say "you are right", so there is no point to the argument. (In the same way that Fox News will never say "Obama is right" and MSNBC will never say "Bush was right" because their identities depend on it.)
     
  7. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,134
    Trophy Points:
    113
    Location:
    India
    From my signature I have learnt something. I have learnt a little (I don't claim much) about how ears and brains work. I consider this a vital and very interesting aspect of listening to music. (not necessarily enjoying it, though: these logic-appraisal brain cells don't much contribute to that).

    Oh,

    I misunderstood you. Are you under the impression that blind testing always "proves" there is no difference? Of course that is not the case. Blind testing can be confirmatory too. It is a test, not a foregone conclusion.

    In fact, of course, it never proves that there is no difference, only that it is actually unlikely that, at that time, that person could reliable be said to have heard one. There is no earth-shattering one-person disproof achieved by "failing" (I don't like that word in this context) a blind test: it weighs the scales of likelihood. The earth-shattering absence, though, of successful blind tests of certain audio aspects also carries a weight of likelihood.

    As to your final (?) argument: that looks like the copout to me.
     
  8. Original Ken

    Original Ken Friend

    Pyrate
    Joined:
    Oct 18, 2015
    Likes Received:
    94
    Trophy Points:
    28
    Location:
    Central California
    Home Page:
    The following was posted some months back in a different Forum by member baldr :

    "Back in the early 1970s, before I founded Theta Electronics, the tube audio products company, I had a busy part time biz rebuilding Dynaco Tube Amplifiers. At that time I had converted to the tube based practice for my own system, convinced that tubes sounded better than the solid state gear of that era. In my ramblings, I met John Koval, a man who had designed a modification for the old Qual ESL loudspeakers which made them sound much better. “The mod gets rid of a 5 db bump in the 200-400 Hz region which makes them much flatter” he explained. I told him that I was enchanted with the sound of tube amplifiers and preamplifiers. He explained that as long as the frequency response was the same and the levels were precisely matched, there was no way anyone could tell any amps/preamps apart in blind A/B tests. He had built a custom box that matched levels and randomized any two amplifiers or preamplifiers with a pushbutton to switch between them. Bullschiit, I thought, what about the solid state A/B box and its sonic signature.

    Intrigued, I built a similar box with passive relays and a passive attenuator. Damn, if he wasn't right. It is really difficult to tell differences in an instantaneous blind A/B test between tube gear that I built versus some commercial gear that I was not particularly fond of. I used to bet John beers that I could tell the difference. Usually, I won at 7 out of 10 picks or so – the best I ever did was 9 out of ten. But it was really hard.

    This whole deal made me wonder if I was crazy hearing differences between amps. If what John said was true, and many others have said in the passing 40 years or so, there is no point for an audio hobby involving anything other than transducers. WTF?

    So I tried something new – I still did the A/B tests, matched levels, but allowed long-term listening to each; at least an hour or two with known recordings. Guess what! Suddenly I knew which was what. I tried it out on John B and Mike and Dave and all my other audio buddies. They called it too – tubes vs a bad solid state preamp. Every friggin' time. My enthusiasm had returned. This taught me that the human ear is an integral, NOT differential device.

    So much for the blind A/B instantaneous naysayers. All that matters is frequency response, they say. People can't hear anything much above 20KHz in their prime, less later. The ear has a short memory, it is all bias, blah, blah. They should take up a different hobby, say stamp collecting.

    Thanks to Dr. Heil, the inventor of the Heil AMT speaker who shared this experiment with me over 40 years ago, Consider this: I am 67 years old – my high end extends to just under 15KHz (not bad for and old fart). I can play back two pulses 200 microseconds in length separated by 20 microseconds and clearly hear two pulses. Not unusual until one considers that 20 microseconds corresponds to a square wave of 50KHz. And then, there is the time domain – home of spatial cues which audio measurement traditionalists ignore. I believe that in the quest for the best sound, an open mind is the most important asset. I will even listen to cables, even though I believe in my heart that all technology about cables is well known. Who knows, even an old fart like me could be surprised.

    Until then, yet another retelling of my old John Koval saga is 40 year old news to me."
     
  9. ultrabike

    ultrabike Measurbator - Admin

    Staff Member Pyrate MZR
    Joined:
    Sep 25, 2015
    Likes Received:
    8,960
    Trophy Points:
    113
    Location:
    Irvine CA
    Moved some post from "Bass Quality" and created the "Expectation Bias" thread.

    So, IMO, expectation bias is very real. Part of being a human being. Sometimes is good. Sometimes not.

    Had to learn a little about it because I had to take Social Psychology for my electives quite a few years ago while at college. It's a pretty interesting subject.
     
    Last edited: Oct 21, 2015
  10. atomicbob

    atomicbob dScope Yoda

    Pyrate BWC MZR
    Joined:
    Sep 27, 2015
    Likes Received:
    18,653
    Trophy Points:
    113
    Location:
    On planet
    Anyone that has spent time at the board in the control room can relate to having a band member ask for more (*insert parameter here*) while everyone is auditioning the latest version of the mix. Mix engineer twists a knob, moves a fader or changes something on the console. Upon next listen everyone agrees the change is better, including the mix engineer. Everyone heard the change. As the mix engineer proceeds to document the change s/he discovers the parameter changed was on an unused channel at the moment. Nothing was really changed. Expectation bias in action.
     
  11. Perot

    Perot Facebook Friend

    Contributor
    Joined:
    Oct 2, 2015
    Likes Received:
    207
    Trophy Points:
    43
    Location:
    South Carolina
    I worked at a friend's bar doing the live sound for a while and noticed a similar thing. Someone on stage would ask for more 'x' in their monitor; sometimes I would do as asked sometimes not. More often than not when I did nothing they would say "sounds better, thanks." Of course sometimes they would just keep saying more.-
     
    Last edited: Oct 21, 2015
  12. JewBear

    JewBear Almost "Made"

    Joined:
    Sep 28, 2015
    Likes Received:
    270
    Trophy Points:
    63
    IMHO I think the problem with both these positions is that all equipment matters to some extent. But after a certain price threshold, the differences are non detectable if the equipment is not purposely colored or poorly implemented. That threshold is different for different gear. I'd say for cables, it's ~$40, for dacs ~$600, for amps ~$1000. Obviously a $10 dac will sound different to a $600 dac but I highly doubt two competent $700 dacs with the same topology will sound different.

    I think there is a tendency for people to overspend and then try to justify the overspending to themselves. Self rationalization is a powerful force when it comes to coloring your experiences. For example, I am much more willing to give equipment the benefit of the doubt if I just spent $1000 on it. Otherwise, I have to admit that I wasted money.

    This whole argument reminds me of the argument they used to have as to whether small amounts of radiation were dangerous. The danger is relative to the dose, but there's no way you'd ever be able to blame the radiation until it got to a certain threshold. It doesn't mean it doesn't have some minuscule effect, it's just undetectable against the backdrop of your normal life.
     
  13. Griffon

    Griffon 2nd biggest asshole on SBAF

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    1,309
    Trophy Points:
    93
    Location:
    Hipster capital of Canada
    Expectation bias and placebo always come into play, and I think no matter how objective one consider oneself to be, there will still be some inevitable expectation bias and placebo. I remember meeting an old audiophile gentleman at a local HiFi shop, who hated mp3 with a strong passion. Then I played some music from my AK Jr (which was loaded with mp3) and he used all sorts of flowery words to describe the music.

    I don't hate manufactuers who price their products with a high tag, but with a high tag I do expect that product to achieve something uniquely impossible to achieve by products in the lower price range. To meaningfully differentiate that unique improvement, I for one strongly believe in that double blind testing will be the bottom line. But some manufactuers just make a high price tag and use the associated expectation bias to make their stuff sound good. This is what I hate. This is the subjectivist extreme.The other objectivist extreme also have their expectation bias. We all know the stories like "O2 ODAC are the final frontier in sonics", or "if 2 DACs have the same measurements in A B C then there is no sonic difference". But in the end, the two camps are just believing what they want to believe, and are just hearing what they want to hear.
     
  14. Thad E Ginathom

    Thad E Ginathom Friend

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    14,134
    Trophy Points:
    113
    Location:
    India
    Yep. Saw that over on HF. And yes, Mike is another founder of the digital age that I respect.

    However, as I also commented on that thread, he had dismissed blind testing as if it were some sort of speed dating thing. That is not a fair condemnation, but encourages one of the common misconceptions, which is that blind testing has to be done in few-second samples. Really, I think that somebody with Mike's experience should have known better than to say that.

    Perhaps, like me, in sighted evaluation, you might have also, sometimes, focussed in on a few moments of music, perhaps even one sound, comparing A to B. Sometimes we do that. Sometimes we listen for an hour, or hours. There is no reason for blind testing to be any different. The tester should be comfortable. Many of the objections to blind testing are misconceptions, which I guess it suits some people to perpetuate. Others are from people with egos who can't stand to have their certainty taken away from them. As I have already explained, even though it is their motto, in reality they do not trust their ears.

    There shouldn't be two camps. There is no need for arguments like this. The tools are there to be used. Part of the reason that this stuff turns into an argument is that vested interests do have something to loose: we are never going to see the cable industry, for instance, form an independent authority for assessing and blind-test-listening to their products. Nope: we are going to continue to read the reviewers, writing that "A" was good, the improvement could be heard in "B," but wow! "C was well worth its price tag." It is not honest, and it makes a fool of the whole industry.

    Another misconception about blind testing is that it is subjective v. objective. It is not. It is entirely subjective: it is about listening to the music, and what one can hear, and as Ken said, ultimately, that is in the mind (which is the reason for my signature). Blind listening just removes other influences and distractions.

    I don't come to forums like this to try to browbeat people into doing a blind test every time they turn on their systems. It would be hypocritical anyway: I am too lazy to actually do the thing properly! Still, I have seen my own certainties vanish in a not-blind-but-slightly-confused (I tried to forget how many times I'd mouse-clicked) test of 16/44 against 24/96. Absolutely every difference I thought was there in one sample turned out to be there in the other, also, ie it wasn't a difference, when I looked for it, sighted or otherwise. And my hearing was better back then.

    I like reading about people's subjective experiences. That is why I come to places like this. It is true that I plain-don't-believe some of the things that people say they hear--- but if I am reading their post about a piece of equipment it is because I want to know their overall feelings.

    I envy Ken his ears. Ken, you are even older (hope you don't mind me putting it like that!) than I am, but have a whole octave on my hearing, which has had some HF defects for a long time, and now tails off from 8kHz. hey ho, I can still enjoy the music, but, apart from the laziness, I now feel disqualified from serious test listening. Hiss? I probably couldn't hear it anyway. Still, in get-togethers with audiophiles, my comments and opinions have not been out on a limb. The difference has sometimes been that I try to take into account how my brain works, and, for some reason, many of them prefer not to.
     
    Last edited: Oct 22, 2015
  15. k3oxkjo

    k3oxkjo Friend

    Pyrate
    Joined:
    Sep 30, 2015
    Likes Received:
    64
    Trophy Points:
    18
    Location:
    N. J.
    Home Page:
    The thing about expectation bias is that once we know about it we can, at least to some extent, do something about it if we want. People mitigate their biases all the time, usually through a process of education and self-examination. For those of us who are fascinated with the science and art of music reproduction, it's important to remember the words of Richard Feynman; The first principle is that you must not fool yourself and you are the easiest person to fool. Personally, I try and keep in mind that price, prestige and other people's opinions don't matter once the rubber meets the road and one is evaluating a piece of equipment for himself.

    I have some sympathy for Thad's representation of Blind testing as an open ended process. But the usual conception of Blind testing was the "scientific experiment" concept with rapid A/B within the context that time was somewhat of the essence. I remember one of the first published mass Blind testing experiments reported in Stereo Review, I believe. In this test, a "competently designed receiver" and "a high end tube amplifier" were A/Bed, the statistics totaled and judged for probability that "differences" could be noted between the two products. It was "proved" to the author's satisfaction that no differences were statistically provable, and that was that.

    What was interesting to me was that this sort of testing was just assumed to be relevant to assessing audio equipment. And behind this lurks something else I found interesting. Due to the higher output impedance of tube amps, one would expect response errors on the order of 1 to 2 db (you can see this in graphic form in many Stereophile tube amplifier reviews). From other types of studies, errors of this magnitude are generally considered to be reasonably audible, but somehow the victims... err... subjects of the blind test sited above were not able to perceive these errors (at least according to the author of the study and the terms of the test). When I realized this, I knew something was "rotten in Denmark". My question: how much difference between devices under test must exist before they can be reliably detected in a blind test of the type usually cited? Has anyone tested this (say keep raising response errors until the subjects pass with some degree of consistency)? Normally, "testing the test" would considered be the scientific thing to do...

    Certainly, there are many disciplines where blind testing is invaluable. I just suspect that evaluating audio equipment is not one of them.

    Kevin
     
  16. money4me247

    money4me247 Friend

    Pyrate
    Joined:
    Sep 29, 2015
    Likes Received:
    83
    Trophy Points:
    18
  17. Bill-P

    Bill-P Level 42 Mad Wizard

    Pyrate
    Joined:
    Sep 25, 2015
    Likes Received:
    4,801
    Trophy Points:
    113
    Location:
    San Jose, CA
    Just to say... it was reasonably hard to do this. I could pick out the 1dB increase every single time, at any level, but the 1dB down is where I get caught a lot.

    So something like this was a nightmare for me to pick out, but I finally got it after a bit:

    [​IMG]

    Note: not trial and error. 1st try was something like 7/10 for me, and then I refreshed the page and got 10/10 after about 2-3 tries, so it is definitely possible to reliably tell, just that when the level is "down" rather than "up", it's harder. I guess that's proof that sins of omission are easier to forgive than sins of commission.

    Note 2: the website seems to favor putting a lot of 1dB down and flat in the trials, which IMO means it's rigged so that most people would fail. That's not cool. But anyway...

    On to the subject. I think this is very highly gear-dependent... as well as being in conjunction with other factors like fatigue/stress/etc. I could never finish this test this reliably until I got the HD800. That's not to say it's dependent on the headphone, but it would seem to me that something with the dynamic range capabilities to pull this off is pretty much mandatory. Attempting this with Apple ear buds is an exercise in futility (I also tried that just now), no matter how nice/resolving/smooth the source gears are, and no matter if EQ was applied to flatten FR.

    So I think... sometimes, it may just be the gears undoing your expectations, rather than the expectations themselves being unrealistic to begin with. I actually went into this test again with the HD800 expecting myself to fail, and yet I succeeded in conquering it for the first time ever. That's just something to keep in mind.
     
  18. Tyll Hertsens

    Tyll Hertsens Grandpappy of the hobby - Special Friend

    Pyrate
    Joined:
    Sep 26, 2015
    Likes Received:
    2,805
    Trophy Points:
    93
    Location:
    Bozeman Montana
    Home Page:
    Just a quick thought: Can expectation bias be reduced simply by not giving a shit?
     
  19. Luckbad

    Luckbad Traded in a unicorn for a Corolla

    Pyrate
    Joined:
    Sep 27, 2015
    Likes Received:
    3,408
    Trophy Points:
    113
    Location:
    Holly Springs, NC
    Quote of the month.
     
  20. mickeyvortex

    mickeyvortex Friend

    Pyrate
    Joined:
    Sep 28, 2015
    Likes Received:
    36
    Trophy Points:
    18
    This thread is (inherently) biased.....

    I love bias because I'm human. Problem is engineers are not biologists and vice versa.
     

Share This Page