Blind tests show that louder is better (hence the importance of precise level matching). In practice (and I assume in blind tests, too), this is true up to the point at which too much volume produces effects that are detrimental to sound quality--distortion, for example, or excessive room resonances. This leaves me with a couple of questions.
First, why aren't concerns about sound quality met with the suggestion to turn it up a bit? Second, and more seriously, could it be that changes such as those I've made to my system over the last couple of years (different speakers, repositioning them on proper stands, room treatment) have improved sound quality at least in part by allowing me to listen at higher volumes without feeling that it's too loud? That's been my subjective impression, supported by the anecdotal evidence of a substantial increase in the volume threshold at which someone (normally wife, sometimes child) asks me to turn it down.
I apologize if this observation is so obvious that it's not worth making. I'm curious, though, because it doesn't seem to come up in general discussions about improving sound quality (at least not in the simpleminded way I understand it).