Optimizing low level listening – what’s the best way?

Hi, I just was thinking that I usually listen to music with the volume pot at 9 o'clock, rarely more. And was wondering if something could be done technically to the amp to have better sound quality.I mean, if I take an amp, find a way to lower its output gain, I'll be able to use the amp at "higher" volume, using more of the entry signal, less lost in the hopefully Alps Blue Velvet volume potentiometer.

I don't know if my idea is interesting, or even correct, but was wondering if modding an amp to make it produce less watt would make these watts better in sound quality, or not at all... Many thanks for your ideas.