The Rolls MX41B, DOD AC240 and ART SplitMix4 all sound exactly like what I'm looking for. The site for the Rolls says it only does what I want it to "when all levels are set to maximum". So I will probably try the ART or the DOD first.
I was looking at the Voodoo Labs amp switcher too. That also fits the bill, but these other ones look like they may be cheaper and smaller.