A lot depends on what you intend to do with the file...
If you intend to process the file afterwards, even simple normalization, but especially compression, EQ and limiting (the usual chain to get a 'pro' sheen and level on it), it's better to record as a 24 bit .wav file. I don't think it is worth higher sample rates (48k or 96k) because you are going to have to sample rate convert to 44.1 for a CD or MP3 anyway, but the extra dynamic range and resolution of 24 bit is good for making a better master...
The main thing about going with a .wav for post-pre mastering is that most programs that work with mp3's really convert the file into a .wav (often in the background, so you don't know about it) where you do the EQ etc, then it converts it BACK to an mp3. Encoding mp3's is a lossy process (a little bit of detail and sound is lost in the process), so the fewer times you do it, the better.
My general system is to record at 24/44 .wav, master a bit afterwards, then either convert to 16/44 .wav for burning an audio CD, or convert to mp3 as the LAST step.
However, if the recording is simply for your own use, as a confidence check on what you are playing, even a 192 or 256 kbps mp3 will suffice to give you a pretty good replay of what you just did. No point with all that other stuff if it just for yourself...
But the aux out on the 3K is probably overloading the input of the H2. This is where putting this little volume control in-line will allow you to lower the level to the point where it doesn't. And theoretically (I haven't tested a Yamaha) the aux output should be a tad cleaner than the headphone output...
Hope this helps.