You're correct - only part is set up to use rhythm sets, all other parts use patches.
The only way to use more than one rhythm set is to program them into a patch, and cards like the bass & drums has waveforms designed to get at least a basic drum kit happening, by using each of the 4 tones in a patch for different sounds.
If you're stuck without a manual, I have an overview of the JV/XP structures on my website at http://www.chariot.net.au/~gmarts/jv-overview.htm
regards,
GM Arts