Hi,
It possibly could be polyphony. With sequencing, you typically don't have all of the 64 voices playing at exactly the same point in time. As one note gets released, it frees up polyphony, and so on. With the type of playing you're doing, all of the sounds are playing back together (albeit with some faded out). Even though a sound may have zero volume for the part, it is still using up polyphony. What you can do is set the note priority for each part. You can, to a certain extent control some of the voice stealing in the box.
Also, although Roland in their ads claimed 32-bit RISC processing, the processor is not as fast as it should be IMHO. I own a JV-2080 and I notice midi slop on it (badly :-(). With studio work, I'll record all of the parts that play back from the JV-2080 to audio tracks one at a time.
Had I known this before I bought it (I didn't notice it as I played it in the store), I would not have spent the money. Sound-wise it sounds great and the expansion card capabilities are great. The midi lag on the unit is terrible for today's synths.
HTH
Fernando