Stam,
First the differences in connection-types :
1. XLR
In my view the best connector there is for many applications. Both contacts (male + female) are well protected by the metal housing. I never had one problem with these XLR's (Neutrik)
2. Jack (6.3 mm)
A low cost connector-type that often has contact-failures because of corrosion. The male types have contacts that are often touched by human fingers (covered with body fluids
). In fact this is a low quality one)
3. RCA (cinch)
Super low cost type. Works fine for 'static' audio setups (Install & never change wiring)
About the audio-quality :
XLR's are mostly used in balanced environments what means that the cable screens have NO function in the transport of (low-voltage) signals. The cable has 2 signal wires (+ and -)
Radio interference or other electromagnetic fields do not harm the original signals.
Both the Jack & RCA are used in un-balanced cicuits, where EM-waves do have a chance to harm signals.
In my home-studio setup is a Mackie 1202 VLZ mixer and it has a ALT3/4 jacks output (kind of a subgroup)
I use this group for recording to my AP2496 RCA line-input. This wiring is permanent because I make the selection on the 1202 mixer what channel(s) to record.
There is no 'absolute' difference in quality between the XLR, Jack or RCA as long as the connectors/wiring is OK.