Hi Dave

There are many options for installing Linux, including all the ones you mention.

USB Stick/HDD: to get the best out of this it would need to be USB 3 as while it would work with USB 2 it would be very slow. (If your computer does not have USB 3 you can purchase a plug-in PCIe card to add it for minimal cost)

The Linux file can be downloaded in the ISO format which can be burned direct to a DVD, thus giving you a fully bootable disc.

If you have a later version of Windows you can use Disk Management to create a new partition on your HDD, (You will need to reduce the size of an existing partition first) or you can use one of the many free Disk utility programs available. (I use http://www.partitionwizard.com/free-partition-manager.html )

When you boot from the Linux Disk it will give you the option to either replace Windows or run Linux on the separate partition, thus giving you a dual boot system.

In ALL circumstances back everything up before doing any modifications.

I would not recommend you get rid of Windows as there are a lot of Music programs (Including many big names) that will not run on any form of Linux even using Wine. (Those that use a separate dongle such as East West & Steinberg are probably the most notable)

Hope this helps

Bill