It's always the bang for the buck (BFTB) part that's fun.
Considering you have a good recorder, you probably don't need the Tascam Pocketstudio 5 which has a GM synth built-in. This should get more discussion here because it's a simple arranger too! Good BFTB.
http://www.tascam.com/product_info.php?pid=274&nav=pocketstudio You obviously have a computer so also good BFTB would be softsynths and sequencing programs. I would recommend Band in A Box because it's easy and comes with a basic Roland GM softsynth program. Also Power Tracks Pro is one of the best BFTB sequencer packages if you want to do more.
http://www.pgmusic.com. You can do a lot with softsynths now. But first, here's a jewel I haven't shared here until now - I found a nice freeware generic low latency ASIO driver program called ASIO4ALL that will work with any standard Windows soundcard:
http://michael.tippach.bei.t-online.de/asio4all/ This means that you can use any midi or usb controller to play softsynths with very low latency.
My BFTB award goes to Propellerhead's Reason, which is a pro-level softsynth/sample/sequencing studio. It gets double BFTB because it doesn't need a monster computer to run it (doesn't hurt though):
http://www.propellerheads.se/ Then you'll want to consider a midi or USB controller keyboard and there's lots to choose from. Several combinations of what I've mentioned would do the job and cost under $500.