You could upgrade your memory and video card but than you may have to upgrade your power supply also. I wouldn't go any higher than a HD 5770 video card or you'll just be wasting your money as your system will bottle neck because your processor is too slow. Probably more like a HD 5670 would be more in tune with your system as the power needed for this card is low. So you could upgrade for about $200 but it may be sending good money after bad seeing as your willing to spend about $600, I'd spend a couple hundred more and go with a new system. Memory express or NCIX would be glad to put something together for you. Just remember to price match all the components.
Here's a couple of sites to do your price matching with.
http://www.pricebat.ca/
http://www.shopbot.ca/
Because I'm bored here's a system to consider before price matching.
Case $116 and PSU
http://ncix.com/products/index.php?s...c&promoid=1141
MB $103
http://ncix.com/products/index.php?s...S&promoid=1141
Processor i3 $130
http://ncix.com/products/index.php?s...l&promoid=1141
Memory $115
http://www.ncix.com/products/?sku=42745
Hard drive 500GB $54
http://ncix.com/products/index.php?s...D&promoid=1141
Video Card HD5750 $120 (this is optional as the processor comes with a built in card but I don't know if it's much good)
http://ncix.com/products/index.php?s...r&promoid=1141
DVD drive $19
http://ncix.com/products/index.php?s...g&promoid=1141
Windows 7 64 bit $106
http://ncix.com/products/index.php?s...t&promoid=1141
PC assembly and testing $50
http://www.ncix.com/products/index.p...facture=NCIXPC
Total $813
This would give you a great computer with the option to upgrade easily.