I use Xp-Dev for my SVN host and they have bug tracking as well. The service is free and very very good, in my opinion. You also do not have to be open source to use it.
http://xp-dev.com
Additionally to what has been said, a VPS is much harder to set up and run on a daily basis, it may be...