I've seen very serious suggestions and admissions by people that using a VPS is an amazing way to remain anonymous and protect privacy. For almost everyone, a VPS will not provide anonymity. It takes a static IP address and locks it to your identity. You can make an identity anonymous from real identity, but nothing is completely anonymous and you can never use the VPS under your "real" identity. SSH keys alone are unique and would probably be a point of failure here unless one were to create a second key. You cannot ever use the VPS as a SOCKS proxy or VPN for anything that you do under your real identity. For example, if you go to some gaming website under a username that is part of your "real" identity, it would be very easy for the moderators to connect your "anonymous" identity. Everything you do would be traceable to your identity. You would not be able to do anything undercover and truly anonymous.
A VPS is still not completely useless. If you don't care that it is generally not anonymous, you can work through it with remote access tools like SSH or a remote desktop client. A setup like this will be fail-closed rather than fail-open. You can easily use a command line IRC client and browser such as Lynx, but command line browsing might not be viable in most situations. You could use a remote desktop connection to use a graphical browser.