I just started following this thread and it's an interesting issue which we've been battling with as well.
Initially, we started with shared hosted servers (Interland) and then moved on to dedicated Rackspace servers. While these do cost us a couple of thousand dollars per month now, but the up-time, reliability and customer support is well worth it. So, for all customer facing systems like website, store management, order management, CRM, forums, we use the Rackspace server.
For internal development, we have a local server that is connected 24/7 with two broadband connections (primary a 4MBPs shared broadband, with a backup secondary 1MBps from another provider). This server runs Windows Server 2003, and has Kaspersky Antivirus. We also have ZyXel UTM box that acts as router/firewall and also offers content filtering. So, internally we host our SCM (we use SVN) and project management tools here (we use AceProject). This is exposed via a live URL that allows us external access (but in a DMZ zone). Maintaining this internal setup needs a dedicated IT person, but it allows us utmost flexibility in configuration and privacy issues (as our entire code is then hosted internally). Since we're an ISV, so our developers can easily manage this setup, with some vendor help. Bio-metric systems futher protect such access in our office.
However, if you lesser code/data and lesser privacy issues, you can use third party SCMs like CVSDude or Hosted-Projects (we've used them earlier), Basecamp or Deskway for project management, Google for emails/apps, a Wiki service and an online hosted bug repository (we've used Mantis and it works just fine, looks apart). Many CRM offerings have also started hosted services; thereby, not needing you to install any infrastructure for the same with near 99.8% availability. If you do not want to use Google for emails, many companies also offer email boxes for like $1 a month per user. So, for 20-25 users max, including all apps, you might be looking at 2-3K$ per year including all these services.
What we've found difficult to manage is internal access restrictions - like what set of developers get access to what sites, protocols etc. Since new recruits are most targeted at this, setting up such restrictions (keeping in mind that you can always use proxy or other methods to get around it) is tough. But, again with the right training and approach - manageable, though a little time consuming.
If you need any more information about out setup, I would be glad to help.