Location - yes and no. Your business would need an address, but it could just as easily be the address of one employee. If you can trust absolutely everyone to work at home and have meetings either at some one's home or over a meal. Face time and an office can improve productivity or impress a prospective client, to be sure, but it's a tremendous amount of overhead. Easily upwards of $2000 a month, so unless it will increase your productivity or profit by more than that, skip it.
Phone - It's probably best to not give out private cellphone and home numbers as business contacts. Instead, get a Skype number. It'll run you all of $3 a month per number and you can forward it for free.
E-mail - not hosting an e-mail server will save you plenty, and time not spent doing IT is more time your developers can spend developing. The absolute best e-mail client in the world is GMail, and you can map it to your own domain for free or sign up for Google Apps and get 25 gigs of storage (per account) plus all the nifty tools that go along with it for only $50/year/account.
Computers - any developer is going to have their own, so you don't necessarily need to buy new ones unless you're hiring people on. Obviously if you're doing work on a computer you're going to want a decently powerful machine with a pretty good sized screen, but if all you need is a word processor with a web connection, you can get a netbook for $300 or so.
Web Site - you can set one up free through Google Sites and map a custom domain to it, but if this is a web development company, they're going to want to have written their own. Probably. Off-site hosting is pretty cheap these days, if you're paying more than $50 a month, you're overpaying. You can get decent hosting for as little as $7 a month, especially if you're not anticipating a lot of bandwidth usage at first.
Software - Again, on the cheap. Linux is a free OS, and there are plenty of programs that run on it, including Open Office, which I would use in lieu of Microsoft Office any day of the week. Office has hung itself from the rope of its own proprietorship. Documents are now .docx instead of .doc and are not compatible with any other word processor. With OOo, you can save things as a .doc in a variety of formats and export to PDF with relative ease. The menus are more familiar than the new Office Layout, and it's slick (in version 3 anyway). Office is bulky.
You can do photo manipulation with GiMP, which is not quite as versatile as PhotoShop, but it costs nothing (as opposed to $400, or whatever PS retails for these days) and is decently powerful.
You can do inter-office communication with GTalk and share projects over Google Docs or Sites pretty easily. Coordinate schedules with Google Calendar. Coordinate projects with any number of free repositories out there (SVN comes readily to mind--and we've recently started using Mercurial at the office). There are a number of free IDE's (if you're working in Ruby or Java or HTML), but there are a few frameworks (.net and development kits for video game systems) that will require some money. Hell, if you're doing web development, you can cover a lot of ground with JEdit and Firefox's Code Auth plugin.
Advertise on Google--what it costs you per click depends on your market, but if you have a $50 monthly ad budget, you can still use PPC. You don't have to take out print ads in every market. Also, with a web presence, you can sell to anyone in the country, if not the world.
So this is an extreme and thoroughly idealistic, perhaps even marginally unrealistic example. But, on the other hand, it shows that the barriers to entrepreneurship are receding. You can start a business in your basement for less than $1000 without breaking a sweat. I've said it before and I'll say it again, the Internet is bringing back the American dream.
Anything I missed?