Now before we begin, you might ask: why would I want to do this? Here's a few possible answers:
Like me you're a cheapskate and you don't want to drop $180 for an Airport Extreme Base Station (or even $50+ for a third-party 802.11 base station).
You have an old Mac (or better yet Mac Mini) that you want to serve as a home server, so why not let it be your router as well?
Like me you've burned out a cheap consumer router with a high number of connections (think bittorrent) and would rather give the Mac direct access to the net to eliminate NAT errors and/or improve bandwidth.
Because you can.
Now there are also some potential downsides to this as well, these include:
Security. Connecting your Mac through a NAT router to your cable model or DSL gives you an extra firewall protecting your Mac from the meanies out on the net.
Your Mac router has to be on anytime you want to use the Internet from another computer/device.
Okay, with that out of the way let's look at some of your options. All of the following scenarios assume you have your incoming internet (cable modem, DLS, etc.) connected to your Router Mac's built-in ethernet port.
As I mentioned before, setting up a Mac as a wireless base station is perhaps the easiest option. All you have to do is go to the Internet tab of Sharing Preference Pane and start it (as pictured at the top). You will also, however, want to click on the "AirPort Options..." button and set up some sort of security as follows. Note that if you're planning to connect with a Windows computer (or some other non-Apple device) you'll need to stick to 5 or 13 character passwords. If you're all Apple, the passwords can be whatever you choose.
A perhaps little known, and lesser used, networking feature of OS X is IP over Firewire. Macs can both share and receive network traffic through their FireWire ports. Setting it up is pretty much the same as setting up the wireless option. Just select the check box next to "Built-in FireWire," and you're good to go.
On the second Mac, you'll need to a make sure the FireWire networking port is active in the Network Pane of the System Preferences. Generally you'll just want to keep this set to: "Using DHCP."
FireWire networking was a bigger deal back before all Macs were shipping with gigabit ethernet. When most Macs were limited to 10/100 ethernet, FireWire was a faster (400Mpbs) option, however, FireWire can still be a good option since portable and consumer Macs (MacBook (Pro)s, iMacs, Mac minis) only have one built-in ethernet port (see the next section). So if you want to set up a wired network between these Macs (or even better, between older Macs) without buying any additional hardware, then FireWire can be a very quick and speedy option.
Adding Another Ethernet Port
Here's where things get interesting and setup becomes a little more difficult. What if 802.11 or FireWire are not options for whatever reason (e.g. wireless is too slow, you already have a Cat 5 network, etc.)? If you want to have your Mac serve as a router to a ethernet network you'll need to add a second ethernet interface to connect to that network. If you're fortunate enough to have a Mac Pro, it has two built-in ethernet ports, so you don't have to add anything. If you have an older PowerMac without a second ethernet port you can add in a PCI card. However, if you have a portable or a consumer Mac practically the only way to add a second ethernet interface is with a USB to ethernet adapter.
Fortunately, Sustainable Softworks (the makers of Mac router software I'll discuss later) offers a free Tiger-compatible Universal Binary OS X driver for several popular models of USB to ethernet devices using the Pegasus or AX8817X chipset, including models from D-Link, Linksys, and Netgear.
With this driver you can easily add a second 10/100 ethernet interface to your Mac through a USB 2.0 port (the driver also works with older Macs with only USB 1.1, but of course it will be limited by USB 1.1's theoretical 12Mbps bandwidth). I had a Netgear FA120 lying around, so that is what I used. Once you install the driver, the new network interface should appear in your Network Preference Pane. On mine you see that it is listed as en4 (because I have Parallels installed it has created two other ethernet interfaces, en2 and en3, en1 is Airport. On your machine, if you have not installed any other interfaces, it is likely that the new USB interface will appear as en2).
Now, you might think that once you've gotten this far all you'd have to do is go back to the Internet tab of Sharing Preference Pane, select the interface and start it up. While that may sometimes work (it's never worked for me, but I've found reports of it working), I have found that it's necessary to set up the network manually when sharing my internet over this interface. My circumstances may be special in part because the device I'm connecting to my Mac is a Xbox 360, but the same holds for some other devices hooked up though this ethernet interface including my MacBook.
Now, in my experience, networking settings often seem sort of like voodoo. I'm perfectly willing to admit that this may be simply because I don't fully understand what I'm doing. Nonetheless, in what follows I'll just report my own experiences in getting the interface working, in hopes that it will be useful to others.
So back in the Network Preference Pane, I set up my secondary ethernet interface as follows:
I've turned off DHCP and set the address to: 192.168.0.1 and the subnet to 255.255.255.192 In making these choices I was following a hint that appeared at macOSXhints back in 2002. Once this is done I went back to the Internet tab of Sharing Preference Pane, select en4 and start it.
Now on my other device (in this case a Xbox 360) I set it up as follows:
IP address: 192.168.0.2
subnet: 255.255.255.192 (this is the same subnet of the en4 USB interface, above)
gateway/router: 192.168.0.1 (this is the address of the en4 interface)
(here's where the voodoo comes in, I have no idea why this DNS setting works. You would think it should be 192.168.0.1, or even the DNS server of my ISP, but those simply do not work. I suspect that it has something to do with sharing over Airport as well).
I've also found that this will only work properly if Internet Sharing is active for my Airport interface. If I turn off the Airport interface (en1), the wired interface (en4) also dies. With these settings in place, everything seems to work properly. Keep in mind you may also need to enable certain services (which will vary with your application) in the Services tab of the Sharing Pane:
As well as open certain ports in the Firewall tab of the Sharing Pane:
More Advanced Options
Using OS X's built-in Internet Sharing is probably good enough for most purposes, but if you have more complex applications you may want to consider some more advanced options. Of course, OS X Server will allow you much more flexibility, but also costs much more. As I mentioned before Sustainable Softworks also offers a free standing router application of OS X (Client) called IPNetRouterX for $100. Both of these options are obviously much more expensive, but also more versatile.
Setting up your Mac as a router is not for everyone, but for some applications it's mighty convenient. It is sometimes is a bit of a beast to get set up, but if you keep at it (sometimes, sadly, just with trial and error) you can probably get it going. Feel free to share your own experiences in the comments.