arstechnica.com: The Ars guide to building a Linux router from scratch

arstechnica.com: The Ars guide to building a Linux router from scratch

After finally reaching the tipping point with off-the-shelf solutions that can’t match increasing speeds available, we recently took the plunge. Building a homebrew router turned out to be a better proposition than we could’ve ever imagined. With nearly any speed metric we analyzed, our little DIY kit outpaced routers whether they were of the $90- or $250-variety.
Naturally, many readers asked the obvious follow-up—”How exactly can we put that together?” Today it’s time to finally pull back the curtain and offer that walkthrough. By taking a closer look at the actual build itself (hardware and software), the testing processes we used, and why we used them, hopefully any Ars readers of average technical abilities will be able to put together their own DIY speed machine. And the good news? Everything is as open source as it gets—the equipment, the processes, and the setup. If you want the DIY router we used, you can absolutely have it. This will be the guide to lead you, step-by-step.

What is a router, anyway?

At its most basic, a router is just a device that accepts packets on one interface and forwards them on to another interface that gets those packets closer to their eventual destination. That’s not what most of us are really thinking when we think of “a router” in the sense of something we’ll plug into our home or office to get to the Internet, though. What do we need to have before any homebrew device looks like a router?

For most of us, the important bits will be routing, NAT, DHCP, and DNS. In this build, we’ll use the same free and open source implementations of these technologies that power huge chunks of the Internet infrastructure itself.

Routing

This one’s easy—the Linux kernel itself handles it for us without any additional software. We’re not looking for any complex multiple-route stuff that would need Border Gateway Protocol or the like, so we can basically just enable forwarding between interfaces, set up a few rules to tell us when not to forward between interfaces, and call it a day.

Network Address Translation

FURTHER READING


IPV6 CELEBRATES ITS 20TH BIRTHDAY BY REACHING 10 PERCENT DEPLOYMENT
All I want for my birthday is a new IP header.
In a nutshell, NAT lets you access routeable (Internet) IP addresses from non-routeable (local, private) IP addresses. A router does this by accepting traffic from the LAN, substituting its own public IP address for the LAN IP address the packet came from, and sending the packet on to the Internet. When replies to the packet come back to the router, the router looks up which LAN IP address the original packet came from, puts that IP address back into the packet in place of its own, and forwards the packet back to the original machine.
We need NAT because we don’t have enough public IP addresses for every personal computer and device out there, not by a long shot. IPv6 will change all that, but when we make the final switchover to IPv6… well, your guess is as good as mine. We aren’t going to even try to cover IPv6 in this article; it’s a complicated matter. For now, we’ll note that we need NAT, the Linux kernel makes that available as the MASQUERADE function of iptables, and move on.

Dynamic Host Configuration Protocol

Yep, that’s what “DHCP” stands for. The DHCP server in a consumer router hands out IP addresses, default gateway information, network and broadcast addresses, and DNS server addresses. We want our router to do the same thing for us—which the Internet Software Consortium’s own reference DHCP service will do just fine.

Domain Name Service

You don’t strictly need to have local DNS service on your router. You could just tell all of your clients to use a publicly available DNS resolution service, like Verizon’s 4.2.2.4 or Google’s 8.8.8.8. Still, it’s certainly nice to have. We’re going to go old school here and use the same DNS server application that the Internet itself does: BIND, the Berkeley Internet Name Daemon. Having our own DNS server locally means that we’ll typically get DNS queries resolved faster, more accurately, and in strict accordance with caching and expiration protocols set in the domain names themselves when compared to using somebody else’s DNS server. It also means less total DNS traffic since we’re doing that caching locally.

Router hardware

As we disclosed last time, this decision was pretty simple: a dual-gigabit-NIC mini-PC from Alibaba and an inexpensive SSD from Newegg. (Update, 4/18: Though that vendor seems to have raised its prices on Alibaba, you can still get a basically identical mini-PC option through Amazon.)

Those particular choices sparked a lot of comments. Was this the absolute least expensive possible setup? No, definitely not. So why did I use the pieces I did? For one thing, I wasn’t actually sure where the homebrew would end up on the performance spectrum. I wanted to get an efficient but relatively powerful CPU… the kind you’d find in, say, an Intel-based Chromebook.

Looking for a high-performing CPU ruled out older Atom-based nettops, which are notoriously anemic for x86 CPUs. It didn’t necessarily rule out parts like PCEngines’ AMD Bobcat-based apu1d4, but that just wasn’t quite what I wanted to play with on the first go-round. The Ivy Bridge Celeron I got was exactly what I wanted—about as high performance as you can get while staying easy on the power bill.

There was one other personal “pro” about buying from an Alibaba vendor. My kids attend a Mandarin language immersion school, and it was kind of awesome discovering a handwritten note on the inside of the packaging, showing it to my kids, and telling them this was actually written by a person in China who was building the new thing Daddy got. (I’m told that the handwriting translates to “high performance computer with two networks,” but I don’t speak Chinese, and my kids’ teacher doesn’t speak nerd. There could be something lost in the translation.)


Enlarge / It’s a small world after all.
Jim Salter
With the other parts selected, let’s start with the 120GB SSD. That’s laughable overkill, right? Well, yes, it’s absolutely way more storage than you need for a router. But I wanted solid state both for fast boot times and for not being reliant on a spinning chunk of rust. I also wanted a brand I know, since really dodgy SSDs can be, well, really dodgy. That ruled out the $20 no-brand SSDs I could have found on Alibaba itself. And when I went shopping for SSDs from known brands, what I found was that dropping from 120GB to 32GB would only have saved me $5, if that. I went with the 120GB Kingston, and as a nice bonus, I now don’t have to worry about accumulated kernels from security updates filling up my root partition and screwing things up.

In theory, I could have used a cheap SD card instead of the SSD. But in practice, the Alibaba page didn’t specify whether the BIOS was capable of actually booting from SD or not, so I didn’t want to rely on it. (I did check when I got the router in hand, and yes, it will boot from SD just fine). SD cards also tend to be noticeably slower than SSDs. This doesn’t matter for the operation of the router, but it does matter for reboot times. (Plus, hey, I wanted to spoil myself! 120GB Kingston SSD it was.)

Since the Partaker mini-PC came complete except for the SSD, there really wasn’t much of a hardware “build” process on my end. Basically, all it took was plugging in one SATA and one SATA power cable. Whether that’s a feature or a bug depends on just how badly you want to feel like you turned a bunch of screws.

 Jim Salter
This image is titled “router-bad-SD-card-placement.jpg.”


The manufacturer provided four mounting holes for the storage on the removable cover plate of the router, but since I chose an SSD that weighs effectively nothing, I elected to ignore them. There’s enough space to just sort of nestle it over to the side of the motherboard, and that way I don’t have to worry about accidentally yanking on a cable when I remove the access plate for the router.
That’s really it to the build process on this one: remove PC from box, open PC, plug in SSD, close PC, call it a day. Call it 10 minutes, really.

Listing image by Jim Salter

Router operating system

This is where the process gets a little more fun. I wanted to use plain-Jane Ubuntu Server for my router since I work professionally with Ubuntu and am extremely familiar with it. On the minus side, that doesn’t come with any bells and whistles. On the plus side, that doesn’t come with any bells and whistles! If you like your infrastructure lean and mean (and you don’t feel itchy without a GUI) this is a pretty cool way to go. Even if you don’t like it that way and you want to force yourself to learn how to make iptables go, this’ll do it.

FURTHER READING


TEN YEARS OF UBUNTU: HOW LINUX’S BELOVED NEWCOMER BECAME ITS CRITICIZED KING
Ars looks back at the decade in Ubuntu, from Warty Warthog to 25 million users worldwide.
If you ultimately prefer to skip the hairy ins and outs of iptables, there are several perfectly cromulent options available to you for pre-baked router distributions with whiz-bang GUIs and all the work done for you. The most notable of these are the BSD-based pfSense and the Linux-based OpenWRT and/or DD-WRT.

There’s no need for a blow-by-blow walkthrough of the Ubuntu Server install, because it’s ludicrously simple. You go to the download page, you download an image (I chose, and recommend, the latest LTS release for its longer support), and you make a USB thumbdrive out of it. The actual install process is a next-fest. You boot, you install, you next your way through. You only have one disk, and the defaults are all fine. The only meaningful choices to make are your username and password and whether you want automatic security upgrades. (Spoiler: Yes, you do.)

Once that’s done, reboot, log in, and now you’re staring at a bash prompt.

Configuring network interfaces

The first thing to do is figure out which one of your network interfaces is which. The simplest way to do this is to go ahead and mark them (if they aren’t already marked) LAN and WAN, making sure you only have the WAN interface actually plugged in. Do an ifconfig from the bash prompt on your router and see which interface has an IP address—presto, that’s your WAN! Assuming you only have two interfaces (in my case, they were p1p1 and p4p1, although your mileage will very much differ with different hardware), you now also know which interface is your LAN interface. But if you’re still not sure, you can always change which port your network cable is plugged into and check for an address again.

After establishing which is which, you’ll configure these network interfaces with a quick sudo nano /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The WAN interface, marked Lan1 on the case
auto p4p1
iface p4p1 inet dhcp

# The LAN interface, marked Lan2 on the case
auto p1p1
iface p1p1 inet static
address 192.168.99.1
netmask 255.255.255.0
The loopback section will already be there, and likely you’ll have something for one or both of your actual NICs there as well. Once you’re done, you want something that looks like the above. Why do my comments say that the interfaces are marked “Lan1” and “Lan2”? That’s merely the way the case came on my particular box, and I hadn’t stickered over them yet. Comment appropriately for your own case, but please do comment and make sure your comments match what you can actually see.

Once you’re done, you’ll want to restart your router and make sure the changes actually worked.

Enabling forwarding in /etc/sysctl.conf

The first step to making Linux route is enabling forwarding between interfaces. This is really simple: sudo nano /etc/sysctl.conf and uncomment (remove the leading pound sign from) the line that says net.ipv4.ip_forward=1. Ctrl-X to exit, Y you want to save, and you’re outta there.

# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
# Enabling this option disables Stateless Address Autoconfiguration
# based on Router Advertisements for this host
#net.ipv6.conf.all.forwarding=1
The change you made will be live after a reboot, or you can speed things up by doing a quick sudo sysctl -p to force the changes you made to take effect immediately.

Creating and starting a skeleton ruleset

Now that your PC knows it’s supposed to be a router, the next step is making sure it’s choosy about what it forwards and when it does so. To do that, we want to build a firewall ruleset and make sure it’s applied before the network interfaces go live.

First, let’s sudo nano /etc/network/if-pre-up.d/iptables to create a startup script and populate it:

#!/bin/sh
/sbin/iptables-restore &1 | tee ~/tests/$1/$1-10K-ab-t180-c10-client-on-LAN.txt
ab -rt180 -c100 -s 240 http://192.168.99.100/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c100-client-on-LAN.txt
ab -rt180 -c1000 -s 240 http://192.168.99.100/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c1000-client-on-LAN.txt
ab -rt180 -c10000 -s 240 http://192.168.99.100/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c10000-client-on-LAN.txt

ab -rt180 -c10 -s 240 http://192.168.99.100/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c10-client-on-LAN.txt
ab -rt180 -c100 -s 240 http://192.168.99.100/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c100-client-on-LAN.txt
ab -rt180 -c1000 -s 240 http://192.168.99.100/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c1000-client-on-LAN.txt
ab -rt180 -c10000 -s 240 http://192.168.99.100/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c10000-client-on-LAN.txt

ab -rt180 -c10 -s 240 http://192.168.99.100/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c10-client-on-LAN.txt
ab -rt180 -c100 -s 240 http://192.168.99.100/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c100-client-on-LAN.txt
ab -rt180 -c1000 -s 240 http://192.168.99.100/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c1000-client-on-LAN.txt
ab -rt180 -c10000 -s 240 http://192.168.99.100/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c10000-client-on-LAN.txt
Make sure you have appropriately sized JPEGs at /var/www/10K.jpg, /var/www/100K.jpg, and /var/www/1M.jpg, and you’re ready to go. If you saved the script as test.sh, you invoke it as ./test.sh routername and give it a while. Some 48 minutes later, you’ll have a bunch of raw apachebench results in appropriately named files under ~/tests/routername, which will include the throughput numbers we looked at in last month’s article along with a fair amount more information that tells you more about the webserver than it does about the network the transactions took place on. (Obviously, the same script with a different IP address and with filenames ending in “-client-on-WAN” was used for the download-side of the testing.)
This is a fairly rough-and-ready test, of course. The underlying metrics that we’re really pushing are the maximum packets per second (PPS), the total network bandwidth available, and the maximum number of entries in the router’s NAT table. I kinda like my rough-and-ready test, though, since it gives me a fairly direct set of real-world numbers rather than arbitrary synthetics. Downloading data over HTTP is something I’ll really be doing a lot of. For example, if a bunch of TCP connections are being arbitrarily reset, that will show up in these benchmarks, where they might not in a more “advanced” suite that more directly tests pure, raw PPS.

Still more to come

If a raw iptables setup isn’t to your liking or even if you just don’t like my choice of hardware, that’s OK. In the next story in this router series, we’ll test some of the most popular items from the comments: a brand-new Mikrotik hAP_ac router (with wired performance better than the wired-only routerboard 450), a PCEngines apu14d, a Ubiquiti EdgeRouter Pro, and a Ubiquiti USG (performance like an EdgeRouter Lite, but this ties into the easy-mode Unifi management interface). We’ll also take a look at running pfSense and OpenWRT on the homebrew and see how (or if) they impact its throughput.

Jim Salter (@jrssnet) is an author, public speaker, small business owner, mercenary sysadmin, and father of three—not necessarily in that order. He got his first real taste of open source by running Apache on his very own dedicated FreeBSD 3.1 server back in 1999, and he’s been a fierce advocate of FOSS ever since. He also created and maintains the Sanoid platform.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s