bryan wann :: binary fury

I recently started using Ubiquiti’s EdgeRouter Lite to replace the routing functionality of my Airport Extreme at home and for some other networking projects. I really like this little box, $99 gets you a 3-port gigabit router and it can really churn through packets. It runs EdgeOS, which is based upon Debian along with the Vyatta routing stack.

EdgeOS provides a nice web UI as well as a cli shell for editing the configuration. If you’ve never used Vyatta, its configuration and cli look and feel are just like Juniper’s JunOS. Like JunOS, you can also get into a system OS bash shell.

I did some poking around and discovered that the router has 2 GB of flash storage (via USB thumb drive internally) and ordinary Debian packages can be installed. Further, once inside a bash shell there is a command-line interface to manipulate the running Vyatta router configuration. This made me wonder, can I run Chef on this thing to manage it?

Historically doing any sort of scripted management on a router has been a giant pain in the ass, either involving remote expect scripts or XML APIs, because there’s no way to run software directly on the router. Chef expects to run locally on a system and be able to run commands to converge the system to the desired state. If you try to bolt Chef onto a Cisco IOS router for example, you’d at the very least need some sort of proxy layer that takes Chef resources and translates those into IOS commands and runs them remotely. It’s hacky and ugly, I wouldn’t want to run it in production. With an EdgeRouter, its 2 GB of storage and Debian underpinnings, Chef can indeed run directly on the router eliminating the need for any sort of proxy layer!

The EdgeRouter has a Cavium 64-bit MIPS CPU so the standard omnibus Chef client packages won’t work because they’re intended for i686/x86_64. However Chef can be installed from RubyGems. This is the same way I install Chef on Raspberry Pis running Raspbian (which is ARM-based, a/k/a armv6l), and in fact the same knife bootstrap script I used for Raspbian worked for EdgeOS.

Bootstrapping takes a fresh EdgeOS router and installs Chef on it via ssh from another system. RubyGem packages are installed, Ohai and Chef client gems are installed, and the Chef client.rb and validation key are copied over. Once this is done, Chef can be ran directly on the router.

Bootstrapping EdgeOS

I’ve uploaded my knife bootstrap template to GitHub (https://github.com/bwann/chef/tree/master/knife). Drop this into your ~/.chef/bootstrap directory, then feed it to knife with the -d option. By default the username/password for EdgeOS is ‘ubnt‘ and ‘ubnt‘.

knife bootstrap -d debian7-edgeos-raspbian -x ubnt --sudo 192.168.1.20

Bootstrapping takes several minutes to run, with installing and updating gems taking 100% of a core for a while. Various things are fetched remotely from the internet with this template, such as Debian package updates and RubyGems.

Using Chef

I’ve only just started using Chef on EdgeOS and so far haven’t gotten to manipulating the Vyatta configuration. I imagine this will involve writing providers to handle running commands via shell. This is tricky because there are some config files (e.g. /etc/snmp/snmpd.conf, zebra) that are managed by EdgeOS and changes to them would be overwritten, therefore they’d have to be managed via Vyatta API. There’s documentation for the Vyatta shell API on the unofficial Vyatta wiki: http://www.vyattawiki.net/wiki/Cli-shell-api. Once I figure this out I’ll write up more about it.

In the meantime I’ve used chef to do things like this:

• Install packages, e.g. iftop
• Manage OpenSSL certificates for my private CA
• Install OpenTSDB tcollectors for system data collection, e.g. CPU/memory/interface counters

Example ohai output

Ohai reports the platform/platform_family as debian/debian, so there’s no clear distinction here that we’re running on a router. So far my way around this in recipes has been keying on node['os_version'].end_with('UBNT').

  "kernel": {
    "machine": "mips64",
    "name": "Linux",
    "os": "GNU/Linux",
    "release": "2.6.32.13-UBNT"
    "version": "#1 SMP Wed Oct 24 01:08:06 PDT 2012",
  },
  "os_version": "2.6.32.13-UBNT",
  "platform": "debian"
  "platform_family": "debian",
  "platform_version": "6.0.6",

Other EdgeRouter hacking

A few other people have been hacking on EdgeRouters, even getting FreeBSD running on them. I was most surprised by internal flash storage being just a removable USB thumb drive sitting in a socket on the board. It has a 2 GB stick in it, which should easily be replaceable by something larger if so desired. The U-Boot bootloader is over in another piece of flash on the board.

I found a handy wiki page over at http://www.crashcourse.ca/wiki/index.php/EdgeRouter_Lite which has some good information and links on the internals of the EdgeRouter.

OpenTSDB tcollectors

tcollectors work just fine running on the EdgeRouter. I’m using them to collect counters every 15 seconds and push to TSD. There doesn’t seem to be any noticeable CPU lag when the Chef client or tcollectors do their thing. All network interfaces show up to the kernel, so all three ethernet ports, VPN tunnels, and IPv4-IPv6 tunnels can be measured.

Here’s an example of charting bytes in/out (red, green, left axis) of my DSL connection and CPU idle (blue, right axis):

Tags: chef, ubnt