Feed on
Posts
Comments

Operations reading list

I love books. These days I buy most of my books for Kindle, but I still buy paper for books I really like and want to keep around. Tech books are notorious for being obsolete a couple of years after printing, but there are still several timeless books I use for reference and would recommend for anyone in UNIX/Linux systems engineering or networking, new or jaded veteran. Some are older than others, but here’s a few that have served me well:

Systems:

If you deal with the internet you must have a very solid understanding of the protocols involved, from ARP to TCP. By the time you’ve been in the industry for several years, you’ll encounter problems with every part of the stack covered by this book, along with lower levels such as Ethernet. tcpdump and other packet sniffers will be your best friend and you should use them liberally. My first edition of this book only covers IPv4 but the second edition covers IPv6 now, which you should be using!

(a/k/a “APUE”) The Internet is built on UNIX and C. This is more of a reference book rather than one you’d sit down and read, but I enjoy reading random bits when I’m curious or want more background on something. The book covers a lot detail of how the UNIX userland environment works with the kernel, giving snippets of C code to show exactly how something like syscalls are implemented under the hood. Ever ran strace and wondered what open(), write(), mkdir(), bind(), connect(), fork(), SIGUSR1 are? This book will show you in simple C code what’s going on.

Three recent additions this past year:

APUE was geared at a general System V / BSD UNIX audience. This book is very similar to APUE, but geared toward a Linux audience. It goes into the same level of detail and explaining things in C code as APUE. It’s a huge book coming in at 1,500+ pages so make room on your bookshelf for it.

Brendan has given many talks and authored several pieces on systems performance, benchmarking, and really digging in deep to troubleshooting bottlenecks. He authored DTrace and if you’ve ever seen the interesting “guy screaming at hard drives” (which shows effect of vibration on disk latency) video on YouTube, that’s him. You can’t change something if you can’t measure it, and this book explains how to get valid data to analyze performance of applications, CPU, memory, disks, kernels and networking. It also covers applications in a cloud environment and gives good insight on how virtualized kernels or system calls can impact performance.

In particular, I really like this book because it covers things from both a Linux and Solaris kernel perspective. I’ve used both over my career and while my Solaris is rusty this gives useful comparisons to get me through problems. I’ve heard Brendan speak a couple of times and his slides (and talk) from SCaLE 11x on Linux Performance Analysis are a great read. There are some very useful illustrations that show which tool to use for the job, e.g. in troubleshooting an issue do I use perf? iostat? sar? tcpdump? netstat? nicstat? strace?

I first ran across Sherri Davidoff by listening to her talk at DEF CON 21 about the do-it-yourself cellular sniffer^W IDS and later found her book. Most systems people are blissfully ignorant beyond the Ethernet interface of their servers. This doesn’t cut it anymore in a land of distributed systems, so you need to understand how to troubleshoot issues on the network too. This book is primarily written for doing forensic analysis and gathering evidence of events for an investigation, but there are still a lot of parallels in troubleshooting a production environment. Some of the same techniques for carefully collecting evidence and gathering logs are fantastic for writing up a root cause analysis, so some bad thing doesn’t happen again[tm].

I like this book because it covers traffic and packet analysis, a TL;DR of network protocols in real life, and the various network devices that data can flow through. This is the only practical book I’ve read that explains why you’d want to do flow analysis (e.g. NetFlow, sflow) to detect problems or see application activity, along with examples of using nfdump/NfSen. It covers intrusion detection, snort, switches, routers, firewalls, logging, tunneling, all good stuff.

Networks:

In a previous life I was dedicated to network engineering in a managed hosting environment for a few years with lots of snowflake customers. I touched a wide swath of different types of gear from multiple vendors, hardware load balancers, VPNs, firewalls, L2/L3 switches, routers, huge L2 domains with hundreds of VLANs. Enough to do the job, but not a master at any. I caused my share of outages with spanning tree before I got a real grasp of what was going on. These books are a bit dated since Cisco and IOS isn’t as dominate as it once was (thank god), but they still have useful network stuff that transfers to other platforms.

My go-to book for Cisco firewalls back in the day. I dealt a lot with all three platforms and it was often quicker to just grab this book than dig around on Cisco’s website for configuration examples. My book is all marked up with notes and bookmarks for packet processing order of operations, NAT and SNAT configuration, failover pairs, and logging. It was good because it usually gave the equivalent PIX, ASA, and FWSM (Cat 6500 Firewall Service Module) commands together when explaining how to configure something.

Oddly absent from this book was a treatment of VPNs, there’s barely any mention of IPsec. I have the companion book “The Complete Cisco VPN Configuration Guide” but was disappointed at its coverage of IPsec and SSL/DTLS VPNs, especially when it came to troubleshooting on firewalls. A good hunk of the book is centered around the Cisco VPN 3000 Concentrator which is way obsolete now.

This was my savior in learning the guts of layer 2 Ethernet and spanning tree in its various flavors. STP, PVST+, Rapid STP, MST, BPDUs, STP roots, enough trees to make a forest. Then there’s VLANs, VLAN trunking, 802.1q tagging, Q-in-Q, private VLANs and multicast. Then it goes into covering CatOS and IOS on the beloved, trusty workhorse of the 2000s, the Catalyst 6500 series of switches. I never did get that CCNP.

  • Designing Content Switching Solutions, by Naseh and Kahn

This book is positively dated now, but if you find yourself still managing an ancient Cisco load balancer (e.g. CSS 11501, CSM for 6500, or firewall load balancing), this is your book. Beyond this it gets into HTTP/RTSP/streaming/layer 7 load balancing, SSL offloading and global load balancing. Now that I think about it, don’t buy this book. Offloading SSL to a hardware load balancer is a terrible thing you don’t want to do. Your farm of Intel Xeons can handle the crypto overhead much better than a puny RISC processor from 2001. The world is much better now and standard Linux servers are the new load balancer.

It’s a classic that practically everyone in the 1990s learned BGP from. Heck, it even includes a CIDR conversion table in the front flap and explains what NAPS were. Nevertheless, it explains various scenarios and topologies where you’d use BGP internally and externally, and how the protocol behaves to control routes. The world has moved to running MPLS within the backbone, but BGP is still alive and kicking on the edges. In fact at work we use BGP right down to the rack switch and inject VIPs onto the network via BGP.

Notable mentions:

Sometimes I just want to read a book with Kindle on an airplane or at breakfast, because I’m that kind of guy.

I hate and love Kerberos, mainly because I was clueless and tossed into the deep end to support it. I want to love it more because distributed authentication and authorization are super useful in a large systems environment and I don’t know how I’d live without it now, so I bought this to read. So far it doesn’t disappoint in how to setup Kerberos realms, KDCs, slaves, and all that fun stuff.

I don’t put on my DBA hat very often and usually touch MySQL seldom enough I have to go remember how to set up replication. If I supported it again, this would probably be the book I’d be reading.

With EdgeOS 1.6 for the EdgeRouter line, Ubiquiti upgraded the Debian distribution from squeeze to wheezy. Along with a more modern 3.10 kernel this gets us a newer version of Ruby too, 1.9.1. During upgrades the system is blown away so I lost my Chef client. I had problems re-bootstrapping my routers until I finally realized that /etc/apt/sources.list still pointed at squeeze repos. I asked Ubiquiti about this and they say it’s intentional that you have to update sources.list to fetch from the new repo.

How to fail
Stepping back in time before I figured this out, this is what transpired.

When I would try to build gems things went sideways; the running system was wheezy but it was trying to install packages from the squeeze distribution. As such there were a lot of version conflicts and packages just refused to install. For the record, these are the sort of errors I was running into with a repo mismatch (mainly around libc6 and libc6-dev when trying to install ruby):

root@gw2:/home/ubnt# apt-get install ruby ruby-dev git ruby1.8-dev
...
The following packages have unmet dependencies:
 ruby1.8-dev : Depends: libc6-dev but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
root@gw2:/home/ubnt#

Trying to install libc6-dev fails:

root@gw2:/home/ubnt# apt-get install libc6-dev
The following packages have unmet dependencies:
 libc6-dev : Depends: libc6 (= 2.11.3-4) but 2.13-38+deb7u6 is to be installed
             Depends: libc-dev-bin (= 2.11.3-4) but it is not going to be installed
             Recommends: gcc but it is not going to be installed or
                         c-compiler
E: Unable to correct problems, you have held broken packages.
root@gw2:/home/ubnt#

You can pretend the problem doesn’t exist and ignore the problem by installing ruby 1.8 without the -dev package, but this will blow up on you later when you try to build gems such as ohai:

root@gw2:/tmp/rubygems-2.4.1# gem install ohai --no-rdoc --no-ri --verbose
...
/usr/lib/ruby/gems/1.8/gems/ffi-1.9.6/spec/ffi/variadic_spec.rb
/usr/lib/ruby/gems/1.8/gems/ffi-1.9.6/spec/spec.opts
Building native extensions.  This could take a while...
/usr/bin/ruby1.8 -r ./siteconf20141213-14984-bl1gm-0.rb extconf.rb
extconf.rb:4:in `require': no such file to load -- mkmf (LoadError)
	from extconf.rb:4
ERROR:  Error installing ohai:
	ERROR: Failed to build gem native extension.

    Building has failed. See above output for more information on the failure.
extconf failed, exit code 1

Gem files will remain installed in /usr/lib/ruby/gems/1.8/gems/ffi-1.9.6 for inspection.
Results logged to /usr/lib/ruby/gems/1.8/extensions/mips-linux/1.8/ffi-1.9.6/gem_make.out
root@gw2:/tmp/rubygems-2.4.1#

Aaaaand this fails because mkmf (Ruby MakeMakefile module) is provided by the ruby-dev package we couldn’t install earlier.

root@gw1:/home/ubnt# dpkg-query -L ruby1.9.1-dev | grep mkmf
/usr/lib/ruby/1.9.1/mkmf.rb
root@gw1:/home/ubnt#

So the lesson here is to make sure you’re fetching package from the correct repo. If you’ve found yourself in this situation, you’ll want to back things out and install the correct versions. First thing you want to do is dpkg --purge ruby1.8 libruby1.8 remove ruby 1.8. Then fix your apt sources and start all over again.

Chef/Ruby version caveat
One thing worth mentioning here is that you won’t be able to run the latest hotness, Chef client 12. The wheezy distro only has ruby 1.9.1, and Chef 12 requires ruby 2.0. The best I’ve been able to install from rubygems is Ohai v7.4.0 and Chef client 11.16.4.

gem install ohai --no-rdoc --no-ri --verbose -v 7.4.0
gem install chef --no-rdoc --no-ri --verbose -v 11.16.4

In honor of xkcd 979, I’m posting this so future generations of Courier-IMAP users won’t have to Bing for a solution in vain (and hit lots of useless advice). In the process of finally getting around to upgrading my 2008-era courier-imap 4.1.1 setup to the shiny new 4.15 hotness and putting things in Chef templates, I encountered this error in /var/log/maillog:

imapd-ssl: couriertls: /etc/pki/tls/private/blah-certkey.pem: error:0906D06C:PEM routines:PEM_read_bio:no start line

My certificate file has three things in it, my SSL certificate, the intermediate CA certificate, and the private key. After making sure I didn’t have wonky ^M, line feeds or malformed certificate START/END headers, I started bisecting the old config with my new template. I discovered I was missing the dhparams parameter configuration which is new in 4.15:

TLS_DHPARAMS="/usr/lib/courier-imap/share/dhparams.pem"

This file is generated by the courier-imap-mkdhparams cronjob. I read the release notes before upgrading but clearly forgot to check for this after upgrading. Added this to my template, now Courier IMAP is a happy camper.

I recently started using Ubiquiti’s EdgeRouter Lite to replace the routing functionality of my Airport Extreme at home and for some other networking projects. I really like this little box, $99 gets you a 3-port gigabit router and it can really churn through packets. It runs EdgeOS, which is based upon Debian along with the Vyatta routing stack.

EdgeOS provides a nice web UI as well as a cli shell for editing the configuration. If you’ve never used Vyatta, its configuration and cli look and feel are just like Juniper’s JunOS. Like JunOS, you can also get into a system OS bash shell.

I did some poking around and discovered that the router has 2 GB of flash storage (via USB thumb drive internally) and ordinary Debian packages can be installed. Further, once inside a bash shell there is a command-line interface to manipulate the running Vyatta router configuration. This made me wonder, can I run Chef on this thing to manage it?

Historically doing any sort of scripted management on a router has been a giant pain in the ass, either involving remote expect scripts or XML APIs, because there’s no way to run software directly on the router. Chef expects to run locally on a system and be able to run commands to converge the system to the desired state. If you try to bolt Chef onto a Cisco IOS router for example, you’d at the very least need some sort of proxy layer that takes Chef resources and translates those into IOS commands and runs them remotely. It’s hacky and ugly, I wouldn’t want to run it in production. With an EdgeRouter, its 2 GB of storage and Debian underpinnings, Chef can indeed run directly on the router eliminating the need for any sort of proxy layer!

The EdgeRouter has a Cavium 64-bit MIPS CPU so the standard omnibus Chef client packages won’t work because they’re intended for i686/x86_64. However Chef can be installed from RubyGems. This is the same way I install Chef on Raspberry Pis running Raspbian (which is ARM-based, a/k/a armv6l), and in fact the same knife bootstrap script I used for Raspbian worked for EdgeOS.

Bootstrapping takes a fresh EdgeOS router and installs Chef on it via ssh from another system. RubyGem packages are installed, Ohai and Chef client gems are installed, and the Chef client.rb and validation key are copied over. Once this is done, Chef can be ran directly on the router.

Bootstrapping EdgeOS

I’ve uploaded my knife bootstrap template to GitHub (https://github.com/bwann/chef/tree/master/knife). Drop this into your ~/.chef/bootstrap directory, then feed it to knife with the -d option. By default the username/password for EdgeOS is ‘ubnt‘ and ‘ubnt‘.

knife bootstrap -d debian7-edgeos-raspbian -x ubnt --sudo 192.168.1.20

Bootstrapping takes several minutes to run, with installing and updating gems taking 100% of a core for a while. Various things are fetched remotely from the internet with this template, such as Debian package updates and RubyGems.

Using Chef

I’ve only just started using Chef on EdgeOS and so far haven’t gotten to manipulating the Vyatta configuration. I imagine this will involve writing providers to handle running commands via shell. This is tricky because there are some config files (e.g. /etc/snmp/snmpd.conf, zebra) that are managed by EdgeOS and changes to them would be overwritten, therefore they’d have to be managed via Vyatta API. There’s documentation for the Vyatta shell API on the unofficial Vyatta wiki: http://www.vyattawiki.net/wiki/Cli-shell-api. Once I figure this out I’ll write up more about it.

In the meantime I’ve used chef to do things like this:

  • Install packages, e.g. iftop
  • Manage OpenSSL certificates for my private CA
  • Install OpenTSDB tcollectors for system data collection, e.g. CPU/memory/interface counters

Example ohai output

Ohai reports the platform/platform_family as debian/debian, so there’s no clear distinction here that we’re running on a router. So far my way around this in recipes has been keying on node['os_version'].end_with('UBNT').

  "kernel": {
    "machine": "mips64",
    "name": "Linux",
    "os": "GNU/Linux",
    "release": "2.6.32.13-UBNT"
    "version": "#1 SMP Wed Oct 24 01:08:06 PDT 2012",
  },
  "os_version": "2.6.32.13-UBNT",
  "platform": "debian"
  "platform_family": "debian",
  "platform_version": "6.0.6",

Other EdgeRouter hacking

A few other people have been hacking on EdgeRouters, even getting FreeBSD running on them. I was most surprised by internal flash storage being just a removable USB thumb drive sitting in a socket on the board. It has a 2 GB stick in it, which should easily be replaceable by something larger if so desired. The U-Boot bootloader is over in another piece of flash on the board.

I found a handy wiki page over at http://www.crashcourse.ca/wiki/index.php/EdgeRouter_Lite which has some good information and links on the internals of the EdgeRouter.

OpenTSDB tcollectors

tcollectors work just fine running on the EdgeRouter. I’m using them to collect counters every 15 seconds and push to TSD. There doesn’t seem to be any noticeable CPU lag when the Chef client or tcollectors do their thing. All network interfaces show up to the kernel, so all three ethernet ports, VPN tunnels, and IPv4-IPv6 tunnels can be measured.

Here’s an example of charting bytes in/out (red, green, left axis) of my DSL connection and CPU idle (blue, right axis):

Balcony plant waterer

I finally reached a critical mass of various plants, flowers, and succulents on my apartment balcony that required multiple trips to the sink when I needed to water them. While browsing the hardware store I was eyeing the garden drip irrigation systems and thought why not? Originally I wanted to use some sort of a jug which I could fill with water from the bathtub and then gravity feed out to my plants. I didn’t find anything off-the-shelf I could use and shelved the idea. Later I was at Harbor Freight and picked up a cheap 12V submersible pump with the intent to make a little fountain for the garden. I apparently got a defective panel because the one included didn’t do anything when I plugged in the pump. Fortunately I was going to hook the pump to my existing balcony setup anyways, so no loss.

Eventually the idea came to me to use a Tidy Cat litter jug and this submersible pump to water my flowers. I go through litter every few weeks and the jugs are sturdy plastic with a hinged lid and handle, and designed to hold 38 pounds of clay, so perfect for lugging water outside. I took the pump to the hardware store and found that its outlet slipped perfectly into 1/2″ garden sprinkler tubing. I had no idea how I’d wind up using it so I settled on using a 1/2″ tubing to connect the pump to a tee up outside of the jug. From there it split off to feed a pair of manifolds, which broke out into 1/4″ connectors for feed lines. When I want to refill the jugs, I lift the tube out and take the pump out, leaving it behind.

Each flowerpot has a 1 gallon per hour drip fitting and the planters have 2-3 drip fittings. There’s plenty of pressure to push water along, even with 1/4″ tubing going upwards. However, it’s not automated at all at this point. I still have to plug in the motor for several minutes to water the plants, which still beats going back and forth to the sink. I do have an OpenSprinkler controller I bought at Maker Faire that I’ll likely try next. This will let me automate watering and maybe not have to think about it. My plants have shown an amazing comeback after getting good watering!

I’ve been late to the game of nerding out on the telephone network and phreaking in general and it’s been fun browsing over old maps. I was browsing the “Central section” microwave map on the AT&T Long Lines Places and Routes website and was surprised to see my home town, Kinta, on the map. I grew up with a tall microwave tower on top of what we know as Beaver Mountain a few miles north of Kinta. I sort of knew it was owned by AT&T (now SBC, now at&t), but I had no idea until now it was part of the fabled AT&T Long Lines division. I also frequently passed the tower at Krebs on the way to McAlester, as it was just off highway 31. From looking at the larger map, it appears both were part of a microwave relay link between St Louis, MO and Fort Worth, TX.

The microwave horns and what not were stripped some time ago on the tower on Beaver Mountain, it’s just sort of standing there now. The tower at Krebs was also stripped of it’s microwave kit and was replaced by various random antenna.

The Internet seems pretty sparse with information about either of these two towers, I’m curious to know if there’s more about them anywhere. The map I found was dated October 1966, so presumably they were in service then, but I have no idea when they were decommissioned. They’d make awesome observation decks!

Edit (Feb 2018): According to FCC Antenna Structure Registration (#1010196), the tower now belongs to “Communication Enhancement”, as site #20031. It stands at 263′ tall.

Over a recent couple of weekends I built a thing to measure the current and voltage of the components of a 12V solar/PV power system. Ever since I started using solar power at Burning Man I wanted to get a good measure of how much I was consuming and how much was left. At the time I settled on a Morningstar charge controller with LCD voltage meters to tell me.

This new setup adds data recording+logging into the mix, something I’ve always wanted. Here, the measurement modules are attached to a Raspberry Pi via i2c and a simple python script collects the data and sends it wirelessly to OpenTSDB for storage and charting.

The hardware and other bits, excluding the Pi, ran me around $50-$60. Allegedly this setup should handle about 3 amps, although if somebody wants to do the arithmetic and make an appropriate shunt they should be able to measure much higher currents.

At the heart of the setup are three INA219 (“high side DC current sensor”) modules from Adafruit which do the measuring, one each for the solar input, the battery, and load. They’re about 1″ square, have a terminal on the board for wiring in-line in electrical circuits to measure, and have a small shunt and chip that measures the voltage and current. The bulk of my time was spent making a wiring harness for this setup to plug into my existing solar setup and soldering on all the connectors. I personally insist on everything I have use Power Pole connectors so if one were less inclined they could avoid a lot of work. Working with the rest was relatively quick work. Adafruit has some excellent tutorials on wiring up and working with the INA219, which is where I got the majority of my information.

Each of the INA219 modules speak i2c to the RPi and the SDA/SCL pins daisy-chained with continuous jumpers back to the respective i2c pins on the RPi (P3 and P5) to form an i2c bus. Two have solder junctions to set the i2c address, resulting in the three being accessible on the bus at 0x40, 0x41, and 0x44. Also to power the INA modules themselves, another set of continuous jumpers connect Vcc and GND back to the 5v and GND pins of the Raspberry Pi. Adafruit has a generous tutorial on how to connect and verify i2c is working. For the most part it was painless and simple, enable the i2c kernel module and the three INA modules just showed up when I ran “i2cdetect -y 1”.

The few software libraries for the INA219 and i2c assume use with an Arduino, but some dude over on Github did a nice Python port of the INA219/i2c library. However, there looks to have been a bug in the original Arduino version regarding two’s complements and the code on Github needs a slight patch to properly measure negative voltages. (note to self, submit a patch). Once that’s sorted out, it’s just simple Python method calls to get instant voltage and current information.

A quick example to query the three INA219 modules looks like:

#!/usr/bin/python

from Subfact_ina219 import INA219

for i2caddr in ['0x40', '0x41', '0x44']:
    ina = INA219(address=int(i2caddr,16))

    print "== %s " % i2caddr
    print "Bus voltage: %.3f V" % ina.getBusVoltage_V()
    print "Shunt voltage: %.3f mV" % ina.getShuntVoltage_mV()
    print "Current: %.3f mA" % ina.getCurrent_mA()
    print "Voltage: %.3f V " % ( ina.getBusVoltage_V() +
        ina.getShuntVoltage_mV() / 1000 )

To get the voltage of the thing you’re measuring, you must add the bus voltage and shunt voltage to get all the components of the measurement. Also, you must wire grounds of the battery, solar panel, load, and the Raspberry Pi together to be able to measure individual voltages. (Something I forgot to realize for a long time.) Otherwise you only get electrical current data, voltage is indeterminate.

root@solar:/home/pi# python power.py
== 0x40
Bus voltage:   12.420 V
Shunt voltage: -37.270 mV
Current:       -373.000 mA
Voltage:       12.383 V
== 0x41
Bus voltage:   12.548 V
Shunt voltage: -37.160 mV
Current:       -372.000 mA
Voltage:       12.511 V
== 0x44
Bus voltage:   12.416 V
Shunt voltage: 1.000 mV
Current:       10.000 mA
Voltage:       12.417 V

Here, module 0x40 is represents the load, 0x41 is the battery, and 0x44 are the solar panel. There’s a string of 12V LED lights hooked up right now, drawing 373 milliamps. Likewise there’s a 373 mA draw on the battery. The solar panel is in shade so it’s only contributing 10 mA right now. The negative values are a handy indicator of the direction of the current, i.e. when the battery is charging the values are positive, when it’s being discharged the values are negative.

For the small solar setup, I’m using a solar panel and charge controller from Harbor Freight. Their 7A charge controller features a low voltage disconnect (something I required for this setup), is compact and pretty inexpensive. It fits tidily into the plastic container which temporarily houses the RPi and modules. This will work with any charge controller. Another container holds a small 12V battery. [Edit: I have my doubts about the LVD feature of this charge controller after it apparently let my battery get down to 6 VDC under load overnight. >:( ]

Note: at the time the Raspberry Pi is not solar/battery powered, nor can it switch on/off any of the loads. I have an USB power cord running inside to power the setup. It should be easy to rig up a voltage converter, but that’s a project for another time. This would make it a real standalone unit and be quite handy and adapted into other real applications.

For the data storage, I wrote a Python tcollector script for OpenTSDB which polls the INA modules every 15 seconds and records the values to a database. Using the built-in plotting tools, I can quickly visualize current or historic solar production or power consumption. The shameless cut-n-paste tcollector I wrote is over on my Github page.

The script prints a ‘ina219’-prefixed metric name, timestamp, tags and data suitable for use in a time series database:

ina219.shunt_voltage 1396837555 0.050 address=0x40 name=load
ina219.voltage 1396837555 12.672 address=0x40 name=load
ina219.bus_voltage 1396837555 12.672 address=0x40 name=load
ina219.current 1396837555 0.000 address=0x40 name=load
ina219.shunt_voltage 1396837555 -0.690 address=0x41 name=battery
ina219.voltage 1396837555 12.671 address=0x41 name=battery
ina219.bus_voltage 1396837555 12.672 address=0x41 name=battery
ina219.current 1396837555 -7.000 address=0x41 name=battery
ina219.shunt_voltage 1396837555 0.130 address=0x44 name=solar
ina219.voltage 1396837555 12.672 address=0x44 name=solar
ina219.bus_voltage 1396837555 12.672 address=0x44 name=solar
ina219.current 1396837555 1.000 address=0x44 name=solar

Two-hour plot of battery voltage while connected to load of lamps for an hour:

Eventually I’d like to have a battery charge monitor, but this involves getting the temperature of the battery to take into account.

Yes! Web-scale home electricity tracking, because why not!

For the longest time I’ve been using the Kill-A-Watt meter to measure home power usage, especially all of my computer gear. The problem is, you only get a LCD readout, no way to store that info over time. A friend told me how he was using Insteon (basically X10 home automation that doesn’t suck) kit at home and they had an iMeter device that would collect power usage which you could download via an Ethernet-based Insteon SmartLinc controller. Each iMeter cost about $35 and the 2412N controller was about $90. Spendy if you want to measure just one thing, but it’s certainly way cheaper than any of the alternatives. I bought a set to put on my desktop computers and IKEA cluster.

Insteon iMeter and 2412N controller

The Insteon controller has a web server built into it with a basic but functional interface; same for the iPhone app. The client lets you control various Insteon devices or display the watt reading from a single iMeter. Unfortunately the 2412N controller I have doesn’t store values it collects. After examining the JavaScript the web UI uses and some tcpdumps, I was able to pick apart the commands the client makes over HTTP to get data from the meters. Each Insteon device has a six-digit hex ID such as 19.84.17 written on the back, which in the case of multiple iMeters is used to reference a particular one.

A request is made to the controller with a GET request with a request string like /3?02621984170F8200=I=3, then another request for /buffstatus.xml fetches the result of the request. In this case:

  • 0262 is basically “here comes a raw command”
  • 198417 is the ID of the iMeter
  • 82 is “get iMeter status”
  • 00 is a do not care byte, and the rest is trailing info.

Given the Insteon IDs of each iMeter, it’s possible to fetch the wattage readings from them via HTTP. Here is an Insteon developer PDF that describe the various request and response fields.

A command-line example:

[bwann@lapdance ~]$ GET http://smartlinc01/3?02621984170F8200=I=3 ; GET http://smartlinc01/buffstatus.xml
<response><BS>02621988140F82000602501988141E50C42B820002511988141E50C41B820000D201050180009A00097D207DD5</BS></response>

The response is a long string of text fields that contain various things as the remote device id, command sent, low+high bytes for counters of current and total watts used.

I incorporated this into a small python script to handle all the munging and parsing for me. (‘GET‘, by the way, along with ‘POST‘ and ‘HEAD‘ are part of the perl-libwww-perl package and is awesome when you want to quickly run a particular HTTP command without typing it out by hand in telnet).

For the TL;DR of the python code, head over to GitHub where I’ve implemented iMeter collection into a database collection script: smartlinc_imeter.py

Great, now how to collect and graph it?

OpenTSDB

All this year I’ve been playing with the OpenTSDB time series database (which you should really check out) at home to gather server statistics and really love how trivial it makes adding and collecting new metrics. Come up with a string of value+timestamp+counter and optional key=value tags, shoot it over to a TCP socket, and you’re done. Plug in the same metric name into the TSD web interface, and boom, there’s your charts. Way, way, way better than the old song and dance of RRD and whipping up the ubiquitous not-so-simple shell script to store+graph just one damn metric. (Okay, it is a bit more complicated to get started with than that in that now you’re running HBase as the backend, but it’s designed to handle billions of data points a day, it’s easy to get started with and totally worth it. Very fast means it’s a pleasure to work with your datasets. The awesomeness of OpenTSDB is a whole ‘nother post.)

OpenTSDB uses a set of simple tcollector scripts that run on systems and upload various data such as cpu/disk/io usage, and whatnot every 15 seconds. The set of tcollector scripts that come with OpenTSDB on GitHub are actually very well written and simple. Because you can send data to TSD over a TCP request it makes collectors very easy to implement. Even one line shell scripts such as “echo blah | nc 4242” work just fine. It lends itself to rapidly letting you invent new collectors to get new metrics you’re interested in. If you’ve ever used Nagios or MRTG, you’ve surely experienced plug-in hell where somebody has gobbed together 50 lines of perl to ultimately run a single command with system() to get one value. Another nice benefit of the tcollector setup, it automatically re-starts a collector script if one dies or you add a new one, something you normally have to deal with on your own.

I wrote a tcollector script in python that would query the Insteon SmartLinc controller every 15 seconds and request data on the various iMeters in my apartment, parse the results, and then upload the values to TSD. I have two metric names insteon.imeter.watt and insteon.imeter.total, tagged with the device ID. Using this I can now generate charts that refresh every 15 seconds to show me the latest power usage! If I ever add another iMeter or other controllers, I just drop in one line into a config file with the new device ID/IP and it automatically picks it up and the data are immediately available in the chart UI.

[root@basic07 0]# python ./smartlinc_imeter.py
insteon.imeter.watt 1382596852 210 controller=smartlinc01.wann.net device=198687
insteon.imeter.total 1382596852 5464099 controller=smartlinc01.wann.net device=198687
insteon.imeter.watt 1382596852 206 controller=smartlinc01.wann.net device=198814
insteon.imeter.total 1382596852 550299 controller=smartlinc01.wann.net device=198814

The tcollector script is over on Github, and the comments include my notes on how to parse the Insteon commands and requests.

Bad luck Bryan

Takes two weeks off from work, plans to go to Death Valley and maybe Yosemite.

Government shuts down all national parks, barricades roads, evicts campers two days in.

Death Valley

I decided to go to Death Valley for a week to unplug and stayed at the Furnace Creek Ranch. People always wonder why I go out there instead of head off to the beach, and while I love laying on the beach too, sometimes it’s nice to go where it’s dead quiet. In some ways it’s sort of like trekking to the middle of nowhere to Burning Man, except in DV you have the option of staying at hotels. (Las Vegas is 2 hours from DV) Having said it’s quiet, right now is actually the biggest tourist season of the year since the weather is mild, but DV is a huge place and everyone returns to camp/hotel/RV at night. At places like FCR there’s plenty of casual social gathering around the fire pits in the evening, if you want company you can find it. Otherwise, if you’re a night owl like I am and have a vehicle, you have complete free reign of the place after the sun sets. Venture just out of town, you will not see another soul or vehicle until morning.

Nighttime is a big part of what makes this place so amazing. DV just recently got some special blah blah acknowledgements of being one of the darkest places in the northern hemisphere and I believe it. Bring binoculars, find a remote parking lot, lay down a blanket, and be amazed at the stars for hours. They’re very numerous and sharp, like lightbulbs on strings. The only thing you hear is the wind, the tinnitus in your ears, and maaaaybe some coyotes off in the distance. It doesn’t suck!

For an interesting experience I recommend driving out to Badwater (the lowest point in the basin, and North America for that matter) at night in the dead of summer. Even at midnight it’s still 111 F, virtually no wind/breeze, pitch black (save for stars/moon), dead silent, and nothing’s around for 20 miles. I think of it as the dry walk-in equivalent of a sensory deprivation tank as you walk around and are hugged by the warmness.

My DV travel tips, for going to/from the SF Bay:

  • Never been to Yosemite? Cut over the Sierras on highways 108 or 120, catch highway 395 south.
  • Going through Bakersfield and then over? Ignore your GPS telling you to take highway 178 (especially on the way back), it sucks. It’s a windy canyon highway and you’ll be stuck behind somebody riding the brakes all the way down at 45 mph.
  • Avoid shortcutting using the Trona/Wildrose road, it sucks. The road is not maintained well and patched all to hell. I discovered this at 1 am and was not pleased.
  • Those windy turns through the passes you take at night? Yeah, the darkness conceals you’re driving on the edge of a mountain with a 2000′ drop off on one side and there’s no guard rails.
  • Places like Panamint Springs, Stovepipe Wells, Furnace Creek are surprisingly far apart, 20-30 miles.
  • Get gas in Stovepipe Wells during the day. FC has 24/7 gas but it’s easily $1-$2/gallon higher than the rest of the world. Likewise since Panamint Springs is so remote and purely diesel generator powered, their gas is even higher; I think I saw $6.95/gallon this trip.
  • There’s AT&T EDGE coverage around Furnace Creek, but that’s it once you enter the park unless you get on a mountain top.

One thing I was amused by was hanging out in front of the FCR store and watching the tourist machine at work. The place seems to do a pretty good job processing bus loads of retirees. A big tour bus rolls in, stops at the curb and all of the passengers are herded up the steps into a big buffet room. An hour later the bus returns, everyone is herded back on the bus and driven the 0.25-0.50 mile back to the buildings with guest rooms. The bus door opens, they all again are herded into the same ground-floor unit to their rooms. Repeat the reverse the next morning, and then I guess they go off to their next tour destination or back to Vegas.

The after-hours crowd is what makes the ranch fun. The tourists and retirees usually clear out by 8 pm or so, and the night crowd starts trickling in (or continuing on) to the saloon or the fire pits in front of the store. What’s nice is that you don’t have to be a guest, any random person can come hang out. Pretty much everyone I chatted with are well traveled, come from all over, and have fascinating stories. Later on the employees get off shift and join in. Be nice to who cooked and served your cheeseburger, you will likely run into them here! Many seem to thrive on adventure too, lots of them are 20-somethings who signed up to work in national parks, traveling from park to park as they’re assigned.

While I was there I spotted a small fleet of Ford Focus and Fusion cars with Michigan manufacturer plates, all with dashes decked out in all sorts of digital readouts; I’m assuming GM product testing. Thursday night some sort of film crew rolled in from LA with several giant camper trailers, with some tall skinny girl with black hair and a clipboard buzzing around who was clearly running the show. As I left Friday morning I noticed the crew was apparently filming at the Old Borax Works, they had CHP present to block off the highway, trailers were set up on the parking lot, and people were huddled out on various hilltops. I have no idea what they were shooting, but I’d like to see it!

It was also sprinkling when I left Death Valley which was pretty nice and smelled amazing. What I didn’t realize is that this meant SNOW was waiting for me in the Sierras! Somewhere between Lone Pine and Bakersfield the sand on the side of the road turned to snow. Several inches had fell, the place was covered!

« Newer Posts - Older Posts »