bryan wann :: binary fury

I feel remiss if I didn’t mention something that was brought to my attention in a comment on my last post.  Back in February I drove down to Vandenberg Air Force Base to photograph the launch of a test Minuteman III ICBM. This was the second one I’ve done, the last being in July last summer.  These happen 3-4 times a year on just a couple of days notice, with a very long launch window, so you have to be ready to drive down and sit it out for however many hours it could take. There was one in the fall that I missed, I think I didn’t see the press release until it was already done.

So I took a long exposure photograph of the missile launching and heading out over the Pacific. I posted the photos to Twitter because there’s a lot of space nerds on there.  I drove home and didn’t think much about it.  Later I get a heads up that my photos got linked from an RT.com (f/k/a Russia Today) article.

The article was about the missile test, and goes on talking about increased missile tension, accusations being thrown around, and missile treaties between the US and Russia. There was my Twitter post, right in the middle of it all.

Well then!  That was unexpected.  The main photo at the top of the article from Reuters looked very similar to mine, and for a minute I thought they had ripped my photo off without credit.  Looking closer, it’s taken at exactly the same location but shows a different launch.

Fortunately my post didn’t get any crazy attention beyond a few likes and extra followers. I didn’t have any visitors and haven’t had any new shady people try to be my friends. It makes for a good story and gets people to say “wait, what?” until I show them the article.

I feel like I haven’t done much that’s post-worthy the past couple of months. Maybe I have. Oh well, just checking to make sure the website still works. It does!

After my last camping trip, I was hauling around three boxes of leftover firewood, a 20 L water jug, and my camping chair in the back of my truck for a couple of weeks. After I left the Sharks game at SAP Center last night, I noticed it was all gone. It was a lot of stuff to move, I suspect somebody just lifted it out of my truck into theirs. They at least left the big jug of cat litter I had just bought.

Sure, leaving it back there it was bound to happen. It’s way better than having the truck itself broken into. I’m mostly annoyed that they got my water can, but oddly I’m more annoyed they stole my 20′ rope that was used to tie it all down with. I realized this morning they also got the 5′ of chain used for my rifle gong too. Who steals rope and chain?

https://www.flickr.com/photos/kintayearbooks/albums

Earlier this year I came across some yearbooks from my school (Kinta High School) in the 1930s. Some belonged to my grandma, a couple from her brother that was killed right out of high school in WWII, and others loaned from family friends. I learned a few things in the process, such as the location and look of the old school buildings, the tea socials, the integration of the Lewisville school, and the impact of the war that crept into their pages. Earlier books had photos glued to the pages, hand colored title pages, and hand drawn ads!

I decided to scan them in so there were good electronic copies available for people to view. Dad also had quite a collection from the 50s, 60s, and beyond, so I started scanning those in too. In some years a yearbook was not published, usually to lack of budget; often the next year’s book will have a page devoted to the seniors that graduated the year no book was published. As I get time when I go home I’ll scan more.

What I have scanned so far is up on Flickr

I’ve tried to be complete and scan every page at least at 300 dpi, and I have only done little, if any, touch up work to them. The more recent ones aren’t as high a quality that I wanted (at least toward the inner edge), it was difficult to put them on a flatbed scanner without destroying the binding. I want these to be available to anyone without restraint. If they’re taken down from Flickr, I will host them on my own or find some other place for them. Tagging names on all the pages would be a nice to-do item!

Kinta Public Schools, Kinta High School, Kinta, Oklahoma

Throwing another tidbit of recently found knowledge out here. Along the way with playing different certs on my EAP-TLS I wound up removing the 802.1x password entry from the OS X Keychain (at the time thinking it would help my problem). What I discovered after that, even after reverting my RADIUS server config, I couldn’t connect back to my test SSID. OS X just threw a “unable to join network” message immediately and gave up. I couldn’t figure out why I couldn’t connect back, what would be keeping state about this network.

The TL;DR is I went into System Preferences > Network > Wi-Fi and told it to forget about my test SSID. After this I was prompted for the username/password on my network when I retried it.

(Actually I’m not even sure how I got in this situation. I just deleted this again and I was able to rejoin the network?)

Along the way I found out the Wireless Diagnostic tool on OS X is actually nice and useful, you wouldn’t think it from the surface. It collects a ton of logs and even packet captures to review. From what I gathered from the internets I needed to look for “eapolclient” logging. In this case eapolclient was reporting “Acquired: cannot prompt for missing user name”. I didn’t get many leads hunting for this message. It wasn’t until I thought about the forgetting thing that fixed my problem.

eapolclient “cannot prompt for missing user name”

Yay, fixed.

Not really relevant to the post, but I found it amusing

You think you know something just enough to get by, until you have something that challenges your workflow and tools. Then you have to brush away the cobwebs, learn a few things, and work on some scripts. Based upon my recent adventures in dealing with EAP-TLS for wireless, I realized I was doing several things wrong with OpenSSL and my private certificate authority (CA) over the years. I never really spent any time reading good docs beyond creating cert requests and converting certificates, nor learning what all the options and extensions do.

I found a great instruction guide on how to properly set up not only a CA with OpenSSL, but an intermediate CA, revocation lists, and certificates for servers and users: https://jamielinux.com/docs/openssl-certificate-authority/. It goes a good job of explaining just the options you need and why.

Based on this guide and copious amounts of Googling to fix other problems I set about putting my CA house in order. Among other things, I wanted to make sure I was generating appropriate certs and finally get around to stop doing some things by hand. (And insert an intermediate CA to my setup for giggles because my home lab isn’t complicated enough Never mind, dealing with certificate chains and applications are annoying.)

Things I learned while yak shaving certificates, all in one place so hopefully other people can avoid my folly:

(disclaimer: I’m possibly wrong about some of this)

iOS needs a common name on certs to trust them: This is what started it all. On Apple iOS devices (unsure when this started, iOS 11?) if your private root CA certificate does not have a Common Name (CN) value set, you will not be able to trust the certificate on the device. Without a CN, the certificate just does not show up under General > About > Certificate Trust Settings > Enable Full Trust For Root Certificates. You can install the root certificate all day long, but you can’t get device-level trust without this.

Certificate Trust Settings: No CN on root cert

Certificate Trust Settings: root cert has CN

Can’t just casually add a CN: Based on what I read for regenerating a root certificate with the same key, I thought I might be able to issue a new root cert and fix the CN. Turns out you can’t just add the CN on your existing root CA certificate because this breaks the chain of trust of any certificates you’ve signed with it. In particular, OpenSSL is going to throw an “unable to get local issuer certificate” when it tries to verify a signed certificate against your newly altered root certificate:

# Existing cert, newly modified root, verification fails:
# openssl verify -CAfile /opt/pki/CA/certs/wannnet-ca-20180913-cert.pem \
  certs/raptor.wann.net-20170624-cert.pem
certs/raptor.wann.net-20170624-cert.pem: C = US, ST = California, L = Fremont, O = wann.net, CN = raptor.wann.net, emailAddress = pki@wann.net
error 20 at 0 depth lookup:unable to get local issuer certificate

# Existing cert, original root, verification is OK:
# openssl verify -CAfile /opt/pki/CA/certs/wannnet-ca-20170624-cert.pem \
  certs/raptor.wann.net-20170624-cert.pem
certs/raptor.wann.net-20170624-cert.pem: OK

This is because the Issuer: of our root certificates has changed and mismatched what the cert expects:

# Existing server certificate, signed with original root CA (without a CN):
# openssl x509 -in certs/raptor.wann.net-20170624-cert.pem -text | grep Issuer
        Issuer: C=US, ST=California, L=Fremont, O=wann.net, OU=wann.net CA/emailAddress=pki@wann.net

# Issuer: of original root CA certificate:
# openssl x509 -in wannnet-ca-20170624-cert.pem -text | grep Issue
        Issuer: C=US, ST=California, L=Fremont, O=wann.net, OU=wann.net CA/emailAddress=pki@wann.net

# Issuer: of new root CA certificate with a proper CN added:
# openssl x509 -in wannnet-ca-20180913-cert.pem -text | grep Issue
        Issuer: C=US, ST=California, L=Fremont, O=wann.net, OU=wann.net CA, CN=wann.net Root CA/emailAddress=pki@wann.net

This means unless you re-issue every single certificate you have to reflect the new CN on the root cert, you’re going to be carrying around both root CA certificates in your devices’ trust stores until the child certificates eventually expire and you re-issue/re-sign them with the new root certificate. (2 years for me).

Upgrading signature algorithm on the root: However! Allegedly you can regenerate the root CA certificate with different signatures (the SHA1 -> SHA256 signature fracas recently) or new validity periods as long as you use the same key file you originally created the self signed cert with. You just can’t change the subject or CN details. (I wish I had known this before reissuing all of my client certs last year when I needed SHA256 signatures)

Adding subjectAltNames is messy: Somewhere along the way browsers such as Chrome started requiring valid Subject Alternative Name (SAN) on TLS certificates. For our purposes this is a list of DNS names (e.g. CNAMEs) that the certificate is valid with.

There’s not an easy way to add this to certificate requests (yet*), and every example on the Internet has you cracking open openssl.cnf in a text editor every time you want a new cert (what could go wrong?!). For a while I did this, I couldn’t be bothered learning a better way and hated myself each time I made a cert.  There’s also some convoluted bash “one-line” scripts out there that attempt to remedy this, but they’re hard to follow what they’re doing until you understand what they’re doing.

Of course my script is better than everyone else’s script: https://github.com/bwann/pki-tools/blob/master/make-wannnet-csrkey.sh. I’ve tried to simplify the bash and make it a little easier to understand. For the skeptical, here’s some sample output.

This script generates an RSA key and certificate request with the server name/common name as the first argument to the script, and any further arguments adds them to the request as Subject Alternative Names.

[1] As of OpenSSL 1.1.1 just released in September 2018, they’ve tweaked the req -extension option to make this a bit easier: https://github.com/openssl/openssl/commit/bfa470a4f64313651a35571883e235d3335054eb

CAs drop subjectAltNames when signing: By default OpenSSL will drop any user-submitted extensions (such as subjectAltNames) from a certificate request when it comes time to sign the certificate with your CA. This means when you sign a certificate request that includes your alternative names, this undoes all the work you just did to add them. Now you’ve got to re-supply the subjectAltNames via the OpenSSL config somehow as seen above for the signing process. There’s a good reason for this behavior, preventing unwanted user input: a rogue user could submit a certificate request with an extension something like basicConstraints=CA:TRUE, and unless it’s caught at signing time, the root has just issued a CA certificate.

You can get around this in your own CA environment by configuring copy_extensions in your openssl.cnf under the CA_default section. There’s a couple of options for this and the man page (man ca) has clear warnings about the implications. By setting this to copy_extensions=copy, this will copy the subjectAltNames from the certificate request; however you will want to make sure whichever extensions you’re using to sign a certificate, you’ve already nailed down basicConstraints and keyUsage in them so the extensions from the request don’t try to overwrite them.

Key extensions for servers and clients: This is what I first learned while setting up certificates for EAP-TLS. There’s extensions you can add to the X509 certificate that tell what the intended purpose of the certificate is (man x509v3_config). I haven’t ever had to use these because Linux has always been pretty happy (blissfully ignorant?) with what I was using before. Apparently things like Windows and Android cares about these extensions, i.e. they expect a server cert on the server, a client cert on the client app.

Ideally in your openssl.cnf you’d have a section defining options to use while signing a certificate for servers (e.g. [ server_cert ], and another section for client/user certificates (e.g. [ user_cert ], each containing the appropriate extendedKeyUsage settings for each type. Then when it comes time to sign a cert, tell OpenSSL which extension to use.

Android needs magic to install a root CA with system level trust. I haven’t gone through the effort to figure this one out yet, every time I install my root CA certificate on my Nexus it only winds up as a user-level cert, and displays a fabulous “warning third parties are snooping on your network” notification. From what I gather some root-level muckery needs to happen. Surely some MDM software has figured this out already.

Links

https://github.com/bwann/pki-tools/ : Some of my own scripts for generating cert requests, keys, signing them with an OpenSSL CA. Server-centric for now, still ironing out the kinks in user certificates for now and will post whenever they’re decent.

https://www.phildev.net/ssl/ : A friend’s guide to X509 certs that I had forgotten about until I posted this

https://jamielinux.com/docs/openssl-certificate-authority/ : The other CA setup guide I mentioned at the beginning of this post

If you’ve ever tried setting up FreeRADIUS and WPA2-Enterprise, and wondered how the example certs wind up with the X509v3 extended key “TLS Web Server Authentication” on it, the trick here is a config file specifying a numerical OID for the key instead of a text description. (Apparently Windows expects to see these TLS extended keys when connecting to a wireless network with EAP/TLS).

This puzzled me for a while, I’ve never seen this key and couldn’t figure out how they were getting it on their certs. It’s obvious now, but I spent way longer than I care to admit carefully combing over the example *.cnf files that came with the package, my system’s openssl.cnf, another system’s pristine openssl.cnf for any missing extendedKeyUsage or nsCertType directives, and the OpenSSL manual page for x509 looking for any missed defaults. It wasn’t until I carefully followed the Makefile in the example that they were pulling in an extension file via -extfile xpextensions and using an section with -extensions xpserver_ext that were specifying the options as OIDs. I had overlooked this file in earlier spelunking. Mystery solved.

P.S. this dude wrote an incredibly detailed instruction guide on a home lab setup using a Raspberry Pi as a UniFi wireless controller, private CA, FreeRADIUS, and WPA2-Enterprise.  It’s got more screenshots and explanations than most of the internet deserves.  https://dot11zen.blogspot.com/2018/04/wpa2-entreprise-using-unifi-access.html

P.P.S. FreeRADIUS has been incredibly annoying to write an attribute-API-driven Chef cookbook for.

users file? Easy, we’ll just iterate over a loop and dump out the values — not so fast, the last entry on a user can’t have a trailing comma, oh and we use tabs in some spots.

Config file? We’ll define convenience variables in it, using them all over the place and then use them to create more convenience variables in other config files we include. (I get it, for human maintainers this makes it easy) Without writing templates for each and every config file to get rid of them all, it took a while to locate the major ones to maintain compatibility.

Ubuntu? We’ll use completely different paths to store our RADDB, daemon names, TLS certs, system users and groups than CentOS, because we’re Ubuntu. And amazingly CentOS 7.x has FreeRADIUS v3, whereas Ubuntu 16.x has v2, which uses slightly different syntax.

The majority of the work is done now so hopefully I can post the cookbook on Github soon.