Feed on
Posts
Comments

Mobile over the years

Gather round, it’s time for the old person to tell a story about how things used to be!

Mid-Late 1990s: no such thing as mobile, I stuck in my 3com PCMCIA card and dialed somebody like CRIS or long distance into my own ISP’s modems. I have flip phones through US Cellular and they frequently fall back to AMPS.

Late 1990s: I get an AT&T “One Rate Digital” TDMA phone for the low price of $99/month. I can text but I don’t see the point. I saw one fancy US Cellular data modem plugged into a flip phone and thought “not bad”. My cutting edge friends in Silicon Valley had Ricochet modems which was the equivalent to a wireless dialup modem. For work I carried an alphanumeric pager to which NOCOL/Nagios sent short messages to like “%SYS-1-UPDOWN: Frame-Relay1/0/13.0 is DOWN” when shit broke. At the movies I overhear some girl behind me ask “why would anyone carry both a phone and a pager anymore?” I think for a while I may have even carried two cell phones, one on US Cellular (CDMA) and another on AT&T (TDMA/GSM).

Early 2000s: I move to Tulsa, have TDMA voice phone service. 802.11 wireless arrives! On the road, still no internets. Here I had a Nokia 6300-something. I started actually texting people for real purposes. For Nagios I try out this e-mail to SMS text message thing and become conditioned to breaking out into a panic when I hear the Nokia text tone at 2 AM. When I was at TU, I bought a Handspring with an Omnisky Minstrel modem. I don’t even remember what data service I subscribed to, but I could see greyscale internets! It was really cool to sit on my friend’s couch, proceed to use my wireless modem and ssh to one of my ISP’s modem banks and feed it AT commands to make an outbound phone call to my friend’s house phone, then giggle while he answered the phone and realized what I had done. Other friends replayed parts of the Matrix, messaging their friends on Yahoo or AIM moments before knocking on their door and scaring the shit out of them.

2004-2008: 802.11 wireless/ethernet in most hotels. Mobile internet consisted of a USB dongle connected to a small external antenna, war driving and camping out in parking lots leeching from whoever was around. State of Texas offers free wifi at most/all interstate rest areas, brilliant thinking to get tired travelers off the road; I frequently used them on my many road trips across the state. Theoretically my Nokia 6310i could do data, but it was so old I don’t think anyone supported it. For work on-call duties I used a Verizon CDMA data card in a Panasonic Toughbook which worked pretty well all around Austin. Here I learn the amateur radio nerds have been bouncing TV off satellites and packets off the moon since forever. Go to Amsterdam to work, slap in a new pre-paid SIM at the airport and now I have a local number.

2008-2009-ish: sometime I got a 3G iPhone and had an unlimited data plan. Mobile use consisted of taking photos with my camera, downloading to my laptop, connecting my phone, emailing to flickr or whatever. Somehow this was better than leeching wifi with a special antenna. Wireless is everywhere. Going deep into the Nevada desert meant my phone dropped all the way down to GPRS, with the little “o” icon for data connectivity. Pretty impressive that I had at least minimal data in the middle nowhere.

2009: now I have an iPad and iPhone with a camera, still no way to tether data. At least now I can plug my compact flash cards from the camera into the iPad, upload photos that way. I can go pretty much anywhere except rural Oklahoma and get some sort of 3G or EDGE data service. Voice? what’s voice?

2010: I needed tethering for on-call duties, got a Verizon Mifi thingy. This is actually pretty cool for laptops, press a button to turn it on, throw it in your bag, open up your laptop. I think I even get AT&T 3G service in parts of Death Valley, if not EDGE in Furnace Creek. Oddly enough in Oklahoma I can get Verizon CDMA service for Mifi, but zero cell service for my AT&T iPhone. I get a AT&T pico-cell thing for my parent’s house, full voice+SMS+MMS service. I have enough 3G contraptions I don’t even care if places have wifi or not. Work causes my text usage to explode, making me get an unlimited text plan.

2011-2012: everything is mobile. There’s an app for everything. I never talk to anyone except my parents on the phone. I consume lots of email but never send any. I write blog posts about how I used to sit in the parking lot with 802.11 antennas.

I finally gave in, adding fans and a 620 W ATX PSU to the IKEA cluster. Fortunately it remains fairly silent and draws little power.

If I was going to power 8-10 servers in the cabinet, multiple 192 W 12VDC power supplies got clunky and the hard drives generated a ton of heat that couldn’t be passively cooled well. Not only that, I needed to maintain a ~40W buffer for hard drive spinup on each PSU so it was inefficient to go this way. Going the PSU route also simplified a lot of wiring.

I have a single Antec 620 W ATX PSU mounted at the top of the cabinet, using one of the HELMER drawers as a mount. I don’t actually use the ATX power cabling here, I have it shunted with a terminator so it’s always on. I take the two 4-pin 12V feeds and tie these to a pair of little 8-screw terminals. From there each motherboard has a picoPSU (really a DC-DC converter) ranging anywhere from 80W-150W.

The PSU route works really well for a several reasons. The particular one I has detachable connectors, so I can get rid of crap I don’t need in here. It has a big fan, virtually silent. Mounted in the top of the cabinet in a drawer it not only gets rid of wiring clutter on the floor, but the fan helps pull the heat off the switches. Plus, tons of headroom. Napkin math says 10 systems at 50 W peak still gives me about 10% headroom at max so I’m reasonably secure I won’t burn my apartment down. As it stands right now, seven motherboards running a CPU stress test + eight hard drives draws around ~220 W. The entire cabinet has a footprint of 4 cables: ethernet, PSU power and power for two switches.

As far as fans go, I mounted two 120mm fans on the rear door and connected them to systems #1 and #4. I bought the slowest, biggest fans I could get at 1000 RPM. With the motherboards running the fans, they keep it around 870 RPM. The result was obviously huge, components dropped 30-40 C! Hard drives are now no longer just beyond the threshold of painful to hold onto.

With the PSU in the rear of the top drawer and wiring terminals in the front, there was a lot of empty space that was perfect for a pair of 8-port Ethernet switches. At first I thought I was going to have to build some little shelf railing contraption to hold it, then I realized I could take a 5″ deep section of drawer, turn it upside down, cut notches for the power cabling underneath and I had a tidy little shelf for the switches.

In the process I cut the power tray in half, thinking it would be handy to make the PSU totally separable from the wiring terminals. In practice this wasn’t a good idea because sliding the first motherboard underneath makes it super easy to snag SATA cables. Oh well.

I also trimmed a half inch off the end of each system tray to make room for the ethernet cables to flex. Sawing through eight sheets of acrylic with a hand saw for such a small amount was a giant pain in the ass, but the result is worth it. The door can shut flush without bunching up the cabling.

Now I’m considering how I can make some sort of drive tray…

Just how many 2.5″ hard drives can you cram into a 3″ x 9.5″ x 15″ space!

Holy shit, time flies

I can’t believe it’s Friday again already. I feel like I’m swimming through the week now. I don’t really understand what’s going on the past few weeks. Am I so busy that I ignore half the day, is it so routine there’s nothing to remember, or is the universe contracting?

Time flies, new cat

I can’t believe it’s already August. AUGUST. I’ve lived in California for 6 months now and time has just flown by. I’m having trouble accepting that. I feel like I need to be out exploring all the time since it’s all new, but man, it’s nice sitting out on the balcony in the sun and staring at a laptop (and not thinking about work). I’ve been putting off a trip to Death Valley for a couple of months now, it’s always some procrastination or excuse. “New cat” “vet appointment” “need oil change” “woke up too late”…

I adopted a new cat, Charlie, a couple of weekends ago from the Hayward Animal Shelter. He’s a black and white tuxedo cat, white paws/feet and white chest, and about a year old. He was a stray from the rough streets of Hayward, but super friendly. Him and Felix are still battling out who’s dominate, Charlie is certainly not shy about pouncing and rough play. One day I think one has the upper paw, the next day the other one is in charge (my bet is on felix). Felix is much quieter now, he doesn’t flop down to vocalize his boredom, he’s usually off doing other things now.

 

I frequently eat weekend/late night breakfast in diners alone, with iPad in tow to catch up on books I’m reading or browse Wikipedia over few cups of coffee. Despite iPads being out for a couple of years now and practically all over the media, strangers sitting next to me frequently stare inquisitively and ask what it is as if they’ve never seen one. It ALWAYS starts with “What is that?” or “Is that an iPad?” I’ve noticed business and tech people have their own little twist on this, they know what it is but always ask “Is that the newest iPad?”

For whatever reason, sometimes these little conversations derail into batshitcrazy wonderland about random conspiracies and incorrect facts, that are so out there that I don’t even know how to begin to respond. I just have to go back to eating my french toast. It can’t be geography, this even happens in places a mere stone’s throw away from the heart of Silicon Valley.

Two recent examples:

Today somebody asked me if buying ebooks was cheaper than buying real books. We started chatting about how books were a little cheaper and about the convenience of having them delivered electronically. He goes off on how today’s youth expects everything instantly instead of having patience for things. He hates how cell phones are being used for location tracking too. He goes off into the deep end by saying he used to work for BoA and the government installs tracking codes into the mag stripes of credit cards. “Anything you buy, these other corporations can see it. If you go into Wal-Mart and buy a big bottle of drugs, they’re [they == The Man/The System?] going to deny you life insurance forever. That’s why I don’t use credit cards and only pay cash.” I can’t imagine what he’d have to say if he knew I worked at Facebook.

A few Saturdays ago the guy next to me asks, “Is that an iPad?” He starts of talking about how he’d like to get one, but says he won’t because he used to work at Apple on the first Mac, Apple used slave labor now, and they screwed a bunch of people out of jobs by moving the company to Sacramento.

I got nothing.

It also reminds me of a time long ago in 1997 when I was searching for a location to co-locate some equipment for my ISP. I would approach local business owners, explain how much space I needed and how I’d give their business free high-speed Internet access in return (woo 56k frame relay!). One woman after hearing my pitch told me no, because she used to get calls from the internet. “The phone would ring, I’d pick it up and it’d say internet and hang up”. What?

Farewell, Firefox

Firefox is dead to me. In the early days, Firefox was leaps and bounds better than anything else out there, and that was fantastic. Now it feels incredibly slow and every time I turn around I’m being nagged about wanting to apply an upgrade. We’re jumping major version numbers like some sort of Microsoft Office product. Somewhere we went from version 3 to version 9 to version 13. I have an old MacBook Pro maxed out with 2 GB RAM and an SSD, and I basically have to restart FF each day to be able to watch a video or load flash. Even then, it eats all my RAM and works the CPU enough to kick on the fans.

I’ve moved on to Chrome. It’s not a favorite, but I have yet to restart it after two weeks. I can (so far) reliably kill off out of control rendering processes if they do happen to leak a ton of memory. It’s helping stave off upgrading to that new shiny Retina display MacBook Pro, at least for a few weeks!

Ever since the mini-ITX bug bit me this spring, I’ve been intrigued by the little Linux clusters people have been building with them. I absolutely despise loud and bulky home servers (not to mention their energy consumption) and thought this was a great idea to get a lot of computers into a small footprint with little to no noise. My favorite one was Glen Gardner’s mini-cluster as described on mini-itx.com. In my daytime profession I’m constantly dealing with the automation of lots of Linux systems and running distributed apps, so I thought it’d be cool to have my own little setup at home to wrench on. One doesn’t get much experience with load balancing or fault tolerance with a couple of systems. I could’ve just installed a bunch of VMs on a CentOS box (which I’ve already done) or use EC2 instances, but I like getting my hands dirty and felt like building something.

In my setup I wanted it to be more of a general purpose lab, optimize for a greater number of individual systems and not necessarily raw computing power. I wanted local storage for each system so I could someday play with things like HBase/Hadoop. Most of all I wanted it to be silent and consume only a modest amount of energy since it’s likely to be idle a lot. CPU fans may be okay, but so far I’m sticking to fan-less Atom boards.

I discovered picoPSUs, which are DC-DC power supplies, very small and eliminate the rats nest of ATX cabling. Being a ham operator I already had the token bigass 12V Samlex linear power supply laying around so I thought I had power solved. I also already had a stack of SATA hard drives, so this sparked pulling the trigger and buying the rest of the gear. At first I bought a few Shuttle XPCs from Weird Stuff Warehouse for $50 to salvage the motherboards, but turns out they were dead and/or very broken. I also discovered my linear PSU had shorted out and was horribly inefficient (50% ?!) anyways. By now I was sort of committed to the project with half the gear needed. I wound up going with Intel Atom D2500 motherboards and a new 192 W power adapter.

The first idea for mounting was to use threaded bolts between wooden plates to stack motherboards, but this wound up being a horrible idea. If the holes were off the slightest bit, they caused the rods to bind through the motherboard mounting holes. Plus, I still wanted hard drives. I sketched up a design for a little blade enclosure, so each system of motherboard and hard drive(s) could be mounted on a metal blade to be slid into a rack. I decided this was too much work and abandoned the idea.

Fast forward to a couple of weeks ago when I was randomly googling and discovered the “Helmer” linux cluster. Somebody had taken an IKEA HELMER cabinet, cut it up, and stuffed it full of micro-ATX boards. Looking at the dimensions of the drawers I realized they were almost exactly the dimensions of the blades I wanted to build. Then I found another design, and another. It also has the added benefit of being sized to slide underneath a desk to get out of the way. IKEA had solved the chassis problem I had and that weekend I went out to buy one to get hacking.

My design differs from others in that I made doors out of expanded metal to put on the front+back of the case, and went with system “blades” consisting of a mini-ITX board and dual 3.5″ hard drives. My requirements for fanless and quiet dictated that I couldn’t use a bunch of ATX power supplies and needed the airflow for passive cooling. Plus I didn’t want cables spilling out all over the place, both for aesthetic purposes and the fact that my cat loves playing with dangling SATA cables. In the end it really does look like a cabinet you’d find in a datacenter.

The cabinet was designed to hold six drawers (4″ tall) and thus six systems. I wanted to try doubling the density, putting 10-12 systems in the cabinet. This involves putting aluminum angle stock in between the existing rails, providing for 2″ tall systems. For six systems I’ll likely cut down the metal from the drawers and use them for blades, for the remainder I’m using sheets of acrylic. I have no idea at this point how hot things will get and how much the plastic will sag or even melt. I figure this can easily be solved with aluminum cross-members.

Right now I don’t have a real way to turn the boards on/off without shorting the power connector or pulling the plug. I configured the BIOS of each one to always power-on after power loss, and discovered wake-on-LAN works. Each motherboard is also configured to PXE boot first, so if I want to fiddle with a system (e.g. kickstart, memtest) without dragging out a monitor and keyboard I can drop in a PXE config on my tftp server or just let it boot locally.

Overall the project went much smoother than I expected. The HELMER assembled with only a screwdriver and I wasn’t left with bloody fingers from sharp edges like I would a normal 42U cabinet in a datacenter. The only power tool I needed was a drill, I was able to cut all of the expanded metal and aluminum stock with shears and tin snips. (silent construction + apartment = win!) It’s not a cheap project, the cost of the retrofit was basically the same cost as the HELMER and going the picoPSU route was more expensive than an army of ATX PSUs.

So far I only have three blades and an ethernet switch installed and heat doesn’t seem to be an issue yet. It’s virtually silent, although I’ve noticed if the hard drives seek just the right way it causes the door to rattle a bit. Each blade draws 30 W at idle, peaking at 50 W during hard drive spinup or heavy CPU usage, so this limits about 3-4 systems per 192 W PSU. Eventually I may have to give in a add a large (but low rpm!) fan or two on the back door, we’ll see.

If you’re looking for kit, I recommend mini-box.com. They stock several mini-ITX boards, invented the picoPSU, and have some nice simple compact cases. Particularly I like them because they’re local; I can either drive over to pick up parts if I want or USPS delivery is basically next day.

Photos on flickr

Time flies

Looking through old photos I often wonder how I had so much time to do and see things in Texas. Then I realized that I lived there six years and I’ve barely been here a few months. The weekends fly by so fast now.

Hot weather finally arrived and it feels fantastic! Completely clear and sunny, not a speck of cloud in the sky. Yesterday was the first time I wore shorts, and I went to the botanical garden in Berkeley to wander about in the sun all afternoon. It makes me feel really glad I finally moved. I’m told it’ll get cool again soon, but I’m enjoying this now. My jasmine is blooming and between that and the petunias, it makes the balcony smell fresh and outdoorsy. It’s awesome to finally be able to sit outside on a warm night.

Oakland hills firestorm

A couple of weekends ago I happened to catch Richard Misrach’s Oakland-Berkeley firestorm exhibit at the Oakland Museum of California. I was only vaguely aware of the firestorm since it happened in ’91, long before I met anyone here, so it was interesting to learn some local history. Basically something started a fire, and intense, dry hot winds turned it into an extreme wildfire that completely scorched the hills. A curator explained some of the stories he had collected, how some of the staff lost their homes, how people dumped valuable belongings in swimming pools as a last ditch effort to save them. (TIL Wil Wright of SimCity fame lost his house in the fires and used the experience of rebuilding as an inspiration to make The Sims.)

It was amazing to see the level of destruction of the area. There were no smoldering charred ruins left, it was all gone. If it wasn’t brick or metal, it was completely turned to ash and blew away. Cars and bicycles stripped of all rubber, melted tricycles, staircases and decks leading nowhere, trees turned to ashy stumps. In the exhibit hall was a book where people were free to write their stories. One entry written by a child said it looked like “another planet, like the moon”; that was an accurate assessment.

What really made me sad was that among the photographs was one of food+water dishes that somebody had left out presumably for their pets. There were several “missing cat” photos posted and it made me realize that lots of beloved pets lost their lives too. I like to think people were able to evacuate with their pets, but I know there were many people who were unable to return home. There would have been nowhere for animals to run, and I imagine even if the front door were left wide open many pets would be too scared to flee. :(

Shit * Says

@lindvall: Feeling a lot of déjà vu from Shit Silicon Valley Says at Seattle Beta.

« Newer Posts - Older Posts »