Feed on
Posts
Comments

IKEA cluster power retrofit

Aug 26th, 2012 by

I finally gave in, adding fans and a 620 W ATX PSU to the IKEA cluster. Fortunately it remains fairly silent and draws little power.

If I was going to power 8-10 servers in the cabinet, multiple 192 W 12VDC power supplies got clunky and the hard drives generated a ton of heat that couldn’t be passively cooled well. Not only that, I needed to maintain a ~40W buffer for hard drive spinup on each PSU so it was inefficient to go this way. Going the PSU route also simplified a lot of wiring.

I have a single Antec 620 W ATX PSU mounted at the top of the cabinet, using one of the HELMER drawers as a mount. I don’t actually use the ATX power cabling here, I have it shunted with a terminator so it’s always on. I take the two 4-pin 12V feeds and tie these to a pair of little 8-screw terminals. From there each motherboard has a picoPSU (really a DC-DC converter) ranging anywhere from 80W-150W.

The PSU route works really well for a several reasons. The particular one I has detachable connectors, so I can get rid of crap I don’t need in here. It has a big fan, virtually silent. Mounted in the top of the cabinet in a drawer it not only gets rid of wiring clutter on the floor, but the fan helps pull the heat off the switches. Plus, tons of headroom. Napkin math says 10 systems at 50 W peak still gives me about 10% headroom at max so I’m reasonably secure I won’t burn my apartment down. As it stands right now, seven motherboards running a CPU stress test + eight hard drives draws around ~220 W. The entire cabinet has a footprint of 4 cables: ethernet, PSU power and power for two switches.

As far as fans go, I mounted two 120mm fans on the rear door and connected them to systems #1 and #4. I bought the slowest, biggest fans I could get at 1000 RPM. With the motherboards running the fans, they keep it around 870 RPM. The result was obviously huge, components dropped 30-40 C! Hard drives are now no longer just beyond the threshold of painful to hold onto.

With the PSU in the rear of the top drawer and wiring terminals in the front, there was a lot of empty space that was perfect for a pair of 8-port Ethernet switches. At first I thought I was going to have to build some little shelf railing contraption to hold it, then I realized I could take a 5″ deep section of drawer, turn it upside down, cut notches for the power cabling underneath and I had a tidy little shelf for the switches.

In the process I cut the power tray in half, thinking it would be handy to make the PSU totally separable from the wiring terminals. In practice this wasn’t a good idea because sliding the first motherboard underneath makes it super easy to snag SATA cables. Oh well.

I also trimmed a half inch off the end of each system tray to make room for the ethernet cables to flex. Sawing through eight sheets of acrylic with a hand saw for such a small amount was a giant pain in the ass, but the result is worth it. The door can shut flush without bunching up the cabling.

Now I’m considering how I can make some sort of drive tray…

Just how many 2.5″ hard drives can you cram into a 3″ x 9.5″ x 15″ space!

Posted in IKEA Cluster | No Comments »

Holy shit, time flies

Aug 10th, 2012 by

I can’t believe it’s Friday again already. I feel like I’m swimming through the week now. I don’t really understand what’s going on the past few weeks. Am I so busy that I ignore half the day, is it so routine there’s nothing to remember, or is the universe contracting?

Posted in Uncategorized | No Comments »

Time flies, new cat

Aug 5th, 2012 by

I can’t believe it’s already August. AUGUST. I’ve lived in California for 6 months now and time has just flown by. I’m having trouble accepting that. I feel like I need to be out exploring all the time since it’s all new, but man, it’s nice sitting out on the balcony in the sun and staring at a laptop (and not thinking about work). I’ve been putting off a trip to Death Valley for a couple of months now, it’s always some procrastination or excuse. “New cat” “vet appointment” “need oil change” “woke up too late”…

I adopted a new cat, Charlie, a couple of weekends ago from the Hayward Animal Shelter. He’s a black and white tuxedo cat, white paws/feet and white chest, and about a year old. He was a stray from the rough streets of Hayward, but super friendly. Him and Felix are still battling out who’s dominate, Charlie is certainly not shy about pouncing and rough play. One day I think one has the upper paw, the next day the other one is in charge (my bet is on felix). Felix is much quieter now, he doesn’t flop down to vocalize his boredom, he’s usually off doing other things now.

 

Posted in Uncategorized | No Comments »

Shit people say about iPads

Jun 24th, 2012 by

I frequently eat weekend/late night breakfast in diners alone, with iPad in tow to catch up on books I’m reading or browse Wikipedia over few cups of coffee. Despite iPads being out for a couple of years now and practically all over the media, strangers sitting next to me frequently stare inquisitively and ask what it is as if they’ve never seen one. It ALWAYS starts with “What is that?” or “Is that an iPad?” I’ve noticed business and tech people have their own little twist on this, they know what it is but always ask “Is that the newest iPad?”

For whatever reason, sometimes these little conversations derail into batshitcrazy wonderland about random conspiracies and incorrect facts, that are so out there that I don’t even know how to begin to respond. I just have to go back to eating my french toast. It can’t be geography, this even happens in places a mere stone’s throw away from the heart of Silicon Valley.

Two recent examples:

Today somebody asked me if buying ebooks was cheaper than buying real books. We started chatting about how books were a little cheaper and about the convenience of having them delivered electronically. He goes off on how today’s youth expects everything instantly instead of having patience for things. He hates how cell phones are being used for location tracking too. He goes off into the deep end by saying he used to work for BoA and the government installs tracking codes into the mag stripes of credit cards. “Anything you buy, these other corporations can see it. If you go into Wal-Mart and buy a big bottle of drugs, they’re [they == The Man/The System?] going to deny you life insurance forever. That’s why I don’t use credit cards and only pay cash.” I can’t imagine what he’d have to say if he knew I worked at Facebook.

A few Saturdays ago the guy next to me asks, “Is that an iPad?” He starts of talking about how he’d like to get one, but says he won’t because he used to work at Apple on the first Mac, Apple used slave labor now, and they screwed a bunch of people out of jobs by moving the company to Sacramento.

I got nothing.

It also reminds me of a time long ago in 1997 when I was searching for a location to co-locate some equipment for my ISP. I would approach local business owners, explain how much space I needed and how I’d give their business free high-speed Internet access in return (woo 56k frame relay!). One woman after hearing my pitch told me no, because she used to get calls from the internet. “The phone would ring, I’d pick it up and it’d say internet and hang up”. What?

Tags: ,

Posted in Uncategorized | No Comments »

Farewell, Firefox

Jun 15th, 2012 by

Firefox is dead to me. In the early days, Firefox was leaps and bounds better than anything else out there, and that was fantastic. Now it feels incredibly slow and every time I turn around I’m being nagged about wanting to apply an upgrade. We’re jumping major version numbers like some sort of Microsoft Office product. Somewhere we went from version 3 to version 9 to version 13. I have an old MacBook Pro maxed out with 2 GB RAM and an SSD, and I basically have to restart FF each day to be able to watch a video or load flash. Even then, it eats all my RAM and works the CPU enough to kick on the fans.

I’ve moved on to Chrome. It’s not a favorite, but I have yet to restart it after two weeks. I can (so far) reliably kill off out of control rendering processes if they do happen to leak a ton of memory. It’s helping stave off upgrading to that new shiny Retina display MacBook Pro, at least for a few weeks!

Posted in Uncategorized | No Comments »

IKEA hacking: HELMER mini-server cabinet

May 27th, 2012 by

Ever since the mini-ITX bug bit me this spring, I’ve been intrigued by the little Linux clusters people have been building with them. I absolutely despise loud and bulky home servers (not to mention their energy consumption) and thought this was a great idea to get a lot of computers into a small footprint with little to no noise. My favorite one was Glen Gardner’s mini-cluster as described on mini-itx.com. In my daytime profession I’m constantly dealing with the automation of lots of Linux systems and running distributed apps, so I thought it’d be cool to have my own little setup at home to wrench on. One doesn’t get much experience with load balancing or fault tolerance with a couple of systems. I could’ve just installed a bunch of VMs on a CentOS box (which I’ve already done) or use EC2 instances, but I like getting my hands dirty and felt like building something.

In my setup I wanted it to be more of a general purpose lab, optimize for a greater number of individual systems and not necessarily raw computing power. I wanted local storage for each system so I could someday play with things like HBase/Hadoop. Most of all I wanted it to be silent and consume only a modest amount of energy since it’s likely to be idle a lot. CPU fans may be okay, but so far I’m sticking to fan-less Atom boards.

I discovered picoPSUs, which are DC-DC power supplies, very small and eliminate the rats nest of ATX cabling. Being a ham operator I already had the token bigass 12V Samlex linear power supply laying around so I thought I had power solved. I also already had a stack of SATA hard drives, so this sparked pulling the trigger and buying the rest of the gear. At first I bought a few Shuttle XPCs from Weird Stuff Warehouse for $50 to salvage the motherboards, but turns out they were dead and/or very broken. I also discovered my linear PSU had shorted out and was horribly inefficient (50% ?!) anyways. By now I was sort of committed to the project with half the gear needed. I wound up going with Intel Atom D2500 motherboards and a new 192 W power adapter.

The first idea for mounting was to use threaded bolts between wooden plates to stack motherboards, but this wound up being a horrible idea. If the holes were off the slightest bit, they caused the rods to bind through the motherboard mounting holes. Plus, I still wanted hard drives. I sketched up a design for a little blade enclosure, so each system of motherboard and hard drive(s) could be mounted on a metal blade to be slid into a rack. I decided this was too much work and abandoned the idea.

…

Fast forward to a couple of weeks ago when I was randomly googling and discovered the “Helmer” linux cluster. Somebody had taken an IKEA HELMER cabinet, cut it up, and stuffed it full of micro-ATX boards. Looking at the dimensions of the drawers I realized they were almost exactly the dimensions of the blades I wanted to build. Then I found another design, and another. It also has the added benefit of being sized to slide underneath a desk to get out of the way. IKEA had solved the chassis problem I had and that weekend I went out to buy one to get hacking.

My design differs from others in that I made doors out of expanded metal to put on the front+back of the case, and went with system “blades” consisting of a mini-ITX board and dual 3.5″ hard drives. My requirements for fanless and quiet dictated that I couldn’t use a bunch of ATX power supplies and needed the airflow for passive cooling. Plus I didn’t want cables spilling out all over the place, both for aesthetic purposes and the fact that my cat loves playing with dangling SATA cables. In the end it really does look like a cabinet you’d find in a datacenter.

The cabinet was designed to hold six drawers (4″ tall) and thus six systems. I wanted to try doubling the density, putting 10-12 systems in the cabinet. This involves putting aluminum angle stock in between the existing rails, providing for 2″ tall systems. For six systems I’ll likely cut down the metal from the drawers and use them for blades, for the remainder I’m using sheets of acrylic. I have no idea at this point how hot things will get and how much the plastic will sag or even melt. I figure this can easily be solved with aluminum cross-members.

Right now I don’t have a real way to turn the boards on/off without shorting the power connector or pulling the plug. I configured the BIOS of each one to always power-on after power loss, and discovered wake-on-LAN works. Each motherboard is also configured to PXE boot first, so if I want to fiddle with a system (e.g. kickstart, memtest) without dragging out a monitor and keyboard I can drop in a PXE config on my tftp server or just let it boot locally.

Overall the project went much smoother than I expected. The HELMER assembled with only a screwdriver and I wasn’t left with bloody fingers from sharp edges like I would a normal 42U cabinet in a datacenter. The only power tool I needed was a drill, I was able to cut all of the expanded metal and aluminum stock with shears and tin snips. (silent construction + apartment = win!) It’s not a cheap project, the cost of the retrofit was basically the same cost as the HELMER and going the picoPSU route was more expensive than an army of ATX PSUs.

So far I only have three blades and an ethernet switch installed and heat doesn’t seem to be an issue yet. It’s virtually silent, although I’ve noticed if the hard drives seek just the right way it causes the door to rattle a bit. Each blade draws 30 W at idle, peaking at 50 W during hard drive spinup or heavy CPU usage, so this limits about 3-4 systems per 192 W PSU. Eventually I may have to give in a add a large (but low rpm!) fan or two on the back door, we’ll see.

If you’re looking for kit, I recommend mini-box.com. They stock several mini-ITX boards, invented the picoPSU, and have some nice simple compact cases. Particularly I like them because they’re local; I can either drive over to pick up parts if I want or USPS delivery is basically next day.

Photos on flickr

Tags:

Posted in IKEA Cluster | 2 Comments »

Time flies

Apr 22nd, 2012 by

Looking through old photos I often wonder how I had so much time to do and see things in Texas. Then I realized that I lived there six years and I’ve barely been here a few months. The weekends fly by so fast now.

Hot weather finally arrived and it feels fantastic! Completely clear and sunny, not a speck of cloud in the sky. Yesterday was the first time I wore shorts, and I went to the botanical garden in Berkeley to wander about in the sun all afternoon. It makes me feel really glad I finally moved. I’m told it’ll get cool again soon, but I’m enjoying this now. My jasmine is blooming and between that and the petunias, it makes the balcony smell fresh and outdoorsy. It’s awesome to finally be able to sit outside on a warm night.

Tags:

Posted in Uncategorized | No Comments »

Oakland hills firestorm

Feb 25th, 2012 by

A couple of weekends ago I happened to catch Richard Misrach’s Oakland-Berkeley firestorm exhibit at the Oakland Museum of California. I was only vaguely aware of the firestorm since it happened in ’91, long before I met anyone here, so it was interesting to learn some local history. Basically something started a fire, and intense, dry hot winds turned it into an extreme wildfire that completely scorched the hills. A curator explained some of the stories he had collected, how some of the staff lost their homes, how people dumped valuable belongings in swimming pools as a last ditch effort to save them. (TIL Wil Wright of SimCity fame lost his house in the fires and used the experience of rebuilding as an inspiration to make The Sims.)

It was amazing to see the level of destruction of the area. There were no smoldering charred ruins left, it was all gone. If it wasn’t brick or metal, it was completely turned to ash and blew away. Cars and bicycles stripped of all rubber, melted tricycles, staircases and decks leading nowhere, trees turned to ashy stumps. In the exhibit hall was a book where people were free to write their stories. One entry written by a child said it looked like “another planet, like the moon”; that was an accurate assessment.

What really made me sad was that among the photographs was one of food+water dishes that somebody had left out presumably for their pets. There were several “missing cat” photos posted and it made me realize that lots of beloved pets lost their lives too. I like to think people were able to evacuate with their pets, but I know there were many people who were unable to return home. There would have been nowhere for animals to run, and I imagine even if the front door were left wide open many pets would be too scared to flee. :(

Posted in Uncategorized | No Comments »

Shit * Says

Jan 31st, 2012 by

@lindvall: Feeling a lot of déjà vu from Shit Silicon Valley Says at Seattle Beta.

Tags:

Posted in Uncategorized | No Comments »

Adventures in winter moving

Jan 25th, 2012 by

This has certainly been an adventurous move, mainly because of how open-ended it was, which then turned into “oh shit, what” when the snow storm hit Seattle. Despite the panic about last minute obstacles, so far they’ve been overcome. I’m not complaining, just what a ride!

I initially put off moving a week because I needed more time to pack and take care of things. At first, because the leasing office at the new place was closed on Sunday, I pondered loading and leaving Friday evening. Then the big snow storm hit Wednesday and buried the PNW under several inches of snow. The roads weren’t looking good so I considered loading Saturday and spending two nights on the road.

Then I realized Oregon and northern California got hit with the storm too and started checking the weather+road conditions there. The Siskiyou Pass in southern Oregon became interesting because it was frequently snowing heavily. At least once I-5 was completely closed and traffic diverted. There was a severe weather hazard Friday with chains mandatory, which cleared off to a warning Saturday so it looked like it was getting better.

By now I was completely packed and was just waiting. I finally decided to pick up the U-Haul on Saturday. Here’s where allll the plans when out the window and I had to start improvising everything. Because I didn’t know when the roads would be clear I hadn’t scheduled movers to help load the truck. I hopped on Yelp and started down the list, asking who was available Sunday. Most were booked weeks out. I found a crew who booked me at 1 PM but said they’d come earlier if they could.

8 AM the next morning I get a call that they’re on their way! I snapped up off the couch and started getting ready. It took a bit longer than I expected, we finished loading the U-Haul after 1 PM. They did a superb job at wrapping my furniture and packing the truck very well. Everything fit into a 14′ truck with room to spare. The only things I left behind were my pile of Rackable servers and closetserver, mainly because I didn’t have sturdy packing materials for them. A few last minute meals and errands I left Redmond at 5 PM. By now there was a travel hazard on the mountain pass because of heavy snowfall. Because I left so late I’d have to spend the night in Medford, OR anyways.

Felix was well behaved on the trip for the most part. He would complain for several minutes after I put him in the carrier and would occasionally get restless and very vocal. All the times I let him loose in the cab, he hid behind the driver’s seat.

The drive took way longer than expected even with generous time estimates. I rolled into Medford at 4 AM, on a very cold morning. Signs all along I-5 warned of severe weather up ahead in Ashland. I holed up in a Motel 6 for a few hours and slept. My previous estimates said I needed to be there by 2 AM, leave at 7 AM, in order to have time to get the lease signed, get keys, and get unloaded all on Monday.

At 8 AM Monday I woke up and checked the pass conditions. The snow pack had broken up and chains weren’t required anymore, so off I went. The mountain looked amazing, it was blanketed by a layer of fog just thick enough to blot out the sun. All of the big evergreen trees were sagging with snow, everything was white. The sand on the pavement was practically the only color. Slow going but eventually made it over.

I passed by Mount Shasta on into Yreka. Every time I’ve been here, it looks different. I’ve been through at night, at winter sunset, summer sunset, and now covered in snow. Again, amazing scenery along Mt Shasta. Felix was being behaving much better today. Initially he refused to go in the carrier and complained for a while after, but eventually he just laid there snoozing in the sun not saying a thing the entire drive. Even on a bumpy road where he was clearly getting bounced around, he still slept like a champ.

On the drive down I was again on Yelp looking up movers and found a crew who could unload the truck at 1 PM. The drive was still taking forever and I didn’t roll into Fremont until 4 PM, so the movers rescheduled for Tuesday morning at 9 AM. I had the night in my empty apartment and it was kind of nice to sprawl out in the emptiness, I didn’t miss my stuff (but worried that somebody would break into the truck on the street).

Tuesday morning we almost got everything in the apartment — except the couch. It was just too big for the sharp turn to get into the doorway. Because the apartment is on the 2nd floor, going up over the balcony would require serious manpower; they estimated at least six men, yikes. Instead I just loaded it back on the truck to deal with later.

Today (Wednesday) I started calling around and found a storage unit, now worrying about who I’m going to get to help me unload this sucker. I had joked earlier about getting day laborers and now it was looking like reality. I decided I could at least get it off the truck by myself, so if I had a trolley I was gold. The storage place had a big elevator and carts where I could just wheel it around, perfect. Initially I started unloading it myself and got it on a cart to wheel it in, then the owner came out to give me a hand. I’m really grateful he did, as it made it much easier; I’m not sure if I could’ve stood it up through the door by myself.

Nowwwww the next chapter in the story is getting back to Redmond to pick up my truck. Did I mention a SECOND drive, this time non-stop?

Tags:

Posted in Uncategorized | No Comments »

« Newer Posts - Older Posts »