Feed on
Posts
Comments

We apologize for the inconvenience

May 13th, 2009 by

A big thing that I see a lot in the content and service provider world, when some outage or negative event happens the line is always “we apologize for the inconvenience.” This is another saying that irritates me to no end. I say this because over the past 36 hours tifa’s host, The Planet, had a couple of outages. I noticed in all their posts they never once mentioned inconvenience.

Every hosting customer of every company I’ve worked for are almost all universally businesses, using our network connections or servers to run their business. When something happens to their infrastructure, it’s not a mere inconvenience. It’s a risk with a measurable negative effect, that is in essence, causing pain to the bottom line.

Going to the store and finding there’s no milk is an inconvienience. Making plans with a friend who later flakes is an inconvienience. Not being able to access your online banking or buy tickets online is an inconvienince to the consumer, but not for the company who’s job is to provide the online services. For some of them, their entire business is built on your mouse clicks, SMS messages, telephone calls and eyeballs.

I personally will not insult your intelligence by apologizing for an inconvienience to your business, if you won’t insult mine by claiming an event stopped you from earning part of your $500 billion annual revenue on your Windows 2000 server (true story).

The corollary to this is, if you really do have a healthy revenue stream that depends on your internet connection and servers, then for god’s sake, invest in it! Let’s be honest, things can and will happen. It’s basic risk management. People will make honest mistakes, automated systems will be mis-programed, power supplies fail. There’s only so much you can do. The technology and know how to help is certainly out there.

I’ve seen far too many solutions riddled with single-points-of-failures that could’ve survived with even basic things such as bonded/teamed NICs in servers or redundant firewalls. Hell, even doing backups and actually practicing restores is worthwhile.

Otherwise, I have zero sympathy when your only Windows 2000 server is down for patches or when human error disconnects something, and you claim you’re losing millions. (another true story)

Posted in Uncategorized | No Comments »

Learning about IPv6

May 8th, 2009 by

A few observations with IPv6 fiddling all day long:

IPv6 peering is going to be crazy important. Only a couple of our transit providers, Tiscali and Verizon (f/k/a UUnet) will give us native/dual-stack transit. To help ensure independent connectivity we’ll have to turn up v6 peering at our various peering points. With IPv4 the driving factor is not so much connectivity as reducing transit costs, as we have several different v4 paths to a destination. To use ‘Google over IPv6’, where they will serve up an AAAA record for www.google.com for your network, you had better have good v6 connectivity to Google to ensure it’s reachable at all times.

A /32 is a crazy amount of address space. The popular ISP allocation practice now is to give /48 or /56 to each customer. A /32 is 2^32 (4.2 billion) /64s, or an entire IPv4 internet worth of /64s. And there’s 2^32 /32s.

IPv6 PTR records is a lot of decimal places. Hurricane routed me a /64, for which I delegated reverse DNS to myself. When setting up the zone file, you can’t leave out the trailing zeros. This results in:

[bwann@tifa ~]$ host tifa.ipv6.wann.net
tifa.ipv6.wann.net has IPv6 address 2001:470:1f0f:6c4::2

[bwann@tifa ~]$ host 2001:470:1f0f:6c4::2
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.4.c.6.0.f.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa 
domain name pointer tifa.ipv6.wann.net.

Fortunately since I’m delegated only a /64, that saves typing out 16 digits in the zone file:

2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0  IN     PTR     tifa.ipv6.wann.net.

Having lots of zeros in the address takes out a lot of grief, since you can leave off leading zeros and use :: shorthand.

Since an ASA doesn’t do plain GRE tunnels, I brought home my 2620 and will use that to run a v6 tunnel to HE for my entire home network. I still need to get my hands on an ASA for firewalling purposes, since it handles v6 directly. This should give me a feel for how auto configuration works.

Now I’m trying to find the biggest websites that serve up AAAA records. I keep hoping some nerd at Facebook, MySpace or CNN has a small setup somewhere. Haven’t found anything terribly exciting so far. This leads me to believe we have a lot of work in the content provider space to get content accessible via v6. It’s obviously a chicken/egg scenario, but I think solving the server side is a slightly easier problem.

Dave Meyer gave a somewhat frightening talk at NANOG 45 about how dual stack has failed us. The original model around 2000 was to start assigning IPv4 + IPv6 addresses to everything. Over time more and more things would run dual stack, eventually being the majority, just as IPv4 space was exhausted.

The point he was trying to make was that we should’ve made dual stack a high priority back in 2000-2003, but we didn’t. IPv6 uptake hasn’t happened anywhere near on the scale it needs to, now IPv4 is almost exhausted, which prevents the use of future dual stack!

Instead we’ll start to see “carrier grade NAT” devices come along. A large ISP/cableco/telco could come along, put all their customers behind it and give them all IPv6 addresses. The CGN will take care of IPv4 accessibility. Vendors will love it since they’ll get to sell big iron. But, it may be nearly impossible to rip it out later and restore the natural end-to-end connectivity we’ve come to enjoy and expect.

The vast majority of our customers have load balancers in one shape or another. The problem is, not many load balancers actually support v6; the Cisco CSM and CSS don’t, the ACE maybe. Citrix Netscalers do, but they’re very expensive boxes. A comment I read last night was “do you seriously expect you’ll do so much v6 traffic you need to load balance?” While likely very true, not many of our customers will go for it. I don’t fully understand IPv6 anycasting to know if it’s a possible solution or not.

Tags:

Posted in Uncategorized | No Comments »

IPv6 in the home

May 8th, 2009 by

I don’t know where March and April went, but they’re long gone.

Since we finally put in for a IPv6 allocation at work, I decided it was high time to finally get it working at home. It’s still a bit of voodoo but I finally have a few things working.

My home network is on NAT behind a Cisco PIX 501. This prevents me from using 6to4 or a tunnel broker, because it won’t pass IP protocol 41. I wound up using a Teredo tunnel from my MacBook (using Miredo for OS X) and a Linux box using miredo as well. It feels sort of sketchy because there’s not many public Teredo tunnel servers out there.

Being the nerd I am, after hitting http://ipv6.google.com/, I tried to get on an IPv6 irc server. This was a totally annoying endeavour at least. BitchX on my Linux box didn’t have IPv6 support compiled in, so I needed to rebuild the RPM. Unfortunately the box is running Fedora Core 6 which was apparently taken down from the usual fedora-updates repositories, and I needed a few things like ncurses-devel. Eventually I rebuilt it, then figured out I couldn’t connect.

Giving up, I turned to my MacBook. The Darwin ports of BitchX builds without IPv6 either, and I didn’t want to learn how to manually rebuild the package. I turned to Xirc, which was horrible; it would crash on connection attempt to an IPv6 host. I finally had success with Colloquy!

Then I turned my attention to tifa. She’s not running behind NAT, so I could easily play with my HE tunnels (with the downside of breaking IPv4 and locking myself out). For whatever reason I couldn’t add a ::/0 default route with iproute2, but I could with net-tools. Now I have a nailed up tunnel providing full access to the various services (i.e. web!) running, and a jumping off point for testing more things. Hooray!

Posted in Uncategorized | No Comments »

Visitors; Coolant woes really start

Apr 9th, 2009 by

It’s been a busy past couple of weeks! Victoria visited me for a week, then the same day she left my sister and her friend arrived for another five days. Lots of running around and heavy eating. I’m glad to have had visitors, it’s a welcome change. Weather has finally settled down and it’s sunny out.

Between all of them we went to Fredericksberg, Lady Bird Johnson Wildflower Center, Congress ave bats, the capitol, UT tower, and many eating establishments in between.

I finally took my truck to the mechanic again to work on my mystery coolant leak and my usual basket of other problems. It’s to the point I’m putting in a half gallon every couple of weeks. I would think it’d be fairly obvious where it’s at, but no luck today. Really, I’m not any better off than I was. I need to have them fill it full of dye, put it in neutral, put a brick on the gas pedal and run it at 3 K RPM for a couple of hours.

He did point me to a TSB on that engine that describes a casting defect on cylinder heads built by Castech which is known to cause coolant leaks with no evidence. It claims in the oil drain holes a porous crack can develop, letting coolant weep out. To verify, it involves getting inside the valve covers. Once inside it should be pretty obvious from what I can tell. The telltale manufacturer’s name should be on the top, and there should be clean/shiny spots around the crack while the rest of the head is oil-stained. There could also be coalesced oil+coolant “butterscotch” everywhere.

There’s a few obvious problems with this. It’ll take a couple of hours of labor just to get into it to see if it’s even made by Castech. It may not even be the cylinder head, it could be a head gasket or intake manifold gasket. Any of these could be weeping at high temperatures and it burns off without a trace. Once in there, if it is identified to be a leaking head, that means new heads. Worse, the vented coolant could’ve corroded my rods and lifters.

From a previous estimate, I know replacing a head gasket would be 10 hours of labor+parts at $1100. Fine. New GM Goodwrench cylinder heads are $344 each, so that’s at least $1100+$688+gaskets+oil+more labor. If it involves rods and lifters, it gets ridicuously expensive. It takes 16 lifters, OEM are $11 each, AC Delco are $44 each. By my estimate, the worst case is around $4k in parts and labor.

The upside to the visit, he’s very convinced my lingering P0135 code is caused by a flaky replacement Bosch O2 sensor I put in. As soon as I walked into the parts store and said I wanted a new sensor, the guy said “you must’ve just failed an inspection.” I mentioned I was replacing a Bosch, and both of them audibly tsk’ed. So, one expensive new O2 sensor later, we’ll see what happens.

Two other interesting notes from today. They inspected my brakes, my pads were at 60% wear — and they’re original parts, 160k miles on them. I clearly don’t stop much. The other, I discovered GM actually makes coolant seal tablets. More interesting, they’re made from ginger root and walnut shells. There’s a couple of tech notes where they recommend this, they’re designed for DEXCOOL and aluminium engines. The only problem I can find is that they also recommend draining the coolant every 24k miles to help maintain the pH balance.

I’m certainly not a fan of cutting corners, but I’m very tempted to try the tabs for now. I’m still angry about my financing deal on this truck. I don’t want to give up on it yet by trading it in. The rest of the truck is in great condition. I may have to get my Maserati coupe sooner than I expected.

Tags:

Posted in Uncategorized | No Comments »

Spring cleaning

Mar 22nd, 2009 by

It’s been a peaceful, warm weekend. Yesterday was breakfast, brunch, then naptime. Today I finished a project I started three weeks ago, cleaning the truck. Finished waxing it and taking clay to the baked on road crap. Ripped out the scanner and CB since I don’t ever use them and re-mounted the 857. A little green part no bigger than a Cheeto was what caused my tailgate to jam, and a little piece of that no bigger than my pinky nail is what caused it to fail.

Tags:

Posted in Uncategorized | No Comments »

Ethane, is there anything it can’t do

Mar 16th, 2009 by

This weekend I was reading about petrochemical feedstocks. Ethane (C2H6) is produced from natural gas. Ethane is used to produce ethylene. Ethylene is also used in plant biology, specifically to ripen things. So the next time you put bananas in a brown paper bag with other fruit to help it ripen, it’s the ethylene being offgassed from the banana that helps the ripening process. Neat.

Tags:

Posted in Uncategorized | No Comments »

New couch

Mar 10th, 2009 by

Couch delivered. Delivery guys hate me so much because it was difficult getting it into the apartment. Note to self, to get it out, walk it out horizontally with the back facing down, cushion side toward the kitchen, past the first counter top. Between the counter tops, stand it up on it end on a trolly, cushions facing the counter. Compress the cushions and push toward the door, feet against the wall. Inside the doorway, rotate toward the door, lift up off the dolly. Pick up and tilt out the front door. Lift over the corner of the stairwell.

Posted in Uncategorized | No Comments »

Out with the old, in with the new

Mar 9th, 2009 by

I gave up on black and ordered a light brown couch today. My buyer took possession of the futon today as well, so the livingroom is feeling empty. The light switch in my bathroom was arching and trying to burn down my apartment, so I went to Home Depot. I bought my 54 cent switch and also walked out with a patio bisto set. I wanted this last fall, but none were to be found. In the meantime, it’s taken the place of the futon!

Posted in Uncategorized | No Comments »

Furniture shopping

Mar 3rd, 2009 by

I am all about supporting the economy lately. I’ve been looking for a new couch lately to replace my futon. I went to Star Furniture this weekend and spent over an hour sitting and laying on things. I found a Flexsteel couch I really liked, but found out tonight it would be a 120 day delay before I could get one delivered. This weekend I also spent 15-20 minutes laying on a Tempur-pedic mattress and actually felt good four a couple of hours afterwards. Found a good deal on Craigslist for one, so it’s been deciding between couch or mattress.

Posted in Uncategorized | No Comments »

More firewall hate

Feb 16th, 2009 by

Oh my god, this has gotten so hairy, unpredictable and complicated that it’s actually easier to write an expect script to log into a massive amount of firewalls to gather interface data.

Posted in Uncategorized | No Comments »

« Newer Posts - Older Posts »