Feed on
Posts
Comments

Engine coolant drama

The truck has been in the shop all week and I’ve realized it’s a great way to save money and diet. All of a sudden, shopping and cheeseburgers are not convienient. Lots of bike riding in the meantime. This morning I feasted at Taco Shack and loaded my backpack full of groceries from Randalls. This afternoon I went for a cooker ride around the neighborhood. My weight is varying wildly, which I blame on the heat.

Things aren’t looking well for the truck. They still haven’t found the leak, but discovered cylinders on the same side of my busted O2 sensor is running hotter than the other. The concern now is that the initial lifter rattling is a sign that the engine has a sludge problem, due to coolant seeping in somewhere. Even if the leak is fixed, is it too late? We don’t know yet how bad it is, nor if it is recoverable.

I haven’t been able to decide what I want to do with it. I got a couple of quotes on new & used engines. A crate engine+install is around $4200, whereas a used engine with 41k miles is $2900 installed. I keep coming up with different numbers for replacing heads, could be anywhere from $1000 to $3000. At that point it’s worth the extra grand to put a new engine in.

Alternatively, I could just put that money toward buying a new one. I really, really do not want another truck payment. 2009 Silverados are $32k, wtf? 2007-2008s are still in the mid-$20s. I’ve pondered buying a 4WD Z71 if I did get another, since I’ve been in a few spots where that’d be handy. Ironically, I found a 2006 Z71 at a Mercedes/Lexus/Maserati dealer in Dallas that was a trade-in.

Right now I’m leaning toward a new engine, just to get it over with. If the shop comes back and says they can make it go away with a $1k head gasket replacement, that works for me!

I’m anti-vehicle this week. The bicycle is doing very well at getting me around. Unfortunately it takes careful effort to go someplace without being a sweaty, smelly mess. I have a horrible craving for a bacon cheeseburger at Roaring Fork or Trudy’s.

I learned two things this week. The first being the reason why my 2611 broke at home after a day. I had an ACL on my egress interface that blocked DHCP traffic so my router could never renew its lease.

The other was how to get back to my apartment from 360 on a bicycle. If I go to the top of 360 & 183 and take Jollyville south, this meets Mesa which is only 4-5 miles from home via the back way. A mile of gradual climing, then relatively flat on Mesa. It’s a longer distance, but it’s certainly easier than a 202 b.p.m. pulse, crank-bending frontal assault on the hill that is my driveway.

For lovers of storing IP addresses as integers in MySQL tables, there’s a snag when doing this for IPv6 addresses. A 128-bit number is too big to stuff into a BIGINT type. All Fs integer equivalent is 340,282,366,920,938,463,463,374,607,431,768,211,455. An unsigned BIGINT tops out at 18,446,744,073,709,551,615.

These are your options:

  • Store it as a VARCHAR(39) in full ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff form. Which of course, is a horrible idea since you can’t easily sort and it’s prone to garbage-in, garbage-out problems because of the various forms of valid shorthand. (Admittedly, I’ve done this in places with v4 addresses and have almost fully recanted in my code!)
  • Denial. “IPv6 is just a fad.” “I’ll have a new job before I have to deal with it.”
  • Wring your hands. File countless MySQL bug reports asking for native IPv6 and IPv6-capable INET_ATON, INET_NTOA functions. (The former still needs to be added in MySQL 5.x)
  • Fan the flames. Proclaim you never had this problem in PostgreSQL because of native data types that handle IPv6 and CIDR, and all MySQL ninnies should escape their new Oracle overlords and convert.
  • Split up a v6 address at the 64-bit boundary, stuff the two halves into dual BIGINT columns. You could even go as far as saying if one BIGINT column is null, then it must be a v4 address.
  • Store it in a DECIMAL(39). You’ll still need to invent a way to convert the address to integer and back. To do this in perl, Net::IP’s intip() method will at least give you IPv6 -> integer. To get it back, you’ll have to roll your own inet_ntoa6 function. Don’t forget Math::BigInt. Alternatively, here’s some MySQL stored functions for INET_ATON6 and INET_NTOA6. You can either use these directly, or use them as inspiration for your own inet_ntoa6().

Since I have apps that require storing v6 addresses now, I’ve chosen the latter.

5,000 mile 6to4 journey

So here’s a traceroute to the anycast 6to4 address from my parents’ place. It winds up in Frankfurt, Germany! Their upstream, MBO is multihomed to Level3 and Sprint. I assume they’re defaulting to Level3, since the path to the Sprint 6to4 is 120 ms.

traceroute to 192.88.99.1 (192.88.99.1), 64 hops max, 40 byte packets
 1  * * *
 2  192.168.1.1 (192.168.1.1)  2.534 ms  1.691 ms  1.416 ms
 3  invalid.mannford.ok.mbo.net (64.111.243.1)  52.790 ms  50.316 ms  38.353 ms
 4  10.255.2.1 (10.255.2.1)  43.310 ms  50.755 ms  48.458 ms
 5  10.255.0.1 (10.255.0.1)  46.152 ms  44.434 ms  46.582 ms
 6  core7200-1-ser-5-0.mannford.ok.mbo.net (216.150.101.13)  46.222 ms  45.412 ms  51.211 ms
 7  core7513-1.mbo.net (216.150.100.31)  50.435 ms  47.729 ms  50.265 ms
 8  gw-espire.mbo.net (216.150.100.45)  48.338 ms  58.902 ms  48.981 ms
 9  so-10-2.hsa1.Dallas1.Level3.net (209.246.148.209)  59.113 ms  62.695 ms  64.900 ms
10  vlan51.csw1.Dallas1.Level3.net (4.68.122.30)  79.931 ms  66.449 ms  76.519 ms
11  ae-63-63.ebr3.Dallas1.Level3.net (4.69.136.153)  55.804 ms  60.723 ms  71.074 ms
12  ae-7.ebr3.Atlanta2.Level3.net (4.69.134.22)  77.580 ms *  79.353 ms
13  ae-2.ebr1.Washington1.Level3.net (4.69.132.86)  95.636 ms  106.422 ms  98.889 ms
14  ae-91-91.csw4.Washington1.Level3.net (4.69.134.142)  102.153 ms  85.858 ms  92.550 ms
15  ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157)  111.244 ms 113.997 ms  117.641 ms
16  ae-43-43.ebr2.Frankfurt1.Level3.net (4.69.137.57)  181.691 ms 
    ae-42-42.ebr2.Frankfurt1.Level3.net (4.69.137.53)  181.550 ms 
    ae-43-43.ebr2.Frankfurt1.Level3.net (4.69.137.57)  183.324 ms
17  ae-92-92.csw4.Frankfurt1.Level3.net (4.69.140.30)  187.870 ms 
    ae-62-62.csw1.Frankfurt1.Level3.net (4.69.140.18)  181.219 ms 
    ae-72-72.csw2.Frankfurt1.Level3.net (4.69.140.22)  186.367 ms
18  ae-2-79.edge3.Frankfurt1.Level3.net (4.68.23.75)  176.536 ms 
    ae-1-69.edge3.Frankfurt1.Level3.net (4.68.23.11)  179.577 ms  186.361 ms
19  tenge-1-3.cr2.NBG1.content-core.net (212.162.19.34)  193.185 ms 187.075 ms  188.304 ms
20  Tenge1-3-57.cr2.FRA3.content-core.net (212.123.123.202)  196.904 ms 211.622 ms *

A big thing that I see a lot in the content and service provider world, when some outage or negative event happens the line is always “we apologize for the inconvenience.” This is another saying that irritates me to no end. I say this because over the past 36 hours tifa’s host, The Planet, had a couple of outages. I noticed in all their posts they never once mentioned inconvenience.

Every hosting customer of every company I’ve worked for are almost all universally businesses, using our network connections or servers to run their business. When something happens to their infrastructure, it’s not a mere inconvenience. It’s a risk with a measurable negative effect, that is in essence, causing pain to the bottom line.

Going to the store and finding there’s no milk is an inconvienience. Making plans with a friend who later flakes is an inconvienience. Not being able to access your online banking or buy tickets online is an inconvienince to the consumer, but not for the company who’s job is to provide the online services. For some of them, their entire business is built on your mouse clicks, SMS messages, telephone calls and eyeballs.

I personally will not insult your intelligence by apologizing for an inconvienience to your business, if you won’t insult mine by claiming an event stopped you from earning part of your $500 billion annual revenue on your Windows 2000 server (true story).

The corollary to this is, if you really do have a healthy revenue stream that depends on your internet connection and servers, then for god’s sake, invest in it! Let’s be honest, things can and will happen. It’s basic risk management. People will make honest mistakes, automated systems will be mis-programed, power supplies fail. There’s only so much you can do. The technology and know how to help is certainly out there.

I’ve seen far too many solutions riddled with single-points-of-failures that could’ve survived with even basic things such as bonded/teamed NICs in servers or redundant firewalls. Hell, even doing backups and actually practicing restores is worthwhile.

Otherwise, I have zero sympathy when your only Windows 2000 server is down for patches or when human error disconnects something, and you claim you’re losing millions. (another true story)

Learning about IPv6

A few observations with IPv6 fiddling all day long:

IPv6 peering is going to be crazy important. Only a couple of our transit providers, Tiscali and Verizon (f/k/a UUnet) will give us native/dual-stack transit. To help ensure independent connectivity we’ll have to turn up v6 peering at our various peering points. With IPv4 the driving factor is not so much connectivity as reducing transit costs, as we have several different v4 paths to a destination. To use ‘Google over IPv6’, where they will serve up an AAAA record for www.google.com for your network, you had better have good v6 connectivity to Google to ensure it’s reachable at all times.

A /32 is a crazy amount of address space. The popular ISP allocation practice now is to give /48 or /56 to each customer. A /32 is 2^32 (4.2 billion) /64s, or an entire IPv4 internet worth of /64s. And there’s 2^32 /32s.

IPv6 PTR records is a lot of decimal places. Hurricane routed me a /64, for which I delegated reverse DNS to myself. When setting up the zone file, you can’t leave out the trailing zeros. This results in:

[bwann@tifa ~]$ host tifa.ipv6.wann.net
tifa.ipv6.wann.net has IPv6 address 2001:470:1f0f:6c4::2

[bwann@tifa ~]$ host 2001:470:1f0f:6c4::2
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.4.c.6.0.f.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa 
domain name pointer tifa.ipv6.wann.net.

Fortunately since I’m delegated only a /64, that saves typing out 16 digits in the zone file:

2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0  IN     PTR     tifa.ipv6.wann.net.

Having lots of zeros in the address takes out a lot of grief, since you can leave off leading zeros and use :: shorthand.

Since an ASA doesn’t do plain GRE tunnels, I brought home my 2620 and will use that to run a v6 tunnel to HE for my entire home network. I still need to get my hands on an ASA for firewalling purposes, since it handles v6 directly. This should give me a feel for how auto configuration works.

Now I’m trying to find the biggest websites that serve up AAAA records. I keep hoping some nerd at Facebook, MySpace or CNN has a small setup somewhere. Haven’t found anything terribly exciting so far. This leads me to believe we have a lot of work in the content provider space to get content accessible via v6. It’s obviously a chicken/egg scenario, but I think solving the server side is a slightly easier problem.

Dave Meyer gave a somewhat frightening talk at NANOG 45 about how dual stack has failed us. The original model around 2000 was to start assigning IPv4 + IPv6 addresses to everything. Over time more and more things would run dual stack, eventually being the majority, just as IPv4 space was exhausted.

The point he was trying to make was that we should’ve made dual stack a high priority back in 2000-2003, but we didn’t. IPv6 uptake hasn’t happened anywhere near on the scale it needs to, now IPv4 is almost exhausted, which prevents the use of future dual stack!

Instead we’ll start to see “carrier grade NAT” devices come along. A large ISP/cableco/telco could come along, put all their customers behind it and give them all IPv6 addresses. The CGN will take care of IPv4 accessibility. Vendors will love it since they’ll get to sell big iron. But, it may be nearly impossible to rip it out later and restore the natural end-to-end connectivity we’ve come to enjoy and expect.

The vast majority of our customers have load balancers in one shape or another. The problem is, not many load balancers actually support v6; the Cisco CSM and CSS don’t, the ACE maybe. Citrix Netscalers do, but they’re very expensive boxes. A comment I read last night was “do you seriously expect you’ll do so much v6 traffic you need to load balance?” While likely very true, not many of our customers will go for it. I don’t fully understand IPv6 anycasting to know if it’s a possible solution or not.

IPv6 in the home

I don’t know where March and April went, but they’re long gone.

Since we finally put in for a IPv6 allocation at work, I decided it was high time to finally get it working at home. It’s still a bit of voodoo but I finally have a few things working.

My home network is on NAT behind a Cisco PIX 501. This prevents me from using 6to4 or a tunnel broker, because it won’t pass IP protocol 41. I wound up using a Teredo tunnel from my MacBook (using Miredo for OS X) and a Linux box using miredo as well. It feels sort of sketchy because there’s not many public Teredo tunnel servers out there.

Being the nerd I am, after hitting http://ipv6.google.com/, I tried to get on an IPv6 irc server. This was a totally annoying endeavour at least. BitchX on my Linux box didn’t have IPv6 support compiled in, so I needed to rebuild the RPM. Unfortunately the box is running Fedora Core 6 which was apparently taken down from the usual fedora-updates repositories, and I needed a few things like ncurses-devel. Eventually I rebuilt it, then figured out I couldn’t connect.

Giving up, I turned to my MacBook. The Darwin ports of BitchX builds without IPv6 either, and I didn’t want to learn how to manually rebuild the package. I turned to Xirc, which was horrible; it would crash on connection attempt to an IPv6 host. I finally had success with Colloquy!

Then I turned my attention to tifa. She’s not running behind NAT, so I could easily play with my HE tunnels (with the downside of breaking IPv4 and locking myself out). For whatever reason I couldn’t add a ::/0 default route with iproute2, but I could with net-tools. Now I have a nailed up tunnel providing full access to the various services (i.e. web!) running, and a jumping off point for testing more things. Hooray!

It’s been a busy past couple of weeks! Victoria visited me for a week, then the same day she left my sister and her friend arrived for another five days. Lots of running around and heavy eating. I’m glad to have had visitors, it’s a welcome change. Weather has finally settled down and it’s sunny out.

Between all of them we went to Fredericksberg, Lady Bird Johnson Wildflower Center, Congress ave bats, the capitol, UT tower, and many eating establishments in between.

I finally took my truck to the mechanic again to work on my mystery coolant leak and my usual basket of other problems. It’s to the point I’m putting in a half gallon every couple of weeks. I would think it’d be fairly obvious where it’s at, but no luck today. Really, I’m not any better off than I was. I need to have them fill it full of dye, put it in neutral, put a brick on the gas pedal and run it at 3 K RPM for a couple of hours.

He did point me to a TSB on that engine that describes a casting defect on cylinder heads built by Castech which is known to cause coolant leaks with no evidence. It claims in the oil drain holes a porous crack can develop, letting coolant weep out. To verify, it involves getting inside the valve covers. Once inside it should be pretty obvious from what I can tell. The telltale manufacturer’s name should be on the top, and there should be clean/shiny spots around the crack while the rest of the head is oil-stained. There could also be coalesced oil+coolant “butterscotch” everywhere.

There’s a few obvious problems with this. It’ll take a couple of hours of labor just to get into it to see if it’s even made by Castech. It may not even be the cylinder head, it could be a head gasket or intake manifold gasket. Any of these could be weeping at high temperatures and it burns off without a trace. Once in there, if it is identified to be a leaking head, that means new heads. Worse, the vented coolant could’ve corroded my rods and lifters.

From a previous estimate, I know replacing a head gasket would be 10 hours of labor+parts at $1100. Fine. New GM Goodwrench cylinder heads are $344 each, so that’s at least $1100+$688+gaskets+oil+more labor. If it involves rods and lifters, it gets ridicuously expensive. It takes 16 lifters, OEM are $11 each, AC Delco are $44 each. By my estimate, the worst case is around $4k in parts and labor.

The upside to the visit, he’s very convinced my lingering P0135 code is caused by a flaky replacement Bosch O2 sensor I put in. As soon as I walked into the parts store and said I wanted a new sensor, the guy said “you must’ve just failed an inspection.” I mentioned I was replacing a Bosch, and both of them audibly tsk’ed. So, one expensive new O2 sensor later, we’ll see what happens.

Two other interesting notes from today. They inspected my brakes, my pads were at 60% wear — and they’re original parts, 160k miles on them. I clearly don’t stop much. The other, I discovered GM actually makes coolant seal tablets. More interesting, they’re made from ginger root and walnut shells. There’s a couple of tech notes where they recommend this, they’re designed for DEXCOOL and aluminium engines. The only problem I can find is that they also recommend draining the coolant every 24k miles to help maintain the pH balance.

I’m certainly not a fan of cutting corners, but I’m very tempted to try the tabs for now. I’m still angry about my financing deal on this truck. I don’t want to give up on it yet by trading it in. The rest of the truck is in great condition. I may have to get my Maserati coupe sooner than I expected.

Spring cleaning

It’s been a peaceful, warm weekend. Yesterday was breakfast, brunch, then naptime. Today I finished a project I started three weeks ago, cleaning the truck. Finished waxing it and taking clay to the baked on road crap. Ripped out the scanner and CB since I don’t ever use them and re-mounted the 857. A little green part no bigger than a Cheeto was what caused my tailgate to jam, and a little piece of that no bigger than my pinky nail is what caused it to fail.

This weekend I was reading about petrochemical feedstocks. Ethane (C2H6) is produced from natural gas. Ethane is used to produce ethylene. Ethylene is also used in plant biology, specifically to ripen things. So the next time you put bananas in a brown paper bag with other fruit to help it ripen, it’s the ethylene being offgassed from the banana that helps the ripening process. Neat.

« Newer Posts - Older Posts »