Feed on
Posts
Comments

This has certainly been an adventurous move, mainly because of how open-ended it was, which then turned into “oh shit, what” when the snow storm hit Seattle. Despite the panic about last minute obstacles, so far they’ve been overcome. I’m not complaining, just what a ride!

I initially put off moving a week because I needed more time to pack and take care of things. At first, because the leasing office at the new place was closed on Sunday, I pondered loading and leaving Friday evening. Then the big snow storm hit Wednesday and buried the PNW under several inches of snow. The roads weren’t looking good so I considered loading Saturday and spending two nights on the road.

Then I realized Oregon and northern California got hit with the storm too and started checking the weather+road conditions there. The Siskiyou Pass in southern Oregon became interesting because it was frequently snowing heavily. At least once I-5 was completely closed and traffic diverted. There was a severe weather hazard Friday with chains mandatory, which cleared off to a warning Saturday so it looked like it was getting better.

By now I was completely packed and was just waiting. I finally decided to pick up the U-Haul on Saturday. Here’s where allll the plans when out the window and I had to start improvising everything. Because I didn’t know when the roads would be clear I hadn’t scheduled movers to help load the truck. I hopped on Yelp and started down the list, asking who was available Sunday. Most were booked weeks out. I found a crew who booked me at 1 PM but said they’d come earlier if they could.

8 AM the next morning I get a call that they’re on their way! I snapped up off the couch and started getting ready. It took a bit longer than I expected, we finished loading the U-Haul after 1 PM. They did a superb job at wrapping my furniture and packing the truck very well. Everything fit into a 14′ truck with room to spare. The only things I left behind were my pile of Rackable servers and closetserver, mainly because I didn’t have sturdy packing materials for them. A few last minute meals and errands I left Redmond at 5 PM. By now there was a travel hazard on the mountain pass because of heavy snowfall. Because I left so late I’d have to spend the night in Medford, OR anyways.

Felix was well behaved on the trip for the most part. He would complain for several minutes after I put him in the carrier and would occasionally get restless and very vocal. All the times I let him loose in the cab, he hid behind the driver’s seat.

The drive took way longer than expected even with generous time estimates. I rolled into Medford at 4 AM, on a very cold morning. Signs all along I-5 warned of severe weather up ahead in Ashland. I holed up in a Motel 6 for a few hours and slept. My previous estimates said I needed to be there by 2 AM, leave at 7 AM, in order to have time to get the lease signed, get keys, and get unloaded all on Monday.

At 8 AM Monday I woke up and checked the pass conditions. The snow pack had broken up and chains weren’t required anymore, so off I went. The mountain looked amazing, it was blanketed by a layer of fog just thick enough to blot out the sun. All of the big evergreen trees were sagging with snow, everything was white. The sand on the pavement was practically the only color. Slow going but eventually made it over.

I passed by Mount Shasta on into Yreka. Every time I’ve been here, it looks different. I’ve been through at night, at winter sunset, summer sunset, and now covered in snow. Again, amazing scenery along Mt Shasta. Felix was being behaving much better today. Initially he refused to go in the carrier and complained for a while after, but eventually he just laid there snoozing in the sun not saying a thing the entire drive. Even on a bumpy road where he was clearly getting bounced around, he still slept like a champ.

On the drive down I was again on Yelp looking up movers and found a crew who could unload the truck at 1 PM. The drive was still taking forever and I didn’t roll into Fremont until 4 PM, so the movers rescheduled for Tuesday morning at 9 AM. I had the night in my empty apartment and it was kind of nice to sprawl out in the emptiness, I didn’t miss my stuff (but worried that somebody would break into the truck on the street).

Tuesday morning we almost got everything in the apartment — except the couch. It was just too big for the sharp turn to get into the doorway. Because the apartment is on the 2nd floor, going up over the balcony would require serious manpower; they estimated at least six men, yikes. Instead I just loaded it back on the truck to deal with later.

Today (Wednesday) I started calling around and found a storage unit, now worrying about who I’m going to get to help me unload this sucker. I had joked earlier about getting day laborers and now it was looking like reality. I decided I could at least get it off the truck by myself, so if I had a trolley I was gold. The storage place had a big elevator and carts where I could just wheel it around, perfect. Initially I started unloading it myself and got it on a cart to wheel it in, then the owner came out to give me a hand. I’m really grateful he did, as it made it much easier; I’m not sure if I could’ve stood it up through the door by myself.

Nowwwww the next chapter in the story is getting back to Redmond to pick up my truck. Did I mention a SECOND drive, this time non-stop?

Moving to California

So I’m moving to California to seek out new opportunities. This should be no surprise to anyone who’s known me, since I’ve rattled on about it incessantly in one form or another (not to mention basically driving/flying down every chance I get). I’ve wanted to live there my entire career since that’s where the action always was and it’s a beautiful place, but it was always a big jump from wherever I was. I almost moved there last summer but instead landed in Redmond; I couldn’t really afford to live there for my remote gig and I didn’t know anyone. Since then things have changed, and I’m in a better position to try it and I’m well motivated.

Finding a new place was tricky. I have no idea where I’d spend the majority of my time there, and it turns out the Bay Area is pretty big. I swear there’s time/distance dilation there. As a random example, traveling one way it only took five minutes to get to my destination, but going the other direction seemed to take 30 (even without traffic). I really love the hills and park system of the East Bay, but any job I work at would very likely be on the peninsula somewhere or in the South Bay. Over New Years vacation I drove around a lot, checking out areas of interests and neighborhoods. At first I thought San Mateo was a good compromise, but the rental prices still made me sick.

On a friend’s advice I looked at Fremont on the day I left and liked what I saw. Really nice area and rentals were relatively cheaaaaap, although nightlife is non-existent from what I can tell. I decided I probably wouldn’t be going into SF all the time so the extra $$$ to live closer wasn’t worth it. Fremont is also a bit sprawl-y, but it’s good since I have a large vehicle. I looked at the traffic maps and decided was acceptable. I flew back down the next weekend to look at more units and finally signed on one. Bonus: it faces Mission Peak. It remains to be seen if this where I want to stay long term, traffic/social/life/work may dictate otherwise, so at least I was able to get a shorter lease for when I want to uproot.

I thought about trying to move this weekend, but looking over what needed to be done and getting quotes from movers, it just wasn’t possible. Plus I wanted to hang out with my peeps more before I split town! Looks like I’ll be moving myself, by the time I rent a truck, drive down, fly back, drive my pickup down, I’m still saving 50% off of what using movers would cost me. It may all be a horrible idea to not use movers, we’ll see! The plan is to move this upcoming weekend, but the details are still in flux.

It’s all exciting and surreal and nerve wracking!

On a less serious note, I miss you Continental Airlines. You never angered me, not even once, in 90,000 miles of travel which is pretty amazing. Being from Austin made it super easy to catch your flights out of Houston. Your express customs lanes and priority baggage loading were so awesome! Even international coach was great and domestic is okay. I decided I’m going to have to make an effort to fly you more (but you’re still a cog in the whole air travel experience). [Well okay, there was that one time you made me run through EWR to catch my flight to AMS. And that one time you delayed me 6 hours going to IAD, but you gave me free first class.]

Alaskan Air, you’re a super easy way to get up and down the coast and have been a life saver at last minute flights but during descent on every flight you make me sit through a solid 3 minutes of scripted marketing crap about your Visa and miles program over the PA that I can’t drown out or ignore. I’ve already paid to get in your airplane, leave me alone. If my SEA->OAK flight is 30 minutes long, that means I spent 10% of it listening to you talk, right?

Delta, I use you because I have to whenever I have to go to Oklahoma. You make me watch a long ass seat-back video of all your employees thanking me, then an annoying smug woman with cheap humor doing the safety lecture, followed by a bunch of commercials that I can’t turn off.

TSA in Tulsa today, wow, you were the most frisky pat down I’ve ever received since you rolled out the millimeter wave machines. I still feel the “back surface” of your hands groping my crotch and running down my leg. shiver. I pondered how fake/real offended I was about it but I’d rather blog about it instead.

American, well, not so much to say good or bad about you. I think some of your planes are older than me.

It was really fantastic to go back home for Thanksgiving, but I’m remembering I can pretend to make a difference by voting with my wallet.

The past 24 hours have been surreal. I unexpectedly had to put Patch to sleep this morning.

Over the past month or two Patch had slowly been losing weight and being more reserved. I always attributed this to Felix bullying her, since he was getting big enough to push her out of the way at her food bowl and would play rough with her. Whatever she was doing, he had to be right there. I thought I was making progress, I gave her food in the bedroom and trained him to leave her alone while she ate. I sprayed him with a water bottle when he played rough with her. I separated them during the day while I was at work to see if that helped, and spent extra time with her alone. I was trying different wet foods and was succeeding in getting her to eat a little meat with the gravy she always loved to lap up. I generously gave her treats which she would always eat, hoping the extra calories would help her gain weight. I was so excited when one night when I was able to coax her into playing, she hadn’t been that playful since getting Felix!

Day-to-day she would always meow-nyaaaah at me as usual, hop up and drink from the sink while I was getting ready in the morning, go out on the balcony, hop in bed, sit on the couch with me, and come out to see my visitors, so it was hard to suspect that anything else was wrong. The only thing different was that she walked a little more cautiously and was in her cat bed more, which is where she usually retreated when the kitten was playing.

Then last week I couldn’t get her interested in treats. Her ears would perk up and she’d come to me when I’d say “Patch, treats!” but she didn’t do more than sniff them. I rarely saw her at the food dish, and I could barely get her to eat more than a tablespoon of gravy at a time from her food. Saturday afternoon I wanted to get to the bottom of it so I made an appointment at the DVM for Monday morning.

Yesterday (Monday) morning the vet agreed she was in rough shape, took some blood for testing, and gave her some antidepressants to help stimulate her appetite and administered some LRS for dehydration. We discussed the possibility of inserting a feeding tube as a last resort if the bloodwork was clean. As soon as I took her home and let her out, she started eating a good size dollop of the A/D pâté they gave me (and she has never showed interest in pâté food). That was very promising.

I had left her alone in the bedroom while I went to work so Felix wouldn’t bother her after the vet visit. When I came home she looked like she hadn’t moved all day long. She hadn’t eaten anything nor could I get her interested in food again. I sat on the bed and put her on my lap, she just laid there resting her head on my leg. This was unusual too since she very rarely laid on my lap. I stayed there with her for a couple of hours and in the meantime I had missed some frantic phone calls from the vet.

The bloodwork came back and it showed she was extremely anemic and there was no sign that she was generating any red blood cells. The vet referred me to another hospital and urged me to go right then, as she may need an emergency blood transfusion and would need the help of an internal medicine team to figure out what was wrong.

Around 9 PM I took Patch to Seattle Veterinary Specialists. After examining her and reviewing test results with an internist, the new diagnoses was very bleak. Something had stopped her bone marrow completely and something was destroying her red blood cells. There were four possible diagnosis for the loss: it could be FIP, a gastrointestinal ulcer, neoplasia, or an immune system deficiency. If it was one type of infection, it could be treated with antibiotics but she’d have it forever and it could flare up again. They said none of these were really a strong fit based on her symptoms, it would take more tests to confirm. Unfortunately, the tests would take several days, if not weeks, to confirm. She’d need a blood transfusion, hospital stay, ultrasounds, and a wide variety of treatments to make it that long. Worse yet, even if correctly identified three of those would be terminal and nothing would guarantee her marrow would start working again. On her own, she would only have a few days.

I couldn’t bear the thought of putting her through a bunch of work and stress just to lose her later. The vet said I didn’t have to do anything right then, I could take her home and decide what to do. I decided to bring her home to have one last night with her to say goodbye. The day started off so different, and now this is how it was ending. I curled up on the bed with her, talked to her, and cried all night long. At this point she was not moving beyond just rolling to lay on her other side. After a couple of hours she still seemed restless so I gave her some of the pain meds they gave me. She laid her head down and slept the rest of the night. I wondered when I’d be taking her back the next day.

This morning (Tuesday) I woke up and she was in even worse shape. She hadn’t moved, she was just sprawled out flat on the comforter not making any effort to remain upright. I was afraid she had already died since she didn’t initially respond to me saying her name or petting her. She finally perked up a bit, but it was clear she was exhausted and didn’t have long. Picking her up, she was completely limp and didn’t offer any resistance. I got dressed and took her back to SVS because I knew it would be a quiet environment and I could get in quickly. She said a few deep throaty “nyaaaaas” on the way to the truck but otherwise didn’t move much. She laid limp in my arms as the doctor administered the drugs. She went quickly and quietly.

It’s been really hard. She was my buddy that kept me company, always laying with me when I was working, sleeping next to me, exploring everything around her. Always well behaved, soft and delicate. When I left the hospital I broke down crying in the parking lot when I reached in my pocket for my keys and felt her collar, still warm from her wearing it ten minutes earlier.

It all feels so sudden. One day she’s pawing at a string, not showing any signs of distress, and now she’s gone. In retrospect, it may not have been behavioral issues leading to her weight loss, she may have been ill for quite some time, or maybe both. It’s possible the reason she was being bullied was because she didn’t have the energy to fight back. I feel guilty for not taking her in sooner, but I’m told it wouldn’t have really mattered; she’d still have a terminal affliction and the goal would be to keep her comfortable for however long she had. I guess at least I’d known it was coming. I hope I gave her all the attention she wanted. It feels weird walking into to the bedroom and not seeing her happily snoozing on the foot of the bed or on a pillow. When I scan the living room I expect her to be on her red cat bed in the corner watching me or on top of the cat tree, and she’s not there.

Rest in peace my sweet kitty, I love you and I’ll miss you.

OpenNMS Experience Redux

I’m doing my second greenfield OpenNMS deployment, and so far it’s going a lot smoother than last time. This time it’s server-centric which is a whole new can of worms to figure out. I don’t yet know how feasible it will be to monitor services like Exchange or SQL Server.

My last setup never really got off the runway because I didn’t have enough random write I/O bandwidth to do massive graphing on all of my switch ports and I left before I could beg for new hardware. Since then I’ve learned there are just flat out physical limits to disks, you can only do so many things per second. Even SSDs have absolute limits on how many operations/second they can do. The way to win is to persist everything to a giant ramdisk, then copy all that to your real disks every X minutes you feel safe living without. All of a sudden those thousands random 8k writes to JRB files every second turn into one giant sequential write to disk. It copies surprisingly very quickly. It’s not that far fetched, you can easily put 64 GB of RAM in a server for less money than buying a new 16 drive system and it’ll give you hella performance over safety.

Anyways, this time I’m going from 1.8.3 to 1.9.92, and it’s considerably better! Not only were two of my long standing gripes (adding relativetime to graphs and TableTracker writing to stderr) fixed, and a slew of bugs fixed, but it just feels like a much nicer piece of software now. I kept track of the release notes over the summer and it’s obvious there were a lot of good updates going in and progress being made. I’m sure there are many things that are fixed that weren’t completely obvious to me as being broken. Props to all those guys. One of the big things I wanted then was IPv6 support, but now I don’t have a v6 network to monitor. :(

A few things I’ve learned either through experience (or things that were obvious to other people finally made sense to me):

– You’ll be happier and scale larger if you use ramdisks

– The services (e.g. ICMP, HTTP, SNMP) listed in the node ‘Availability’ table is a completely independent operation of the process for data collection (i.e. graphing). You can graph things not listed there, and you can monitor availability without graphing it.  I’ve known this for a while but it seems like it’s lost on a lot of people still.

– Maybe I’m stupid but I was always lead to believe there were two types of categories being used, one in categories.xml (the home page) and another in surveillance views. I think this was because of poor examples. The default categories configuration refers to things as rules instead of filters, and there’s nothing in there that leads you to believe you can do “categoryName == Servers” and/or even “nodelabel like ‘%city%'”.

– If you want to ping/monitor external sites: create a new provisioning group, strip down all the collectors down to ICMP and HTTP, add a policy to categorize everything as ‘external-monitors’. Go setup a poller package to enable StrafePing for “<filter><![CDATA[(IPADDR != ‘0.0.0.0’) & (categoryName == ‘external-site-monitor’) ]]></filter>”. You’re a good citizen for not repeatedly over-discovery-scanning some poor website and your NOC peeps will understand from the category why amazon.com is in the node list.

– Poller packages are additive, it doesn’t have to be all-or-nothing. If an IP address or category matches more than one poller package, both get applied. This was another fascinating realization (by accident of course) because one of my big peeves was figuring out a easy way to get really custom polling without cut-n-pasting huge swaths of configuration.

– Automatically adding categories to new nodes via a foreign source policy is the way to go. You can rig it up so that if you have different provisioning groups per city/site, everything can be tagged with a site category by default without having to select from a drop-down. Then, based on the hostname/node label, add more categories to indicate its role, etc. The OpenNMS Provisioning PDF is an outstanding reference for doing policies, it is much clearer and has better examples than the provisiond wiki page.

– Tip: when creating a foreign policy and you want to use a regex, prepend the expression with a ~ (tilde), e.g. “~.*” or “~^hostname\d+.*”. I don’t think this was explained in any documentation and I happened to stumble upon it on a random mailing list. Otherwise your stuff will not match and you won’t know why!

– Integrating the web UI to auth/authorize off of Active Directory via LDAP looks hairy and makes me want to drink more before trying it. It looks like there were a few examples thrown into the wiki but who knows if they work, they certainly don’t look simple and easy. A simple “hello world” example to just to prove the thing works in a basic capacity would be nice.

– When all else fails, think how Java would think.

All this being said, realize it’s definitely not an application you casually install with “yum install opennms” and walk away. It takes quite of bit of knowledge to get it working and thought on how you want to organize things to benefit both operational and business-level needs. Even then, it seems to take several iterations of “I want to do this” to “the OpenNMS way is impossible or has to be done this way” and then “okay, redesigned the need with that in mind, try it this way”. Otherwise you’ll be doing hacky complicated things that get out of hand and are impossible to manage.

My new big gripe is that OpenNMS as a concept has serious shortcomings when it comes to monitoring disk and storage resources. Maybe the big kids are monitoring 50,000 cable modems (which it works really well for) plus a token web farm and there just isn’t much need for an extensive storage interface. I mean I can certainly graph space and disk I/O, but I would really love to look at a node’s page and be at a glance able to see a short numerical list of space used/free on all my partitions and individual RAID statuses. Some of us have 50,000 hard disks and exibibabblebytes to monitor instead! Same with BGP peering, I’d love to have been able to look at the router’s node page and see at a glance how many of my BGP sessions were up/down (peering routers easily had hundreds). Server load balancing too, now that I think about it more. You can improvise custom Availability checks for a few one-offs, but they’re usually static to one device’s configuration. Even if I was motivated to take a crack at writing my own, it doesn’t seem like a place with any hooks to do something custom.

I don’t think that OpenNMS itself is the problem, but just that all the tools for getting data about storage and RAIDs suck balls. Just this week I learned that there’s a patch in net-snmp 5.3 on CentOS 5.7 that lets it support filesystems > 4 TB, but that patch is not in net-snmp 5.4 on OpenIndiana. SNMP access to PERC status? ha ha. ZFS metrics via SNMP? ha ha ha. NFS service poller? doesn’t exist. iSCSI or FCoE ping? what what

The only way to get around this is to rig up some middleware solution to do custom aggregation of this information and then present it in a standardized way to OpenNMS. Even then, there still needs to be an interface (like IP addresses/interfaces) to show it.

The other long standing gripe is the need for an easy way to aggregate a bunch of performance data from multiple nodes into one large graph. Suppose I have a datacenter full of power strips and I want to have an graph that shows total power consumption, or multiple transit links to the same provider, total disk throughput across many disks. KSC reports can’t do this (that I know of) and the alternative is rigging up a virtual node and symlinking things all over the place then creating a custom graph report. blah.

The last time I looked at thresholding, alarms and notifications it was so complicated and dreary it made me want to drink. It’s most certainly not as easy as throwing a 15 line file in /etc/nagios/objects. In fact, I’m keeping Nagios around for exactly that reason. Here’s to hoping it makes more sense this time!

# zpool list
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
rpool    136G  32.0G   104G    23%  1.00x  ONLINE  -
tank    47.1T  45.9T  1.26T    97%  1.00x  ONLINE  -
tank10  48.9T  48.1T   870G    98%  1.00x  ONLINE  -
tank11  48.9T  48.2T   783G    98%  1.00x  ONLINE  -
tank12  48.9T  48.0T   922G    98%  1.00x  ONLINE  -
tank13  48.9T  47.8T  1.12T    97%  1.00x  ONLINE  -
tank14  48.9T  47.8T  1.15T    97%  1.00x  ONLINE  -
tank15  48.9T  47.8T  1.15T    97%  1.00x  ONLINE  -
tank16  48.9T  48.2T   783G    98%  1.00x  ONLINE  -
tank17  48.9T  47.9T  1.01T    97%  1.00x  ONLINE  -
tank18  48.9T  48.2T   783G    98%  1.00x  ONLINE  -
tank19  48.9T  48.2T   783G    98%  1.00x  ONLINE  -
tank2   48.9T  47.7T  1.22T    97%  1.00x  ONLINE  -
tank20  48.9T  47.8T  1.10T    97%  1.00x  ONLINE  -
tank21  48.9T  46.8T  2.13T    95%  1.00x  ONLINE  -
tank22  48.9T   320K  48.9T     0%  1.00x  ONLINE  -
tank3   48.9T  47.6T  1.38T    97%  1.00x  ONLINE  -
tank4   48.9T  47.9T  1.07T    97%  1.00x  ONLINE  -
tank5   38.1T  37.4T   697G    98%  1.00x  ONLINE  -
tank6   48.9T  48.2T   783G    98%  1.00x  ONLINE  -
tank7   48.9T  47.3T  1.61T    96%  1.00x  ONLINE  -
tank8   48.9T  45.6T  3.32T    93%  1.00x  ONLINE  -
tank9   55.9T  54.6T  1.38T    97%  1.00x  ONLINE  -

LIFE magazine

Sunday I was wandering around Pike Place Market and wound up in the paper store. I was flipping through their racks of LIFE magazine and was fascinated by the headlines/covers of the issues from the 60s. There were several issues about space, Vietnam, McNamera, Nixon, and Johnson. It’s one thing to read so much about those, but even more awesome to actually hold something in hand. It was also odd to realize this was one of the primary ways that news and entertainment was spread then, now it’s all on the web.

Time Capsule

I just dug my old Dell desktop out of the closet to see if there’s anything on it I want to keep before tossing it.

Looks like it was last fired up in September 2006 and I was in ham mode, messing with MixW. It’s running Windows 2000, Firefox 1.5.0.7, QuickTime 7.0.4, Yahoo! Messenger 7.0.0.120, Flight Simulator 2002. Pentium III 1 GHz, 512 MB RAM, 40 GB hard drive, fun times!.

I remember when Windows 2000 was cool, it’s soft Verdana font soothed my wounds from Windows 98. Now it looks all sorts of jagged and unrefined.

Felix

I have a new kitty in the family, 2 months old. His shelter name is Shaun, but I’m not sure if I’m keeping that. (Update 9/17: I’m calling him Felix) He’s very vocal, spent all night meowing like murder was happening. He’s a little better today, but still gets very lonely. I laid in the office floor with him until 2-3 AM to quiet him down. I forgot how much energy kittens have, when he plays he just goes all out plowing into everything. I was also surprised at how much to feed a kitten; 1 can per pound body weight per day, so that works out to basically 1 can, 3 times a day.

Patch is somewhat concerned by the noises coming from the office, but otherwise seems indifferent. Today I gave them a little introduction, holding Shaun in the same room while Patch looked on.

In the process of trying to get up to keep Shaun from running out of the office, I stumped my little toe really hard on the bookcase. It smarted for a while and started swelling up. After dinner it was purple and blue on both sides. Today the swelling and discoloration spread farther, enough to make me go to urgent care. They took some x-rays and decided it wasn’t broken, just sprained. The doctor showed me how to split it with tape to my other toes and said that’s all that should be done with it.

Last month I finally decided to re-join the modern age and get cable. I’ve been waiting for FiOS from Frontier ever since I moved in, and my apartment building was finally wired up for it. I was flabbergasted when they told me the install charge for TV+internet over FiOS was $500, but if I just wanted internet, there was no install fee. It turns out, according to the internet, Frontier is losing money left and right on channels and desperately wants to get out of the cable TV business.

I flipped over to comcast.com. I didn’t have a high opinion of them because of the data caps, but just wanted cable. I was able to put in my order within a few minutes without having to call. When I got my DVR/cable box and hooked it up, it didn’t work. I spent a while troubleshooting and got nowhere. I dreaded having to call and wait around for a tech to be dispatched or something (and I was traveling a lot), so I put it off. I put it off for a couple of weeks.

When I finally called and navigated through the IVR tree, it was a surprisingly quick “oh let me send a reset signal” and that’s all it took to make it come to life. Less than three minutes on the phone. After a couple of hours I still wasn’t getting all of my channels. I needed to call again, but was still dreading hassle and put it off.

When I called again at 1 AM this morning, I got a female apparently in Seattle, who was very friendly and sincere. She reset my cable box and walked me through verifying channel subscriptions. Then she credited me a whole month of service because it had never worked since install. Less than ten minutes too, no too bad!

You win this time Comcast, but I’m keeping my eye on you.

« Newer Posts - Older Posts »