Feed on
Posts
Comments

Here’s one for the future troubleshooting seekers. I was testing IPv4 UEFI PXE booting a Supermicro A1SAi motherboard after applying the Atom 2550 fix and couldn’t get the thing to load the network bootstrap program (NBP). I’m not at all saying this is the only reason for hitting a PXE-E99 error, this is just what I hit today.

This blipped by on VGA console so fast I had to use slo-mo on my phone to capture it. (With the Atom 2550 fix, console redirection is lost, so no serial console scrollback).

Checking Media Presence.....
Media Present....
Start PXE over IPv4. Press ESC key to abort PXE boot.
Station IP address is 192.168.135.29

Server IP address 192.168.130.10
NBP filename is /efi64/syslinux.efi
NBP filesize is 0 Bytes
PXE-E99: Unexpected network error.

My PXE environment is pretty set it stone, it’s configuration managed and doesn’t get changed willy-nilly.

Things that came to mind:

  • Wrong EFI binary? Did this particular firmware want some weird 32-bit EFI program? Possible, but I had an near identical motherboard with an older firmware that loaded the x86-64 just fine. Plus I’m near 100% certain I’ve used this same binary on this same motherboard before it died.
  •  TFTP server broken? No, I was able to fetch syslinux.efi on several other machines on the same LAN just fine.
  •  UEFI not support routing / off-network TFTP server? This seemed feasible, but yet I’m absolutely certain I’ve PXE installed these on different LANs than the TFTP server.
  •  Does the UEFI firmware not like it when two DHCP servers respond? While both of my DHCP servers run identical configuration files and reservations, this was easy to test by stopping one temporarily. Didn’t help.

Tcpdumping on both the TFTP server and the router facing the LAN this motherboard was attached on showed there was absolutely no attempts on the wire made to fetch anything over TFTP.

The only thing left was DHCP. The “unexpected network error” got me digging into the DHCP responses being sent back:

01:35:21.075721 IP (tos 0x0, ttl 64, id 4266, offset 0, flags [DF], proto UDP (17), length 354)
    192.168.130.12.67 > 192.168.135.1.67: [bad udp cksum 0x8bbe -> 0xd3ef!] BOOTP/DHCP, Reply, length 326, hops 1, xid 0xafb187ee, Flags [Broadcast] (0x8000)
          Your-IP 192.168.135.29
          Server-IP 192.168.130.10
          Gateway-IP 192.168.135.1
          Client-Ethernet-Address 0c:c4:7a:32:27:e0
          file "/efi64/syslinux.efi"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 192.168.130.12
            Lease-Time Option 51, length 4: 86400
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Default-Gateway Option 3, length 4: 192.168.130.1             <<<<<<<<<<<<<<<<<<<<<<
            Domain-Name-Server Option 6, length 12: 192.168.135.1,192.168.130.10,192.168.130.12
            Hostname Option 12, length 16: "basic10.wann.net"
            Domain-Name Option 15, length 8: "wann.net"
            BR Option 28, length 4: 192.168.130.255                       <<<<<<<<<<<<<<<<<<<<<<
            NTP Option 42, length 8: 192.168.130.10,192.168.130.12

After a while what stood out to me was that my DHCP server was returning a Default-Gateway of 192.168.130.1 in the RFC1497 Options section compared to the Gateway-IP/GIADDR set earlier in the packet. Also the broadcast address being set in the Options too.

It would seem that I have a misconfiguration in my ISC DHCP server somewhere. That’s what’s causing the wrong gateway to be returned, and makes sense in that the UEFI loader gets an address but can’t reach anything off the local subnet. Apparently all the previous times I’ve PXE booted these systems I’ve always used IPv6 and never hit this problem until I tested with IPv4. I hacked on my config to temporarily set the gateway manually to what it should be for the test host, and it PXE booted off the network just fine, using the TFTP server on the other LAN.

As to how my DHCP server configuration is wrong, I haven’t figured it out yet. I never put time to understanding how classes and groups were supposed to go, my config looked something like this:

option ntp-servers 192.168.130.10, 192.168.130.12;
option domain-name-servers 192.168.130.10, 192.168.130.12;

subnet 192.168.130.0 netmask 255.255.255.0 {
  next-server 192.168.130.1;
  option routers 192.168.130.1;

  host a {
     hardware ethernet ...
     fixed-address ...
  }
  host b {
    ...
  }
}

subnet 192.168.135.0 netmask 255.255.255.0 {
  next-server 192.168.135.1;
  option routers 192.168.130.1;

  host c {
     hardware ethernet ...
     fixed-address ...
  }
  host d {
    ...
  }
}

And in this example, for whatever reason booting host “D” would get the router from the other subnet. Every subnet example I’ve seen shows putting “options routers” in a subnet scope. I do have some some groups and classes in there that’s clearly fowling things up but I don’t see how.

From what I’ve been reading, class and host are top-level scopes and shouldn’t go inside subnet.

So I just re-wrote my DHCP configuration into what I believe now are the right scopes:

...
option ntp-servers 192.168.130.10, 192.168.130.12;
option domain-name-servers 192.168.130.10, 192.168.130.12;
next-server 192.168.130.10;

# For things that match this class, override global options
class "pxeclients" {
  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";

  if option arch = 00:07 {
    filename "/efi64/syslinux.efi";
  } else {
    # PXELINUX >= 5.X is the new hotness with HTTP/FTP
    filename "/bios/lpxelinux.0";
  }
}

subnet 192.168.130.0 netmask 255.255.255.0 {
  option routers 192.168.130.1;
  include "/etc/dhcp/homenet-130.inc";
}

subnet 192.168.135.0 netmask 255.255.255.0 {
  option domain-name-servers 192.168.135.1,192.168.130.10,192.168.130.12;
  option routers 192.168.135.1;
}

subnet 192.168.136.0 netmask 255.255.255.0 {
  option routers 192.168.136.1;
  include "/etc/dhcp/homenet-136.inc";
}

group homenet {
  host a {
    fixed-address 192.168.130.x
    ...
  }
  host b {
    fixed-address 192.168.130.x
    ...
  }
}

group otherstuff {
  host c {
    fixed-address 192.168.135.x
  }
  host d {
    fixed-address 192.168.135.x
  }
}

Instead of putting a “host” inside of the “subnet” scope, I put them all at the top level. Apparently dhcpd just “knows” that a host belongs to a subnet based upon the fixed-address matching the subnet+mask given in the subnet declaration, instead of trying to use the “subnet” scope to organize “host” entries.

After doing this, factoring out some duplicate classes to set the filename, UEFI PXE booting my test system worked on the first try! I’m past the point of caring now to bisect my original config, this was a detour from doing other things. Besides, I should be burning all of this down and finishing my migration to the Kea DHCP server.

Supermicro / Intel Atom 2550 fix

I’ve loved using the A1SRi motherboards in ikeacluster for years, they offered a lot of RAM power with a fan-less embedded CPU. I’ve ran them for several years and had a couple eventually succumb to the Atom C2000 clock failure problems. The BMC would work but the motherboard would just decide one day to stop booting.

For years I wrote these off as dead and what I thought as beyond the period at which people were getting Supermicro to replace them. I was sad there was not a great replacement. I seem to recall the Denverton mini-ITX boards were either late or were considerably more expensive. There were some other A1SRi boards on ebay, but they were expensive and who knows when they’d die.

A while back I had read on some forums where people were working on a fix to revive these motherboards. I finally got around to reading up on the Serve The Home thread and a Truenas thread to try it out. On my boards I ran a 200 ohm jumper between pins 1 and 9 on the TPM header and that seemed to do the trick:

TPM header jumper

It still has the problem where the BMC is still alive and I can control VGA+keyboard input, but Linux ipmitool can’t query the BMC nor does serial redirection work. But I’m happy I was able to revive two of my motherboards! It also looks like even as late as 2023 people were getting these boards replaced by Supermicro, so I might have to give that a try.

Update 27-Oct-2024:

huh, I thought I had read this fix breaks connectivity between the OS and the BMC, but serial-over-LAN and things like ipmitool lan print work:

ipmitool sol activate after resistor fix

ipmitool viewing BMC sensors and network settings

I’ll take it!

The whole Let’s Encrypt thing has the side effect of making me cranky every few months as I go around checking what expired, what automatically renewed, and what needs more babysitting.

Today I want to bitch about the Ubiquiti UniFi controller software. As far as I can tell even the mere CONCEPT of updating TLS certificates STILL does not exist anywhere in the controller or the support documentation. Sure they make a nice web UI to manage your 11ty-dozen wireless APs, cameras, doorbells, LED panels, key readers, and whatever thing they’re pushing this month, but keeping the web UI secure and up to date in a post-Snowden world? Nah, screw you. Not even a clumsy annoying web way to do it, no “click here to re-generate a self-signed certificate”, not even a sanctioned command line way to do it. You’re utterly on your own to figure it out. I guess this is one carrot of forcing people to use their cloudy UI.com service.

This has lead to countless people like me reinventing the wheel since 2016 and poking at the Java keystore directly with the old ACE.jar and keytool tools. You did naturally assume it’s a Java keystore the first time you encountered self-signed or expired certs warnings, right?

It’s even worse now when you layer all the Let’s Encrypt tools on top of it, because virtually all of them assume you’re on some form of Ubuntu or Linux. You won’t know it until you try to run a deploy script or read the code. I’m running it on MacOS which is a sanctioned platform and gets regular releases. The official acme.sh/deploy/unifi.sh claims it supports self-hosted, but it really assumes self-hosted on Linux. I’m afraid to know what the Windows people have to deal with.

What I wound up doing is using and tweaking the unifi_ssl_import.sh script from https://github.com/stevejenkins/unifi-linux-utils. This takes care of exporting a PKCS12 file and importing it into the Java keystore. It assumes Certbot and Linux, but it easily adapted to Acme.sh paths on MacOS. Thank god this isn’t some gigantic monolith of bash and it is fairly straightforward. I run only this script and it takes care of updating the UniFi keystore.

It is not automatic upon renewal, and doesn’t automatically restart the Unifi software. Those are problems for another day, maybe in 100 more days.

-UNIFI_HOSTNAME=hostname.example.com
+UNIFI_HOSTNAME=${HOSTNAME}

# Add this to override all of the Fedora/CentOS/Ubuntu/CloudKey paths
#
+# MacOS paths
+UNIFI_DIR="${HOME}/Library/Application Support/UniFi"
+JAVA_DIR="${UNIFI_DIR}"
+KEYSTORE="${UNIFI_DIR}/data/keystore"

# Script assumes Certbot paths, tweak for acme.sh
+# MacOS, this time for acme.sh
+ACMEBASE="${HOME}/.acme.sh/${UNIFI_HOSTNAME}"
+PRIV_KEY="${ACMEBASE?}/${UNIFI_HOSTNAME}.key"
+SIGNED_CRT="${ACMEBASE?}/${UNIFI_HOSTNAME}.cer"
+CHAIN_FILE="${ACMEBASE?}/ca.cer"

# Add -legacy option to openssl in two spots
+    openssl pkcs12 -export -legacy\

Maybe someday I’ll get around to sending in a PR to add MacOS support for the deploy script, but not today. I’ve already spent too much time shaving this yak and have other things to do.

Unhelpful responses from the peanut gallery on this issue:

  • Just type in “thisisunsafe” every time in Chrome! fucking hell, this isn’t even attempting to solve the problem. would you tell your director or CISO to do this?
  • Just proxy it behind Apache/Nginx/Linux!  no. now I have to support and configure two things.
  • Just run it on Linux! Bro, I swear a raspberry pi is all you need, bro please! no. see above, now I have to support an entirely different piece of hardware and OS.
  • Just don’t run the web UI! bro, their entire product revolves around running a web UI, how do YOU run it?

Or you know, Ubiquiti could actually provide a mechanism for uploading a new certificate+key pair.

TL;DR: Controller said RAID1 was lost after disks being powered on for first time after 20 years, I didn’t believe it. Booted into Linux and dd’d the last good disk. Recovered the UFS filesystem, I have 20 year old artifacts to sift through.  Always take images of your drives before mucking with them.

The main database server / admin server for my old ISP was a Dell PowerEdge 1550 1U server running Solaris 8 x86, on three 36 GB Seagate Cheetah SCSI U160 hard drives. It was shut down in 2004 when I folded the company, but I hung on to the drives in case I needed the records for disputes or something, and repurposed the server as a colocated shell server. I almost took the system to e-waste a few months ago when I was purging a bunch of other old rackmount servers from my storage unit, but decided to hang on to it for whatever sentimental reason a little longer.

Recently I was digging through old files to find old ISP setup notes. I found what I needed on my laptop, but it made me remember I still had the ISP drives and I should see if I had any more vintage notes and squirrel away an image of the OS so I could finally ditch the hardware. I had no intention of ever firing this stuff up again and considered it a forgotten memory. The old hard drives have been in my drive collection in the bedroom, so that’s about as good as storage as they get.

In search of RAID

During the time at the ISP the server was using a Dell/Adaptec PERC hardware RAID controller, so I’d need that to revive the data. I took the controller out when I switched to Linux with software RAID using the on-board Adaptec AIC-7899 SCSI controller, and I have no idea what I did with it. I probably e-wasted it a long time ago. So first thing I needed to do was find out what kind of PERC card it had and go find one on eBay. My system was so old I couldn’t even look up the service tag on Dell’s website anymore. The PowerEdge 1550 has been lost to time, there’s very few photos of it online, and none that I found with a PERC installed to reference. I guessed from some service notes and went with a Dell 493 PERC 3/DC card, which sounded vaguely familiar and was around the right vintage.

I made sure the system could actually power on and put in a set of Linux disks from the colo days. Other than a dead CMOS battery, the system eventually booted into Linux as a test just fine. I have no idea why but it takes several minutes for POST to run and load the Adaptec 7899 BIOS, I don’t remember it being this achingly slow.

Next it came time to try the Solaris hard drives. I had no idea what RAID configuration I used, I kind of assumed I probably did a RAID 5. No idea of the order of the drives. I wasn’t even sure which version of Solaris was on there. I first powered up the system without the drives, went into the PERC firmware and reset all the logical device configuration to defaults. I popped in the Solaris drives and right away on boot the PERC BIOS spun up two drives.

Going into the PERC BIOS again, it had imported a RAID1 configuration from the drives. Two drives were in a logical group, one marked ONLINE and one marked FAIL. The third drive was marked as HOT SPARE. That was a promising start!

A brief glimmer of hope after 20 years

I didn’t put a lot of care into trying to recover this, it was more of a nice-to-have. #YOLO. I let the system boot, told the PERC to proceed with the degraded logical volume group. Up pops the blue Solaris Boot Subsystem screen! Right at this same time the PERC alarm starts SCREECHING because of the failed drive and it was LOUD. I had forgotten all about this and there were no buttons or anything anywhere to silence it. There’s no way I could work on this thing in an apartment with that going off.

I hit the power button to turn off the system, turned it back on and went back into the PERC menu to silence the alarm. Except now in the PERC BIOS all drives were marked FAILED! wtf!

 

Artists re-enactment of RAID failure

I wasn’t completely convinced the drives died all of a sudden after one power-off and thought it was more likely there was some sort of bad state stored in the RAID configuration from the power-off. I fiddled with it for a while, trying to remove the config from the card and re-importing it, moving drives around in drive slots, and it kept coming back as FAILED. One of the disks had to still be working to read the RAID config I thought. I also didn’t know the numbering of the drive slots, so I wasn’t sure which two were the data drives and which was the hot spare anymore. Did I mix the old hot spare into an order it expected to find a RAID member? Did one RAID member just die?

So I put it all aside for a few weeks to ponder.

What to do

If it was a RAID1 I thought in theory both drives should have a usable set of data outside the RAID metadata, provided they were still mechanically functional. Even if the sync was broke and one had a slightly older set of writes, this was fine for this archeology dig. The question was if the RAID metadata would throw off any tools to poke at the filesystem. Message board posts all suggested if anything hooking the drives up to a non-RAID SCSI controller to take the hardware RAID out of the picture and taking images of the drive if they showed up, that way they could be experimented on with recovery tools. This was slightly more complicated in that the Solaris 8 filesystem is the older UFS, not ZFS or EXT3/4. Several commercial packages promised they could recover UFS for a modest three digit sum.

I decided on hooking the Dell drive backplane directly to the onboard Adaptec SCSI controller and booting Linux. If the drives showed up I could at least dd a copy of them to fiddle with later and would have more tools to poke at the SCSI bus.

Getting Linux over was going to be work, the system didn’t support booting from USB. It had an IDE CD-ROM drive, a 3.5″ floppy drive, and could network PXE boot. While I have a functioning PXE environment and actually PXE installed CentOS on this system when I had it in colo, I long since removed my old CentOS 5 files. Rigging up a PXE bootable Live ISO image just for this sounded like a lot of work. Ubuntu 14 server was the latest i386 version I could find that still fit on a CDR disc. Miraculously I still had five blanks laying around. The only CD burner I owned was in my Windows 95 machine, so instead of shelling out money on Amazon for another external burner, I went to a lot of effort to just burn it using the 486 (at 2x!).

Of course when it came time to boot, the CD drive in the Dell was not working anymore. I wound up throwing together enough PXE glue anyways to boot the CentOS 6.10 i386 installer in rescue mode. This kernel should well be new enough to have all the 2000-era Adaptec drivers built-in.

Struck data!

One by one I tried all three hard drives. The first one oddly showed part of a serial number to the Adaptec BIOS, but otherwise was undetected by Linux. The second drive showed up! An fdisk -l detected two partitions, “Solaris boot” and “Linux swap / Solaris” !!!

I popped in a USB stick which at least showed up as a mass storage device to Linux and I began a dd of the hard disk to it. About 15 minutes later I checked progress on another vty and quickly realized it had only copied a few dozen megabytes and this was probably using USB 1.1 or maybe 2.0 and it was going to take all night to copy this drive. Would the hard disk survive this long? I threw together a dd | ssh command and let it copy a couple of images across the network to another system. It’s a Pentium III 933 MHz system, so not a complete slouch.

Eventually after a couple of hours the dd over the network succeeded without any sort of errors, so I had at least one copy of whatever was on that disk. I have no idea if that was a working member of the RAID1, or if once upon a time it was part of the RAID1 and I demoted it to hot spare without wiping it, or what. The 3rd disk was completely dead, it didn’t show up on the Adaptec at all. So it seems I did lose one disk during my initial power-off.

After I was satisfied I got a good as copy possible, I let the good disk boot in the system by itself to see what would happen. The blue Solaris bootloader screen loaded, then dropped into the configuration assistant. It didn’t seem to find a kernel on disk to boot, but otherwise the disk acted fine.

Over on another Linux system I ran “strings” on the 36 GB image I captured and it clearly had some viable data in it. I saw a bunch of email, sendmail config, html, mysql commands, and other stuff I recognized. Now the question was how to mount this sucker under Linux. I did some reading and Linux does have UFS support, including Sun x86. I learned that Solaris slices are different than typical Linux partitions in that they’re more a set of logical extended partitions within a standard partition. The Linux kernel with the UFS module loaded understands this and as I saw with the Solaris drive inserted over on the Dell, it will enumerate all the possible slices as extra disk partitions, e.g. sda1 sda2 sda3 sda4 sda5 ... sda15 even if tools like Linux fdisk and parted only see a boot and data partition.

Linux recognizing Solaris disk slices

Here’s what fdisk looked like when reading the captured dd image itself:

root@basic06:~# fdisk -l ./image-sda2
Disk sda2: 33.9 GiB, 36328801280 bytes, 70954690 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x69747261

Device Boot Start End Sectors Size Id Type
sda2p1 1851867950 2396369563 544501614 259.7G 6f unknown
sda2p2 1397314113 3266884704 1869570592 891.5G 20 unknown
sda2p3 0 0 0 0B 6f unknown
sda2p4 20480 20480 0 0B 0 Empty

Partition table entries are not in disk order.

Trying to mount UFS from Linux

(See 9/26 update where a newer kernel fixed all this) I tried a variety of ways trying to mount the UFS filesystem on Linux with no luck. Neither “mount -t ufs -oro,ufstype=sunx86” on an extended device id for a slice such as /dev/sda10 worked, nor on the raw image file of just the 2nd Solaris data partition nor image of the entire disk. I tried some examples of calculating offsets to mounting specific slices or possibly avoid any RAID metadata and those didn’t work. I got a variety of wrong fs type, bad option, bad superblock, or ufs: ufs_fill_super(): bad magic number errors with these attempts. losetup and friends didn’t seem to work for me either, which to be fair I’ve never used.

Another idea I had was to copy the image to a USB stick on another system and letting the kernel detect it as a drive again. Trying to mount it this way didn’t work while I was booted into CentOS 6, I thought maybe a newer kernel would help. I let it copy to USB while I went on to try the next thing, installing Solaris. (I wound up not using this)

Installing a Solaris 8 VM

I gave up and installed Solaris 8 Intel in a VirtualBox VM to see if I could mount the image there.. It’s been yeaaaaars since I’ve touched Solaris, much less v8, but I got something working. I had to convert the dd image to a .VDI image so VirtualBox could actually present it as a drive to the VM. (“VBoxManage convertdd image1-sda image1-sda.vdi --format VDI“).

Within Solaris I had to run devfsadm after boot to get it to recognize this as another IDE drive. It showed up as /dev/dsk/c0d1, and “format” listed a bunch of slices when it was mounted!

Finally, success!

At long last I was finally able to mount the individual slices! and there was intact filesystems with my files!

Browsing around it looked familiar, all bits and pieces of a working system. It looks like this stuff is somehow from about 2003, so this may be leftover from a drive swap, I don’t know.

I also forgot Solaris doesn’t have anything like ssh or rsync out of the box, or I forgot where to install it. So I’m going old-school and running a “tar | rsh” to another system to sift through it more.

I am curious to go looking for the hardware RAID metadata on this disk, is it at the beginning, the end? What does it look like?

Update 9/26:

Fiddling with the whole disk image on a CentOS 7 system with a 5.3.5 kernel, I have success mounting the UFS filesystem, whereas this was failing over on Ubuntu 18 with a 4.15 kernel:

# Mounting with a loop device
[root@basic03 ~]# losetup --partscan --find --show ./staff1-9pf-sda
/dev/loop0

[root@basic03 ~]# dmesg -T
[Thu Sep 26 23:46:44 2024] loop: module loaded
[Thu Sep 26 23:46:51 2024]  loop0: p1 p2
  p2: <solaris: [s0] p5 [s1] p6 [s2] p7 [s3] p8 [s4] p9 [s5] p10 [s6] p11 [s8] p12 >

[root@basic03 ~]# fdisk -l /dev/loop0
Disk /dev/loop0: 36.4 GB, 36420075520 bytes, 71132960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *       16065       48194       16065   be  Solaris boot
/dev/loop0p2           52610    71007299    35477345   82  Linux swap / Solaris
[root@basic03 ~]#

[root@basic03 mnt]# mkdir s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

# Mounted Solaris slice 0 containing / using the linux /dev/loop0p5 partition
[root@basic03 ~]# mount -oro,ufstype=sunx86 /dev/loop0p5 /mnt/s0

[root@basic03 ~]# mount | grep mnt
/dev/loop0p5 on /mnt/s0 type ufs (ro,relatime,ufstype=sunx86,onerror=lock)

# Solaris / directory!
[root@basic03 ~]# ls -l /mnt/s0
total 39
lrwxrwxrwx  1 root root     9 Sep 24  2001 bin -> ./usr/bin
drwxr-xr-x  2 root root   512 Sep 24  2001 boot
drwxr-xr-x  3 root 60001  512 Sep 24  2001 cdrom
drwxr-xr-x 12 root sys   3584 Sep 27  2001 dev
drwxr-xr-x  6 root sys    512 Sep 24  2001 devices
drwxr-xr-x 30 root sys   3584 Jun  8  2003 etc
drwxr-xr-x  3 root root   512 Sep 24  2001 export
...

 

Improvised drive sled

[photos: flickr – Macintosh Quara 700 drive sled]

The Quadra 700 I acquired had the internal plastic assembly that held the floppy drive and hard drives, but didn’t have the sled that the hard drive went in and clipped into the system. These are hard to find on top of an already hard to find system, lore seems to be when recyclers yank the hard drives, they discard the sleds. The Quadra 700, IIcx, and IIci all use the same sled, model numbers starting with 805-5078 or 815-5078. I checked several component places and eBay, and nobody had any for sale. You can still put the hard drive in the Quadra, there’s just nothing holding it in place preventing it from flopping around.

Fortunately lots of photos of the thing exists and it’s just a U-shaped piece of sheet metal with some holes and tabs stamped in it. It seemed easy enough to just go make one. I broke out the ruler and caliper and made some measurements of my own system. Later I discovered this post on 68kmla by Phipli who had a drawing of the drive sled which gave me the outside dimensions and let me fine tune my own measurements a bit more. Then I discovered the 3D printed version by branchus on Thingiverse (I love his Mac repair streams). I don’t have a 3D printer, and didn’t feel like going out to learn Fusion360, how to use a 3D printer, and tracking down one of our libraries just for this when I already have the metal and a metal brake. A local 3D service quoted me $38 to print one, which felt steep.

Nibbling the holes

I started with a piece of 18 gauge aluminum 85mm x 196mm. I didn’t yet know how much I needed to compensate my measurements for the bending so I started working from the inside going out. I made sure the inner dimension was at least 103mm wide to allow a 3.5″ drive to slip in. First I nibbled out holes for the raised square bit at the bottom and the sides.

I bent a small scrap piece to figure out the bend would eat about 1.5mm, and used it to find the position and dimensions of the vertical holes where tabs would lock in. I finished marking all of these on the metal. If you can read my scribbles, that’s all of the dimensions give or take a mm.

My Dremel was utterly dead so I took shortcuts with the rest of the cutting. The OEM part had a D-shaped cut over the side humps presumably to let part of the side remain straight upright while the rest flexed, I omitted this completely. I thought a 1/2″ cold chisel would be perfect for knocking out the side tabs that would lock into the plastic assembly in the case. After a few whacks with a hammer I didn’t punch through the aluminum like I had hoped for, so I opted for the jankiest part of this whole thing by hammering a screwdriver through it!

Being punched through actually worked pretty well at giving me protruding tabs on the outside surface, smoothing a bit here and there with pliers to get it just right. Next up I used my metal brake to fold the sides up. I didn’t trim the tops of the vertical pieces like the original to provide finger tabs, it seemed to fit fine without them.

The thing actually fit into the system almost exactly the first try. I had to do a bit more nibbling on the square hole on the middle and square off my bend and it fit nicely. I messed up drilling the screw holes, so they aren’t pretty, but they work.

All in all, the thing works. I can pinch the sides to take the drive in and out, and it locks the hard drive in place. I’d say pretty good for a Saturday afternoon of tinkering around without the right set of tools. Now I can continue lurking eBay sales hoping for an original sled, or get around to having one 3D printed someday. If I had a OEM sled I would be temped to get better measurements and send off somewhere to laser/plasma cut a few dozen sleds to hand out but eh I don’t want to be in the shipping biz.

Final product

Around 1995 I had a Zoom VFX V.32bis 14.4k modem as my main workhorse. It was a white plastic shell with a smoky brown translucent front face. I decided to buy one recently for old times sake:

Zoom VFX V.32 bis 14,400 bps fax modem

I also came into possession of a Telebit Netblazer PN (which I need to finish working on and write up about it), which lead me to searching for manuals and more information about it. I stumbled across this eBay listing for a Telebit Teleblazer:

Telebit TeleBlazer V.34 modem

It’s the exact same case! Back, front, shell, underneath, face, font of the V.32bis / V.34 badge, it’s all the same! In one photo of I think the box it mentioned being based on a Rockwell chipset too. Previous Telebit modems such as the Txxxx series, Worldblazer, all had their own blocky look. At first I thought the seller had the wrong modem, but after looking at the pictures it’s very much a Telebit branded product with “Telebit TeleBlazer” on the bottom. Funnily there was one auction for $250 and another for $18 for the same kind of TeleBlazer.

This lead me to do even more digging. My Zoom VFX V.32bis was made by Zoom Telephonics Inc in 1991-1992 or so. It’s based on the Rockwell RC144DP data pump. There’s also another VFX V.32bis with a different solid, slant-front, white case that came out later I think because it was used in their later 28.8k, 33.6k and 56k models. I’ve only seen the translucent brown plastic case on VFX 14.4k modems, never anything newer.

Telebit was well renown for producing modems with fast transfers ahead of their time using their own modulation system and throwing a Motorola 68000 at it for processing oomph. The Netblazer I have has a modem chip produced by AT&T. Apparently around 1993 Telebit was trying to put out a V.34 modem like everyone else and just decided to buy Octocom Systems, who was developing their own V.34 modem. Telebit also wanted to put out their own low-cost V.34 modem to compete, so I’m guessing that’s probably how they wound up using a Rockwell chipset.

What’s interesting is that Zoom Telephonics Inc was based in Boston, MA in 1992. Octocom Systems was based in Wilmington, MA, about 15 miles outside of Boston. Did proximity have anything to do with this case story? Were there ex-Zoom employees who went over to Octocom and took their case design with them? Did Zoom sell a bunch of pallets of leftover cases to Ocotocom or Telebit? Did Zoom and Telebit share the same ODM and Telebit said gimme the cheapest case you got and ship it?

I never did find any interesting stories or gossip to explain why they used the same case. I’d also be curious to tear open a Telebit TeleBlazer to see if it even uses something like the Rockwell RC288 datapump, which everyone seemed to be using by then. But I’m not $40 curious to buy one. Further, if their V.34 modems are based Rockwell chip, is there any Telebit magic left in there, or it just a Telebit sticker on the box?

Also, I’m not getting any good nostalgic memories of this VFX modem I bought, it’s been a dog. In my testing it fails to connect a lot of the time and locks up. I don’t know if it’s because the components are aged or if this thing got damaged somehow. The modem speaker only has one sound, LOUD, no matter if I use ATM1L0 or ATM1L3. The owners manual for the VFX V.32bis can be found over on archive.org (ZV32BIS.ZIP), it has some interesting subtleties such as only MNP enabled out of the box and you have to go find the command to enable V.42/V.42bis/LAPM support. For whatever reason even if the connection is using MNP5, the DC/EC lights on the front don’t come on, you have to be using V.42/V.42bis before they light up. Fortunately it’s data compression is in hardware, it’s not one of those janky Rockwell RPI chipsets that required a driver to punt EC/DC off to the PC’s CPU. I had completely forgotten those cheap bastards existed.

Custom aux/roll/null/DCD cable

This is part of the project to connect my Wildcat! BBS to a retro X.25 network, but it also applies more broadly to “reverse telnet” operation of a Cisco router where you telnet/ssh to a router at a given port to access a serial device hanging off of the aux or a terminal line. I don’t think there’s a lot of people seeking this solution, but I’m writing about it for when I eventually forget. This post mainly covers the serial connection and Cisco bits, I’m still clueless about the whole X.25 part.

This isn’t quite as simple as slapping a null modem cable between a serial port on the BBS machine and the aux port on the router, altho that’s part of it and would work. The problem is gracefully disconnecting the reverse telnet/SSH session when the visitor is done so the next person can log in. This is done to improve user experience and increase line availability.

Normally when using a reverse telnet session, it’s expected that a user send a ^] to close the telnet connection or a Ctrl-Shift-6 to break out. Until a the user sends a break/escape or a session-timeout happens, nobody else can use this BBS line. And it’s just not a good experience to tell somebody who’s gone to all the effort to connect to your board to oh yeah do this extra step too please. In the worst case this probably means somebody could tailgate in on the end of last person’s session somehow.

TL;DR:

  • Aux cable + modem adapter (with pin 1 and 6 DTR/DCD connected)+ null modem adapter + gender changers
  • “line aux 0” set to “modem printer”
  • chat script to send a string
  • BBS software configured to see said string and start a call, not “auto-answer”

Wildcat! is an MS-DOS program (at least the 4.x version I’m using) that is designed to use RS-232 serial ports to talk to a modem. The manual does discuss connecting to an X.25 PAD, namely a Microtronics CSI-X.25 PAD, so non-modem serial (i.e. using a direct, null modem cable) usage is expected to work.

Wildcat! and probably most BBS software expects to “answer” a serial line and sending a login prompt to the visitor in one of three ways: “auto answer”, detect when the RS-232 CD (carrier detect) line is raised; “ring detect”, detect when RS-232 RI (ring indicate) is raised; and “ring result”, look for specific text strings such as ‘RING’ to indicate an incoming call from a modem. In the latter two cases, Wildcat! will send an “ATA” command to the modem to answer the call. After that, in all cases Wildcat! expects to see a CD signal on the serial port which tells it there’s an active user on the line. If CD is abruptly dropped, Wildcat! will assume the caller has disappeared and will “hang up” its side. If the visitor selects “Goodbye” from the menu screen, Wildcat! will send the DTR (data terminal ready) line low briefly, which is intended to tell the modem to disconnect.

Normally if you connect a PC serial port to a BBS PC serial port with a direct null modem cable, with Wildcat! configured to auto-answer, then start a communications program such as Qmodem, Qmodem will raise DTR as it’s a terminal that’s now ready to process I/O. Wildcat! will see this and immediately send a login prompt to the terminal. However, if somebody logs into the BBS and selects the “Goodbye” menu option to leave, Wildcat! will wrap up the call and get ready for the next caller — in this case with our hardwired connection, in the Qmodem terminal window we’ll immediately see another login prompt. It’s not until Qmodem is exited that Wildcat! finally resets and waits for the next user. (Or you yank the serial cable from the PC).

BBS null modem / X.25 PAD connection

The Wildcat! Sysop Guide really only refers to a one other serial port configuration that doesn’t involve modems, that’s for for hooking up an X.25 PAD. This would allow users to come in via X.25 network such a Telenet or Tymnet, go through the PAD, which acts as a basic terminal server connected to the BBS via multiple serial cables. Which is kind of convenient for me since this is ultimately what I want to do, but with different hardware. If you wanted to configure the BBS to accept connections from something via null modem or a terminal server, you’d have to pick through this section and pull out the bits that look relevant.

The important part of this section of the manual are the details needed to build a wcMODEM .MDM modem profile file to use for the node that’ll be used for the direct connections. For example, creating a file called like DIRECT.MDM with the specified serial port info, options, and removing the modem commands. Then in the batch file that starts the Wildcat! instance for that node, add in a “WCMDM=DIRECT” to have it load the profile for that node.

I looked up the Microtronix CSI-X.25 PAD mentioned in the manual to get an idea of how it actually handed off serial connections. At the bottom of this post I’ve added some details about the history that I could find, I wasn’t able to find any manuals. Apparently the CSI-X.25 is a box with a number of DB-25 RS-232 ports off the back. It says the PAD is configured to “act like a modem that is in auto answer mode .. simply raises carrier detect (CD) when a call comes in”. It mentions other things here like it supports RTS/CTS hardware flow control, and probably running the serial lines at 9600 or 19200 bps. I’m going to go on a limb and assume it probably supports all serial pins, for example it knows when Wildcat! drops DTR to end the session.

It’s worth mentioning MSI did internally support another terminal server setup. For BBS Direct offered by Concentric, I’m told there was a Xylogics terminal server that received callers via IP/frame relay, and handed off via stack of serial cables to the MSI HQ BBS. I guess they made it all work with their BBS software out of the box.

Cisco operations

You can connect the aux port or an async serial breakout cable from a Cisco router to the serial port of a BBS as well. This could be used to provide inbound telnet/ssh connectivity to a MS-DOS BBS that has no concept of TCP/IP. What I’ve discovered is it’s not great when a user ends their session. It’s the same problem as a PC null modem connection, as soon as the user says “goodbye”, Wildcat! ends their session, and gets ready for the next caller. Except Wildcat! can’t drop the serial connection, you’ll see it eventually sending ++++ ATH0 AT&C1D1 commands desperately trying to get rid of the caller and blindly initializing a modem. Then another login prompt is sent.

As mentioned, until a the user sends a break/escape or a session-timeout happens, nobody else can use this BBS line.

What needs to happen is two things: 1) When the session first starts, the Cisco needs to raise DTR to activate the line and raise CD so Wildcat! knows there’s a visitor there. 2) when a visitor says “goodbye” to the BBS, the Cisco needs to see DTR being temporarily lowered by Wildcat! as a signal to boot the reverse telnet session.

Cabling

This setup only works on the aux port or terminal lines via WIC or NM card. To make any of this work start with the serial cable being used. Cisco used to ship along with their baby blue console cables two adapters, a RJ-45/8P8C to DB-9 “terminal” adapter for connecting a PC to the console port for initial configuration, and a RJ-45/8P8C to DB-25 “modem” adapter for connecting a modem to the console or aux port. The difference between the two is the “terminal” adapter took care of setting up a null modem connection (i.e. crossing RX/TX) for you, however the DCD and RI pins are completely left unconnected as they’re not needed. The “modem” adapter is straight through, but connects DCD to DTR, but only comes in DB-25 form.

Apologies: As an aside it’s maddening following pins that get rolled from the aux port to the Cisco blue roll (not Ethernet crossover) cable to the various adapters. It gets confusing to me which signal to talk about too, since they’re all ultimately wired together — do I say pin 6, or do I say DSR or DTR? So if I say DSR and probably mean DTR signal, forgive me.

You’ll either need to use the DB-25 modem adapter in addition to a null modem adapter, and probably a gender changer too somewhere, or edit the DB-9 terminal adapter to add a DCD pin. This turns into quite a stack of connectors. For this experiment my BBS only has DB-9 serial ports coming out the back, so I wound up making my own combo roll + DB-9 + null modem + add DCD cable. I imagine with the newer Cisco console cables that have a molded DB-9 adapter attached, you’ll need a null and a way to fix DCD.

Remember, Wildcat! expects DCD to be up so we have to have that pin connected to something. Only using RI won’t work either, while that may signal that there’s a new connection, Wildcat! still requires DCD afterwards.

It remains to be seen what kind of adapters are needed to do this for something like a CAB-OCTAL-ASYNC from a NM-16A.

Believe it or not, all of these cable combos below do the same thing. Mine is much simpler and prettier but I don’t want to solder more connectors like it.

My ultimate awesome Aux RJ-45 + roll + null + DTR/CD DB-9 cable

A roll cable, DB-25 adapter, gender changer, DB 9/25, cable and a null oh my

Another abomination

My awesome cable pinout, using a regular RJ-45 Ethernet cable, chop off one end and connect as follows (ignore all the labels and just pay attention to the pin numbers):

RJ-45  (Aux)   -  DB-9
1 w/o  (RTS)   -  pin 8 (CTS)
2 o    (DTR)   -  pin 6 (DSR, pin 6 also jumpered to pin 1)
3 g/w  (TXD)   -  pin 2 (RXD)
4 bl   (GND)   -  pin 5 (GND)
5 bl/w (GND)   -  pin 5 (GND, blues are grounds, connect together)
6 g    (RXD)   -  pin 3 (TXD)
7 br/w (DSR)   -  pin 4 (DTR)
8 br   (CTS)   -  pin 7 (RTS)
-n/a-          -  pin 1 (jumpered to pin 6)

Plug directly into aux port of router.

Aux port configuration

TL;DR: Through much trial and error I settled on configuring my aux port as “modem printer” and “script connection RINGRING” which I’ll explain why.
Cisco IOS provides a variety of options for setting up the aux port for serial operations, and there’s a whole document describing modem signal and line states. Here’s a document for aux pinouts too.

vintage-gw2(config)#line aux 0
vintage-gw2(config-line)#modem ?
  CTS-Alarm       Alarm device which only uses CTS for call control
  DTR-active      Leave DTR low unless line has an active incoming connection or EXEC
  Dialin          Configure line for a modern dial-in modem
  Host            Devices that expect an incoming modem call
  InOut           Configure line for incoming AND outgoing use of modem
  Printer         Devices that require DSR/CD active
  always-on       Configure line for a modern always-on modem
  answer-timeout  Set interval between raising DTR and CTS response
  autoconfigure   Automatically configure modem on line
  dtr-delay       Set interval during which DTR is held low
  onhold          Set the V.92 modem on hold timer duration

vintage-gw2(config-line)#

Ideally we need a config option that does /something/ different on the serial line when a reverse telnet session is started, that way we have signal (a literal electrical signal) to the BBS that there’s a new visitor on the line. Then we could wire that up to DCD so that pin is alive when there’s a reverse telnet session in progress. Also we do not care at all about “inbound” or “exec” sessions, that’s for something connecting TO the Cisco from a serial port.

I’ve gone through every single one of these options with an RS-232 LED breakout and there are exactly two options, “modem Host” and “modem DTR-active” that actually change state. They raise/lower pin 6 for DTR. Normally it’s low/off, but when a remote telnet session comes in, DTR is raised. All other pins 4, 7, 8 all stay the same. One would assume they could connect pin 6 to pin1 so that when DTR is raised it also raises DCD, and Wildcat! could be set up to auto-answer. While that is technically true and does get the visitor to the BBS, it doesn’t solve our original problem of graceful session endings.
I did find other people sell Cisco DB-9 connectors with DSR/DTR connected to DCD (pin 1 and 6), so I’m not crazy in imagining this need.

Nice, but this isn’t the problem we’re trying to solve!

Bye bye bye

Now we need a way to signal back from the BBS to the Cisco that the session is ended, the serial line has been dropped, go disconnect the reverse telnet session.
When a person does “goodbye” from Wildcat!, Wildcat! lowers DTR from the BBS side. When connected to a null modem adapter, this means DSR on the Cisco side changes — except when using “modem Host” or “modem DTR-active” nothing is paying attention to DSR! The Cisco has no idea Wildcat! is telling it the user has hung up and keeps DTR high.

The only option I found is “modem Printer“. Apparently there used to be an option called “modem cts-active” that got replaced by “modem Printer“, but “modem Printer” isn’t really documented in the Modem Signal and Line States document. Anyways IOS says “modem printer” is “Devices that require DSR/CD active“. That’s exactly what we want here, when Wildcat! lowers DTR, it lowers DSR on the Cisco side and yeets the reverse telnet session!

But, this conflicts with our previous step in that with “modem printer” our DTR and thus CD is always asserted! Wildcat! will not be able to auto-answer and we’ll never get a new caller!

RINGRING banana phone

With “modem printer” configured on the aux port to gracefully disconnect visitors, and our DCD line is hardwired to be constantly active, we need another way to signal to Wildcat! that there’s an inbound caller.

What I did here was configure Wildcat! to use “ring result” and gave it a completely made up string to look for, “RINGRING“.

wcMODEM .MDM file for my fake X.25 PAD

Made up RINGRING string

Then on the Cisco side, I configured a simple “chat-script RINGRING "" RINGRING“, and on the aux port, “script connection RINGRING“.

Now when a reverse telnet session starts up, the Cisco sends the text “RINGRING” down the serial port. Wildcat! sees this and answers the line, all transparent to the user. The visitor can use the BBS all they want, I even tested this downloading a 10 MB file with Ymodem and it all worked.

Along with the right cable and aux settings, then when the user says goodbye from the BBS, their reverse telnet session gets gracefully disconnected.

For whatever reason this is not completely perfect, both the Cisco and Wildcat! seem to be trying to fiddle with serial lines for several seconds before things settle down and the next visitor can log in. But it’s a heck of a lot better than what it was!

Final Cisco config:

!
chat-script RINGRING "" RINGRING
!
line aux 0
 session-timeout 5
 no motd-banner
 script connection RINGRING
 modem answer-timeout 5
 modem Printer
 rotary 1
 no exec
 transport input pad telnet ssh
 autohangup
 stopbits 1
 speed 38400
 flowcontrol hardware
!

It also has the nice benefit that if the BBS is down or if a visitor is already using the BBS, the Cisco sends a “Connection refused” instead of black-holing the caller into nothingness of an empty session. I tried setting up some sort of “connection in use, try again later” thing, but doesn’t work like this.

For whatever reason I have the .MDM file set up to force “yes this is a reliable connection give me Ymodem/G” option, but it doesn’t take effect. I tried configuring the Cisco to send something like “FAKELAPM” as a string to tell Wildcat! it supported error correction and enable it, or send “VMP” to pretend it’s an OS/2 virtual modem, but neither worked. Oh well.

This is what the aux port looks while idle:

vintage-gw2#show line aux 0
   Tty Line Typ     Tx/Rx    A Modem  Roty AccO AccI  Uses  Noise Overruns  Int
      1    1 AUX  38400/38400 - printer   1    -    -   142     34 1786/5780   -

Line 1, Location: "", Type: ""
Length: 24 lines, Width: 80 columns
Baud rate (TX/RX) is 38400/38400, no parity, 1 stopbits, 8 databits
Status: Ready, No Exit Banner, CTS Raised, Modem Signals Polled
Capabilities: EXEC Suppressed, Hardware Flowcontrol In,
  Hardware Flowcontrol Out, Modem CTS-Required, Hangup on Last Close
  MOTD Banner Suppressed
Modem state: Ready
Modem hardware state: CTS* DSR*  DTR RTS
Rotary address 51010000
Special Chars: Escape  Hold  Stop  Start  Disconnect  Activation
                ^^x    none   -     -       none
Timeouts:      Idle EXEC    Idle Session   Modem Answer  Session   Dispatch
               00:10:00       00:05:00                       none     not set
                            Idle Session Disconnect Warning
                              never
                            Login-sequence User Response
                             00:00:30
                            Autoselect Initial Wait
                              not set
Modem type is unknown.
Session limit is not set.
Time since activation: never
Editing is enabled.
History is enabled, history size is 20.
DNS resolution in show commands is enabled
Full user help is disabled
Allowed input transports are pad telnet ssh.
Allowed output transports are pad telnet rlogin lapb-ta mop v120 ssh.
Preferred transport is telnet.
Shell: enabled
Shell trace: off
No output characters are padded
No special data dispatching characters

And this is what it looks like with a user on (not much difference):

vintage-gw2#show line aux 0
   Tty Line Typ     Tx/Rx    A Modem  Roty AccO AccI  Uses  Noise Overruns  Int
*     1    1 AUX  38400/38400 - printer   1    -    -   143     34 1786/5780   -

Line 1, Location: "", Type: "SCREEN"
Length: 59 lines, Width: 174 columns
Baud rate (TX/RX) is 38400/38400, no parity, 1 stopbits, 8 databits
Status: Ready, Connected, Active, No Exit Banner, CTS Raised
  Modem Signals Polled
Capabilities: EXEC Suppressed, Hardware Flowcontrol In,
  Hardware Flowcontrol Out, Modem CTS-Required, Hangup on Last Close
  MOTD Banner Suppressed
Modem state: Ready
Modem hardware state: CTS* DSR*  DTR RTS
Rotary address 51010000
Special Chars: Escape  Hold  Stop  Start  Disconnect  Activation
                ^^x    none   -     -       none
Timeouts:      Idle EXEC    Idle Session   Modem Answer  Session   Dispatch
               00:10:00       00:05:00                       none     not set
                            Idle Session Disconnect Warning
                              never
                            Login-sequence User Response
                             00:00:30
                            Autoselect Initial Wait
                              not set

 

Microtronix CSI-X.25 PAD

Cableshare Inc X.25 Data Concentrator

The PAD that’s mentioned in the Wildcat! has definitely been lost to time. I can find very little about this, no manuals, and maybe a couple of times it was for sale 30 years ago. Doing some sleuthing apparently it was originally made by Cableshare Inc. in London, Ontario as the “X.25 Data Concentrator”, and then starts showing up as the Microtronix CSI-X.25, who is also based in London, Ontario. I’m assuming CSI == Cableshare Inc.   I found exactly two articles even talking about it, an IEEE Communications Magazine “New Products” article from March 1984 that has the only photo of it, and a Computerworld from December 12, 1983 announcing it at $2,700 per port for a four port config. Sounds like off the back it had up to 16 DB-25 ports for connecting to thing.

So if you see one or its manuals, send it my way! Microtronix is still around, looks like they did a couple more X.25 devices, but have long left it behind.

Macintosh SE

I think I may have used a classic Macintosh once in my life, at a Kinko’s copy location of all places. We didn’t have them in school, we went from Commodore CBMs, to Apple IIe, to IBM PC 8088 clones. At the ISP I borrowed a customer’s PowerBook overnight so I could get experience with System 8 and to write how-to instructions for setting up dial-up accounts. It was nice but I didn’t bite. By 2003 when I finally bought my first Mac, a PowerBook G4, I started off on OS X.

At VCF I was playing with some of the classic macs on display and later saw some at the consignment sale. I thought why not, I am an adult, I can buy one if I want to. (Which is how I wound up buying USR Courier modems). I knew literally nothing about classic Macs, quickly googling what the difference between an SE, a Classic, and a 128k. I decided on an SE, it had an Asante Ethernet card, it seemed like a good deal so now I own an SE.

After getting home and using it, I quickly learned about the 800k floppy drives in them and I had nothing that could write disks for it. I was beginning to wish I had held out for a SE/30 with a FDHD, but here we are. It had 800k floppies of System 6 and the ethernet card driver, that was it. This is where I learned about BlueSCSI and using SCSI Zip drives to copy files to it, so I ordered an external BlueSCSI.

The next day at VCF I was browsing the sales again and this time there was a IIsi for sale, I think for like $30. I kinda wanted a color mac but learned the classic compacts didn’t have color. The small form factor won me over, it was running System 7, had a 1 GB hard drive in it, and another Ethernet card. I could at least stick it somewhere and it wouldn’t take a lot of space.

 

Macintosh IIsi

Now I suddenly owned two Macs! I thought as a bonus I could use it to write 800k floppies for the SE, but that doesn’t seem to be the case. Those damn 800k drives. When I took the IIsi home and opened up to examine it, it’s clear somebody took very good care of it. Not only did it have the huge 1 GB replacement hard drive, all of the components were very clean. I found out later the logic board had been re-capped, had a new battery, and the guts of the power supply had been replaced with a PicoPSU adapter. Very nice.

The IIsi has been a blast to use, there’s just something nice about the System 7 interface. I maxed it out with 64 MB of RAM which seemed to help the speed a bit. It had copies of apps such as After Dark, Lemmings, Oregon Trail,  I was able to get some floppy disk images from sites such as Macintosh Garden to load on ZTerm, Network Software Installer, and a few other tools. Then I started copying larger files to the BBS and downloading them to the IIsi using a modem. Eventually my AAUI transceiver cable came in and I was finally able to hook up Ethernet.

My BlueSCSI finally came in last week, so now I’ve been able to make more progress with the SE. I’ve been able to get it online. I’ve been inside it once to check things out, it does not look like a trivial machine to take apart like the IIsi. Watching some videos it seems I’ll need to replace the battery on it and possibly recap it, and the seller’s tag noted the floppy drive worked but needed to be lubricated, so all that is probably up next for it.

One more thing

Then I got a Quadra 700. I knew about the whole Jurassic Park thing, and while I love that movie, that aspect didn’t really appeal to me. When I saw the 700 at VCF I thought it was the neatest mini-tower kind of case, smaller than a PC mini-tower even, it spoke to me. That beige, those lines in the case, and Apple rainbow apple on the front, mmmm. Then I found out they’re a big collectors item because of the whole said movie thing. Prices for previous eBay auctions were all over the place, some beat and yellowed to hell, some in mint condition, from a few hundred for parts chassis to well over $1k for fully kitted systems with the PowerPC card, and they seemed to come along once a month or so.

I set an email alert, not expecting to see a system come by for a long time. Then by sheer luck several days later I happened to be up late at night browsing eBay when a Quadra was added, it looked in decent shape so I jumped on it. This is gonna be another round of picking up the bits to build it up, so far I’m in the process of getting RAM, VRAM, a drive sled, and a hard drive.

Tandy CCR-82

[photos: flickr – Tandy CCR-82 tape recorder]

Between VCF West and the Electronics Flea Market I have been inundated with projects! I wasn’t intending to mess with cassette tape on the TRS-80, despite having a cassette of Talking Eliza. At VCF I ran across a Tandy tape recorder in good shape in a box for a great price, so sure why not. It had the original box, manual, packing material, and TRS-80 interface cable. We had one of these, I think an prior generation, once upon a time but I guess it got sold with our old Model 3.

I stuck new batteries in it, hit play and I could feel the motor turning but nothing. Opening it up it was immediately obvious what was wrong, the drive belts were completely stretched out and lost tension. Browsing Youtube for the CCR-82 I happened upon this one from ACs 8-Bit Zone where he replaces the belt and does some other troubleshooting, and I used it as my guide. Later I found Console5 sold a belt kit for the CCR-82 which I bought.

Console5 CCR-82 belt kit

I got all the bits and put it aside for a few weeks. I thought I’d make a repair view, but decided against it since one already existed. Finally tonight I dove in to trying to revive mine.

In AC’s video it shows disconnecting various wires from the PCB with a soldering iron to lift the PCB out of the way. I thought I could get away with doing this and was able to replace the third (counter) belt, but ran into problems with the middle drive belt. It had completely turned into an extremely sticky rubber string that I had to pick off bit by bit with tweezers, which got everywhere. I needed to get the PCB out of the way to clean things up more, fortunately it was just a few quick dabs with the iron to get the four wires loose.

I cleaned the gears up with 91% IPA and got the belts on without any issue. I cleaned up the old grease on all the mechanisms and put new silicone grease on, everything seemed tip top. I flipped it over and play/fast-forward/rewind all worked even with a tape but record was physically not working. Opening it back up and checking the release mechanism I realized the tape I had was write-protected, sticking some tape on the holes later, record worked!

Now I need to figure out how to make it work with the TRS-80. Fiddling around I found I had a version of BASIC that supported CLOAD/CSAVE. I could CSAVE a BASIC program to tape, play it back in all of its FM/MFM audio glory. But for some reason when I tried to CLOAD, I could hear something click inside the TRS-80, the tape would play for a second, stop and the prompt would return on the system. I don’t know if I’m not getting audio back to the computer or there’s some other reason it doesn’t seem to load completely.

This will be the next thing to figure out, but at least the tape recorder is finally working!

I was recently expanding my analog voice empire and noticed my Cisco ATA191 was blinking like it was rebooting, and coming back. Looking at logs it was indeed warm rebooting and SIP re-registering every few minutes. I ruled out a duplicate IP address, and it never missed a ping. I was wondering if the VoIP provider was having issues, or if adding a new extension somehow broke things, so I rolled back changes I made. After a few hours of maddening searching I saw link flaps on my switch and thought ah ha it’s just a flaky cable. No, shortly after fixing the cable the ATA still kept rebooting. My other ATAs like the Cisco SPA122 and Grandstream weren’t having problems and stayed registered the whole time.

In the debug logs on the ATA I’d see something like this, like something was happening to cause the unit to want to reboot. The “reboot reason 800000” and “reason 0x800000” looked interesting but I didn’t turn up any useful information. Had I had a support contract they could probably tell me quickly, but I’m on my own.

Jul 28 20:09:22 ata01 Network[468]: [netCtrl]: raMonitorMain(), send AUTO_CFG_CHANGE to WAN
Jul 28 20:09:22 ata01 [161241.038349] sysevt_comm_sendto: (54, rc)=>
Jul 28 20:09:22 ata01 System[468]: [rcDbg]==== event_process start-832 , module(wan_module, evid=0x702) ====
Jul 28 20:09:22 ata01 Network[468]: wan_event_process(1279)..recv AUTO_CFG_CHANGE...
Jul 28 20:09:23 ata01 Network[468]: [netCtrl]: runDhcpv6App(), infoOnly = 1, prefixLen = 64
Jul 28 20:09:23 ata01 System[468]: [rcDbg]==== event_process end-832 ====
...
Jul 28 20:09:25 ata01 vsock: nmlink_server_task(), message received: 14
Jul 28 20:09:25 ata01 vsock: nmlink_server_task(), voice app restart
Jul 28 20:09:25 ata01 vsock: system request reboot, type 1, reason 0x800000, graceful 0
Jul 28 20:09:25 ata01 vsock: [cc_pre_reboot_check]: NO CALL, send unregister here...
...
Jul 28 20:09:26 ata01 vsock: SIP_regTsEventProc(event: 28)
Jul 28 20:09:26 ata01 vsock: setRegState(), line{1} REG State(1->0) pCause=unREG
Jul 28 20:09:26 ata01 vsock: SIP_regTsEventProc(event: 32)
Jul 28 20:09:27 ata01 vsock: fpar2_update_flash() Finish SYS Saved, infoCnt=0 sysCnt=3
Jul 28 20:09:27 ata01 vsock: fpar2_update_flash() PAL-PARM Saved, pid=208, type=2, attr=0x0, name=SIP Reg Call ID State
Jul 28 20:09:28 ata01 vsock: fpar2_update_flash() Finish PAL Saved, palCnt=1 infoCnt=0 sysCnt=3
Jul 28 20:09:28 ata01 vsock: reboot_check(341), reboot reason 800000
Jul 28 20:09:29 ata01 vsock: hal_board_warm_reboot (145, tid=0xc47ff460) do system sync
Jul 28 20:09:29 ata01 vsock: SAFE_MON_main() ccTick:6266->6366, ccCnt=6146->0, monNum=2
Jul 28 20:09:30 ata01 vsock: hal_board_warm_reboot () fp-size=1048576 date=2024-07-28T20:09:28
Jul 28 20:09:30 ata01 vsock: hal_board_warm_reboot (163, tid=0xc47ff460) terminate VoIP service

The “AUTO_CFG_CHANGE” bit followed a dump of dhcpv6c details made me think something related to SLAAC, DHCPv6, router advertisements. Especially when I was watching the unit now reboot every 5 minutes now. I had just powered up a new, freshly wiped Cisco 2821 on my LAN with some basic IPv6 config and realized the thing must be sending out IPv6 RAs or maybe some sort of CallManager auto-provisioning thing the ATA was picking up on.

Logging into the Cisco I did a  ‘ipv6 nd ra suppress‘ on the interface plugged into my LAN. It’s not connected to anything else and therefore has no connectivity to offer. Lo and behold the auto reboots stopped!

It was only the Cisco ATA191 that had this problem. The Cisco SPA122 and Grandstream HT802 are on the same LAN and they had no problems at all. I have only one other router on this LAN, and it’s been speaking IPv6 for years just fine. The ATA is configured to get an IPv6 address from DHCPv6 along with a static list of DNS servers. (Now I remember the SPA122 is lame and doesn’t even support IPv6). I can reproduce this by re-enabling RAs on the Cisco and the problem with the ATA191 comes right back and starts warm rebooting again.

This feels like a bug, a device receiving two sets of RAs shouldn’t go janky like this. I dug into this for a while today doing some packet captures from the ATA’s switchport. I could see RAs from my normal router and the Cisco router on the wire, but nothing was really lining up time-wise with the log messages I was seeing. RAs might go by and then like 60-120 seconds later it decides to reboot itself. I can’t even come up with any off the cuff theories, it’s not like the Cisco was advertising anything crazy.

I stopped looking into this problem but maybe if somebody else stumbles upon this it’ll give them some insight to carry on and find a root cause. To be clear I am absolutely not advocating for disabling IPv6 here!

Cisco 2821 Noctua fan replacement

[photos: flickr – Cisco 2821 fan replacement]

I have both a Cisco 2921 and this Cisco 2821 to play with. The 2921 is considerably louder even at idle and not really suitable for my 24/7 homelab production. The 2821 is much quieter so I wanted to use it, but was still enough white noise to notice. If it had brand new fans it may have been quiet enough, but these were of 2005 vintage and had some noise to them. The existing fans were Delta AFB0812SH-F00R, 4000 RPM, 80 mm, 12 VDC, with a 3-pin connector.

Instead of just buying a set of the same OEM fans, I tried a set of Noctua NF-A8 FLX fans. I didn’t think I needed to go all the way to their ultra low noise versions, just some with a lower RPM. At first they didn’t work, then I noticed the red cable on the Delta fans was on the outside, and on the middle on the Noctua fans. Using a paperclip to push out the terminals, I re-arranged them with the red on the left, black in the middle, and yellow on the right. The fans worked, at full RPM.

IOS was cranky about it, repeatedly with %ENVMON-4-FAN_LOW_RPM in the logs and show environment reporting “Low RPM”.

vintage-gw2#show environment

 Main Power Supply is AC

 Fan 1 Low RPM
 Fan 2 Low RPM
 Fan 3 Low RPM

 Fan Speed Setting: Normal

 System Temperature: 29 Celsius (normal)

 Environmental information last updated 00:00:10 ago

The datasheet for the Delta fans shows the white cable is a tach / frequency generator output, and the Noctua fan also has a tach output. At idle the Delta fan was running around 2520 RPM. When I measured the Nocuta it was running at 1400 RPM, so this may be too low for what the router was expecting. I’ve seen on reddit other people have encountered this same problem with other Cisco routers, trying various wiring hacks, with no satisfactory solutions. It may need a circuit to artificially output double the frequency so the Cisco things it’s running faster, or just short the thing to 12 V and be done with it. At least here at home I don’t think it’s going to overheat.

7/29: Playing with my oscilloscope today, I see these kind of waveforms on the tach/fan speed pins of the Delta and Noctua fan. Also at boot the Cisco kicks the output voltage to 12 volt and then settles in around 7.2 volt at idle.

Delta fan tach pin output

Noctua speed output

Update 9/2:

If you get really fed up with the %ENVMON-4-FAN_LOW_RPM messages want want to yeet them into the void, hashtag YOLO, ignore all the consequences, you can set up a logging discriminator:

logging discriminator nolog msg-body drops Fan
logging buffered discriminator nolog 4096
logging console discriminator nolog
logging monitor discriminator nolog

Many years ago I was out of the country turning up a new site. Along with racks of servers we had a pair of cabinets shipped to us that contained Cisco 6509s, patch panels, with 300+ ports pre-mounted, pre-cabled, and tested from our main US site. This was a weeks long project and we were a couple of days from being completely done and going home, and the site was already taking some customer traffic. The company CEO stops by to check things out, barely looks at one of the 6509s and says “that’s mounted wrong”. We never noticed it, the group that originally racked it never noticed it, but indeed the rear of the 6509’s shelf was one bolt position too low. Not a RU low, just a half inch.

Some people strive for perfection, some people strive for done is good enough. Word came down we had to fix it. This would entail draining all the customer traffic (an 8-12 hour wait itself), unplug all 336 patch cables, fibers, de-rack the 6500, fix the shelf, and reinstall everything. This would have added a day to our trip at the least, provided nothing else went wrong in the process. My manager and I decided to take matters into our own hands.

We went out to our rental cars and got our jacks for changing flat tires. We stuck them under the back of the 6509, gently lifted up the chassis enough to unscrew and fix the supporting shelf (the chassis rack ears were still secure to the rack), and let it back down. While it was serving customer traffic. Problem was solved within an hour, we were very happy with ourselves.

Needless to say our bosses were not happy about our bit of improvisation. But it worked and not a packet was lost that day!

Older Posts »