Feed on
Posts
Comments

Years behind schedule I finally got around to replacing ISC DHCP with Kea DHCP so I could finally have proper IPv6 host reservations. What I just learned, and should have learned years ago, that several of my motherboards such as the Supermicro A1SAi and Intel NUC while they support UEFI PXE booting, they do not support TFTP servers outside of their local /64 network. Doh! They will happily get an address via DHCPv6 on a DHCPv6 server on another network via a relay, that’s not a problem, but if the TFTP server is not on the same LAN the NBP download process times out and fails. It would seem that Linkedin learned this years ago too. This is similar in effect to my misconfigured DHCP server the other day, but not the same cause.

The only solution is to either have a TFTP server on the same LAN as the target system, or keeping around legacy IPv4 networking so that the target system can use UEFI IPV4 PXE to boot something like syslinux.efi, or GRUB2, or iPXE, which in turn has IPv6 support, and can finish downloading the kernel and initramfs over IPv6.

At first I thought I was doing something wrong in Kea (and I verified this with the old ISC DHCP), but no, packet captures prove that during UEFI PXE boot the system is making zero effort to send out Router Solicitations. It also tries to do Neighbor Discovery for IPv6 addresses that it should be sending to the default gateway, which implies it’s not honoring Router Advertisements that tell the system its prefix and prefix length. Or, it has some wild ideas as what it thinks are “on-link”, which is how IPv6 determines if something is on the same L2 network.

An example

Here’s a target system, Supermicro A1SAi-2550 with MAC address 0c:c4:7a:32:27:6, trying to UEFI PXE boot over IPv6:

First, the Kea DHCP6 server configuration, just says here your IP address is 2001:470:8122:1::9, and go fetch grub2 using tftp at 2001:470:1f05:2c9::10:

    "client-classes": [
      {
        "name": "grub2_tftp_efi",
        "test": "option[61].hex == 0x0007",
        "option-data": [
          {
            "name": "bootfile-url",
            "data": "tftp://[2001:470:1f05:2c9::10]/efi/bootx64.efi"
          }
    ...
    ...
    "subnet6": [
    ...
    ...
    "hostname": "basic09.wann.net",
                  "hw-address": "0c:c4:7a:32:27:6c",
                  "ip-addresses": [ "2001:470:8122:1::9" ],
                  "client-classes": [ "ikeacluster" ]
    ...

On boot, this is displayed on console:

>>Checking Media Presence......
>>Media Present......
>>Start PXE over IPv6..
  Station IP address is 2001:470:8122:1:0:0:0:9

  ....long 20 second wait...

  Server IP address is 2001:470:1F05:2C9:0:0:0:10
  NBP filename is efi/bootx64.efi
  NBP filesize is 0 Bytes
  PXE-E18: Server response timeout.

This tells us the target system did a successful DHCPv6 Solicit/Advertise/Request/Reply (S.A.R.R.) to Kea, it understood the bootname-url option in the DHCP6 response. But then got zero bytes.

From the standpoint of the DHCPv6 and TFTP servers, there’s not much to see. The SARR process happens, and that’s it. Nothing tries to hit the tftp server at all.

From a packet capture of the router (:89:f0) facing the Supermicro system (:27:6c) we see:

Solicit XID: 0xe8a7e3 CID: 000100013875b6020cc47a32276c                               ok
Advertise XID: 0xe8a7e3 CID: 000100013875b6020cc47a32276c IAA: 2001:470:8122:1::9     ok
Request XID: Oxe9a7e3 CID: 000100013875b6020cc47a32276c IAA: 2001:470:8122:1: :9      ok
Reply XID: 0xe9a7e3 CID: 000100013875b6020cc47a32276c IAA: 2001:470:8122:1::9         ok
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:6c                what!
Neighbor Solicitation for fe80::ec4:7aff:fe32:276c from fc:ec:da:4a:89:f0             < router :f0 asks who :6c is
Neighbor Advertisement fe80::ec4:7aff:fe32:276c (sol, ovr) is at 0c:c4:7a:32:27:6c    < :6c replies
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:6c                what!
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60                what!
Neighbor Solicitation for 2001:470:1f05:2c9::10 from 0c: c4:7a:32:27:6c
Neighbor Solicitation for fe80::feec:daff:fe4a:89f0 from Oc:c4:7a:32:27:6c            < :6c unicast-asks who's :89:f0
Neighbor Advertisement fe80::feec:daff:fe4a:89f0 (rtr, sol)                           < :f0 replies I am he, also I'm a router
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from 0c:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from 0c:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:C4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:C4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Neighbor Solicitation for 2001:470:1f05:2c9::10 from Oc:c4:7a:32:27:60
Release XID: Oxeaa7e3 CID: 000100013875b6020cc47a32276c IAA: 2001:470:8122:1::9    < :6c I give up
Reply XID: Oxeaa7e3 CID: 000100013875b6020cc47a32276c

We see the Supermicro go through the whole SARR process. DHCP6 by design does not carry any router details or subnet/prefix information design. It’s up to the target system to listen for Router Advertisements to find the prefix of the associated subnet of the LAN. In other words, the Supermicro assumes it is 2001:470:8122:1::9/128 until something tells it otherwise. Here the Supermicro did not make any sort of Router Solicitation. I’ve filtered it out for brevity, but the router was indeed sending out RAs every 4 seconds so it had ample time and had at least 5 go by in this time frame.

My hypothesis is that maybe the IP stack did receive an RA but just decided everything is “on-link” anyways? Or has a wildly wrong prefix misconfigured and thinks everything in the world is on the same network. For giggles I did try to fetch from a Comcast 2601:646:: network so it’s in at least a different /14, didn’t help. In any event, the Supermicro starts sending out Neighbor Discovery requests for the TFTP server at 2001:470:1f05:2c9::10 over and over, which is a completely different subnet on a completely different LAN.

It tries this for many seconds and eventually gives up. It’s nice enough to release the DHCPv6 lease before it returns to the boot menu.

How to fix?

I don’t know if there is a fix for this, at least one available to me. I’ve already tried upgrading the Supermicro BIOS which jumped it way ahead from a 2014 vintage to 2019. I’m sure Supermicro’s solution is “buy something newer”.

In the meantime I’m going to go back to booting GRUB2 over IPv4 and be mad about it.

A peek inside PXE – TianoCore EDK II

Googling for anything related to PXE booting is futile. Pages and pages of people way off the mark and no real definitive information. The UEFI 2.1 and 2.7 Implementation specifications are useful, they go into a lot of detail as to what should happen, but it’s up to others to actually write the code. Somehow I did stumble upon TianoCore EDK II, from what I gather was Intel’s original EFI reference code that was open sourced and now has grown into its own reference UEFI codebase. TianoCore is the community, EDK II is the reference implementation.

I have no idea if Supermicro’s UEFI code is based off of EDK, it seems fairly sorta similar from what little I can see at least. Maybe not, because EDK supports UEFI HTTP boot and my Supermicro doesn’t. I give the Intel NUC more possibility that it could be using code from the same pedigree.

EDK II code is a fascinating read, especially the NetworkPkg/UefiPxeBcDxe code that shows an actual PXE implementation, “Start PXE over IPv6” and all. It answers a few questions, such as what’s the real format for bootfile-url options in DHCP (tftp://ip.address./path/path/file), or why the leading / slash gets chopped off paths, or the variety of code paths that get you to different PXE-Exx error codes.

Another cute thing I learned from the EDK code, and I’ve seen it on the NUC, is that every dot it prints after “Start PXE over IPv6” means the stack has sent a packet on the network.

[photos: flickr – Vintage dial-up modem teardowns]

[photos: flickr – Analog telephone adapters]

For several months I’ve been buying old popular models of dial-up modems from the 1990s to test how they fare over VoIP connections along with different analog telephone adapters. To my great annoyance maybe a quarter of them didn’t include an AC power adapter, so I had to do a bunch of sleuthing to figure out if the modem took AC or DC power, the voltage, the expected amperage, what type and size of power connector. What worked for one model is no guarantee it works for another similar one.

USR Courier I-modem AC transformer guts

For instance even between my USR Courier V.Everything modems, models 1868, 2806, 3453C, they came with AC step-down transformers that output 20 VAC, 9 VAC, or 15 VAC. The USR Courier I-modem AC adapter claims it has a 20 VAC output, but after getting weird output measurements on the pins, I cut open the the impossible-to-find AC transformer to find it has a diode which seems to imply it’s outputting half-wave rectified DC-ish power and a much easier to find DC-only supply might work.

It looks like Retro Web doesn’t allow for documentation of external devices like modems, there’s no good collection of this information that I’m aware of. To help future generations avoid this problem, I started photographing and noting the details of every power supply in my collection. And for history’s sake I decided to open up the modems and make high-quality-ish photos of them too. Hopefully this will let people find cheap replacements for modems they buy or in the case of the Courier I-modem, find a workaround replacement because they are very rare.

At least one, such as the first gen USR Courier I-modem, had leaking electrolytic capacitors so I’ve taken extra photos of the caps to get size information. Unfortunately I am not yet an expert on circuit design, DSPs, and ROMs, so I don’t have much illuminating commentary or stories to tell about these modems.

For now I have all the teardown photos in a single, large Flickr album, organized by modem name/model.

I haven’t decided how I want to organize these, if I want to put together a modem wiki over on Tuxedocatbbs.com, or go for a more structured approach like Retro Web did. I have more information that goes along with them, either manuals I’ve scanned or dug up, replacement capacitor sizing, along with init strings used during my testing.

As for the testing itself, that’s a whole ‘nother post. I used Qmodem on my 486 to make thousands of calls to my BBS and do a 64 KB Ymodem download. For actually calling, handshaking, and connecting, surprisingly all of the modems have almost a 100% success rate over VoIP without any speed restrictions. Disabling V.92 quick connect is usually the only tweak I’ve had to make. However actually trying a download is where things start telling different stories and results vary widely. Preliminary test data and results are over on the BBS website: https://tuxedocatbbs.com/stats/ccr.txt

As of 11/2024 I have these modems up:

  • Cardinal 28.8k V.34 external 020-0458
  • Hayes Smartmodem Optima 9600 “Optima 96” 2003 AM
  • Hayes Smartmodem 2400
  • Hayes Smartmodem Optima 288
  • Motorola ModemSURFR 33.6
  • Motorola Premier 33.6
  • MultiTech MultiModem II MT1432BA
  • MultiTech MultiModem II MT2834BA
  • MultiTech Multimodem MT5634ZBA
  • SupraFAXmodem 144 LC
  • SupraFAXmodem 288
  • SupraFAXmodemPlus 2400
  • Telebit Netblazer PN V.32bis
  • US Robotics 56k V.90/x2 (basically Sportster)
  • US Robotics USR5637 USB
  • US Robotics Courier 56k Business Modem 3453C
  • US Robotics Courier I-modem ISDN with V.Everything
  • US Robotics Courier I-modem with ISDN/V.34
  • US Robotics Courier V.Everything 1868
  • US Robotics Courier V.Everything 2806
  • US Robotics Sportster 56k with x2
  • Viva 9600/4800 2400 bps data fax
  • Zoom VFX V.32bis

Have you seen this modem?

Wang 9648/24e

One of my very first modems was a Wang 9648/24e, a 2400bps fax/modem that I bought at Walmart around 1993. I have only found exactly one photograph of this model on the Internet.  Barely anyone seems to remember Wang, much less that Wang made modems. It wasn’t particularly good nor bad, just a pokey 2400. I even used it for years during the ISP for credit card batch processing because higher speed modems had problems connecting to the processor. I tossed mine years and years ago, but if you come across one send it to me! I thought the Viva 9600/4800 was a rebranded version but after buying one it only looks vaguely similar and is most definitely nowhere near the same thing.

Update: 9:17 PM

Literally hours after I posted these, one just sold on eBay three hours ago! I’ve been keeping an eye out for it but guess I didn’t have a saved search for it.

Update 9:20 Oh it’s actually a 9696/24e which I’ve actually never heard of and looks slightly larger, so not exactly the same, but still so close!

 

Petcube Bites 2.0 teardown

This is my second Petcube Bites and after a few years of operation it stopped dispensing treats. Treats started getting jammed between the rotating loader head thingy and the slot loading to the launcher chute and I’d have to empty it out and pick out the offending treat, only to have it jam the next time around. Using the little reducer didn’t seem to matter. The unit would growl and whirrrrr for 10-15 seconds before it timed out, it sounded like it had a stripped gear inside.

No other option other than throwing it away, I opened it up to take a look:

2024-11 Petcube Bites 2 Teardown

Go to Flickr gallery

[photos – flickr: Petcube Bites 2.0 teardown]

The unit was designed simpler than the 1st generation Petcube Bites. That one had a spring loaded flipper thing that I seem to recall just stopped working and I couldn’t fix it after I took it apart too. The 2.0 just has two motors, one for the “loader” at the top and another for the “launcher” on the side. I knew the 2.0 would launch treats with some force across my apartment, after opening it up I found out why. The launcher motor spins up at a pretty good clip the whole time while waiting for the loader to feed in treats, then turns off. The whirrrrring sound I heard seems to be the launcher motor running empty until it times out.

The first gen had sensors in the launcher chute which I assume is to tell if a treat dropped or not. One the 2nd gen both motors have a wheel that passes through opto-interrupters, which I’m wondering measures slight changes in RPM to figure out if a treat has been fed through.

 

Here’s one for the future troubleshooting seekers. I was testing IPv4 UEFI PXE booting a Supermicro A1SAi motherboard after applying the Atom 2550 fix and couldn’t get the thing to load the network bootstrap program (NBP). I’m not at all saying this is the only reason for hitting a PXE-E99 error, this is just what I hit today.

This blipped by on VGA console so fast I had to use slo-mo on my phone to capture it. (With the Atom 2550 fix, console redirection is lost, so no serial console scrollback).

Checking Media Presence.....
Media Present....
Start PXE over IPv4. Press ESC key to abort PXE boot.
Station IP address is 192.168.135.29

Server IP address 192.168.130.10
NBP filename is /efi64/syslinux.efi
NBP filesize is 0 Bytes
PXE-E99: Unexpected network error.

My PXE environment is pretty set it stone, it’s configuration managed and doesn’t get changed willy-nilly.

Things that came to mind:

  • Wrong EFI binary? Did this particular firmware want some weird 32-bit EFI program? Possible, but I had an near identical motherboard with an older firmware that loaded the x86-64 just fine. Plus I’m near 100% certain I’ve used this same binary on this same motherboard before it died.
  •  TFTP server broken? No, I was able to fetch syslinux.efi on several other machines on the same LAN just fine.
  •  UEFI not support routing / off-network TFTP server? This seemed feasible, but yet I’m absolutely certain I’ve PXE installed these on different LANs than the TFTP server.
  •  Does the UEFI firmware not like it when two DHCP servers respond? While both of my DHCP servers run identical configuration files and reservations, this was easy to test by stopping one temporarily. Didn’t help.

Tcpdumping on both the TFTP server and the router facing the LAN this motherboard was attached on showed there was absolutely no attempts on the wire made to fetch anything over TFTP.

The only thing left was DHCP. The “unexpected network error” got me digging into the DHCP responses being sent back:

01:35:21.075721 IP (tos 0x0, ttl 64, id 4266, offset 0, flags [DF], proto UDP (17), length 354)
    192.168.130.12.67 > 192.168.135.1.67: [bad udp cksum 0x8bbe -> 0xd3ef!] BOOTP/DHCP, Reply, length 326, hops 1, xid 0xafb187ee, Flags [Broadcast] (0x8000)
          Your-IP 192.168.135.29
          Server-IP 192.168.130.10
          Gateway-IP 192.168.135.1
          Client-Ethernet-Address 0c:c4:7a:32:27:e0
          file "/efi64/syslinux.efi"
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 192.168.130.12
            Lease-Time Option 51, length 4: 86400
            Subnet-Mask Option 1, length 4: 255.255.255.0
            Default-Gateway Option 3, length 4: 192.168.130.1             <<<<<<<<<<<<<<<<<<<<<<
            Domain-Name-Server Option 6, length 12: 192.168.135.1,192.168.130.10,192.168.130.12
            Hostname Option 12, length 16: "basic10.wann.net"
            Domain-Name Option 15, length 8: "wann.net"
            BR Option 28, length 4: 192.168.130.255                       <<<<<<<<<<<<<<<<<<<<<<
            NTP Option 42, length 8: 192.168.130.10,192.168.130.12

After a while what stood out to me was that my DHCP server was returning a Default-Gateway of 192.168.130.1 in the RFC1497 Options section compared to the Gateway-IP/GIADDR set earlier in the packet. Also the broadcast address being set in the Options too.

It would seem that I have a misconfiguration in my ISC DHCP server somewhere. That’s what’s causing the wrong gateway to be returned, and makes sense in that the UEFI loader gets an address but can’t reach anything off the local subnet. Apparently all the previous times I’ve PXE booted these systems I’ve always used IPv6 and never hit this problem until I tested with IPv4. I hacked on my config to temporarily set the gateway manually to what it should be for the test host, and it PXE booted off the network just fine, using the TFTP server on the other LAN.

As to how my DHCP server configuration is wrong, I haven’t figured it out yet. I never put time to understanding how classes and groups were supposed to go, my config looked something like this:

option ntp-servers 192.168.130.10, 192.168.130.12;
option domain-name-servers 192.168.130.10, 192.168.130.12;

subnet 192.168.130.0 netmask 255.255.255.0 {
  next-server 192.168.130.1;
  option routers 192.168.130.1;

  host a {
     hardware ethernet ...
     fixed-address ...
  }
  host b {
    ...
  }
}

subnet 192.168.135.0 netmask 255.255.255.0 {
  next-server 192.168.135.1;
  option routers 192.168.130.1;

  host c {
     hardware ethernet ...
     fixed-address ...
  }
  host d {
    ...
  }
}

And in this example, for whatever reason booting host “D” would get the router from the other subnet. Every subnet example I’ve seen shows putting “options routers” in a subnet scope. I do have some some groups and classes in there that’s clearly fowling things up but I don’t see how.

From what I’ve been reading, class and host are top-level scopes and shouldn’t go inside subnet.

So I just re-wrote my DHCP configuration into what I believe now are the right scopes:

...
option ntp-servers 192.168.130.10, 192.168.130.12;
option domain-name-servers 192.168.130.10, 192.168.130.12;
next-server 192.168.130.10;

# For things that match this class, override global options
class "pxeclients" {
  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";

  if option arch = 00:07 {
    filename "/efi64/syslinux.efi";
  } else {
    # PXELINUX >= 5.X is the new hotness with HTTP/FTP
    filename "/bios/lpxelinux.0";
  }
}

subnet 192.168.130.0 netmask 255.255.255.0 {
  option routers 192.168.130.1;
  include "/etc/dhcp/homenet-130.inc";
}

subnet 192.168.135.0 netmask 255.255.255.0 {
  option domain-name-servers 192.168.135.1,192.168.130.10,192.168.130.12;
  option routers 192.168.135.1;
}

subnet 192.168.136.0 netmask 255.255.255.0 {
  option routers 192.168.136.1;
  include "/etc/dhcp/homenet-136.inc";
}

group homenet {
  host a {
    fixed-address 192.168.130.x
    ...
  }
  host b {
    fixed-address 192.168.130.x
    ...
  }
}

group otherstuff {
  host c {
    fixed-address 192.168.135.x
  }
  host d {
    fixed-address 192.168.135.x
  }
}

Instead of putting a “host” inside of the “subnet” scope, I put them all at the top level. Apparently dhcpd just “knows” that a host belongs to a subnet based upon the fixed-address matching the subnet+mask given in the subnet declaration, instead of trying to use the “subnet” scope to organize “host” entries.

After doing this, factoring out some duplicate classes to set the filename, UEFI PXE booting my test system worked on the first try! I’m past the point of caring now to bisect my original config, this was a detour from doing other things. Besides, I should be burning all of this down and finishing my migration to the Kea DHCP server.

Supermicro / Intel Atom 2550 fix

I’ve loved using the A1SRi motherboards in ikeacluster for years, they offered a lot of RAM power with a fan-less embedded CPU. I’ve ran them for several years and had a couple eventually succumb to the Atom C2000 clock failure problems. The BMC would work but the motherboard would just decide one day to stop booting.

For years I wrote these off as dead and what I thought as beyond the period at which people were getting Supermicro to replace them. I was sad there was not a great replacement. I seem to recall the Denverton mini-ITX boards were either late or were considerably more expensive. There were some other A1SRi boards on ebay, but they were expensive and who knows when they’d die.

A while back I had read on some forums where people were working on a fix to revive these motherboards. I finally got around to reading up on the Serve The Home thread and a Truenas thread to try it out. On my boards I ran a 200 ohm jumper between pins 1 and 9 on the TPM header and that seemed to do the trick:

TPM header jumper

It still has the problem where the BMC is still alive and I can control VGA+keyboard input, but Linux ipmitool can’t query the BMC nor does serial redirection work. But I’m happy I was able to revive two of my motherboards! It also looks like even as late as 2023 people were getting these boards replaced by Supermicro, so I might have to give that a try.

Update 27-Oct-2024:

huh, I thought I had read this fix breaks connectivity between the OS and the BMC, but serial-over-LAN and things like ipmitool lan print work:

ipmitool sol activate after resistor fix

ipmitool viewing BMC sensors and network settings

I’ll take it!

The whole Let’s Encrypt thing has the side effect of making me cranky every few months as I go around checking what expired, what automatically renewed, and what needs more babysitting.

Today I want to bitch about the Ubiquiti UniFi controller software. As far as I can tell even the mere CONCEPT of updating TLS certificates STILL does not exist anywhere in the controller or the support documentation. Sure they make a nice web UI to manage your 11ty-dozen wireless APs, cameras, doorbells, LED panels, key readers, and whatever thing they’re pushing this month, but keeping the web UI secure and up to date in a post-Snowden world? Nah, screw you. Not even a clumsy annoying web way to do it, no “click here to re-generate a self-signed certificate”, not even a sanctioned command line way to do it. You’re utterly on your own to figure it out. I guess this is one carrot of forcing people to use their cloudy UI.com service.

This has lead to countless people like me reinventing the wheel since 2016 and poking at the Java keystore directly with the old ACE.jar and keytool tools. You did naturally assume it’s a Java keystore the first time you encountered self-signed or expired certs warnings, right?

It’s even worse now when you layer all the Let’s Encrypt tools on top of it, because virtually all of them assume you’re on some form of Ubuntu or Linux. You won’t know it until you try to run a deploy script or read the code. I’m running it on MacOS which is a sanctioned platform and gets regular releases. The official acme.sh/deploy/unifi.sh claims it supports self-hosted, but it really assumes self-hosted on Linux. I’m afraid to know what the Windows people have to deal with.

What I wound up doing is using and tweaking the unifi_ssl_import.sh script from https://github.com/stevejenkins/unifi-linux-utils. This takes care of exporting a PKCS12 file and importing it into the Java keystore. It assumes Certbot and Linux, but it easily adapted to Acme.sh paths on MacOS. Thank god this isn’t some gigantic monolith of bash and it is fairly straightforward. I run only this script and it takes care of updating the UniFi keystore.

It is not automatic upon renewal, and doesn’t automatically restart the Unifi software. Those are problems for another day, maybe in 100 more days.

-UNIFI_HOSTNAME=hostname.example.com
+UNIFI_HOSTNAME=${HOSTNAME}

# Add this to override all of the Fedora/CentOS/Ubuntu/CloudKey paths
#
+# MacOS paths
+UNIFI_DIR="${HOME}/Library/Application Support/UniFi"
+JAVA_DIR="${UNIFI_DIR}"
+KEYSTORE="${UNIFI_DIR}/data/keystore"

# Script assumes Certbot paths, tweak for acme.sh
+# MacOS, this time for acme.sh
+ACMEBASE="${HOME}/.acme.sh/${UNIFI_HOSTNAME}"
+PRIV_KEY="${ACMEBASE?}/${UNIFI_HOSTNAME}.key"
+SIGNED_CRT="${ACMEBASE?}/${UNIFI_HOSTNAME}.cer"
+CHAIN_FILE="${ACMEBASE?}/ca.cer"

# Add -legacy option to openssl in two spots
+    openssl pkcs12 -export -legacy\

Maybe someday I’ll get around to sending in a PR to add MacOS support for the deploy script, but not today. I’ve already spent too much time shaving this yak and have other things to do.

Unhelpful responses from the peanut gallery on this issue:

  • Just type in “thisisunsafe” every time in Chrome! fucking hell, this isn’t even attempting to solve the problem. would you tell your director or CISO to do this?
  • Just proxy it behind Apache/Nginx/Linux!  no. now I have to support and configure two things.
  • Just run it on Linux! Bro, I swear a raspberry pi is all you need, bro please! no. see above, now I have to support an entirely different piece of hardware and OS.
  • Just don’t run the web UI! bro, their entire product revolves around running a web UI, how do YOU run it?

Or you know, Ubiquiti could actually provide a mechanism for uploading a new certificate+key pair.

TL;DR: Controller said RAID1 was lost after disks being powered on for first time after 20 years, I didn’t believe it. Booted into Linux and dd’d the last good disk. Recovered the UFS filesystem, I have 20 year old artifacts to sift through.  Always take images of your drives before mucking with them.

The main database server / admin server for my old ISP was a Dell PowerEdge 1550 1U server running Solaris 8 x86, on three 36 GB Seagate Cheetah SCSI U160 hard drives. It was shut down in 2004 when I folded the company, but I hung on to the drives in case I needed the records for disputes or something, and repurposed the server as a colocated shell server. I almost took the system to e-waste a few months ago when I was purging a bunch of other old rackmount servers from my storage unit, but decided to hang on to it for whatever sentimental reason a little longer.

Recently I was digging through old files to find old ISP setup notes. I found what I needed on my laptop, but it made me remember I still had the ISP drives and I should see if I had any more vintage notes and squirrel away an image of the OS so I could finally ditch the hardware. I had no intention of ever firing this stuff up again and considered it a forgotten memory. The old hard drives have been in my drive collection in the bedroom, so that’s about as good as storage as they get.

In search of RAID

During the time at the ISP the server was using a Dell/Adaptec PERC hardware RAID controller, so I’d need that to revive the data. I took the controller out when I switched to Linux with software RAID using the on-board Adaptec AIC-7899 SCSI controller, and I have no idea what I did with it. I probably e-wasted it a long time ago. So first thing I needed to do was find out what kind of PERC card it had and go find one on eBay. My system was so old I couldn’t even look up the service tag on Dell’s website anymore. The PowerEdge 1550 has been lost to time, there’s very few photos of it online, and none that I found with a PERC installed to reference. I guessed from some service notes and went with a Dell 493 PERC 3/DC card, which sounded vaguely familiar and was around the right vintage.

I made sure the system could actually power on and put in a set of Linux disks from the colo days. Other than a dead CMOS battery, the system eventually booted into Linux as a test just fine. I have no idea why but it takes several minutes for POST to run and load the Adaptec 7899 BIOS, I don’t remember it being this achingly slow.

Next it came time to try the Solaris hard drives. I had no idea what RAID configuration I used, I kind of assumed I probably did a RAID 5. No idea of the order of the drives. I wasn’t even sure which version of Solaris was on there. I first powered up the system without the drives, went into the PERC firmware and reset all the logical device configuration to defaults. I popped in the Solaris drives and right away on boot the PERC BIOS spun up two drives.

Going into the PERC BIOS again, it had imported a RAID1 configuration from the drives. Two drives were in a logical group, one marked ONLINE and one marked FAIL. The third drive was marked as HOT SPARE. That was a promising start!

A brief glimmer of hope after 20 years

I didn’t put a lot of care into trying to recover this, it was more of a nice-to-have. #YOLO. I let the system boot, told the PERC to proceed with the degraded logical volume group. Up pops the blue Solaris Boot Subsystem screen! Right at this same time the PERC alarm starts SCREECHING because of the failed drive and it was LOUD. I had forgotten all about this and there were no buttons or anything anywhere to silence it. There’s no way I could work on this thing in an apartment with that going off.

I hit the power button to turn off the system, turned it back on and went back into the PERC menu to silence the alarm. Except now in the PERC BIOS all drives were marked FAILED! wtf!

 

Artists re-enactment of RAID failure

I wasn’t completely convinced the drives died all of a sudden after one power-off and thought it was more likely there was some sort of bad state stored in the RAID configuration from the power-off. I fiddled with it for a while, trying to remove the config from the card and re-importing it, moving drives around in drive slots, and it kept coming back as FAILED. One of the disks had to still be working to read the RAID config I thought. I also didn’t know the numbering of the drive slots, so I wasn’t sure which two were the data drives and which was the hot spare anymore. Did I mix the old hot spare into an order it expected to find a RAID member? Did one RAID member just die?

So I put it all aside for a few weeks to ponder.

What to do

If it was a RAID1 I thought in theory both drives should have a usable set of data outside the RAID metadata, provided they were still mechanically functional. Even if the sync was broke and one had a slightly older set of writes, this was fine for this archeology dig. The question was if the RAID metadata would throw off any tools to poke at the filesystem. Message board posts all suggested if anything hooking the drives up to a non-RAID SCSI controller to take the hardware RAID out of the picture and taking images of the drive if they showed up, that way they could be experimented on with recovery tools. This was slightly more complicated in that the Solaris 8 filesystem is the older UFS, not ZFS or EXT3/4. Several commercial packages promised they could recover UFS for a modest three digit sum.

I decided on hooking the Dell drive backplane directly to the onboard Adaptec SCSI controller and booting Linux. If the drives showed up I could at least dd a copy of them to fiddle with later and would have more tools to poke at the SCSI bus.

Getting Linux over was going to be work, the system didn’t support booting from USB. It had an IDE CD-ROM drive, a 3.5″ floppy drive, and could network PXE boot. While I have a functioning PXE environment and actually PXE installed CentOS on this system when I had it in colo, I long since removed my old CentOS 5 files. Rigging up a PXE bootable Live ISO image just for this sounded like a lot of work. Ubuntu 14 server was the latest i386 version I could find that still fit on a CDR disc. Miraculously I still had five blanks laying around. The only CD burner I owned was in my Windows 95 machine, so instead of shelling out money on Amazon for another external burner, I went to a lot of effort to just burn it using the 486 (at 2x!).

Of course when it came time to boot, the CD drive in the Dell was not working anymore. I wound up throwing together enough PXE glue anyways to boot the CentOS 6.10 i386 installer in rescue mode. This kernel should well be new enough to have all the 2000-era Adaptec drivers built-in.

Struck data!

One by one I tried all three hard drives. The first one oddly showed part of a serial number to the Adaptec BIOS, but otherwise was undetected by Linux. The second drive showed up! An fdisk -l detected two partitions, “Solaris boot” and “Linux swap / Solaris” !!!

I popped in a USB stick which at least showed up as a mass storage device to Linux and I began a dd of the hard disk to it. About 15 minutes later I checked progress on another vty and quickly realized it had only copied a few dozen megabytes and this was probably using USB 1.1 or maybe 2.0 and it was going to take all night to copy this drive. Would the hard disk survive this long? I threw together a dd | ssh command and let it copy a couple of images across the network to another system. It’s a Pentium III 933 MHz system, so not a complete slouch.

Eventually after a couple of hours the dd over the network succeeded without any sort of errors, so I had at least one copy of whatever was on that disk. I have no idea if that was a working member of the RAID1, or if once upon a time it was part of the RAID1 and I demoted it to hot spare without wiping it, or what. The 3rd disk was completely dead, it didn’t show up on the Adaptec at all. So it seems I did lose one disk during my initial power-off.

After I was satisfied I got a good as copy possible, I let the good disk boot in the system by itself to see what would happen. The blue Solaris bootloader screen loaded, then dropped into the configuration assistant. It didn’t seem to find a kernel on disk to boot, but otherwise the disk acted fine.

Over on another Linux system I ran “strings” on the 36 GB image I captured and it clearly had some viable data in it. I saw a bunch of email, sendmail config, html, mysql commands, and other stuff I recognized. Now the question was how to mount this sucker under Linux. I did some reading and Linux does have UFS support, including Sun x86. I learned that Solaris slices are different than typical Linux partitions in that they’re more a set of logical extended partitions within a standard partition. The Linux kernel with the UFS module loaded understands this and as I saw with the Solaris drive inserted over on the Dell, it will enumerate all the possible slices as extra disk partitions, e.g. sda1 sda2 sda3 sda4 sda5 ... sda15 even if tools like Linux fdisk and parted only see a boot and data partition.

Linux recognizing Solaris disk slices

Here’s what fdisk looked like when reading the captured dd image itself:

root@basic06:~# fdisk -l ./image-sda2
Disk sda2: 33.9 GiB, 36328801280 bytes, 70954690 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x69747261

Device Boot Start End Sectors Size Id Type
sda2p1 1851867950 2396369563 544501614 259.7G 6f unknown
sda2p2 1397314113 3266884704 1869570592 891.5G 20 unknown
sda2p3 0 0 0 0B 6f unknown
sda2p4 20480 20480 0 0B 0 Empty

Partition table entries are not in disk order.

Trying to mount UFS from Linux

(See 9/26 update where a newer kernel fixed all this) I tried a variety of ways trying to mount the UFS filesystem on Linux with no luck. Neither “mount -t ufs -oro,ufstype=sunx86” on an extended device id for a slice such as /dev/sda10 worked, nor on the raw image file of just the 2nd Solaris data partition nor image of the entire disk. I tried some examples of calculating offsets to mounting specific slices or possibly avoid any RAID metadata and those didn’t work. I got a variety of wrong fs type, bad option, bad superblock, or ufs: ufs_fill_super(): bad magic number errors with these attempts. losetup and friends didn’t seem to work for me either, which to be fair I’ve never used.

Another idea I had was to copy the image to a USB stick on another system and letting the kernel detect it as a drive again. Trying to mount it this way didn’t work while I was booted into CentOS 6, I thought maybe a newer kernel would help. I let it copy to USB while I went on to try the next thing, installing Solaris. (I wound up not using this)

Installing a Solaris 8 VM

I gave up and installed Solaris 8 Intel in a VirtualBox VM to see if I could mount the image there.. It’s been yeaaaaars since I’ve touched Solaris, much less v8, but I got something working. I had to convert the dd image to a .VDI image so VirtualBox could actually present it as a drive to the VM. (“VBoxManage convertdd image1-sda image1-sda.vdi --format VDI“).

Within Solaris I had to run devfsadm after boot to get it to recognize this as another IDE drive. It showed up as /dev/dsk/c0d1, and “format” listed a bunch of slices when it was mounted!

Finally, success!

At long last I was finally able to mount the individual slices! and there was intact filesystems with my files!

Browsing around it looked familiar, all bits and pieces of a working system. It looks like this stuff is somehow from about 2003, so this may be leftover from a drive swap, I don’t know.

I also forgot Solaris doesn’t have anything like ssh or rsync out of the box, or I forgot where to install it. So I’m going old-school and running a “tar | rsh” to another system to sift through it more.

I am curious to go looking for the hardware RAID metadata on this disk, is it at the beginning, the end? What does it look like?

Update 9/26:

Fiddling with the whole disk image on a CentOS 7 system with a 5.3.5 kernel, I have success mounting the UFS filesystem, whereas this was failing over on Ubuntu 18 with a 4.15 kernel:

# Mounting with a loop device
[root@basic03 ~]# losetup --partscan --find --show ./staff1-9pf-sda
/dev/loop0

[root@basic03 ~]# dmesg -T
[Thu Sep 26 23:46:44 2024] loop: module loaded
[Thu Sep 26 23:46:51 2024]  loop0: p1 p2
  p2: <solaris: [s0] p5 [s1] p6 [s2] p7 [s3] p8 [s4] p9 [s5] p10 [s6] p11 [s8] p12 >

[root@basic03 ~]# fdisk -l /dev/loop0
Disk /dev/loop0: 36.4 GB, 36420075520 bytes, 71132960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *       16065       48194       16065   be  Solaris boot
/dev/loop0p2           52610    71007299    35477345   82  Linux swap / Solaris
[root@basic03 ~]#

[root@basic03 mnt]# mkdir s0 s1 s2 s3 s4 s5 s6 s7 s8 s9

# Mounted Solaris slice 0 containing / using the linux /dev/loop0p5 partition
[root@basic03 ~]# mount -oro,ufstype=sunx86 /dev/loop0p5 /mnt/s0

[root@basic03 ~]# mount | grep mnt
/dev/loop0p5 on /mnt/s0 type ufs (ro,relatime,ufstype=sunx86,onerror=lock)

# Solaris / directory!
[root@basic03 ~]# ls -l /mnt/s0
total 39
lrwxrwxrwx  1 root root     9 Sep 24  2001 bin -> ./usr/bin
drwxr-xr-x  2 root root   512 Sep 24  2001 boot
drwxr-xr-x  3 root 60001  512 Sep 24  2001 cdrom
drwxr-xr-x 12 root sys   3584 Sep 27  2001 dev
drwxr-xr-x  6 root sys    512 Sep 24  2001 devices
drwxr-xr-x 30 root sys   3584 Jun  8  2003 etc
drwxr-xr-x  3 root root   512 Sep 24  2001 export
...

 

Improvised drive sled

[photos: flickr – Macintosh Quara 700 drive sled]

The Quadra 700 I acquired had the internal plastic assembly that held the floppy drive and hard drives, but didn’t have the sled that the hard drive went in and clipped into the system. These are hard to find on top of an already hard to find system, lore seems to be when recyclers yank the hard drives, they discard the sleds. The Quadra 700, IIcx, and IIci all use the same sled, model numbers starting with 805-5078 or 815-5078. I checked several component places and eBay, and nobody had any for sale. You can still put the hard drive in the Quadra, there’s just nothing holding it in place preventing it from flopping around.

Fortunately lots of photos of the thing exists and it’s just a U-shaped piece of sheet metal with some holes and tabs stamped in it. It seemed easy enough to just go make one. I broke out the ruler and caliper and made some measurements of my own system. Later I discovered this post on 68kmla by Phipli who had a drawing of the drive sled which gave me the outside dimensions and let me fine tune my own measurements a bit more. Then I discovered the 3D printed version by branchus on Thingiverse (I love his Mac repair streams). I don’t have a 3D printer, and didn’t feel like going out to learn Fusion360, how to use a 3D printer, and tracking down one of our libraries just for this when I already have the metal and a metal brake. A local 3D service quoted me $38 to print one, which felt steep.

Nibbling the holes

I started with a piece of 18 gauge aluminum 85mm x 196mm. I didn’t yet know how much I needed to compensate my measurements for the bending so I started working from the inside going out. I made sure the inner dimension was at least 103mm wide to allow a 3.5″ drive to slip in. First I nibbled out holes for the raised square bit at the bottom and the sides.

I bent a small scrap piece to figure out the bend would eat about 1.5mm, and used it to find the position and dimensions of the vertical holes where tabs would lock in. I finished marking all of these on the metal. If you can read my scribbles, that’s all of the dimensions give or take a mm.

My Dremel was utterly dead so I took shortcuts with the rest of the cutting. The OEM part had a D-shaped cut over the side humps presumably to let part of the side remain straight upright while the rest flexed, I omitted this completely. I thought a 1/2″ cold chisel would be perfect for knocking out the side tabs that would lock into the plastic assembly in the case. After a few whacks with a hammer I didn’t punch through the aluminum like I had hoped for, so I opted for the jankiest part of this whole thing by hammering a screwdriver through it!

Being punched through actually worked pretty well at giving me protruding tabs on the outside surface, smoothing a bit here and there with pliers to get it just right. Next up I used my metal brake to fold the sides up. I didn’t trim the tops of the vertical pieces like the original to provide finger tabs, it seemed to fit fine without them.

The thing actually fit into the system almost exactly the first try. I had to do a bit more nibbling on the square hole on the middle and square off my bend and it fit nicely. I messed up drilling the screw holes, so they aren’t pretty, but they work.

All in all, the thing works. I can pinch the sides to take the drive in and out, and it locks the hard drive in place. I’d say pretty good for a Saturday afternoon of tinkering around without the right set of tools. Now I can continue lurking eBay sales hoping for an original sled, or get around to having one 3D printed someday. If I had a OEM sled I would be temped to get better measurements and send off somewhere to laser/plasma cut a few dozen sleds to hand out but eh I don’t want to be in the shipping biz.

Final product

Around 1995 I had a Zoom VFX V.32bis 14.4k modem as my main workhorse. It was a white plastic shell with a smoky brown translucent front face. I decided to buy one recently for old times sake:

Zoom VFX V.32 bis 14,400 bps fax modem

I also came into possession of a Telebit Netblazer PN (which I need to finish working on and write up about it), which lead me to searching for manuals and more information about it. I stumbled across this eBay listing for a Telebit Teleblazer:

Telebit TeleBlazer V.34 modem

It’s the exact same case! Back, front, shell, underneath, face, font of the V.32bis / V.34 badge, it’s all the same! In one photo of I think the box it mentioned being based on a Rockwell chipset too. Previous Telebit modems such as the Txxxx series, Worldblazer, all had their own blocky look. At first I thought the seller had the wrong modem, but after looking at the pictures it’s very much a Telebit branded product with “Telebit TeleBlazer” on the bottom. Funnily there was one auction for $250 and another for $18 for the same kind of TeleBlazer.

This lead me to do even more digging. My Zoom VFX V.32bis was made by Zoom Telephonics Inc in 1991-1992 or so. It’s based on the Rockwell RC144DP data pump. There’s also another VFX V.32bis with a different solid, slant-front, white case that came out later I think because it was used in their later 28.8k, 33.6k and 56k models. I’ve only seen the translucent brown plastic case on VFX 14.4k modems, never anything newer.

Telebit was well renown for producing modems with fast transfers ahead of their time using their own modulation system and throwing a Motorola 68000 at it for processing oomph. The Netblazer I have has a modem chip produced by AT&T. Apparently around 1993 Telebit was trying to put out a V.34 modem like everyone else and just decided to buy Octocom Systems, who was developing their own V.34 modem. Telebit also wanted to put out their own low-cost V.34 modem to compete, so I’m guessing that’s probably how they wound up using a Rockwell chipset.

What’s interesting is that Zoom Telephonics Inc was based in Boston, MA in 1992. Octocom Systems was based in Wilmington, MA, about 15 miles outside of Boston. Did proximity have anything to do with this case story? Were there ex-Zoom employees who went over to Octocom and took their case design with them? Did Zoom sell a bunch of pallets of leftover cases to Ocotocom or Telebit? Did Zoom and Telebit share the same ODM and Telebit said gimme the cheapest case you got and ship it?

I never did find any interesting stories or gossip to explain why they used the same case. I’d also be curious to tear open a Telebit TeleBlazer to see if it even uses something like the Rockwell RC288 datapump, which everyone seemed to be using by then. But I’m not $40 curious to buy one. Further, if their V.34 modems are based Rockwell chip, is there any Telebit magic left in there, or it just a Telebit sticker on the box?

Also, I’m not getting any good nostalgic memories of this VFX modem I bought, it’s been a dog. In my testing it fails to connect a lot of the time and locks up. I don’t know if it’s because the components are aged or if this thing got damaged somehow. The modem speaker only has one sound, LOUD, no matter if I use ATM1L0 or ATM1L3. The owners manual for the VFX V.32bis can be found over on archive.org (ZV32BIS.ZIP), it has some interesting subtleties such as only MNP enabled out of the box and you have to go find the command to enable V.42/V.42bis/LAPM support. For whatever reason even if the connection is using MNP5, the DC/EC lights on the front don’t come on, you have to be using V.42/V.42bis before they light up. Fortunately it’s data compression is in hardware, it’s not one of those janky Rockwell RPI chipsets that required a driver to punt EC/DC off to the PC’s CPU. I had completely forgotten those cheap bastards existed.

Custom aux/roll/null/DCD cable

This is part of the project to connect my Wildcat! BBS to a retro X.25 network, but it also applies more broadly to “reverse telnet” operation of a Cisco router where you telnet/ssh to a router at a given port to access a serial device hanging off of the aux or a terminal line. I don’t think there’s a lot of people seeking this solution, but I’m writing about it for when I eventually forget. This post mainly covers the serial connection and Cisco bits, I’m still clueless about the whole X.25 part.

This isn’t quite as simple as slapping a null modem cable between a serial port on the BBS machine and the aux port on the router, altho that’s part of it and would work. The problem is gracefully disconnecting the reverse telnet/SSH session when the visitor is done so the next person can log in. This is done to improve user experience and increase line availability.

Normally when using a reverse telnet session, it’s expected that a user send a ^] to close the telnet connection or a Ctrl-Shift-6 to break out. Until a the user sends a break/escape or a session-timeout happens, nobody else can use this BBS line. And it’s just not a good experience to tell somebody who’s gone to all the effort to connect to your board to oh yeah do this extra step too please. In the worst case this probably means somebody could tailgate in on the end of last person’s session somehow.

TL;DR:

  • Aux cable + modem adapter (with pin 1 and 6 DTR/DCD connected)+ null modem adapter + gender changers
  • “line aux 0” set to “modem printer”
  • chat script to send a string
  • BBS software configured to see said string and start a call, not “auto-answer”

Wildcat! is an MS-DOS program (at least the 4.x version I’m using) that is designed to use RS-232 serial ports to talk to a modem. The manual does discuss connecting to an X.25 PAD, namely a Microtronics CSI-X.25 PAD, so non-modem serial (i.e. using a direct, null modem cable) usage is expected to work.

Wildcat! and probably most BBS software expects to “answer” a serial line and sending a login prompt to the visitor in one of three ways: “auto answer”, detect when the RS-232 CD (carrier detect) line is raised; “ring detect”, detect when RS-232 RI (ring indicate) is raised; and “ring result”, look for specific text strings such as ‘RING’ to indicate an incoming call from a modem. In the latter two cases, Wildcat! will send an “ATA” command to the modem to answer the call. After that, in all cases Wildcat! expects to see a CD signal on the serial port which tells it there’s an active user on the line. If CD is abruptly dropped, Wildcat! will assume the caller has disappeared and will “hang up” its side. If the visitor selects “Goodbye” from the menu screen, Wildcat! will send the DTR (data terminal ready) line low briefly, which is intended to tell the modem to disconnect.

Normally if you connect a PC serial port to a BBS PC serial port with a direct null modem cable, with Wildcat! configured to auto-answer, then start a communications program such as Qmodem, Qmodem will raise DTR as it’s a terminal that’s now ready to process I/O. Wildcat! will see this and immediately send a login prompt to the terminal. However, if somebody logs into the BBS and selects the “Goodbye” menu option to leave, Wildcat! will wrap up the call and get ready for the next caller — in this case with our hardwired connection, in the Qmodem terminal window we’ll immediately see another login prompt. It’s not until Qmodem is exited that Wildcat! finally resets and waits for the next user. (Or you yank the serial cable from the PC).

BBS null modem / X.25 PAD connection

The Wildcat! Sysop Guide really only refers to a one other serial port configuration that doesn’t involve modems, that’s for for hooking up an X.25 PAD. This would allow users to come in via X.25 network such a Telenet or Tymnet, go through the PAD, which acts as a basic terminal server connected to the BBS via multiple serial cables. Which is kind of convenient for me since this is ultimately what I want to do, but with different hardware. If you wanted to configure the BBS to accept connections from something via null modem or a terminal server, you’d have to pick through this section and pull out the bits that look relevant.

The important part of this section of the manual are the details needed to build a wcMODEM .MDM modem profile file to use for the node that’ll be used for the direct connections. For example, creating a file called like DIRECT.MDM with the specified serial port info, options, and removing the modem commands. Then in the batch file that starts the Wildcat! instance for that node, add in a “WCMDM=DIRECT” to have it load the profile for that node.

I looked up the Microtronix CSI-X.25 PAD mentioned in the manual to get an idea of how it actually handed off serial connections. At the bottom of this post I’ve added some details about the history that I could find, I wasn’t able to find any manuals. Apparently the CSI-X.25 is a box with a number of DB-25 RS-232 ports off the back. It says the PAD is configured to “act like a modem that is in auto answer mode .. simply raises carrier detect (CD) when a call comes in”. It mentions other things here like it supports RTS/CTS hardware flow control, and probably running the serial lines at 9600 or 19200 bps. I’m going to go on a limb and assume it probably supports all serial pins, for example it knows when Wildcat! drops DTR to end the session.

It’s worth mentioning MSI did internally support another terminal server setup. For BBS Direct offered by Concentric, I’m told there was a Xylogics terminal server that received callers via IP/frame relay, and handed off via stack of serial cables to the MSI HQ BBS. I guess they made it all work with their BBS software out of the box.

Cisco operations

You can connect the aux port or an async serial breakout cable from a Cisco router to the serial port of a BBS as well. This could be used to provide inbound telnet/ssh connectivity to a MS-DOS BBS that has no concept of TCP/IP. What I’ve discovered is it’s not great when a user ends their session. It’s the same problem as a PC null modem connection, as soon as the user says “goodbye”, Wildcat! ends their session, and gets ready for the next caller. Except Wildcat! can’t drop the serial connection, you’ll see it eventually sending ++++ ATH0 AT&C1D1 commands desperately trying to get rid of the caller and blindly initializing a modem. Then another login prompt is sent.

As mentioned, until a the user sends a break/escape or a session-timeout happens, nobody else can use this BBS line.

What needs to happen is two things: 1) When the session first starts, the Cisco needs to raise DTR to activate the line and raise CD so Wildcat! knows there’s a visitor there. 2) when a visitor says “goodbye” to the BBS, the Cisco needs to see DTR being temporarily lowered by Wildcat! as a signal to boot the reverse telnet session.

Cabling

This setup only works on the aux port or terminal lines via WIC or NM card. To make any of this work start with the serial cable being used. Cisco used to ship along with their baby blue console cables two adapters, a RJ-45/8P8C to DB-9 “terminal” adapter for connecting a PC to the console port for initial configuration, and a RJ-45/8P8C to DB-25 “modem” adapter for connecting a modem to the console or aux port. The difference between the two is the “terminal” adapter took care of setting up a null modem connection (i.e. crossing RX/TX) for you, however the DCD and RI pins are completely left unconnected as they’re not needed. The “modem” adapter is straight through, but connects DCD to DTR, but only comes in DB-25 form.

Apologies: As an aside it’s maddening following pins that get rolled from the aux port to the Cisco blue roll (not Ethernet crossover) cable to the various adapters. It gets confusing to me which signal to talk about too, since they’re all ultimately wired together — do I say pin 6, or do I say DSR or DTR? So if I say DSR and probably mean DTR signal, forgive me.

You’ll either need to use the DB-25 modem adapter in addition to a null modem adapter, and probably a gender changer too somewhere, or edit the DB-9 terminal adapter to add a DCD pin. This turns into quite a stack of connectors. For this experiment my BBS only has DB-9 serial ports coming out the back, so I wound up making my own combo roll + DB-9 + null modem + add DCD cable. I imagine with the newer Cisco console cables that have a molded DB-9 adapter attached, you’ll need a null and a way to fix DCD.

Remember, Wildcat! expects DCD to be up so we have to have that pin connected to something. Only using RI won’t work either, while that may signal that there’s a new connection, Wildcat! still requires DCD afterwards.

It remains to be seen what kind of adapters are needed to do this for something like a CAB-OCTAL-ASYNC from a NM-16A.

Believe it or not, all of these cable combos below do the same thing. Mine is much simpler and prettier but I don’t want to solder more connectors like it.

My ultimate awesome Aux RJ-45 + roll + null + DTR/CD DB-9 cable

A roll cable, DB-25 adapter, gender changer, DB 9/25, cable and a null oh my

Another abomination

My awesome cable pinout, using a regular RJ-45 Ethernet cable, chop off one end and connect as follows (ignore all the labels and just pay attention to the pin numbers):

RJ-45  (Aux)   -  DB-9
1 w/o  (RTS)   -  pin 8 (CTS)
2 o    (DTR)   -  pin 6 (DSR, pin 6 also jumpered to pin 1)
3 g/w  (TXD)   -  pin 2 (RXD)
4 bl   (GND)   -  pin 5 (GND)
5 bl/w (GND)   -  pin 5 (GND, blues are grounds, connect together)
6 g    (RXD)   -  pin 3 (TXD)
7 br/w (DSR)   -  pin 4 (DTR)
8 br   (CTS)   -  pin 7 (RTS)
-n/a-          -  pin 1 (jumpered to pin 6)

Plug directly into aux port of router.

Aux port configuration

TL;DR: Through much trial and error I settled on configuring my aux port as “modem printer” and “script connection RINGRING” which I’ll explain why.
Cisco IOS provides a variety of options for setting up the aux port for serial operations, and there’s a whole document describing modem signal and line states. Here’s a document for aux pinouts too.

vintage-gw2(config)#line aux 0
vintage-gw2(config-line)#modem ?
  CTS-Alarm       Alarm device which only uses CTS for call control
  DTR-active      Leave DTR low unless line has an active incoming connection or EXEC
  Dialin          Configure line for a modern dial-in modem
  Host            Devices that expect an incoming modem call
  InOut           Configure line for incoming AND outgoing use of modem
  Printer         Devices that require DSR/CD active
  always-on       Configure line for a modern always-on modem
  answer-timeout  Set interval between raising DTR and CTS response
  autoconfigure   Automatically configure modem on line
  dtr-delay       Set interval during which DTR is held low
  onhold          Set the V.92 modem on hold timer duration

vintage-gw2(config-line)#

Ideally we need a config option that does /something/ different on the serial line when a reverse telnet session is started, that way we have signal (a literal electrical signal) to the BBS that there’s a new visitor on the line. Then we could wire that up to DCD so that pin is alive when there’s a reverse telnet session in progress. Also we do not care at all about “inbound” or “exec” sessions, that’s for something connecting TO the Cisco from a serial port.

I’ve gone through every single one of these options with an RS-232 LED breakout and there are exactly two options, “modem Host” and “modem DTR-active” that actually change state. They raise/lower pin 6 for DTR. Normally it’s low/off, but when a remote telnet session comes in, DTR is raised. All other pins 4, 7, 8 all stay the same. One would assume they could connect pin 6 to pin1 so that when DTR is raised it also raises DCD, and Wildcat! could be set up to auto-answer. While that is technically true and does get the visitor to the BBS, it doesn’t solve our original problem of graceful session endings.
I did find other people sell Cisco DB-9 connectors with DSR/DTR connected to DCD (pin 1 and 6), so I’m not crazy in imagining this need.

Nice, but this isn’t the problem we’re trying to solve!

Bye bye bye

Now we need a way to signal back from the BBS to the Cisco that the session is ended, the serial line has been dropped, go disconnect the reverse telnet session.
When a person does “goodbye” from Wildcat!, Wildcat! lowers DTR from the BBS side. When connected to a null modem adapter, this means DSR on the Cisco side changes — except when using “modem Host” or “modem DTR-active” nothing is paying attention to DSR! The Cisco has no idea Wildcat! is telling it the user has hung up and keeps DTR high.

The only option I found is “modem Printer“. Apparently there used to be an option called “modem cts-active” that got replaced by “modem Printer“, but “modem Printer” isn’t really documented in the Modem Signal and Line States document. Anyways IOS says “modem printer” is “Devices that require DSR/CD active“. That’s exactly what we want here, when Wildcat! lowers DTR, it lowers DSR on the Cisco side and yeets the reverse telnet session!

But, this conflicts with our previous step in that with “modem printer” our DTR and thus CD is always asserted! Wildcat! will not be able to auto-answer and we’ll never get a new caller!

RINGRING banana phone

With “modem printer” configured on the aux port to gracefully disconnect visitors, and our DCD line is hardwired to be constantly active, we need another way to signal to Wildcat! that there’s an inbound caller.

What I did here was configure Wildcat! to use “ring result” and gave it a completely made up string to look for, “RINGRING“.

wcMODEM .MDM file for my fake X.25 PAD

Made up RINGRING string

Then on the Cisco side, I configured a simple “chat-script RINGRING "" RINGRING“, and on the aux port, “script connection RINGRING“.

Now when a reverse telnet session starts up, the Cisco sends the text “RINGRING” down the serial port. Wildcat! sees this and answers the line, all transparent to the user. The visitor can use the BBS all they want, I even tested this downloading a 10 MB file with Ymodem and it all worked.

Along with the right cable and aux settings, then when the user says goodbye from the BBS, their reverse telnet session gets gracefully disconnected.

For whatever reason this is not completely perfect, both the Cisco and Wildcat! seem to be trying to fiddle with serial lines for several seconds before things settle down and the next visitor can log in. But it’s a heck of a lot better than what it was!

Final Cisco config:

!
chat-script RINGRING "" RINGRING
!
line aux 0
 session-timeout 5
 no motd-banner
 script connection RINGRING
 modem answer-timeout 5
 modem Printer
 rotary 1
 no exec
 transport input pad telnet ssh
 autohangup
 stopbits 1
 speed 38400
 flowcontrol hardware
!

It also has the nice benefit that if the BBS is down or if a visitor is already using the BBS, the Cisco sends a “Connection refused” instead of black-holing the caller into nothingness of an empty session. I tried setting up some sort of “connection in use, try again later” thing, but doesn’t work like this.

For whatever reason I have the .MDM file set up to force “yes this is a reliable connection give me Ymodem/G” option, but it doesn’t take effect. I tried configuring the Cisco to send something like “FAKELAPM” as a string to tell Wildcat! it supported error correction and enable it, or send “VMP” to pretend it’s an OS/2 virtual modem, but neither worked. Oh well.

This is what the aux port looks while idle:

vintage-gw2#show line aux 0
   Tty Line Typ     Tx/Rx    A Modem  Roty AccO AccI  Uses  Noise Overruns  Int
      1    1 AUX  38400/38400 - printer   1    -    -   142     34 1786/5780   -

Line 1, Location: "", Type: ""
Length: 24 lines, Width: 80 columns
Baud rate (TX/RX) is 38400/38400, no parity, 1 stopbits, 8 databits
Status: Ready, No Exit Banner, CTS Raised, Modem Signals Polled
Capabilities: EXEC Suppressed, Hardware Flowcontrol In,
  Hardware Flowcontrol Out, Modem CTS-Required, Hangup on Last Close
  MOTD Banner Suppressed
Modem state: Ready
Modem hardware state: CTS* DSR*  DTR RTS
Rotary address 51010000
Special Chars: Escape  Hold  Stop  Start  Disconnect  Activation
                ^^x    none   -     -       none
Timeouts:      Idle EXEC    Idle Session   Modem Answer  Session   Dispatch
               00:10:00       00:05:00                       none     not set
                            Idle Session Disconnect Warning
                              never
                            Login-sequence User Response
                             00:00:30
                            Autoselect Initial Wait
                              not set
Modem type is unknown.
Session limit is not set.
Time since activation: never
Editing is enabled.
History is enabled, history size is 20.
DNS resolution in show commands is enabled
Full user help is disabled
Allowed input transports are pad telnet ssh.
Allowed output transports are pad telnet rlogin lapb-ta mop v120 ssh.
Preferred transport is telnet.
Shell: enabled
Shell trace: off
No output characters are padded
No special data dispatching characters

And this is what it looks like with a user on (not much difference):

vintage-gw2#show line aux 0
   Tty Line Typ     Tx/Rx    A Modem  Roty AccO AccI  Uses  Noise Overruns  Int
*     1    1 AUX  38400/38400 - printer   1    -    -   143     34 1786/5780   -

Line 1, Location: "", Type: "SCREEN"
Length: 59 lines, Width: 174 columns
Baud rate (TX/RX) is 38400/38400, no parity, 1 stopbits, 8 databits
Status: Ready, Connected, Active, No Exit Banner, CTS Raised
  Modem Signals Polled
Capabilities: EXEC Suppressed, Hardware Flowcontrol In,
  Hardware Flowcontrol Out, Modem CTS-Required, Hangup on Last Close
  MOTD Banner Suppressed
Modem state: Ready
Modem hardware state: CTS* DSR*  DTR RTS
Rotary address 51010000
Special Chars: Escape  Hold  Stop  Start  Disconnect  Activation
                ^^x    none   -     -       none
Timeouts:      Idle EXEC    Idle Session   Modem Answer  Session   Dispatch
               00:10:00       00:05:00                       none     not set
                            Idle Session Disconnect Warning
                              never
                            Login-sequence User Response
                             00:00:30
                            Autoselect Initial Wait
                              not set

 

Microtronix CSI-X.25 PAD

Cableshare Inc X.25 Data Concentrator

The PAD that’s mentioned in the Wildcat! has definitely been lost to time. I can find very little about this, no manuals, and maybe a couple of times it was for sale 30 years ago. Doing some sleuthing apparently it was originally made by Cableshare Inc. in London, Ontario as the “X.25 Data Concentrator”, and then starts showing up as the Microtronix CSI-X.25, who is also based in London, Ontario. I’m assuming CSI == Cableshare Inc.   I found exactly two articles even talking about it, an IEEE Communications Magazine “New Products” article from March 1984 that has the only photo of it, and a Computerworld from December 12, 1983 announcing it at $2,700 per port for a four port config. Sounds like off the back it had up to 16 DB-25 ports for connecting to thing.

So if you see one or its manuals, send it my way! Microtronix is still around, looks like they did a couple more X.25 devices, but have long left it behind.

Macintosh SE

I think I may have used a classic Macintosh once in my life, at a Kinko’s copy location of all places. We didn’t have them in school, we went from Commodore CBMs, to Apple IIe, to IBM PC 8088 clones. At the ISP I borrowed a customer’s PowerBook overnight so I could get experience with System 8 and to write how-to instructions for setting up dial-up accounts. It was nice but I didn’t bite. By 2003 when I finally bought my first Mac, a PowerBook G4, I started off on OS X.

At VCF I was playing with some of the classic macs on display and later saw some at the consignment sale. I thought why not, I am an adult, I can buy one if I want to. (Which is how I wound up buying USR Courier modems). I knew literally nothing about classic Macs, quickly googling what the difference between an SE, a Classic, and a 128k. I decided on an SE, it had an Asante Ethernet card, it seemed like a good deal so now I own an SE.

After getting home and using it, I quickly learned about the 800k floppy drives in them and I had nothing that could write disks for it. I was beginning to wish I had held out for a SE/30 with a FDHD, but here we are. It had 800k floppies of System 6 and the ethernet card driver, that was it. This is where I learned about BlueSCSI and using SCSI Zip drives to copy files to it, so I ordered an external BlueSCSI.

The next day at VCF I was browsing the sales again and this time there was a IIsi for sale, I think for like $30. I kinda wanted a color mac but learned the classic compacts didn’t have color. The small form factor won me over, it was running System 7, had a 1 GB hard drive in it, and another Ethernet card. I could at least stick it somewhere and it wouldn’t take a lot of space.

 

Macintosh IIsi

Now I suddenly owned two Macs! I thought as a bonus I could use it to write 800k floppies for the SE, but that doesn’t seem to be the case. Those damn 800k drives. When I took the IIsi home and opened up to examine it, it’s clear somebody took very good care of it. Not only did it have the huge 1 GB replacement hard drive, all of the components were very clean. I found out later the logic board had been re-capped, had a new battery, and the guts of the power supply had been replaced with a PicoPSU adapter. Very nice.

The IIsi has been a blast to use, there’s just something nice about the System 7 interface. I maxed it out with 64 MB of RAM which seemed to help the speed a bit. It had copies of apps such as After Dark, Lemmings, Oregon Trail,  I was able to get some floppy disk images from sites such as Macintosh Garden to load on ZTerm, Network Software Installer, and a few other tools. Then I started copying larger files to the BBS and downloading them to the IIsi using a modem. Eventually my AAUI transceiver cable came in and I was finally able to hook up Ethernet.

My BlueSCSI finally came in last week, so now I’ve been able to make more progress with the SE. I’ve been able to get it online. I’ve been inside it once to check things out, it does not look like a trivial machine to take apart like the IIsi. Watching some videos it seems I’ll need to replace the battery on it and possibly recap it, and the seller’s tag noted the floppy drive worked but needed to be lubricated, so all that is probably up next for it.

One more thing

Then I got a Quadra 700. I knew about the whole Jurassic Park thing, and while I love that movie, that aspect didn’t really appeal to me. When I saw the 700 at VCF I thought it was the neatest mini-tower kind of case, smaller than a PC mini-tower even, it spoke to me. That beige, those lines in the case, and Apple rainbow apple on the front, mmmm. Then I found out they’re a big collectors item because of the whole said movie thing. Prices for previous eBay auctions were all over the place, some beat and yellowed to hell, some in mint condition, from a few hundred for parts chassis to well over $1k for fully kitted systems with the PowerPC card, and they seemed to come along once a month or so.

I set an email alert, not expecting to see a system come by for a long time. Then by sheer luck several days later I happened to be up late at night browsing eBay when a Quadra was added, it looked in decent shape so I jumped on it. This is gonna be another round of picking up the bits to build it up, so far I’m in the process of getting RAM, VRAM, a drive sled, and a hard drive.

Older Posts »