bryan wann :: binary fury

I’ve been happily nerding out with vintage bulletin board system (BBS) software and UUCP to send/receive e-mail and newsgroups for a while. Something I kept seeing in BBS documentation was mention of the “.BAG format” or “UUCP BAG format” used by satellite providers when delivering Usenet feeds over satellite. I got curious about what exactly were these files and what did they contain? The term BAG or bag file never appeared in any of my UUCP books, Taylor UUCP, nor INN documentation. (Batching is mentioned, but not bags) The file type seemed oddly standardized for such a niche application and several BBS utilities claimed to understand .BAG files.

---

TL;DR: .BAG is simply an ASCII file containing a batch of newsgroup articles, as batched together for example by batcher(8) from INN, for easy delivery over UUCP. Also known as “rnews batching”. Instead of potentially thousands of individual files with one article each, it’s a sequence of larger file with a series of articles inside it. As the file exceeds a given size a new file is written with an incremental filename. It’s not explicitly for UUCP nor is it something that only satellite providers invented. It’s unclear if the bag term originated from satellite providers or DNews.

---

I’m familiar with satellite services such as Planet Connect and PageSat, which delivered a Usenet newsgroup feed, Fidonet, and other files over a over one-way satellite link. These operated at speeds between 19.2 – 128 kilobit/second. In terms of BBS usage, the receiver software would get these so-called .BAG files over the air throughout the day, write them to disk, and then some sort of tosser software would import them into the BBS message conferences. These services weren’t limited to BBSs — ISPs, businesses, and anyone else that wanted Usenet newsgroups could use them for a news feed as an alternative to fetching over their expensive Internet connections.

What I didn’t know was what was the format of these .BAG files were and was it possible to re-create them to re-create a dummy “satellite” service? After all, at the end of the day it seemed like it was just a stream of bytes that came in over a serial connection. I had so many questions: Was this some sort of binary stream with things like variable values? Binary blobs? Were the .BAG files like .ZIP archives that contained different file inside like a mix of usenet, Fido, and shareware programs?  Just ASCII text? Was it just RFC-822 messages in a stream?  Information was scarce on it but after spending quite a while searching around I finally started uncovering information.

One of the first things I found was this post to news.misc by Norman Gillaspie from PageSat in 1993. Here he goes into detail on how their satellite feeds work. Here he mentions “These files are written … with a *.bag (for mailbag) file extension”. This got me wondering if these were just Unix mbox style files that contained usenet articles.

A while later a very enlightening thing I found was from The Unix Heritage Society mailing list from 2018, when somebody else asked the same question “Does anyone know why UUCP “bag” files are called “bag”? The interesting part is when they asked one of the authors of the HoneyDanBer UUCP software and about “bag” they said they had never heard of it, so this was clearly coined somewhere else.

The thread went on to mention it was associated with the DNews software, which was popular for “suck” newsfeeds. (As opposed to IHAVE feeds where a NNTP server offers up all articles to a peer, a suck feed requests articles and stores them on the local news server saving bandwidth of taking a full feed.) For outbound feeds, DNews offered the ability to write batches of articles to a “bag file” or “rnews uucp bag” as of version 2.7. The documentation has a section called “writing uucp bag files“. The format of the .BAG file is even mentioned on the mailing list thread:

    The BAG/UUCP file format is:

    #! rnews nnnn
    ...(article, exactly nnnn bytes, counting each end of line as one byte)
    #! rnews nnnn
    ...(next article)...

This all kind of makes sense, if you’re a satellite data company and need to send files over the air, why invent some new binary file format? It’s the 90s and we haven’t gotten around to building over-complicated tech cathedrals, just do the simplest thing possible. Take the output from your news software as if it were sending to a UUCP site and shove it up to the satellite.

Admittedly as the thread ends and my own research concludes it’s still not clear how the term “bag” exactly originated. I have a hunch it was probably the satellite biz (maybe PageSat?) because news admins appear to have already had their own term for batching (“rnews batch”)? I guess somebody just called bag that and it stuck. I dug through the earliest release notes I could find for DNews and couldn’t find any further information about how their bag support came about.

File naming

I came across this old web page “Building a Satellite-News-Feed (UseNet, PlanetC)” by Juergen Helbing who goes into detail of his woe trying to get Planet Connect going in Germany. He talks about using DNews to import news from Planet Connect to his news server. There were several very interesting bits of information in this post. One was the file names used by Planet Connect:

PlanetC send out two different names of BAG-Files:

news####.zip   (NewsGroups) and
pcbin###.zip   (binary Groups)

This indicates there were different filenames based on types of content. Here news####.zip were zipped .BAG files containing batches text articles, and pcbin###.zip were zipped .BAG files containing batches of text-encoded binary groups. This was written in 1997 and he mentions Planet Connect was sending down about 400 MB of compressed articles a day, which uncompressed was about 1.5 GB per day.

Also interesting was mentioned how the Planet Connect terminal/receiver actually worked. Once the satellite, LNB, decoder, software were all installed, the software instantly starts reading the stream, had enough information to know when one file stopped and started, and just started writing files to disk. I think of this as analogous to a Zmodem/Ymodem-batch download, both of which are protocols that have headers that carry the filenames+sizes. Except it’s a one way transmission with no acknowledgements, and if there are CRC errors (he mentions this) that file is screwed as there’s no way to re-request bad blocks. I wondered if Planet Connect and other services re-broadcast files in case they were corrupt, reading an old FAQ indicates they sent some files at least twice a day but not the newsfeed itself.

Creating a .BAG with INN and send-uucp

I’ve been running the INN news server software in conjunction with my BBS to send test private newsgroup posts over UUCP. I wanted to look closer at the files being sent to see if they used this same batch format. Could a batch outfeed file from INN be treated as a .BAG file to the BBS?

First thing I discovered was my INN wasn’t set up correctly. I needed to be running send-uucp every hour on my news server, to well, batch up articles for an outbound new feed to the BBS over UUCP. This explains why BBS->INN posts worked, but I wasn’t seeing anything going INN->BBS! I also discovered I should have been filtering the news feed on the bang path, so on the first run of send-uucp I was back-feeding thousands of test posts from the BBS back to itself!

But sure enough after running send-uucp, I went and looked in my /var/spool/uucp/tuxedocatbbs/ directory for stuff spooled for the BBS. The resulting data file D.007N contained the same sort of rnews formatting mentioned from the TUHS list and DNews docs.

---

#! rnews 506
Path: news.wann.net!.POSTED.localhost!localhost!bwann
From: bwann@wann.net
Newsgroups: tuxedocat.test
Subject: Test 3/14 post
Date: Fri, 14 Mar 2025 14:56:29 -0700
Organization: wann.net
Message-ID: <c5f44a0f-6a69-3235-e9f4-5eb692d8d52d@wann.net>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=US-ASCII
Injection-Info: uucp.wann.net; posting-host="localhost:::1";
	logging-data="168550"; mail-complaints-to="usenet@uucp.wann.net"
Xref: news.wann.net tuxedocat.test:10874

happy pi day!
#! rnews 505
Path: news.wann.net!.POSTED.localhost!localhost!bwann
From: bwann@wann.net
Newsgroups: tuxedocat.test
Subject: 3/14 test again
Date: Fri, 14 Mar 2025 14:56:54 -0700
Organization: wann.net
Message-ID: <80d63cde-8ccf-b661-fc50-3447d089ea70@wann.net>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=US-ASCII
Injection-Info: uucp.wann.net; posting-host="localhost:::1";
	logging-data="168697"; mail-complaints-to="usenet@uucp.wann.net"
Xref: news.wann.net tuxedocat.test:10875

3.14!  meow

---

Fake .BAG import to BBS

I wanted to test my theory by seeing if I could just take this data file from the UUCP spool and import it to the BBS as a fake .BAG file. Instead of beaming the file up and down over satellite, I just scp’d it to the BBS system and renamed it.

wcGate is the software used by Wildcat! BBS to import/export messages from UUCP to the BBS message base, and includes support for satellite feeds. I created a new dummy UUCP provider called FAKESAT:

 

I copied the raw data file over to the BBS into the incoming Satellite directory configured in MakeGate:

Next I ran wcgate import uucp h:fakesat to try to import my newly created .BAG file:

And it imports my fakesy .BAG file with test newsgroup posts to the BBS! Success!

 

Also good to know wcGate understands duplicate newsgroup posts, so I can go hog wild and set up multiple news feeds and not worry about the same article popping up repeatedly.

For what it’s worth when wcGate exports messages to transmit over UUCP, it uses a similar batch format except the batch is compressed as noted by the #! cunbatch header:

Other file types

I haven’t seen the Planet Connect or PageSat decoder software in operation, but presumably there are other file name series for other thing such as FidoNet posts, stock quotes, and whatever else they sent across the link.

A friend showed me a video that had a clip of a consumer service called SkyLink (offered by the same people at Planet Connect) in operation, which gives some idea of how file downloads from satellite could work. Apparently they had a manifest of files they sent in advance (daily? weekly?) of what would be sent over satellite. Using their software you would mark which files you were interested in, and as they came in throughout the day they would be saved to disk. The example video showed a list of .ZIP, .ARC, and .QWK files among others that could be saved. At the bottom shows a status pane with filenames, block and error counts:

According to an article about Planet Connect in Boardwatch Magazine (Jan 1994), Fidonet files were named `0000FFFF.M01` and required a TICK processor to import them.

It seems like it should be straightforward, if not at least possible, to mock up a dummy satellite service over a null modem serial connection to relive the experience. There needs to be something to assembled the file manifest to periodically “upload”, and then something sitting on the BBS/receiver side sitting there continually decoding as they come through and write them back to individual files. Again, something very similar to Ymodem-batch or Zmodem. This could be used to send a dummy newsfeed, or dummy stock quotes, weather images, or cat memes.

Bonus: while reading up on usenet and uucp, I learned that in 1983 when Australia first joined usenet articles were written to tape and FLOWN to the University of Sydney where they were ingested and distributed. This is also mentioned in the end of the O’Reilly Managing Usenet (1998) book under “Last-Resort Transmission Methods”.

Tags: bbs, satellite, uucp