Monday, January 15, 2024
Amiga + Unix = AMIX
Tuesday, June 13, 2023
Beyond The BBS
Bulletin Board Systems (BBSs) in the 1980s are sometimes compared to a primitive version of the internet, and I think there's something to that. Consider the experiences that a typical BBS caller would have when calling around.
A user might start their call, for example, by engaging in instant messaging by chatting with the SysOp, then sliding into the message bases for a bit of 'social media' with the other hundred or so active users. Looking for new software, the user might then hit up the file transfers area, where perhaps a whole megabyte of games and utilities might be available, and finally head to the text file area for ansi/petscii images, computer related news, or encyclopedic articles.
But then came the actual internet. Instead of chatting with one person, you could chat with all of them. Your social media encompassed people from all over the globe. The availability of software for your computer was measured not in megabytes, but in all-the-bytes; your images and movies were measured in pixels instead of characters per line, and your access to news and information became up-to-the-minute from professional journalists. It was little wonder that BBSes did not fair well against such competition.
Some computing ecosystems handled this well. By "well", I mean that standards for network interface APIs were adopted by their communities and operating systems rather quickly. This allowed any software that wanted to provide solutions involving the internet to do so, without having to worry whether their programs would work on newer or competing internet interfaces. Some obvious examples from the early 90s are PCs with Windows via WINSOCK, Apple computers with MacTCP, Amiga computers with AmiTCP, Atari STiNG, and others.
This adoption was vital for progress in networking software to occur. With only a single viable API to write, e.g., a web browser against, competing browsers could be developed, with each trying to out-do the others in features, which moved the product space forward. And whether you are looking at Internet Explorer vs Netscape, or IBrowse vs AWeb, this is exactly what happened.
Some platforms, however, did not handle the dawn of the internet particularly well. An example of this is the Commodore 8-bit line, particularly the Commodore 64. With multiple and endlessly changing internet interfaces, each one picked up and tossed aside in favor of a Shiny New Way, the result would be a a relative lack in internet software, even of the limited sort that the hardware could handle, and this led to little or no competition or progress.
It is this amusing history - this parade of technologies - that I will be walking you through, before I bring in my own float.
In the late 1980s and early 1990s, the initial solution for C64 users to get on the internet was an ISP "shell" account. This was functionally similar to a BBS, in the sense that a common 1980s modem was the only hardware you needed, and you used the same terminal programs to access the shell account as you did to access many other non-Commodore BBSs. During this time, demand for C64 and C128 terminal programs that supported *NIX terminal types would increase, leading to the popularity of programs like Novaterm or Desterm.
At this time, the C64 "user port", where serial modems were connected, was viewed as being limited to 2400 baud. This limitation was overcome in two different ways.
The first was by using the Creative Micro Designs "SwiftLink" 6551 UART cartridge. This cartridge would consume the computers expansion port to provide a faster dedicated standard RS-232 serial interface, capable of using faster modems available for PCs at up to 38.4 bps.
Another way higher speeds were achieved was through Danial Dallmann's UP9600 mod, which was presented in 1997. This was a modification of the standard C= user port serial interface, which used additional pins to take advantage of that computers' automatic bit-shifting feature. As the name suggests, it enabled the C64 to reliably communicate at 9600 baud. The mod itself could be done to the modem or to the computer, and required only jumpering a few extra pins to each other.
Of course, this entire variety of dial-up internet died with the loss of popularity and availability of ISP shell accounts.
So, we're still in the 1990s, and we've already got 3 hardware interfaces to the internet. This means it's time to talk about PC-based modem emulators and null-modem cables.
In 1996, I was only 3 years separated from my own BBS SysOp days, when I found myself working in a computer science lab at my university. We had several rows of 386 PCs running Slackware linux, all connected to the internet, and each with an external IP address. This meant that anyone on the internet could access any of those machines. Oh, those days of innocence!
Together with some of my old BBSing friends, we engaged in a project to connect my Commodore 128 to one of these machines using a null modem cable, which is simply a serial cable that allows two computers to talk to each other. One end of the cable connected to the PC serial port, while the other end connected to the C128 user port, after going through some voltage converting chips. The C128 would be running my BBS software and watching for the "Carrier Detect" signal on the serial port to go active. Meanwhile, the PC would run a linux terminal program called 'minicom' when a user logged in, which would automatically wake up the C128 BBS program and announce itself to the user. In effect, by remotely connecting to the PC via telnet, and logging in, you would be using a BBS, running on a Commodore 128.
My little college project illustrates the next phase in Commodore 8-bit internet access. Around the turn of the millennium, Leif Bloomquist would present "BBS Server", a windows application that would forward tcp/ip connection from the internet to one of the PCs serial ports. The purpose was the same as my minicom example: to allow Commodore BBSs to be accessed from the internet instead of old phone lines, but also included outgoing connection abilities with Hayes style AT-command emulation.
This was followed by a project from Jim Brain called TCPSER, which also used null modem cables. This was written as a port of BBS Server to linux, and provided the same Hayes/C= 1670 modem emulation that BBS Server does.
The last in this category was Strikelink USB by Alwyz, author of the popular CCGMS term program for the C64. In this case, the Strikelink is a cable that connects the Commodore user port serial interface to the USB port of a PC instead of the standard com port. Like the later Strikelink variations, this was never really produced for sale, but only presented as a project for interested users.
These sorts of solutions are still used today. Strikelink USB, in fact, only goes back a decade. However, the requirement of having a PC physically close to your Commodore computer has probably limited its popularity overall.
The late 90s also saw the first attempts to join the rest of the computing world in connecting to the internet via dial-up SLIP/PPPoE. If you dialed up to the internet in the 90s, and you did not have AOL, you were probably using a SLIP or PPPoE account. There were several ways for the C64 to do this as well.For software, the C64/C128 operating system LUNIX, by Daniel Dallman, had both a SLIP and PPoE client, as well as some very basic internet tools, such as telnet and ftp clients, and a web server. Another internet client was The Wave, for the Wheels operating system, which included a terminal program and web browser.
The ethernet cradles may still be in use, but dial-up internet is not exactly common any more. Also, while LUNIX is still remembered by some, The Wave required both Wheels OS and a Super CPU, which leaves it with a limited audience.
The mid-late 2000s saw the first ethernet bus cartridge for the C64/C128, called the RR-Net, which is short for "Retro-Replay-Net". It was produced by Individual Computers as an 'add-on' for their "Retro Replay" freeze cartridge.
The cartridge contains a Cirrus logic CS8900a ethernet chip, which is set up in 8-bit mode. Registers are then exposed on the C64 expansion port address page to allow packet bytes to be transferred back and forth.
The RR-Net was eventually turned into a proper stand-alone cartridge, completely compatible with the original RR-Net add-on. These can be found under the names RR-Net MK3 from Individual Computers, TFE from Adam Dunkels, and the 64NIC+ from go4retro.com.
A cartridge solution likes this has the benefit of speed, but the downsides of consuming the cartridge port, and requiring that the C64 run a full tcp/ip stack, with its own buffers. Perhaps half a dozen software titles support the RR-Net, including terminals, a browser, and PC transfer tools, the most impressive being Contiki from Adam Dunkels. Contiki is a GUI operating system with a tcp/ip stack called uIP. The entire OS is essentially designed around networking, and its small selection of internet clients and tools.
More recently, the 1541Ultimate-II cartridge was also released with its own ethernet port and interface. The author has provided a couple of tools, including a terminal, which supports this port. The 1541UII is reportedly also adding SwiftLink and modem emulation, to allow the use of other terminal programs. Presumably it is the same interface integrated into the Ultimate64 computer, which is an FPGA reproduction of the computer with built in 1541Ultimate-II features.
As microcontrollers became cheaper and more full featured, we started to see them used to give us even more internet options.
The COMET 64 was released in 2008, along with the launch of its companion web site "commodoreserver.com". While COMET appears to be a user port modem with wired ethernet, it's actually better described as a storage and data exchange device. Much like Q-Link, CommodoreServer features chat and online gaming options. When a special driver is loaded on the C64 computer, the modem also provides an internet disk drive on device 2, with all files stored on the commodoreserver.com web site.
In 2015 we saw the first emergence of the "wifi" modem with the release of the Commodore WiFi Modem by Leif Bloomquist, of BBS Server fame. His user port modem was a complete "BBS Server" package in one box: it could allow incoming connections in order to run a Commodore BBS, as well as allowing outgoing connections to the emerging (re-emerging?) "Telnet BBS" phenomenon.
Great care was given to the user interface of this device, with easy menu driven configuration and use.
This was the device that inspired my own interest in wifi modems.
The next year, the C64Net-WiFi modem would appear, followed soon after by the Strikelink wifi modem project, and later by several other similar devices, which you can still pick up on eBay for $25-50. Like Leif's modem, these would be user port devices, typically with UP9600 support. They all have varying features, but in common they support some variation on the Hayes AT-command set, giving them a level of compatibility with standard C64/C128 terminal programs.
Related to the modem emulator, is the LINK-232 WiFi. This wireless ethernet device is connected to an integrated Link-232 cartridge, which in turn ia a clone of the CMD Turbo-232 cart, the successor to the SwiftLink cartridge mentioned above. Like the other wireless modems, it is compatible with a few exceptional term programs, and supports the AT command set. Aside from terminal programs, it also enjoys support from several Zimodem-compatible apps, and upcoming support from Greg Nacu's 64OS.
Even more recently the WiC 64 device was also made available for the C64. It features a user port interface like other wifi modems, but uses a parallel instead of serial interface. Although this creates yet another software ecosystem, the advantage would be the speed boost over serial.
The last internet platform we'll look at are also the newest: IEC serial port internet devices. The IEC bus is the C64s primary disk drive and printer port, and is supported by the C64 KERNAL as such. These IEC modems have other internet socket communication features, but their primary appeal is clearly for network-based storage.
The first of these devices to appear was the COMET+ and COMET Flyer, the successors to the COMET 64 modem discussed previously. Along with it came another web site, commodoreonline.com, providing similar remote disk/file storage as commodoreserver.com did.
A later addition to network storage is the PETDisk Max, from bitfixer, which has a IEEE-488 interface for PETs, but includes wifi capabilities for fetching data from disk images over the network.
The newest entry in this category is the Meatloaf and Fujinet. While not yet available, Meatloaf is reported to offer similar features to the Flyer, except that it uses more standard web protocols (WebDAV) for dealing with remote disk images and files. It seems is will also optionally include a user-port option for standard modem compatibility. The relationship between Meatloaf and Fujinet is unclear to me, but it's possible the Fujinet for the C64 will be based on Meatloaf.
And that brings us up to the present.
My own story in all this begins in 1984, when I picked up a VICMODEM for my Commodore 64, and immediately became an avid BBS user. By 1985, I was running my own BBS system, initially on 64Messenger, and then switching to CMBBS. I always found one reason or another to be dissatisfied with these programs though, so in 1986 I wrote and ran my own program, called Zelch 64. In 1990, I switched to the newly written Zelch 128, and ran that until 1993. In 1996, as mentioned above, I revived the Zelch 128 BBS as a telnet-board. After 1997 though, I was a typical C= 8-bit user enjoying all the various internet options mentioned previously, until 2016.
The creation of the C64Net WiFi in 2016, by Carlos Santiago of ElectronicsIsFun.com, was my first opportunity to have an impact on internet appliances for the Commodore 8-bits. Carlos himself really wanted a way to access all the disk images he'd downloaded from the internet directly from his C64, and I suspect what he really had in mind was something closer to the Flyer or Meatloaf. However, he also wanted a proper internet modem, and so that's the direction he chose. At the time, I was a user of Leif Bloomquist's wifi modem, and was very excited about the chance to influence the features it would have.
The C64Net WiFi would be based on the Espressif ESP-8266, specifically using the incredibly cheap ESP-01 package. This microcontroller came with 1mb of flash memory, and 80kb of user ram, half of which is available to application programmers. The cartridge was designed to be powered entirely by the host computer using its stock power supply. This presented some serious constraints on the C64 user port, which was solved by using both the 5V and 9VAC rails to power the device. A lot of the circuitry on the right side is for converting the 9VAC to 5VDC. The interface is obviously the user port, and the pin configuration is designed to comply with the standards for Commodore user port modems, including pins for both Carrier Detect, and hardware flow control. It also included the pin configuration to support UP9600, which could be disabled via jumpers for C128 users.
The speed of the C64 user port serial interface is worth discussing at this point. The most popular Commodore 8-bits: the VIC-20, C64, and C128, did not have a serial UART built into the computer. Serial communication was achieved on the computers' 8-bit parallel "user port" interface by bit-banging routines built into the operating system, called the KERNAL.
Commodore initially advertised that the C64 user port could achieve a speed of 300bps, however, it was quickly discovered that, by working around a bug in the KERNAL's bit timing table, 1200bps could easily be supported. The C128 could handle 2400bps, especially in its 2mhz operating mode. By ignoring the KERNAL and using more tightly written bit-banging algorithms in assembly language (Ilker Ficicilar), speeds of 4800-7200bps were achievable. By altering the hardware, UP9600 bumped this up to 9600bps (Dallmann). More recently, 57600bps was achieved by both disabling the VIC-II video chip and timing the bit transfer by counting the cpu cycles instead of using timer interrupts from the computer I/O chips (Jorge Castillo).
So, my Zimodem project began in 2016 to provide firmware for the C64Net WiFi and its ESP-8266. From the very beginning, I had three specific goals for the project, which have been achieved to varying degrees.
The first goal was that the firmware appear to the user as a Hayes/C= 1670 style modem, and have all the features expected from such a command set. The second was that the modem be a useful appliance to computers that are limited to running terminal programs. The last was that the modem be a useful internet platform, allowing it to be easily used by the host computer for writing custom network applications and games.
The firmware for the C64Net uses inverted signals: high for active, low for inactive. This always struck me as intuitive, though the more I've learned about electronics, the more I realize that it's not common, including in RS232.
The AT command set includes all the standard commands and every extended command that made any sense for a WiFi modem, and many that didn't. This includes things like response codes, verbosity, duplex, and setting internal registers for auto-answer and number of rings. That last one may not make much sense, but I tried not to assume what all BBS software might expect or be looking for.
Commands were also added to list available wireless access points and to connect to one with credentials. Although I was a slave to the Espressif libraries for the types of wireless security supported, I've yet to run into any troubles in that department.
ATD, which is the command for making an outgoing call and connecting to a remote phone number, was extended to allow internet hosts and ip addresses. On connection, the modem exits command mode and enters stream mode, where bytes can be sent to and received from the remote system.
Support for both hardware and software flow control was added as well, using the ATF and AT&K commands.
Additional AT commands were invented to overcome the limits of old terminal programs, and expand the supported platforms.
For example, the AT&P command was added to support decoding ASCII/ANSI to PETSCII, including color translation.
Both ATR and ATS3 allow the carriage return character to be tweaked.
"Telnet" code handling was added mostly to make a remote telnet server happy by responding properly to requests and demands. One issue I ran into here was remote telnet servers that, when receiving a telnet escape code, would immediately read the next character. When dealing with network packets, however, there is never a guarantee that all of the bytes of a telnet command would arrive in the same packet. One of the many accommodations I had to make over the years was to try and guarantee that an entire command would always be contained in the same packet.
Baud rate, as well as the ability to change other RS232 settings, such as data bits, stop bits, and parity bits, can be changed with the ATB command. Although 8N1 is practically universal, it wasn't always so, and some older terminals will desire other settings.
An important feature for supporting existing terminals and applications was the addition of a persistent phonebook. With the ATP command, fake "phone numbers" can be assigned to particular internet hosts and ports, with desired terminal settings. The phone numbers are automatically persisted to flash memory. Later, whenever the ATD command is used with an integer, the phone book is checked and the matching host connected to.
This makes it easy to use the Q-Link revival server called "Q-Link Reloaded". For those unaware, Quantum Link was a C64 online service that was a predecessor to AOL.
In addition to the phonebook, 38 different terminal and register settings can be persisted in the onboard flash, meaning that the modem always comes on exactly as you prefer it. This was especially important for machines that prefer odd baud rates or strange data bit and parity settings.
The behavior of various pins of the microcontroller can also read or changed using special AT commands. This was important for dealing with UARTs that required, for example, the DCD pin to always be active.
The firmware supports over-the-air updates, so there's never a need to deal with special programming cables or software to get the latest version. Updates can be checked for and downloaded on demand using the AT&U command. Specific or custom versions can also be reverted to, should one be feeling nostalgic. I often use this for testing versions with one-off features.
The microcontroller does its best to maintain a real-time clock, which can be read using the AT&T command. It will initialize this clock via the Network Time Protocol (NTP) on bootup, and the firmware will allow the TimeZone to be set.
A help file for all the AT commands can be accessed with the AT&H command, which will persist the current versions help file after first access for off-line viewing.
The wifi modem can be configured with multiple incoming socket listeners on different ports. Like the phonebook, these settings can be persisted with their own terminal settings so that they are immediately available after reboots.
Like the "auto-answer" mode on old modems, incoming connections can be configured to send zero or more RING messages and then go automatically into stream mode. This allows Commodore BBS programs that support Hayes or 1670 modems to work out of the box.
A custom "busy message" can also be configured, so that an incoming connection request, when another connection is already using the serial port, will know they need to try back later.
Another feature in supporting the firmware as a networking platform is the ability to make and manage multiple outgoing connections. The ATC command allows a new remote client to be created, without immediately going into stream mode. Instead, the modem stays in command mode, where special AT commands allow the connections to be managed and communicated with.
Whenever data is received from any of the open connections, the data is presented in the form of packets of data, which can be up to 255 bytes. These packets are preceded by a header block, which can contain information such as the connection id that the data came from, the size of the packet, an 8-bit crc of the packet, and the packet sequence number for that connection. If a sequence or crc number for a packet is incorrect, the ATL command can order the modem to re-transmit a previous packet.
The ATT command can be used to transmit either a string message, or a block of binary data, to a specified open connection. It can also be configured to return the CRC8 of the data block it received via serial, for error checking.
Because internet data is likely to be received far faster than a 1mhz computer can process it, there are several new flow control methods that can be turned on to manage incoming data. These are typically variations on standard software (XON/XOFF) flow control that operate at the packet instead of the character level. These are selected using the ATF command.
All of this ended up allowing me to fulfill my dream of being able to write internet application in simple BASIC. Well, *mostly* BASIC.
The problem is that BASIC V2 found in the VIC-20 and C64 does not have good string parsing functions, which makes separating the numeric data out of each packet header especially painful. The INPUT# command in BASIC also has some rather annoying parsing rules of its own. Luckily, during my BBS days, I learned how to create BASIC strings from machine language. I therefore wrote some packet machine language routines, which I call "PML", that I can call from BASIC to quickly parse the information from incoming packets, leaving the data in a BASIC string variable.
With that, then, I cobbled together some super simple internet applications, such as IRC (Internet Relay Chat) for instant-messaging, an FTP (file transfer protocol) application for uploading and downloading files from FTP sites, a wget application for downloading web pages or binary files from web servers, and a d64wget for downloading .D64/D71/etc disk images directly from a web server to a blank floppy. I also wrote a telnetd server that allows an incoming internet user to remotely "take control" of the C64 BASIC prompt and use the computer as if they were sitting in front of it, so long as they limit their activity to text input and output.
I did also write some super simple telnet and petscii terminal programs, just as examples. There are far better terminal programs out there.
The culmination of the entire Zimodem project was embodied, however, when took one of my favorite 2 player PET games, called "Weather!", ported it to the C64, and made it playable over the internet.OPEN4,2,0,CHR$(8):PRINT#4,”at+printp”:CMD4:LIST:CLOSE4
GEOS, also for the C64 and C128, supports proper printer drivers, so it was a relatively simple matter to produce one that supports the network printing features of the modem. GEOS has a graphical WYSIWYG paradigm for printing; all printouts are actually pictures of what was on the screen. Luckily, the IPP protocol specification lists native support for an image format called RAS, which CUPs also supports. The GEOS printer driver, therefore, has only to translate the screen image data into a RAS formatted image before sending it to the modem for printing. This project was called ras4c64net, and can be found at github.
The Amiga also supports printer drivers, but the documentation on how to write them is much less straight forward. I'd love to talk to anyone who's actually written one, as what I found on Aminet and in my books did not click with me. Instead, I relied on the fact that Amiga supports both serial-based printers, and that later versions of AmigaOS come with a PostScript printer driver. I therefore added a feature in the firmware's command mode that would automatically detect when a PostScript file was being dumped into the modem, and immediately go into printing mode.
The Gurumodem was also designed with an SD card interface, and therefore needed some way to access it via the modem. This was done by adding the AT+SHELL command, which provides a command line prompt into which you can enter well-known file and directories commands such as dir, copy, move, makedir, cd, etc. So that users of many platforms would feel comfortable, it also supports aliases of all these commands, such as "$", "ls", or "list" for directory, and similar aliases for other commands.
The shell also includes a simple ftp client for fetching files from the internet to the fat-formatted sd card.
Getting files from the SD card to the host computer is done through the built in shell support for uploading and downloading via the X-Modem protocol, Z-Modem protocol, or KERMIT. The later protocol was added primarily to support the Commodore 900s only stock terminal software.
Shell commands can also be entered directly from command mode by adding a colon after AT+SHELL followed by the command. For example AT+SHELL:list would list the current SD card directory to the modem, while remaining in command mode.
The Commodore SuperPET natively supports a special serial-port file management protocol called "HOSTCM", which was used to send data to and from other central computers. This 'mode' is turned on with the AT+HOSTCM command.
So, that's where things stand with the firmware as of June, 2023. Several new features are actively in the works, or coming very soon.
One is SLIP/PPoE support, which has been on the TODO list almost from the very beginning. The ability to leverage LUNIX and support The Wave browser would be an amazing boon. Integrating with lwip in a way that doesn't trash otherwise normal operation has been tricky, but it's coming.
I also have been eager to add Pulse-dialing support to the modem, along with pin-level emulation of the Commodore 1650/1660. This would be an amazing addition because of all the pre-1986 software that would instantly become compatible with it, including terminal programs, BBS software, and the game Modem Wars.
Lastly, an integrated GOPHER client has been requested, for the extremely limited platform crowd. GOPHER is a fun idea though, as I recall thinking at one point that it was far superior to HTTP and HTML. Apparently, there are still people running GOPHER servers out there, so the usefulness would be immediate.
The creator of the WiFi Retromodem at tempestfpga.com has also recently created an ESP-32 based modem that fits in an old Hayes modem case. To the Zimodem firmware, he added the ability to play old modem sounds out of a speaker, which is absolutely delightful. Perhaps that will be integrated to the main branch soon.
Otherwise, I'm always open to suggestions and bug reports. Please post them on my Zimodem repository "Issues" section at github.
Well, that wraps up this very very long look back at internet solutions for the Commodore 8-bits, and how my own projects have weaved in and out of that tapestry. I hope you enjoyed the parade, and perhaps learned about some technology you'd never heard of. I also sincerely hope you recognize the potential the 8-bits have as networking clients, if we can only settle on an API.
Here are some parting resources:
- Project page: github.com/bozimmerman/zimodem
- C64/128 BBS Software page: bbs.zimmers.net
- The "Zelch BBS": coffeemud.net:6502
- MMORPG for old machines: coffeemud.net:23