off by one for 2009 July 19 (entry 0)

< Abstraction and Trade-offs: The Devil You Know
"You wouldn't like me when I'm angry..." OR "Dr. Jekyll and Jiminy Cricket" >

[Trackback URL for this entry] Sun Jul 19 20:41:53 remote administration in the post-onboard-serial era:

Please Don't Take Away My Serial Port!:

I've been dismayed lately at the lack of serial ports on COTS desktop PCs. I understand eliminating things like parallel ports, game ports, even audio line in... but serial ports are special... and expansion devices are not as good as on-board hardware.

If you ever end up using the PC as a server that you have to administer remotely, you may need a serial console so you can see what's going on at boot time, or to manually select a kernel in the bootloader -- say if your testing kernel fails. When most people need a serial port on a serial-less machine (e.g., to access the serial port on a router from a laptop) they get a serial expansion card or a USB serial dongle. It seems like this should work. But don't count on it.

grub and expansion devices:

Expansion serial ports are barely passable as a solution for Linux/grub users -- and in my opinion are not acceptable for real production systems. In particular, the bootloader and kernel will not be able to send output to expansion devices until after the hardware has been properly recognized and the bootloader or kernel is prepared to handle it. During the time prior to initialization of each system (early in the bootloader and early in the kernel startup sequence), you'll be flying blind.

However, grub does not understand USB *anyway*, so unless you are lucky enough to have a BIOS which supports USB serial ports as native devices (like it does for keyboards and mice), your USB dongle will not allow you to control your bootloader. Period. This is because making a USB serial port work requires a functional USB subsystem, which is more than a bootloader is supposed to handle. As of now, it's not clear if or when grub will support USB. So laptops are screwed.

But a desktop machine can us a PCI card, right? You'd like to think that, wouldn't you? Unfortunately, grub only knows the standard IO ports (memory addresses and IRQs) for COM 1-4 (units 0-3 in grub parlance) -- which means if your PCI serial card appears at a different address, grub will not be able to use it. There is code in the pipe for PCI expansion serial ports in grub, but I'm not sure of its status. It doesn't work in my Hardy Heron Ubuntu, although I'm hopeful that this will work reliably in the future. (If so, then PCI cards could be a good solution for "desktop" PCs.)

The Linux Kernel and Expansion Devices:

OK, so grub support is thin. How about the kernel?

The outlook is better here, inasmuch as there is some hope. The kernel has driver software for using these devices, but again, until your device is initialized, you won't see anything. Getting it initialized early enough to use as a console may require you to use a ramdisk to boot. While most distros already do this, there are valid reasons for wanting to avoid using a ramdisk. Too bad.

If you don't mind using a ramdisk, it may work if you're using a stock kernel and your devices are common enough. If they're uncommon, you may be required to compile alternative drivers into a custom ramdisk, including extra USB stuff. Building custom ramdisks for your hardware is never as simple as it should be, and is a ticking time bomb for maintenance. At some point, the update path will change, you'll get a new kernel but the ramdisk won't be properly modified, you reboot, and suddenly you don't have a console anymore.

But let's say you go this way (out of necessity or because you feel a need for more stress in your life) and you get the ramdisk set up with appropriate drivers for your devices. Now that the kernel can see your hardware, you need to create the appropriate devices. In the old days, that meant creating a physical device and setting permissions on it. But nowadays, devices are created automagically. This is a good thing in general, but it can also be a pain when you only have to fiddle with it every two years.

Anyway, you'll have to monkey around to get your event system (e.g., udev or the older hotplug or what have you) to set up the devices for you properly, including proper permissions. You'll also have to add inittab or event.d rules to start a getty process on the new port if you want a terminal on the console and add rules in securetty if you want to log in as root. Many HOWTOs covering this information are online.

Short version: Expansion COM ports are flaky PITAs. If your computer just had a single built in COM port, it would be easy-peasy, stable and un-cheesy. (Hey HW guys, if you're listening, a serial device could just be an on-board device with bare pins and we can add a ribbon cable if we need it! Everybody wins!)

eXtreme Bootloading!!1!:

There is some good news, however. If you can get grub installed, you can make temporary kernel selections before reboot -- without modifying your menu.lst file. The basic idea is that you set up a "fallback" priority for the kernels and then temporarily select your testing kernel as a "one time boot" using the command grub-set-default. Ok -- things are looking brighter. But what if your kernel panics? Now it's hung. Ah, but you can make Linux reboot upon panic by adding the line "kernel.panic = 15" to /etc/sysctl.conf. (Before it panics!)

... But You Still Want That Smart Power Strip:

Of course, if your kernel deadlocks, you're screwed. The kernel's so hung it can't even reboot itself. Time to use that Linux-friendly IP-enabled power strip you've been meaning to buy for a while. I found the above device being sold under a different label on eBay. The interface isn't the best, but it works, it doesn't require any special software, it has four independent outlets, and it's relatively cheap.

Now, if your machine goes dark when you're remotely testing that bleeding edge kernel, just reboot it! If the disk isn't scrogged, it will happily fall back to the known good kernel and you're golden!

Filed under: technical:administration


Unless otherwise noted, all content licensed by Peter A. H. Peterson
under a Creative Commons License.