Finding embers in the ashes of a crashed system ...

By: Carsten Emde

.. and see the NMI fire

The easy case

When a computer system crashes, it often lets us know what went wrong - there are many names of this very last message ranging from "crash dump" to "BSOD". In any case, this message is intended to help the system programmer locate the underlying problem and fix it. Less often there is no such message, but the system simply stops. Special keyboard codes such as RightAlt + SysRq + command may then be used to get some more information about the state of the system. Under Linux, this must be enabled by setting the related kernel variable to 1 either directly

echo 1 >/proc/sys/kernel/sysrq

of by editing /etc/sysctl.conf

kernel.sysrq = 1

the latter ensuring that the former setting is made at every boot. The command H will output a short instruction where the upper case character in a function name or the character in parentheses denotes the command to be used for a given function:

SysRq : HELP : loglevel(0-9) reBoot Crash terminate-all-tasks(E) memory-full-oom-kill(F) kill-all-tasks(I) thaw-filesystems(J) saK show-backtrace-all-active-cpus(L) show-memory-usage(M) nice-all-RT-tasks(N) powerOff show-registers(P) show-all-timers(Q) unRaw Sync show-task-states(T) Unmount force-fb(V) show-blocked-tasks(W) dump-ftrace-buffer(Z)

For testing purposes, the command character can be written to /proc/sysrq-trigger to check the output channel of the system and make sure that it works should it be needed, e.g.

echo h >/proc/sysrq-trigger

The less easy case

Even more rarely, the keyboard of a crashed system no longer works but with some luck the system still responds to ping requests. In this case, the PREEMPT_RT Linux RTOS patches contain a small patch set that allows to define an ICMP packet pattern to identify a SysRq command transmitted via ping request. If, for example, the pattern 0x01020304 is going to be used, the command

echo 0x01020304 >/proc/sys/net/ipv4/icmp_echo_sysrq

would be needed to communicate this to the kernel. Later on, when the system crashes and the famous wonder formula SysRq SUB (sync disks, unmount disks, reboot) should be sent to the system, the following three lines submitted from a remote peer would do the magic:

ping -c1 -s57 -p0102030473 crashedhost
ping -c1 -s57 -p0102030475 crashedhost
ping -c1 -s57 -p0102030462 crashedhost

where 0x73, 0x75 and 0x62 represent ASCII codes of "S", "U" and "B", respectively.

The most unwanted case

Very rarely, however, the computer just stops and there is no way to get any information out of it, even not with any of the SysRq tricks. Of course, you can always stop the power supply and restart the system. But this normally erases all traces of the crash, and the only remaining certainty is that the system will crash again - sooner or later, and most probably at a situation when it creates maximum trouble. So, wouldn't it be good to detect at least whether the NMI is still executed in a crashed system? Of course, not all processors support NMI, but at least x86-based computers have it. Ok, how can we signal NMI activity of a near-dead computer?

Well, the only way to output information of a crashed system is to use a controller with a data register the content of which is directly reflected at its digital output lines, since the crash already may have destroyed most of the kernel functionality including interrupt handling. A good candidate for a direct silicon-to-socket controller is the parallel port. So let's add a callback function to the NMI handler and use a special (somewhat hacky) driver that detects the location of the I/O data register of the parallel port, if any. This driver also defines some codes such as clear, set, invert, increment, decrement etc. to manipulate the output pins of the parallel port. Many kernels of the OSADL QA Farm systems are patched to provide the related kernel driver setparport, and the related farm systems carry an 8-LED parallel port plug to visualize the output pins of the parallel port. This works equally well with on-board parallel controllers and with PCI and PCI express adapter boards provided that the latter are based on a standard IEEE1284 controller and map the data register into the I/O port space. A simple

modprobe setparport nmicode=515

will load the driver and increment the parallel port output byte at every NMI call.

An example of a PCI express adapter board that was successfully tested for the OSADL Parport Monitor is the Lindy PCIe Serial/Parallel Card 2S1P No. 51188; another one is manufactured by Longshine and has the product number LCS-6322M. Both contain a Moschip MCS9901 controller and are recognized at the PCI bus as NetMos Technology Device 9912. The mentioned 8-LED parallel port plug is available at OSADL or can easily be built using the PCB data including all required Gerber files that may be obtained from OSADL as well.

Ok, why does this help?

When the system silently crashes next time we can determine whether the NMI still is executed or not. If not, there really is very little we can do - such situation may require really expensive analyzers, and even this will only work, if the system is equipped with suitable hardware interfaces. If, however, the NMI still fires, we may use the control lines of the parallel port to enter commands to the system. For this purpose, the five input-only lines ACK, BUSY, PAPER, SELECT and ERROR are best suitable. They are available at pin 10, 11, 12, 13 and 15 of the 25-pin D-Sub connector and are set using the control register which is located at the next byte following the data register. Bit 6, 7, 5, 4 and 3 of the control register are connected to the respective pins. Unfortunately, there is no general recommendation how to best use these pins and what kernel action to start, since this largely depends on the individual conditions of a particular system. But the described mechanism enables us at least to establish some low-level communication with the system, before we finally let the fire go out.