Linux Crash HOWTO
nepatten@us.ibm.com
2002-01-30
Revision HistoryRevision 1.02002-01-30Revised by: NMInitial release.
This document describes the installation and usage of the LKCD (Linux Kernel Crash Dump) package.
----; Table of Contents; 1. Introduction: ; 1.1. Copyright and License; 2. How LKCD Works: ; 2.1. What You Need; 3. Installation of lkcd: ; 3.1. Installing From Source Code; 3.2. Building and Installing LKCD Utilities; 3.3. What Gets Installed; 3.4. Installing LKCD Utilities From RPM; 3.5. Patching the Kernel; 3.6. Build and Install the Kernel; 4. Setup, Test, and Running crash: ; 4.1. Setting up crash dump; 4.2. Testing crash; 4.3. Running crash----
The LKCD (Linux Kernel Crash Dump)
project is a set of kernel patches and utilities to allow a copy
of the kernel memory to be saved in the event of a kernel panic.
The saved kernel image makes forensics on the kernel panic
possible with utilities included in the package. Most commercial
Unix operating systems come with similar crash utilities, but
this package is fairly new to Linux and has to be added on
manually. The LKCD utility is not designed to gather helpful
information in the case of a hardware caused panic or a segment
violation. The complete LKCD package is available for download at
http://lkcd.sourceforge.net/.
This document is copyrighted (c) 2002 by Norman Patten.
Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.1 or any later version published by the Free Software
Foundation; with no Invariant Sections, with no Front-Cover
Texts, and with no Back-Cover Texts. A copy of the license is
available at http://www.gnu.org/copyleft/fdl.html.
Linux is a registered trademark of Linus Torvalds . lkcd is distributed under the copyright of Silicon Graphics Inc.
Send feedback to nepatten@us.ibm.com.
When a kernel encounters certain errors it calls the
"panic" function which results from a unrecoverable error. This panic results in LKCD initiating a kernel dump where kernel memory is copied out to the pre-designated dump area. The dump device is configured as primary swap by default. The kernel is not completely functional at this point, but there is enough functionality to copy memory to disk. After dump finishes copying memory to disk, the system re-boots. When the system boots back up, it checks for a new crash dump. If a new crash dump is found it is copied from the dump location to the file system, "/var/log/dump" directory by default. After copying the image, the system continues to boot normally and forensics can be performed at a later date.
lkcd-kernelxxx.diff file for patching the
kernel. The kernel version supported will change routinely. lkcdutils-xx.src.rpm - this is the utilities source and scripts you will need to setup and read a crash. At the time of this writing there is a i386 binary rpm available from lkcd.sourceforge.net, but you will still need the patches for the startup scripts from the source rpm.
Get the lkcdutils-xxx.src.rpm and install it
using rpm -i kcdutils-xxx.src.rpm . This will place a file called lkcdutils-xxx.tar.gz in the /usr/src/redhat/SOURCES directory. This file is a compressed tar image of the lkcd source tree. Unwind the source in a directory of your choice like "/usr/src" with tar -zxvf kcdutils-xxx.src.rpm . This will create a directory called "kcdutils-xxx" which will contain the LKCD utilities source.
LKCD used the standard GCC compiler and make files. To build
the suite, cd to the LKCD src directory and run ./configure to build configuration files. The next step is to run make to build the utilities, and finally run make install to install the utilities and man pages.
/etc/sysconfig/dump # Configuration file for dump
/sbin/lcrash # The crash utility /sbin/lkcd # Script to configure and save a crash /sbin/lkcd_config # Configuration utility for dump /sbin/lkcd_ksyms # Utility for reconstructing kernel symbols /usr/include/sial_api.h # Header file for the SIAL API /usr/lib/libsial.a # Simple Image Access Language library /usr/man/man1/lcrash.1 # man page for lcrash /usr/man/man1/lkcd_config.1 # man page for lkcd_config /usr/man/man1/lkcd_ksyms.1 # man page for lkcd_ksyms /usr/share/sial/lcrash/ps.sial # ps command implementation of SIAL
You can install the pre-built utilities from rpm by running
rpm -i kcdutils-xxx.rpm . You will still need to patch the kernel and install the startup script patches. However you can bypass building the utilities step.
The next step is patching and rebuilding the kernel. You will
need to patch the kernel source with the
lkcd-xxx.diff file you downloaded from
http://lkcd.sourceforge.net/.
Copy the patch into the same directory as your kernel and run
patch -p0 ` lkcd-kernelxxx.diff . Make sure
the patch you use is the same version as the kernel you
are patching. Next you will need to configure the kernel to enable
crash dump support. By default crash support is turned off after
applying the patch. If you use make menuconfig
or make xconfig, the "LKCD support"
option is under kernel hacking. You will also need
to enable other kernel features you might need. See the
The Linux
Kernel HOWTO for further details.
The next is building and installing the crash enabled kernel. In the
kernel source directory run the following commands in order.
make depend make install make modules make modules_install
This will build and installing the new kernel, you will also need to
copy the Kerntypes file from the kernel source to
the /boot directory. You may also need to edit the
lilo.conf file to point to you new kernel. See
http://www.linuxdoc.org/HOWTO/Kernel-HOWTO.html
for more information on building and installing a kernel.
In order to save a core image that has been written into swap,
the image must be saved prior to swap being re-mounted during boot. To accomplish this, the sysinit startup file needs to be changed. The lkcd source includes a scripts directory which contains patches for various sysinit startup scripts. These patches add the lkcd config and lkcd save commands to enable crash dumps and to save any existing crash dumps upon startup.
To force a panic to test you new crash setup, compile the following
code with cc -c -I/usr/src/linux/include panic.c . After building the panic.o module just insmod panic.o to panic the kernel.
int init_module (void) { panic(" panic has been called"); return 0; }
To view your kernel core file lcrash needs
Example: lcrash /boot/System.map ./dump.1 /boot/Kerntypes
It will take a minute to load the kernel image into memory and drop you into the crash shell. At the crash shell prompt you can type a ? to see the available commands.
No other page links to HowToLinuxCrashHOWTO yet.
