For a good introduction to processes, have a look at the slides on our UnixTutorials page.

Useful Process Related utilities

fuser(1)

Tells you which processes are using a resource, and optionally send them a Signal

kill(1)

Send a Signal to a process by ProcessID

killall(1)

Send a Signal to a process by name

killall5(8)

Send a Signal to all running processes

lsof(8)

Similar to fuser(1)

nice(1)

Run a program with modified scheduling priority

pgrep(1), pkill(1)

Look up or signal processes based on name and other attributes

pidof(8)

List pid(s) of process(es) by name

ps(1)

Display process status

pstree(1)

Display processes as a tree

top(1)

Display processes sorted by certain criteria (default: CPU load)

vmstat(8)

Show VirtualMemory statistics

top

The 'TIME' column in top is the amount of time the program has spent running, not to be confused that the amount of time since the program was started. eg: a program started a month ago may have only run for 1 minute total, so it's TIME column will only show 1 minute of running time.

RSS is the ResidentSetSize?, the amount of memory that the program has which is actually in memory (not swapped out). Note that this also covers memory which is shared between programs and threads. Mozilla for instance shows as using about 20M in 5 processes, but this doesn't mean it is using 100M in total, it means it's using about 20M in total, shared between 5 processes :)

Someone was searching for "WCHAN", so here's a definition, when a process is 'sleeping in the kernel' (in the S state) then WCHAN is the function inside the kernel it is sleeping on. for instance init(8) (at least on my machine) usually is blocked inside "select" from select(2).

top(1)'s summary output:

• The top line has the uptime, the number of users logged in (according to utmp(5)) and the LoadAverage (according to uptime(1))
• The next line has the number of processes, then a break down of sleeping processes (processes blocked waiting for an event), the number of running processes, the number of ZombieProcesses (processes that haven't been cleaned up by their parent process) and the number of stopped processes (processes that are stopped by SIGSTOP)
• The next line has the CPU states, amount used in userspace, the amount of CPU used in the system (kernel and device drivers), the amount of cpu used by nice processes (processes that have a lower than normal priority) and the amount of cpu time that is idle (is spent with the cpu shutdown). In more recent versions (such as "procps version 3.2.0"), this line gives a summary of all cpus, for user, system, nice, idle, waiting on I/O, hard IRQ, and soft IRQ. (For this version of top, pressing "1" toggles between 1 summary for all cpus, and a summary line for each cpu.)

• Then the memory breakdowns

  • Total amount of physical memory that the kernel knows about
  • The amount of physical memory that is in use
  • The amount of physical memory that is not in use (wasted)
  • The amount of physical memory used for buffers (eg: networking)

• The swap breakdown

  • Total amount of swap space
  • Total amount of swap used
  • Total amount of swap free
  • Total amount of physical memory being used as disk cache.

After this comes the list of processes.

Some hints:

• If you're doing a lot of I/O (or especially older IDE I/O) you will probably see your "System %" increase. This means that your CPU is spending it's time talking to the hardware, and perhaps not spending much time doing whatever you want it to. If your system % is high you should perhaps consider upgrading your hardware.
• If the number of zombies is high then you possibly have a poorly written program that doesn't cleanup zombies. use pstree(1) to get an idea which process is not cleaning up it's children.
• See LoadAverage about the Load average and related issues.

The various states of a process can be:

nice

nice(1) lets you make programs "nicer" (ie: have less access to the CPU in proportion to other processes). nice values in Linux range between -20 and +19. The default nice(1) level is "0". Only the root user can lower their niceless level. Higher nice level means it has a lower priority. A process running at -20 is considered "RealTime" and is never preempted.

 nice -n nicelevelchange ''program'

eg:

 nice -n 1 ./program OR nice -1 ./program

will run ./program with one level higher niceness (ie: lower priority compared to other processes).

 nice --5 ./program

will run a process with lower niceness (ie higher priority) of negative 5. (Only the root user can do this).

ps

If you want to grep for a running process (eg foo) use:

 ps ax | grep [f]oo

not:

 ps ax | grep foo

The reason for this is that the latter example will also find the grep foo itself in the process list, while the first one won't.

The most useful ps(1) command is probably

 ps auxww

This gives a lot more information about each process than you get by default.

Here is a poor man's Linux-only ps replacement (for when ps(1) just don't work or can't be relied upon):

 #!/bin/bash
 cd /proc && for p in [0-9]* ; do
        echo -ne "$p\0"
        tr '\0' ' ' < $p/cmdline
        echo -ne '\0'
 done | xargs -r0t printf ' %5g %s\n' | sort -ns

Miscellaneous

If you want to unmount a filesystem but it's in use you can use

 ps -auxwwe |grep mountpoint
 lsof | grep mountpoint
 fuser -vm mountpoint

lsof(8) stands for list of open files.

PerryLorier suggests

  fuser -k -v -m /mnt/nfs

to kill all processes using that mount point.

If your program says "Signal 11", "SegmentationFault", or similar, you can retrieve information about the process when it crashed. First remove the limit on dumping core files (so it will dump core this time around):

 ulimit -c unlimited

Then run the program again. (See builtins(1) and ulimit(3) for more information about this.) This time when it SegmentationFaults it will leave a file called "core" which contains the state of the program when it died. This file can be inspected by

 gdb programname corefilename

To find out where it crashed try bt full at the prompt. You can also print variables to find out what they currently hold, for example print argc will tell you the contents of argc. Of course, quit or [Ctrl?[d? will exit gdb. For more information about the GNU debugger see gdb(1). For more information about this procedure see DeBugging.

Help I'm running out of file handles, what's using them all?

 lsof | awk '{print $2}' | uniq -c | sort -n +1 | join -12 -21 - <(ps ax -o pid,command | sort -n)  | sort -n +1

This one liner lists processes in the form "pid,number of open files,command" with the process using the most files at the end. You can use this to determine which process on your system is leaking file handles, then use strace(1) to figure out why.

Read about Zombie processes.

Saving processes to disc (software suspend)

Hate having processes die because of kernel upgrades? The thought of losing your irssi scrollback just too much for you?

Get Cryopid. Compile (I needed zlib1g-dev on my Ubuntu machine which had build-essentials installed) and then run 'freeze' to save processes to disk. You start the process again by executing the file that it saves - there is no 'thaw' utility

It can't save a screen session, but you can save the processes inside them and rescreen.

CategoryNotes