View Source: ProcessNotes - Waikato Linux Users Group

Edit PageHistory Diff Info LikePages
For a good introduction to processes, have a look at the slides on our UnixTutorials page.

!! Useful Process Related utilities

; %%% fuser(1)           :Tells you which processes are using a resource, and optionally send them a [Signal]
; %%% kill(1)            :Send a [Signal] to a process by ProcessID
; %%% killall(1)         :Send a [Signal] to a process by name
; %%% killall5(8)        :Send a [Signal] to all running processes
; %%% lsof(8)            :Similar to fuser(1)
; %%% nice(1)            :Run a program with modified scheduling priority
; %%% pgrep(1), pkill(1) :Look up or signal processes based on name and other attributes
; %%% pidof(8)           :List pid(s) of process(es) by name
; %%% ps(1)              :Display process status
; %%% pstree(1)          :Display processes as a tree
; %%% top(1)             :Display processes sorted by certain criteria (default: [CPU] load)
; %%% vmstat(8)          :Show VirtualMemory statistics

----

!! top

The 'TIME' column in top is the amount of time the program has spent running, not to be confused that the amount of time since the program was started.  eg: a program started a month ago may have only run for 1 minute total, so it's TIME column will only show 1 minute of running time.

RSS is the ResidentSetSize, the amount of memory that the program has which is actually *in* memory (not swapped out).  Note that this also covers memory which is shared between programs and threads.  Mozilla for instance shows as using about 20M in 5 processes, but this doesn't mean it is using 100M in total, it means it's using about 20M in total, shared between 5 processes :)

Someone was searching for "WCHAN", so here's a definition, when a process is 'sleeping in the kernel' (in the S state) then WCHAN is the function inside the kernel it is sleeping on.  for instance init(8) (at least on my machine) usually is blocked inside "select" from select(2).

top(1)'s summary output:
* The top line has the uptime, the number of users logged in (according to utmp(5)) and the LoadAverage (according to uptime(1))
* The next line has the number of processes, then a break down of sleeping processes (processes blocked waiting for an event), the number of running processes, the number of [ZombieProcess]es (processes that haven't been cleaned up by their parent process) and the number of stopped processes (processes that are stopped by SIGSTOP)
* The next line has the CPU states, amount used in userspace, the amount of CPU used in the system (kernel and device drivers), the amount of cpu used by nice processes (processes that have a lower than normal priority) and the amount of cpu time that is idle (is spent with the cpu shutdown). In more recent versions (such as "procps version 3.2.0"), this line gives a summary of all cpus, for __us__er, __sy__stem, __ni__ce, __id__le, __wa__iting on I/O, __h__ard __I__RQ, and __s__oft __I__RQ. (For this version of top, pressing "1" toggles between 1 summary for all cpus, and a summary line for each cpu.)
* Then the memory breakdowns:
     * Total amount of physical memory that the kernel knows about
     * The amount of physical memory that is in use
     * The amount of physical memory that is not in use (wasted)
     * The amount of physical memory used for buffers (eg: networking)
* The swap breakdown:
     * Total amount of swap space
     * Total amount of swap used
     * Total amount of swap free
     * Total amount of physical memory being used as disk cache.

After this comes the list of processes.

Some hints:
* If you're doing a lot of I/O (or especially older IDE I/O) you will probably see your "System %" increase.  This means that your CPU is spending it's time talking to the hardware, and perhaps not spending much time doing whatever you want it to.  If your system % is high you should perhaps consider upgrading your hardware.
* If the number of zombies is high then you possibly have a poorly written program that doesn't cleanup zombies.  use pstree(1) to get an idea which process is not cleaning up it's children.
* See LoadAverage about the Load average and related issues.

The various states of a process can be:
<?plugin OldStyleTable
|^ __State__ |^ __Meaning__
| S | Sleeping
| W | Swapped out
| R | Running
| D | Blocked in a device driver in the kernel.  Unkillable.
| < | Process is running with a high priority (nice level <0)
?>

----

!! nice

nice(1) lets you make programs "nicer" (ie: have less access to the [CPU] in proportion to other processes).  nice values in Linux range between -20 and +19.  The default nice(1) level is "0".  Only the root user can lower their niceless level.  Higher nice level means it has a lower priority.  A process running at -20 is considered "RealTime" and is never preempted.

<pre>
 nice -n ''nicelevelchange'' ''program'
</pre>

eg:

<pre>
 nice -n 1 ./program OR nice -1 ./program
</pre>

will run ./program with one level higher niceness (ie: *lower* priority compared to other processes).

<pre>
 nice --5 ./program
</pre>

will run a process with lower niceness (ie *higher* priority) of negative 5. (Only the root user can do this).

----

!! ps

If you want to grep for a running process (eg foo) use:

<pre>
 ps ax | grep ~[f]oo
</pre>

not:

<pre>
 ps ax | grep foo
</pre>

The reason for this is that the latter example will also find the __grep foo__ itself in the process list, while the first one won't.

The most useful ps(1) command is probably

<pre>
 ps auxww
</pre>

This gives a lot more information about each process than you get by default.

Here is a poor man's Linux-only ps replacement (for when ps(1) just don't work
or can't be relied upon):

<pre>
 #!/bin/bash
 cd /proc && for p in ~[0-9]* ; do
 	echo -ne "$p\0"
 	tr '\0' ' ' < $p/cmdline
 	echo -ne '\0'
 done | xargs -r0t printf ' %5g %s\n' | sort -ns
</pre>

----

!! Miscellaneous

If you want to unmount a filesystem but it's in use you can use

<pre>
 ps -auxwwe |grep ''mountpoint''
 lsof | grep ''mountpoint''
 fuser -vm ''mountpoint''
</pre>

lsof(8) stands for __l__i__s__t of __o__pen __f__iles.

PerryLorier suggests

<pre>
  fuser -k -v -m /mnt/nfs
</pre>

to kill all processes using that mount point.

If your program says "[Signal] 11", "SegmentationFault", or similar, you can
retrieve information about the process when it crashed. First remove the limit
on dumping core files (so it will dump core this time around):

<pre>
 ulimit -c unlimited
</pre>

Then run the program again. (See builtins(1) and ulimit(3) for more information
about this.) This time when it [SegmentationFault]s it will leave a file called
"core" which contains the state of the program when it died.  This file can be
inspected by

<pre>
 gdb ''programname'' ''corefilename''
</pre>

To find out where it crashed try __bt full__ at the prompt.  You can also print
variables to find out what they currently hold, for example __print argc__ will
tell you the contents of argc.  Of course, __quit__ or [[Ctrl][[d] will exit
gdb.  For more information about the [GNU] debugger see gdb(1).  For more
information about this procedure see DeBugging.

!!Help I'm running out of file handles, what's using them all?
<pre>
 lsof | awk '{print $2}' | uniq -c | sort -n +1 | join -12 -21 - <(ps ax -o pid,command | sort -n)  | sort -n +1
</pre>
This one liner lists processes in the form "pid,number of open files,command" with the process using the most files at the end.
You can use this to determine which process on your system is leaking file handles, then use strace(1) to figure out why.

Read about [Zombie processes|Zombie].

!! Saving processes to disc (software suspend)

Hate having processes die because of kernel upgrades?  The thought of losing your irssi scrollback just too much for you?

Get [Cryopid|http://cryopid.berlios.de/].  Compile (I needed zlib1g-dev on my Ubuntu machine which had build-essentials installed) and then run 'freeze' to save processes to disk.  You start the process again by executing the file that it saves - there is no 'thaw' utility

It can't save a screen session, but you can save the processes inside them and rescreen.

----
CategoryNotes
4 pages link to ProcessNotes:
Last edited on Sunday, September 3, 2006 4:41:25 pm by CraigBox
Edit PageHistory Diff Info LikePages