Solaris Useful Commands

Commands

TRUSS

truss -c (Solaris >= 8): This astounding option to truss provides a profile summary of the command being trussed

$ truss -c grep asdf work.doc

syscall              seconds   calls  errors

_exit                    .00       1

read                     .01      24

open                     .00       8      4

close                    .00       5

brk                      .00      15

stat                     .00       1

fstat                    .00       4

execve                   .00       1

mmap                     .00      10

munmap                   .01       3

memcntl                  .00       2

llseek                   .00       1

open64                   .00       1

                        ----     ---    ---

sys totals:              .02      76      4

usr time:                .00

elapsed:                 .05

It can also show profile data on a running process. In this case, the data shows what the process did between when truss was started and when truss execution was terminated with a control-c. It’s ideal for determining why a process is hung without having to wade through the pages of truss output.

truss -d and truss -D (Solaris >= 8): These truss options show the time associated with each system call being shown by truss and is excellent for finding performance problems in custom or commercial code. For example:

$ truss -d who

Base time stamp:  1035385727.3460  [ Wed Oct 23 11:08:47 EDT 2002 ]

 0.0000 execve(“/usr/bin/who”, 0xFFBEFD5C, 0xFFBEFD64)  argc = 1

 0.0032 stat(“/usr/bin/who”, 0xFFBEFA98)                = 0

 0.0037 open(“/var/ld/ld.config”, O_RDONLY)             Err#2 ENOENT

 0.0042 open(“/usr/local/lib/libc.so.1”, O_RDONLY)      Err#2 ENOENT

 0.0047 open(“/usr/lib/libc.so.1”, O_RDONLY)            = 3

 0.0051 fstat(3, 0xFFBEF42C)                            = 0

. . .

truss -D is even more useful, showing the time delta between system calls:

Dilbert> truss -D who

 0.0000 execve(“/usr/bin/who”, 0xFFBEFD5C, 0xFFBEFD64)  argc = 1

 0.0028 stat(“/usr/bin/who”, 0xFFBEFA98)                = 0

 0.0005 open(“/var/ld/ld.config”, O_RDONLY)             Err#2 ENOENT

 0.0006 open(“/usr/local/lib/libc.so.1”, O_RDONLY)      Err#2 ENOENT

 0.0005 open(“/usr/lib/libc.so.1”, O_RDONLY)            = 3

 0.0004 fstat(3, 0xFFBEF42C)                            = 0

In this example, the stat system call took a lot longer than the others.

truss -T: This is a great debugging help. It will stop a process at the execution of a specified system call. (“-U” does the same, but with user-level function calls.) A core could then be taken for further analysis, or any of the /proc tools could be used to determine many aspects of the status of the process.

truss -l (improved in Solaris 9): Shows the thread number of each call in a multi-threaded process. Solaris 9 truss -l finally makes it possible to watch the execution of a multi-threaded application.

Truss is truly a powerful tool. It can be used on core files to analyze what caused the problem, for example. It can also show details on user-level library calls (either system libraries or programmer libraries) via the “-u” option.

PKG-GET

pkg-get: This is a nice tool (http://www.bolthole.com/solaris) for automatically getting freeware packages. It is configured via /etc/pkg-get.conf. Once it’s up and running, execute pkg-get -a to get a list of available packages, and pkg-get -i to get and install a given package.

COREADM

coreadm (Solaris >= 8): In the “old” days (before coreadm), core dumps were placed in the process’s working directory. Core files would also overwrite each other. All this and more has been addressed by coreadm, a tool to manage core file creation. With it, you can specify whether to save cores, where cores should be stored, how many versions should be retained, and more. Settings can be retained between reboots by coreadm modifying /etc/coreadm.conf.

PROCESS MANAGEMENT

pgrep (Solaris >= 8): pgrep searches through /proc for processes matching the given criteria, and returns their process-ids. A great option is “-n”, which returns the newest process that matches.

preap (Solaris >= 9): Reaps zombie processes. Any processes stuck in the “z” state (as shown by ps), can be removed from the system with this command.

pargs (Solaris >= 9): Shows the arguments and environment variables of a process.

nohup -p (Solaris >= 9): The nohup command can be used to start a process, so that if the shell that started the process closes (i.e., the process gets a “SIGHUP” signal), the process will keep running. This is useful for backgrounding a task that should continue running no matter what happens around it. But what happens if you start a process and later want to HUP-proof it? With Solaris 9, nohup -p takes a process-id and causes SIGHUP to be ignored.

plimit (Solaris >= 8): This command displays and sets the per-process limits on a running process. This is handy if a long-running process is running up against a limit (for example, number of open files). Rather than using limit and restarting the command, plimit can modify the running process.

prstat (Solaris >= 8): prstat is top and a lot more. Both commands provide a screen’s worth of process and other information and update it frequently, for a nice window on system performance. prstat has much better accuracy than top. It also has some nice options. “-a” shows process and user information concurrently (sorted by CPU hog, by default). “-c” causes it to act like vmstat (new reports printed below old ones). “-C” shows processes in a processor set. “-j” shows processes in a “project”. “-L” shows per-thread information as well as per-process. “-m” and “-v” show quite a bit of per-process performance detail (including pages, traps, lock wait, and CPU wait). The output data can also be sorted by resident-set (real memory) size, virtual memory size, execute time, and so on. prstat is very useful on systems without top, and should probably be used instead of top because of its accuracy (and some sites care that it is a supported program).

trapstat (Solaris >= 9): trapstat joins lockstat and kstat as the most inscrutable commands on Solaris. Each shows gory details about the innards of the running operating system. Each is indispensable in solving strange happenings on a Solaris system. Best of all, their output is good to send along with bug reports, but further study can reveal useful information for general use as well.

vmstat -p (Solaris >= 8): Until this option became available, it was almost impossible (see the “se toolkit”) to determine what kind of memory demand was causing a system to page. vmstat -p is key because it not only shows whether your system is under memory stress (via the “sr” column), it also shows whether that stress is from application code, application data, or I/O. “-p” can really help pinpoint the cause of any mysterious memory issues on Solaris.

pmap -x (Solaris >= 8, bugs fixed in Solaris >= 9): If the process with memory problems is known, and more details on its memory use are needed, check out pmap -x. The target process-id has its memory map fully explained, as in:

# pmap -x 1779

1779:   -ksh

 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File

00010000     192     192       -       - r-x--  ksh

00040000       8       8       8       - rwx--  ksh

00042000      32      32       8       - rwx--    [ heap ]

FF180000     680     664       -       - r-x--  libc.so.1

FF23A000      24      24       -       - rwx--  libc.so.1

FF240000       8       8       -       - rwx--  libc.so.1

FF280000     568     472       -       - r-x--  libnsl.so.1

FF31E000      32      32       -       - rwx--  libnsl.so.1

FF326000      32      24       -       - rwx--  libnsl.so.1

FF340000      16      16       -       - r-x--  libc_psr.so.1

FF350000      16      16       -       - r-x--  libmp.so.2

FF364000       8       8       -       - rwx--  libmp.so.2

FF380000      40      40       -       - r-x--  libsocket.so.1

FF39A000       8       8       -       - rwx--  libsocket.so.1

FF3A0000       8       8       -       - r-x--  libdl.so.1

FF3B0000       8       8       8       - rwx--    [ anon ]

FF3C0000     152     152       -       - r-x--  ld.so.1

FF3F6000       8       8       8       - rwx--  ld.so.1

FFBFE000       8       8       8       - rw---    [ stack ]

-------- ------- ------- ------- -------

total Kb    1848    1728      40       -

Here we see each chunk of memory, what it is being used for, how much space it is taking (virtual and real), and mode information.

DISK SPACE

df -h (Solaris >= 9): This command is popular on Linux, and just made its way into Solaris. df -h displays summary information about file systems in human-readable form:

$ df -h

Filesystem             size   used  avail capacity  Mounted on

/dev/dsk/c0t0d0s0      4.8G   1.7G   3.0G    37%    /

/proc                    0K     0K     0K     0%    /proc

mnttab                   0K     0K     0K     0%    /etc/mnttab

fd                       0K     0K     0K     0%    /dev/fd

swap                   848M    40K   848M     1%    /var/run

swap                   849M   1.0M   848M     1%    /tmp

/dev/dsk/c0t0d0s7       13G    78K    13G     1%    /export/home