Solaris performance monitoring
The following is a brief overview of common performance monitoring tools for Solaris, including built-in tools, freeware add-ons, and commercial applications.
Souce: http://www.sunperf.com/perfmontools.html
Built-in Tools
vmstat
The vm in vmstat stands for Virtual Memory, of course. So let us begin our discussion by going over virtual memory on Solaris. There are several important kernel parameters: lotsfree, freemem, desfree, minfree, fastscan, slowscan, nscan, desscan, pagesio, physmem, handspreadpages, maxpgio, cachefree. The major portion of the behavior of the VM subsystem is described by the following graph. There is a “linear„ relationship between the scan rate and free memory.
Many FAQ’s, documents and article have answered the “Where did my memory go?„ question. Here it is again. Solaris does not continuously run the paging daemon. It only runs when there is a “need„ i.e. when the freemem drops below lotsfree. When memory falls below desfree, Solaris scans for pages, which haven’t been used for a while, 100 times a second. Free memory is wasted memory. Using every bit of memory to cache data makes sense when you read the same data over and over again. This is especially true for binary executables.
Imagine a clock, with two hands. The first hand clears a flag on a page of memory. If that page of memory is accessed, it gets set again. The second hand checks the flag to see if it is set. If it is set, the page is “in use„ or has been used “recently„. If it is not set the page has not been used recently, and will be scheduled to be paged out of memory. The parameter handspreadpages determines how many pages of memory are between the first and second hand. A minor fault occurs when a page is marked for being paged to disk, but gets accessed before the VM subsystem actually pages it. A major fault or reclaim is when the memory page is paged to disk and is accessed again and has to be paged into memory. Minor faults are of no real consequence since the data is still in memory and really amounts to changing a flag for that page of memory. If minor faults are large, it is due to handspreadpages being set too low. Reclaims are a sign of a memory shortage, since memory is being paged out only to be brought back in again.
Here’s the burning question, “what indicates a shortage of what?”
root:sparc > vmstat 5 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr cd cd cd f0 in sy cs us sy id 0 0 0 329064 27332 3 64 2 0 1 0 0 0 0 1 0 16 1517 1259 2 4 94 0 0 0 325276 2280 45 526 0 6 6 0 0 0 3 4 0 63 23601 21485 20 47 33 0 0 0 325276 2276 46 517 1 4 4 0 0 0 0 2 0 43 23753 21134 25 48 27 0 0 0 325276 2276 47 528 0 13 13 0 0 0 0 4 0 59 24152 21759 20 42 38 0 0 0 325276 2288 47 539 0 8 8 0 0 0 0 3 0 51 24155 21547 19 41 40 0 0 0 325276 2284 46 527 0 5 5 0 0 0 0 2 0 48 24232 22012 18 45 37 0 0 0 325276 2280 46 481 0 8 8 0 0 0 0 3 0 53 24264 22292 21 45 34 0 0 0 325276 2300 47 527 0 8 8 0 0 0 0 3 0 52 23821 21239 22 46 33 0 0 0 325300 2332 45 526 0 4 4 0 0 0 2 4 0 59 23398 20848 22 42 36
The more important columns are procs: r,b,w, page: re, pi, po, de, sr, cpu: us, sy, id. The r,b,w under proc stands for runnable, blocked and runnable but swapped out. Runnable means the process is in the queue waiting for a CPU to free up and has work to do (is not sleeping). Blocked is for processes waiting for a resource that is busy (see biowait(9f)). Essentially a driver has called biowait and this process is suspended. Having processes in the w column is a bad sign. Having entire processes swapped out even though they are runnable is a sign of severe memory shortage. Unfortunately, if a process does get swapped out and then goes to sleep, it will remain in the w column until it receives a signal to wake it up. Under the page heading the columns re, pi, po, de, and sr stand for reclaims, minor faults, pages paged in, pages paged out, estimated memory shortfall, and scan rate, respectively. Large reclaims and minor faults are not definitive by theirself. They may indicate that the VM subsystem is tuned incorrectly. Pages in and out can also be misleading since Solaris will treat reading an executable from disk as a page in. Also if an application uses mmap to read a file more quickly, then those reads will be counted as page ins. The de column is the kernel’s guess at how much memory needs to be freed. The most important column is sr. If the scan rate is high you have sever memory shortage. Unfortunately most references say 320 is high. I disagree. I have seen boxes go above 10000 on scan rate. This, just like everything else, is proportional to how big your system is. In the past (and present) the VM subsystem has been clamped way too tight. The system would prevent itself form paging and lock up since fastscan was set to a maximum of 8192. You could not scan for pages to page out faster. Hence the box ran out free memory and could do no new work.
mpstat
root:sparc > mpstat 1 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 28 0 1 112 2 436 1 0 110 0 540 1 3 0 95 1 24 0 2 3 0 390 1 0 137 0 492 1 2 0 96 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 248 0 3 144 10 4790 15 10 5813 0 11328 23 65 0 12 1 290 1 22 10 3 15438 5 7 1505 0 11132 17 25 1 57 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 303 0 3 142 9 12257 14 8 2641 0 12321 26 60 0 14 1 231 0 21 23 3 8919 16 10 4151 0 11199 41 18 0 41 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 392 0 4 139 6 7497 16 12 2503 0 12293 22 67 0 11 1 369 0 23 14 3 10837 8 19 1810 0 9658 22 25 0 53 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 365 0 4 133 6 14145 10 12 1706 0 12270 20 63 0 17 1 169 0 20 19 3 6716 10 9 4029 0 10983 36 30 0 34 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 187 0 2 132 11 3070 11 11 4338 0 11461 31 57 0 12 1 346 1 9 19 3 17596 10 11 910 0 11392 11 29 3 57 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 343 0 35 194 44 1834 44 12 4344 0 10547 31 59 0 10 1 386 0 24 63 4 16694 125 12 600 0 10397 13 27 0 60 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 345 0 3 119 6 17230 9 12 713 0 11701 13 66 0 21 1 210 0 6 22 3 3121 13 12 5833 0 10846 43 37 0 20
mpstat is one of my favorites. It has everything for looking at your processors. mp stands for Multi-Processor. Hence, it gives statistics on each processor. The columns are as follows: CPU id, minor faults, major faults, cross calls, interrupts, thread interrupts, context switches, involuntary context switches, migrations, spin on mutex lock, spin on reader-writer mutex lock, system calls, user cpu time, system cpu time, wait cpu time, and idle cpu time. Some of these are the same as with vmstat. I will look at the new ones which are important in my opinion. A cross happens when an interrupt arrives on one CPU, but is intended for a system call executing on a different CPU. An involuntary context switch happens when a process has work to do, but its time slice has elapsed. A large percentage of context switches being involuntary is an indicator of being CPU bound. A migration occurs when process is switched from one CPU to a different CPU after a context switch. Typically a process will stay on a particular CPU, but will occasionally switch CPU’s. When this happens the cache has to be synced from the original CPU to the new CPU. This can lead to inefficiency with a large number of processors. A spin on a mutex happens when a mutex lock cannot be obtained on the first or subsequent trys. A large number of spins means processes are trying to use the same resource which is becoming a bottleneck. The wt column is the percentage of time slices the CPU spends running biowait. See the discussion for blocked processes for vmstat above.
iostat
root:sparc > iostat -nxtc tty cpu tin tout us sy wt id 0 13 1 0 0 98 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.2 0.1 1.8 1.2 0.0 0.0 22.4 9.6 0 0 c0t0d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 fd0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t2d0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 sparc:vold(pid205)
iostat is my favorite way of looking at disk activity. The column that I pay most attention to is the wsvc_t column. This is the average service time in milliseconds. A high number is a sign that the disk is becoming a bottleneck. A rule of thumb is >35 is cause for investigation. Large numbers in the r/s and w/s is an indication of a too small block size. This could also be a poorly tuned application that is making many small reads/writes instead of a few large reads/writes. The kr/s and kr/s give you a good indication of how much bandwidth you are using. For a single Ultra Wide Differential SCSI disk I would expect to get 10MB/s in throughput. For a correctly configured stripe, I would expect to get 10MB/s x number of disks in the stripe. On a read from RAID 5 you should get similar performance. On write the cache will help and you should get close to the same performance while the cache is not being over run.
netstat
root:sparc > netstat -I iprb0 5 input iprb0 output input (Total) output packets errs packets errs colls packets errs packets errs colls 41860 0 66310 0 0 42371 0 66821 0 0 8 0 7 0 0 8 0 7 0 0 7 0 7 0 0 7 0 7 0 0 10 0 10 0 0 10 0 10 0 0 8 0 8 0 0 8 0 8 0 0 7 0 7 0 0 7 0 7 0 0 8 0 8 0 0 8 0 8 0 0 8 0 8 0 0 8 0 8 0 0 7 0 8 0 0 7 0 8 0 0 8 0 7 0 0 8 0 7 0 0
netstat is a good for getting a general feel for network traffic. Unlike iostat it does not give you throughput numbers. This is tragic since it is hard to judge whether an interface is pegged by the number of packets. Each packet can vary in size greatly. If you have Sun Trunking software you can use nettr to get this information.
For example
nettr -stats 0 device=qfe interval=1
An undocumented option for netstat is the -k option. It dumps a huge amount of kernel parameters and statistics. This is extremely helpful in diagnosing problems. Unfortunately it is undocumented so you will just have to search the header files for info. maybe Sun will release this info but it seems unlikely.
root:sparc > netstat -k | grep nocanput retry_error 0 first_collisions 0 nocarrier 0 inits 7 nocanput 0
One key indicator that I have run into is nocanput being non-zero. If this is the case your streams queue is too small. It should be set to 400 per GB of memory. Put a similar line in your /etc/system file. This assumes you have 4GB RAM.
set sq_max_size=1600
I use netstat as a diagnostic tool more than a performance tool. netstat -an | grep ESTA | wc -l will tell you how many connections you have. This would be very important to a web server administrator for example. netstat -an | grep LISTEN is a good way to check for software that should be turned off for security reasons. netstat -nr is a good way to check for routing information.
nfsstat
NFS tuning is going to require a fairly lengthy discussion, so I will hold off for now. For now the best tool is nfsstat which gives a copious amount of information.
root:sparc > nfsstat Server rpc: Connection oriented: calls badcalls nullrecv badlen xdrcall dupchecks 36 0 0 0 0 3 dupreqs 0 Connectionless: calls badcalls nullrecv badlen xdrcall dupchecks 0 0 0 0 0 0 dupreqs 0 Server nfs: calls badcalls 36 0 Version 2: (0 calls) null getattr setattr root lookup readlink 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% read wrcache write create remove rename 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir statfs 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% Version 3: (36 calls) null getattr setattr lookup access readlink 1 2% 14 38% 0 0% 0 0% 14 38% 0 0% read write create mkdir symlink mknod 2 5% 0 0% 0 0% 0 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 0 0% 0 0% 0 0% 0 0% 0 0% 3 8% fsstat fsinfo pathconf commit 0 0% 1 2% 1 2% 0 0% Server nfs_acl: Version 2: (0 calls) null getacl setacl getattr access 0 0% 0 0% 0 0% 0 0% 0 0% Version 3: (0 calls) null getacl setacl 0 0% 0 0% 0 0% Client rpc: Connection oriented: calls badcalls badxids timeouts newcreds badverfs 414 0 0 0 0 0 timers cantconn nomem interrupts 0 0 0 0 Connectionless: calls badcalls retrans badxids timeouts newcreds 15 1 0 0 0 0 badverfs timers nomem cantsend 0 12 0 0 Client nfs: calls badcalls clgets cltoomany 15 1 15 0 Version 2: (14 calls) null getattr setattr root lookup readlink 0 0% 13 92% 0 0% 0 0% 0 0% 0 0% read wrcache write create remove rename 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% link symlink mkdir rmdir readdir statfs 0 0% 0 0% 0 0% 0 0% 0 0% 1 7% Version 3: (0 calls) null getattr setattr lookup access readlink 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% read write create mkdir symlink mknod 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% remove rmdir rename link readdir readdirplus 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% fsstat fsinfo pathconf commit 0 0% 0 0% 0 0% 0 0% Client nfs_acl: Version 2: (1 calls) null getacl setacl getattr access 0 0% 0 0% 0 0% 1 100% 0 0% Version 3: (0 calls) null getacl setacl 0 0% 0 0% 0 0%
Free Tools
Xkstat
This is a X Windows utility to monitor various subsystems on Solaris servers/workstations. The following can be monitored: virtual memory, network interfaces, disk io, and CPU utilization. It is based on the user accessible interface to the kstat driver. This is unique to Solaris, as far as I know. The kstat driver provides statistics which are kept on every subsystem. This is not by any means an exhaustive tool, there are too many variables to present at once. If one wanted a complete dump of all statistics provided, it would be extremely difficult (probably humanly impossible) to sift through all the data real time. For an example, issue the following command to get a snap shot: netstat -k. This is an absolute gold mine of information on the state of the system.
The syntax for running Xkstat is:
Xkstat -{net | vm | disk | cpu} [sleeptime] Xkstat -other ks_module ks_instance ks_name name [sleeptime] Xkstat -otherc ks_module ks_instance ks_name name [sleeptime] Xkstat -dump file_name
The options are for network, virtual memory, disk IO and CPU utilization, respectively. The optional sleep time can be specified as a decimal number. Hence the intervals can be smaller than one second, except for CPU. The default is 1 second. CPU information is only updated every second, hence it doesn’t make sense to query more often. Beware that the smaller the sleep time the more of a load the tool will put on the system. What load you can withstand will depend on how many CPU’s you have. Most of the load will fall under /usr/openwin/bin/Xsun (Solaris’s Window Manager for CDE) which should be running on a workstation rather than the server.
$ /usr/ucb/ps -aux | head -1 USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND $ /usr/ucb/ps -aux | grep '[X]kstat' n 8432 1.0 0.9 2492 996 pts/5 S 11:43:17 0:00 ./Xkstat -vm n 8434 0.4 0.9 2492 1004 pts/5 S 11:43:17 0:00 ./Xkstat -disk n 8428 0.3 0.9 2492 1004 pts/5 S 11:43:17 0:00 ./Xkstat -net n 8430 0.1 0.9 2492 992 pts/5 S 11:43:17 0:00 ./Xkstat -cpu total 1.8 % CPU Pentium 200 MHz $
One of the main design goals is for this tool to be “light” on CPU/memory usage. The last thing a performance tuning tool should do is degrade performance. For that reason I have coded it entirely using XLib only. It could be fancier with a lot of Motif (or some other toolkit, GTK, FLTK) buttons and widgets. However, I guarantee it would use more than 0.1-1.0 % CPU and the footprint would be much bigger.
Along the caption bar is the host name and architecture of the box it is running on.
At the top of each column is the name of the interface for which the graphs apply. The arrows at the bottom allow one to scroll left or right if several interfaces exist. The network parameters monitored are opackets, ipackets, oerrors, ierror, collisions, obytes, rbytes, brdcstxmt, brdcstrcv, unknowns, oflo, uflo. These are the names of the variables in the kstat structures. The best way to find information on what these mean is to look for the parameter in the header files under /usr/include or /usr/include/sys.
opackets - output packets. ipackets - input packets. oerrors - output errors. ierror - input errors. collisions - collisions. obytes - bytes of data output. rbytes - bytes of data read. brdcstxmt - broadcast packets transmitted. brdcstrcv - broadcast packets received. unknowns - packets of unknown type. oflo - over Flows. uflo - under flows.
The virtual memory info is both static variables and dynamic variables. The static variables are presented on the left and the dynamic variables are presented in the graphs on the right. Again, the names are taken from the variable names in the kstat structure or vm structure. Information can be found for each variable by searching the header files, for example:
$ find /usr/include -name '*.h' -exec grep physmem {} \; struct memlist *physinstalled; /* amt of physmem installed */ struct memlist *physavail; /* amt of physmem avail for use */ extern pgcnt_t physmem; /* physical memory (pages) on this CPU */ void add_physmem(page_t *, size_t); $ for X in kernelbase econtig lotsfree desfree minfree fastscan slowscan pp_kernel > do > find /usr/include -name '*.h' -exec grep $X {} \; > done * with installed memory and the eprom variable 'eprom_kernelbase'. * which it saves in the variable _kernelbase. * _kernelbase with the correct value in i86pc/os/startup.c * i86 and i86pc files use kernelbase instead of KERNLEBASE, which is extern uintptr_t kernelbase; * The eprom variable 'eprom-kernelbase' can be used to increase the amount of extern const uintptr_t _kernelbase; #define KERNELBASE _kernelbase extern pgcnt_t lotsfree; /* max free before clock freezes */ extern pgcnt_t swapfs_desfree; extern pgcnt_t desfree; /* minimum free pages before swapping begins */ extern pgcnt_t swapfs_minfree; /* amount of availrmem (in pages) */ * tmpfs_minfree is the amount (in pages) of anonymous memory that tmpfs * tmpfs could allocate up to 16MB - tmpfs_minfree. The default value * for tmpfs_minfree is btopr(TMPMINFREE) but it can cautiously patched extern long tmpfs_minfree; /* Anonymous memory in pages */ int minfree; /* %age to reserve for root */ int32_t fs_minfree; /* minimum percentage of free blocks */ int vfs_minfrags; /* calc. from fs_minfree */ extern pgcnt_t segspt_minfree; /* low water mark for availrmem in seg_spt */ extern pgcnt_t minfree; /* no of pages to try to keep free via daemon */ * MAX(ani_max - ani_resv, 0) + (availrmem - swapfs_minfree) (k_anoninfo.ani_max + MAX((spgcnt_t)(availrmem - swapfs_minfree), 0)) MAX((spgcnt_t)(availrmem - swapfs_minfree), 0)) extern pgcnt_t fastscan; extern pgcnt_t slowscan; $
The two most important parameters to watch are nscan and desscan. desscan will be a minimum of 25. nscan is the number of pages actually scanned. If these two variables are constantly high, it is time to pony up for some more memory. fastscan and slowscan are poorly tuned for systems with large amounts of memory on Solaris 2.6. If you have more than 256 MB of memory, they are clamped too low. fastscan is important in that this is the maximum number of pages that will be scanned per clock tick (100 clock ticks per second) after the available memory has fallen below minfree. If memory falls below throttlefree, the box will lock until it can free more memory. The scan rate is linear from slowscan to fastscan as memory decreases from lotsfree to minfree.
Unfortunately Solaris represents disks, CDROM’s and other drives internally as some type (usually by the name of the driver) and an instance number. This is not the same as the cXtXdXsX that administrators are used to seeing. All hope is not lost though. With some clever ingenuity one can put the pieces together. Let us take cmdk7 for an example. First we find its entry in the file /etc/path_to_inst. Then we examine the links in the /dev/dsk directory and look for the same device path. The information presented is for the entire disk not just the first slice. This will be different for each type of disk, i.e. ide, scsi, fibre channel, … etc.
$ cat /etc/path_to_inst | grep cmdk "/pci@0,0/pci-ide@7,1/ata@0/cmdk@0,0" 0 "cmdk" "/pci@0,0/pci-ide@7,1/ata@0/cmdk@1,0" 1 "cmdk" "/pci@0,0/pci-ide@7,1/ata@1/cmdk@1,0" 8 "cmdk" "/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0" 7 "cmdk" $ ls -l /dev/dsk/c1d0* lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0p0 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:q lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0p1 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:r lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0p2 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:s lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0p3 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:t lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0p4 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:u lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s0 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:a lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s1 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:b lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s10 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:k lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s11 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:l lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s12 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:m lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s13 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:n lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s14 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:o lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s15 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:p lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s2 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:c lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s3 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:d lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s4 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:e lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s5 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:f lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s6 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:g lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s7 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:h lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s8 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:i lrwxrwxrwx 1 root root 50 Aug 5 1999 /dev/dsk/c1d0s9 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:j $
On the left is a bar graph of the CPU usage at the moment. Solaris divides the usage up into idle, user, kernel and wait. Hence I have represented it that way here. It is maintained as the number of clock ticks that are spent doing one of the 4 categories of tasks. This is updated once per second. Hence if you specify a sleep time less than 1 second, it will increase it to 1. If you have more than 36 CPU’s, one overall graph will be displayed. This is to try to not present more information than is useful. On the right side is various virtual memory, thread and hardware parameters that are interesting to me. These can be found in /usr/include/sys/sysinfo.h.
Just in case I left out something you need to monitor, I put in two generic options. The option -other is for time dependent counters i.e. counters which are incremented over time. The option -otherc (the c is for “constant”) is for variables which are not counters i.e. the variable is a rate or kernel parameter. This works for KSTAT_TYPE_NAMED variables. To get a list of the available parameters, issue the following command:
Xkstat -dump file
If the parameter you want to monitor is listed under a module that is Data Type NAMED then you can monitor it. See the man page for kstat_lookup(3K). Example:
Xkstat -other eepro 0 eepro0 opackets64 Xkstat -otherc unix -1 system_pages nscan 5
As with all software this needs to be tested in your environment to ensure it operates as expected. No expressed or implied warranty is given. This software is as-is with no statement of merchantability or fitness for any purpose.
Souce: http://www.sunperf.com/perfmontools.html
Popularity: 5% [?]
Related posts:
