Periodically I need to update my systems with new NTP server information. Since I have no idea if the remote servers are well kept, I tend to run ntpdate in test mode prior to starting ntpd or adding ntpdate to root’s crontab. To run ntpdate in test mode, you can use the “-d” option:

$ /usr/sbin/ntpdate -d 192.168.1.1

12 Feb 17:36:36 ntpdate[24068]: ntpdate 4.1.2@1.892 Tue Feb 24 06:32:26 EST 2004 (1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
server 192.168.1.1, port 123
stratum 3, precision -18, leap 00, trust 000
refid [192.168.1.1], delay 0.02592, dispersion 0.00000
transmitted 4, in filter 4
reference time:    c97b6a71.b4251202  Mon, Feb 12 2007 17:36:33.703
originate timestamp: c97b6ab2.f2ec3995  Mon, Feb 12 2007 17:37:38.948
transmit timestamp:  c97b6a74.f210f0e9  Mon, Feb 12 2007 17:36:36.945
filter delay:  0.02609  0.02592  0.02594  0.02596
       0.00000  0.00000  0.00000  0.00000
filter offset: 62.00322 62.00318 62.00317 62.00316
       0.000000 0.000000 0.000000 0.000000
delay 0.02592, dispersion 0.00000
offset 62.003182

In addition to printing the timestamps, it also provides the offset the clock will be adjusted by. Certain applications dislike the time moving forward or back, which makes the ntpdate test option even more useful.

The Linux kernel uses a slab based allocator to allocate kernel memory. Inside each slab is a collection of objects that have been allocated by one or more kernel subsystems. To monitor slab utilization in realtime, most modern day Linux distributions ship with the slaptop utility. When slabtop is run without any arguments, it displays a nice slab usage summary, and provides details of how various slabs are being used:

$ slabtop

 Active / Total Objects (% used)    : 220410 / 234629 (93.9%)
Active / Total Slabs (% used)      : 4728 / 4728 (100.0%)
Active / Total Caches (% used)     : 91 / 139 (65.5%)
Active / Total Size (% used)       : 16609.59K / 18017.14K (92.2%)
Minimum / Average / Maximum Object : 0.01K / 0.08K / 128.00K

OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
80591  80494  99%    0.02K    397      203      1588K avtab_node
53562  53562 100%    0.03K    474      113      1896K size-32
35108  35099  99%    0.05K    524       67      2096K buffer_head
7378   7377  99%    0.27K    527       14      2108K radix_tree_node
7182   7182 100%    0.14K    266       27      1064K dentry_cache
6499   6497  99%    0.05K     97       67       388K selinux_inode_security
5460   5398  98%    0.04K     65       84       260K sysfs_dir_cache
3953   3425  86%    0.06K     67       59       268K size-64
3663   3644  99%    0.41K    407        9      1628K inode_cache
3654    479  13%    0.02K     18      203        72K biovec-1
2904   1882  64%    0.09K     66       44       264K vm_area_struct
2580   2442  94%    0.12K     86       30       344K size-128
2080    924  44%    0.19K    104       20       416K filp
2070    744  35%    0.12K     69       30       276K bio
1944    660  33%    0.05K     27       72       108K journal_head
1896   1892  99%    0.59K    316        6      1264K ext3_inode_cache
1595    525  32%    0.02K     11      145        44K anon_vma
1288   1209  93%    0.04K     14       92        56K Acpi-Operand
 845    730  86%    0.02K      5      169        20K Acpi-Namespace
 648    511  78%    0.05K      9       72        36K avc_node
 540    471  87%    0.43K     60        9       240K proc_inode_cache

These is a sweet utility, and is another one of those tools that should be in every Linux administrators tool belt.

Periodically situations arise where applications will write to memory that isn’t mapped into their address space. On UNIX systems, this results in a SIGSEGV signal being sent to the offending process. If for some reason you can’t get a core file, you can run the application under the control of the catchsegv utility. The following example shows the results that are displayed when a SIGSEGV signal is received, and the program was run under the control of the catchsegv script:

$ catchsegv ./coreme 10000000

Calling malloc() to allocate 10000000 bytes of heap space
*** Segmentation fault
Register dump:

EAX: fffffffc   EBX: bff12634   ECX: bff12634   EDX: 00bd3ff4
ESI: bff12634   EDI: 00000000   EBP: bff12648   ESP: bff12488

EIP: 0034a402   EFLAGS: 00200246

CS: 0073   DS: 007b   ES: 007b   FS: 0000   GS: 0033   SS: 007b

Trap: 00000000   Error: 00000000   OldMask: 00000000
ESP/signal: bff12488   CR2: 00000000

Backtrace:
/lib/libSegFault.so[0x3d512f]
??:0(??)[0x34a420]
??:0(main)[0x8048453]
/lib/libc.so.6(__libc_start_main+0xdc)[0xab0f2c]
??:0(_start)[0x8048361]

Memory map:

0034a000-0034b000 r-xp 0034a000 00:00 0 [vdso]
003d3000-003d6000 r-xp 00000000 08:01 1426324 /lib/libSegFault.so
003d6000-003d7000 r-xp 00002000 08:01 1426324 /lib/libSegFault.so
003d7000-003d8000 rwxp 00003000 08:01 1426324 /lib/libSegFault.so
00a47000-00a52000 r-xp 00000000 08:01 1427470 /lib/libgcc_s-4.1.1-20070105.so.1
00a52000-00a53000 rwxp 0000a000 08:01 1427470 /lib/libgcc_s-4.1.1-20070105.so.1
00a7e000-00a97000 r-xp 00000000 08:01 1427446 /lib/ld-2.5.so
00a97000-00a98000 r-xp 00018000 08:01 1427446 /lib/ld-2.5.so
00a98000-00a99000 rwxp 00019000 08:01 1427446 /lib/ld-2.5.so
00a9b000-00bd2000 r-xp 00000000 08:01 1427447 /lib/libc-2.5.so
00bd2000-00bd4000 r-xp 00137000 08:01 1427447 /lib/libc-2.5.so
00bd4000-00bd5000 rwxp 00139000 08:01 1427447 /lib/libc-2.5.so
00bd5000-00bd8000 rwxp 00bd5000 00:00 0
08048000-08049000 r-xp 00000000 08:03 1442166 /home/matty/coreme
08049000-0804a000 rw-p 00000000 08:03 1442166 /home/matty/coreme
095bd000-095e2000 rw-p 095bd000 00:00 0
b75f4000-b7f7f000 rw-p b75f4000 00:00 0
b7f8c000-b7f8e000 rw-p b7f8c000 00:00 0
bfefe000-bff13000 rw-p bfefe000 00:00 0 [stack]

This is a nifty utility, and can be useful for viewing the environment of a process at the time the segmentation violation occurred.

While the days of conflicting IRQ lines are almost behind us, it is still useful to see the interrupt request lines associated with the devices in a given system. To view IRQ to device mappings on Solaris systems, the mdb ::interrupts DCMD (requires a recent build of Nevada) can be used:

$ mdb -k
Loading modules: [ unix krtld genunix specfs ufs pcipsy ip sctp fctl nca md sd zfs
                         random logindmux ptm cpc fcip sppp nfs ]
> ::interrupts

      Device   Shared  Type    MSG #   State   INO     Mondo    Pil    CPU
       glm#0      no   Fixed    ---    enbl    0x14    0x7d4      4      0
       hme#0      no   Fixed    ---    enbl    0x21    0x7e1      6      0
        su#1      no   Fixed    ---    enbl    0x2a    0x7ea     12      0
        su#0      no   Fixed    ---    enbl    0x29    0x7e9     12      0
        se#0      no   Fixed    ---    enbl    0x2b    0x7eb     12      0
       m64#0      no   Fixed    ---    enbl    0xf     0x7cf      9      0
     power#0      no   Fixed    ---    enbl    0x25    0x7e5     14      0
      uata#0      no   Fixed    ---    enbl    0x20    0x7e0      4      0

On Linux systems you can cat /proc/interrupts to get IRQ information:

$ cat /proc/interrupts

           CPU0
0:     162248          XT-PIC  timer
1:        335          XT-PIC  i8042
2:          0          XT-PIC  cascade
5:      31561          XT-PIC  eth0
6:          1          XT-PIC  uhci_hcd:usb1
7:          1          XT-PIC  parport0
8:          1          XT-PIC  rtc
9:        165          XT-PIC  acpi, ide2, uhci_hcd:usb2
12:        278          XT-PIC  i8042
14:       8285          XT-PIC  ide0
15:       5596          XT-PIC  ide1
NMI:          0
LOC:          0
ERR:          0
MIS:          0

I recently came across the procinfo utility, and use it periodically to view system utilization and configuration data on the Linux servers I support. Procinfo is a monitoring utility that interfaces with the Linux /proc file system, and displays data such as CPU utilization, memory utilization, interrupts serviced and information on the modules that are currently loaded into the kernel. The full list of options is documented in the procinfo “-h” (print a help screen) option:

$ procinfo -h

procinfo version 18 (2001-03-02)
usage: ./procinfo [-fsmadiDSbhv] [-nN] [-Ffile]

      -s      display memory, disk, IRQ & DMA info (default)
      -m      display module and device info
      -a      display all info
      -f      run full screen

      -i      show all IRQ channels, not just those used
      -nN     pause N second between updates (implies -f)
      -d      show differences rather than totals (implies -f)
      -D      show current memory/swap usage, differences on rest
      -S      with -nN and -d/-D, always show values per second
      -r      show memory usage -/+ buffers/cache
      -Ffile  print output to file -- normally a tty
      -v      print version info
      -h      print this help

To see this nifty utility in action, you can run it without arguments to get the basic display:

$ procinfo

Linux 2.6.16-1.2122_FC5 (bhcompile@hs20-bc1-3) (gcc 4.1.0 20060304 ) #1 Sun May 21 15:01:01 EDT 2006 1CPU [fedora]

Memory:      Total        Used        Free      Shared     Buffers
Mem:        515816      509384        6432           0       21924
Swap:       522104          76      522028

Bootup: Sat Jun 24 10:12:53 2006    Load average: 1.83 1.10 0.55 1/63 5420

user  :       0:01:21.51   1.8%  page in :        0
nice  :       0:00:16.94   0.4%  page out:        0
system:       0:01:24.18   1.8%  swap in :        0
idle  :       1:06:15.81  87.1%  swap out:        0
uptime:       1:16:05.69         context :   350192

irq  0:   1136198 timer                 irq  9:        78 Intel 82801BA-ICH2,
irq  1:       232 i8042                 irq 10:    183597 eth0
irq  2:         0 cascade [4]           irq 12:       310 i8042
irq  6:         6                       irq 14:    110425 ide0
irq  8:         1 rtc

In addition to displaying status information, you can also display the modules, devices and file systems that are present in the kernel by invoking procinfo with the “-m” (display module and device info) option:

$ procinfo -m

Linux 2.6.16-1.2122_FC5 (bhcompile@hs20-bc1-3) (gcc 4.1.0 20060304 ) #1 Sun May 21 15:01:01 EDT 2006 1CPU [fedora]

Kernel Command Line:
ro root=LABEL=/ rhgb quiet

Modules:
220 *ipv6            19 *autofs4         16 *hidp            34  rfcomm          23 *l2cap           43 *bluetooth
133 *sunrpc           5  ipt_REJECT       3  xt_tcpudp       12 *x_tables        20  dm_mirror       50 *dm_mod
12  lp              25  parport_pc      34 *parport         56  floppy           8  nvram           28  uhci_hcd
30  snd_intel8x0    82 *snd_ac97_codec   2 *snd_ac97_bus     4  snd_seq_dummy   28  snd_seq_oss      7 *snd_seq_midi_e
46 *snd_seq          9 *snd_seq_device  44  snd_pcm_oss     16 *snd_mixer_oss   75 *snd_pcm         22 *snd_timer
10  ne2k_pci        49 *snd             10 *8390             9 *soundcore       10 *snd_page_alloc 113 *ext3
51 *jbd          

Character Devices:                      Block Devices:
1 mem              13 input             1 ramdisk
4 /dev/vc/0        14 sound             2 fd
4 tty              29 fb                3 ide0
4 ttyS            116 alsa              9 md
5 /dev/tty        128 ptm             253 device-mapper
5 /dev/console    136 pts             254 mdp
5 /dev/ptmx       180 usb
6 lp              189 usb_device
7 vcs             216 rfcomm
10 misc            254 pcmcia                             

File Systems:
[sysfs]             [rootfs]            [bdev]              [proc]              [binfmt_misc]       [debugfs]
[securityfs]        [sockfs]            [usbfs]             [pipefs]            [futexfs]           [tmpfs]
[inotifyfs]         [eventpollfs]       [devpts]            ext2                [ramfs]             [hugetlbfs]
iso9660             [mqueue]            ext3                [rpc_pipefs]        [autofs]

As with most utilities, you can also invoke procinfo with the “-f” (run full screen) option to get a full screen display that is periodically refreshed (you can control the refresh rate with the “-n” (pause N seconds) option).

When an application is running, it is sometimes useful to see which stack frame the application is currently in. This is especially useful when debugging hung processes, since you can see which function the program is currently executing, and the code path taken to get to that point. Both Linux and Solaris provide utilities to capture live stack traces from processes. The Solaris tool to grab this information is called pstack. Pstack takes a process id as an argument, and will print stack traces for each thread in the process id passed as an argument:

$ pstack 539

539:    slapd -f /opt/openldap-common/etc/slapd.conf -u openldap -g openldap -
-----------------  lwp# 1 / thread# 1  --------------------
fee412d0 lwp_wait (2, ffbffaac)
fee3ccd8 _thrp_join (2, 0, 0, 1, ffbffaac, fee6cbc0) + 34
00032f30 slapd_daemon (0, 130118, 0, 0, 0, 1) + cc
00023bb8 main     (9, ffbffc2c, a0, 199920, 0, 199338) + ed4
000228b0 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 2 / thread# 2  --------------------
fee40e7c pollsys  (f83fd000, 3, 0, 0)
fede25b0 pselect_large_fdset (f83fd018, 3, f83fd208, f83fd200, 20, f83fd20c) + 2ec
fede29f8 select_large_fdset (13, f83ffd88, 0, 0, 0, 199500) + a0
00032300 ???????? (134c00, f83ffb88, 0, 173c00, 170400, 176d80)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 3 / thread# 3  --------------------
fee40150 lwp_park (0, 0, 0)
fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0000, 2280b8, 0) + 10
fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
000df5b4 ???????? (1acba8, f7c00000, 0, 0, df42c, 0)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 4 / thread# 4  --------------------
fee40150 lwp_park (0, 0, 0)
fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0400, 2280b8, 0) + 10
fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
000df5b4 ???????? (1acba8, f7400000, 0, 0, df42c, 0)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)

The Linux utility to grab comparable information is called gstack, and it works nearly identically to pstack:

$ gstack 2195

#0  0x00194402 in __kernel_vsyscall ()
#1  0x00469e7d in ___newselect_nocancel () from /lib/libc.so.6
#2  0x00cc1c5b in main () from /proc/2195/exe

These utilities have helped me time and time again, and are two extremely useful pieces of software.