Introduction to Linux - A Hands on Guide | Linux Bible | Linux From Scratch | A Newbie's Getting Started Guide to Linux | Linux Command Line Cheat Sheet | More Linux eBooks



Sunday, 24 August 2014

SAR Command for Linux System Performance Monitoring

    System Activity Report, also known as sar, is one of the most important utility for Linux system administrators when it comes to performance monitoring of a Linux system. sar provides an overview of the Linux system with various crucial metrics which include Processor, Memory, I/O Devices and Network related information. With sar, one can gather and store the information whenever there is an issue with the Linux server, and then use this data in order to deal with similar issue in future by comparing these system statistics with the ones at that point of time. In brief, historical analysis can be made a lot easier, when sar is used.



With sar, one can get information regarding following metrics:
  • Overall CPU Utilization
  • Individual CPU Utilization
  • Memory Utilization
  • Swap Utilization
  • Block Device Statistics
  • I/O Related Details
  • System Buffer and Context Switch Related Information
  • Network Related Statistics
  • Memory Allocation
In order to use this command, you will have to install 'sysstat' package.

# For Debian Based Systems
sudo apt-get install sysstat

# For Red Hat Based Systems
yum install sysstat
# OR
rpm -ivh sysstat-11.0.0-1.x86_64.rpm
To check the version of sar installed, run sar -V.

LinuxBox root [root] > sar -V
sysstat version 11.0.0
(C) Sebastien Godard (sysstat <at> orange.fr)

1. Overall CPU Statistics

To view the overall (collective) CPU statistics, you can use option -u as follows:

Syntax:

sar -u [INTERVAL] [NO. OF CHECKS]
Example:

 sar -u 1 5
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

09:52:57 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:52:58 AM     all      0.06      0.00      6.39      0.00      0.00     93.55
09:52:59 AM     all      0.00      0.00      6.70      0.00      0.00     93.30
09:53:00 AM     all      0.19      0.00      6.32      0.00      0.00     93.49
09:53:01 AM     all      0.06      0.00      6.70      0.00      0.00     93.24
09:53:02 AM     all      0.13      0.00      6.39      0.25      0.00     93.24
Average:        all      0.09      0.00      6.50      0.05      0.00     93.36
So, this "1 5" will display 5 results checked in the interval of 1 second each.

  • %user: This indicates the percentage of total time spent by the CPU in serving different user level processes.
  • %system: This indicates the percentage of total time spent by the CPU in serving different kernel level processes.
  • %iowait: This indicates the percentage of total time spent by the CPU in waiting for different I/O requests.
  • %nice: This indicates the percentage of total time spent by the CPU in serving user level processes with nice priority.
  • %steal: This indicates the percentage of total time spent by the CPU in serving virtual CPUs.
  • %idle: This indicates the percentage of total time spent by the CPU being in idle state.

2. Statistics for Individual Core

Above example shows, cumulative CPU statistics i.e performance delivered by all the cores collectively. If you wish to see the performance offered by each and every individual CPU core, use option -P as follows:

Syntax:

sar -P [ALL/CORE NUMBER] [INTERVAL] [NO. OF CHECKS]
In order to view performance of ALL the cores, mention "ALL" in the first argument as follows:

Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:10:06 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:10:07 AM     all      4.00      0.00      9.43      0.06      0.00     86.51
10:10:07 AM       0      0.00      0.00    100.00      0.00      0.00      0.00
10:10:07 AM       1      0.98      0.00      1.96      0.00      0.00     97.06
10:10:07 AM       2      0.00      0.00      0.00      0.00      0.00    100.00
10:10:07 AM       3      0.00      0.00      0.99      0.00      0.00     99.01
10:10:07 AM       4     10.10      0.00      1.01      0.00      0.00     88.89
10:10:07 AM       5      0.00      0.00      0.00      0.00      0.00    100.00
10:10:07 AM       6      3.06      0.00      3.06      0.00      0.00     93.88
10:10:07 AM       7     20.79      0.00     13.86      0.00      0.00     65.35
10:10:07 AM       8     20.79      0.00      8.91      0.00      0.00     70.30
10:10:07 AM       9      1.00      0.00     14.00      1.00      0.00     84.00
10:10:07 AM      10      0.00      0.00      0.00      0.00      0.00    100.00
10:10:07 AM      11      2.00      0.00      3.00      0.00      0.00     95.00
10:10:07 AM      12      3.88      0.00      1.94      0.00      0.00     94.17
10:10:07 AM      13      1.01      0.00      1.01      0.00      0.00     97.98
10:10:07 AM      14      0.00      0.00      0.00      0.00      0.00    100.00
10:10:07 AM      15      1.00      0.00      3.00      0.00      0.00     96.00

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      4.00      0.00      9.43      0.06      0.00     86.51
Average:          0      0.00      0.00    100.00      0.00      0.00      0.00
Average:          1      0.98      0.00      1.96      0.00      0.00     97.06
Average:          2      0.00      0.00      0.00      0.00      0.00    100.00
Average:          3      0.00      0.00      0.99      0.00      0.00     99.01
Average:          4     10.10      0.00      1.01      0.00      0.00     88.89
Average:          5      0.00      0.00      0.00      0.00      0.00    100.00
Average:          6      3.06      0.00      3.06      0.00      0.00     93.88
Average:          7     20.79      0.00     13.86      0.00      0.00     65.35
Average:          8     20.79      0.00      8.91      0.00      0.00     70.30
Average:          9      1.00      0.00     14.00      1.00      0.00     84.00
Average:         10      0.00      0.00      0.00      0.00      0.00    100.00
Average:         11      2.00      0.00      3.00      0.00      0.00     95.00
Average:         12      3.88      0.00      1.94      0.00      0.00     94.17
Average:         13      1.01      0.00      1.01      0.00      0.00     97.98
Average:         14      0.00      0.00      0.00      0.00      0.00    100.00
Average:         15      1.00      0.00      3.00      0.00      0.00     96.00
Pretty lengthy output, isn't it? This is because I am using 16 Core Processor.

One thing should be observed that, the numbering of the cores starts from '0'. So, 1st core will be CPU0, 2nd core will be CPU1, and so on.

So, I don't want to see all this stupid stuff. I am just interested in knowing what my 13th core i.e CPU12 is doing. In this case, I won't use "ALL", instead I would use "12"- the core number.

sar -P 12 1 1
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:16:56 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:16:57 AM      12      2.02      0.00      1.01      0.00      0.00     96.97
Average:         12      2.02      0.00      1.01      0.00      0.00     96.97

3. Memory Details

One might be interesting in knowing "how much memory is my server using?". Dude, Use option-r as follows:

Syntax:

sar -r [INTERVAL] [NO. OF CHECKS]
Example:

sar -r 1 2
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:21:18 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
10:21:19 AM    234240  32574492     99.29    498808  27477196   9626636      9.63
10:21:20 AM    233108  32575624     99.29    498808  27477196   9626672      9.63
Average:       233674  32575058     99.29    498808  27477196   9626654      9.63
  • kbmemfree: Free memory in kilobytes
  • kbmemused: Used memory in kilobytes.
  • %memused: Percentage of used memory.
  • kbbuffers: Memory used as buffers by the kernel in kilobytes.
  • kbcached: Memory used to cache data by the kernel in kilobytes.
  • kbcommit: Memory in kilobytes needed for current workload.
  • %commit: Percentage of memory needed for current workload in relation to the total amount of memory (RAM+swap).

4. Swap Usage Statistics

To view the swap usage details, use option -S as follows:

Syntax:

sar -S [INTERVAL] [NO. OF CHECKS]
Example:

sar -S 1 2
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:31:41 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
10:31:42 AM  66859392    249464      0.37     35760     14.33
10:31:43 AM  66859392    249464      0.37     35760     14.33
Average:     66859392    249464      0.37     35760     14.33
  • kbswpfree: Free swap space in kilobytes.
  • kbswpused: Used swap space in kilobytes.
  • %swpused: Percentage of used swap space.
  • kbswpcad: Cached swap memory in kilobytes.
  • %swpcad: Percentage of cached swap memory in relation to the amount of used swap space.

5. Monitoring I/O Activity

To view information related to I/O devices, use option -b as follows:

Syntax:

sar -b [INTERVAL] [NO. OF CHECKS]
Example:

sar -b 1 2
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:38:20 AM       tps      rtps      wtps   bread/s   bwrtn/s
10:38:21 AM     34.00      0.00     34.00      0.00    576.00
10:38:22 AM      6.00      0.00      6.00      0.00     96.00
Average:        20.00      0.00     20.00      0.00    336.00
  • tps: Transfers per second issued to physical devices.
  • rtps: Read requests per second issued to physical devices.
  • wtps: Write requests per second issued to physical devices.
  • bread/s: Data read from the devices in blocks per second.
  • bwrtn/s: Data written to devices in blocks per second.

6. Context Switch Reports

Syntax:

sar -w [INTERVAL] [NO. OF CHECKS]
Example:

sar -w 1 2
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:42:57 AM    proc/s   cswch/s
10:42:58 AM      0.00   3603.00
10:42:59 AM      0.00   4052.53
Average:         0.00   3826.63
  • proc/s: Processes created per second.
  • cswch/s: Context switches per second.

7. Display Run Queue and Load Average


Syntax:

sar -q [INTERVAL] [NO. OF CHECKS]
Example:

sar -q 1 2
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:48:33 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
10:48:34 AM         0      1240      0.03      0.10      0.35
10:48:35 AM         0      1240      0.03      0.10      0.35
Average:            0      1240      0.03      0.10      0.35
  • runq-sz: Run queue length (number of tasks waiting for run time).
  • plist-sz: Number of tasks in the task list.
  • ldavg-1: System load average for the last minute.
  • ldavg-5: System load average for the past 5 minutes.
  • ldavg-15: System load average for the past 15 minutes.

8. Network Statistics


Syntax:

sar -n [VALUE] [INTERVAL] [NO. OF CHECKS]
The VALUE can be:
  • DEV: For network devices like eth0, bond, etc.
  • EDEV: For network device failure details
  • NFS: For NFS client info
  • NFSD: For NFS server info
  • SOCK: For sockets in use for IPv4
  • IP: For IPv4 network traffic
  • EIP: For IPv4 network errors
  • ICMP: For ICMPv4 network traffic
  • EICMP: For ICMPv4 network errors
  • TCP: For TCPv4 network traffic
  • ETCP: For TCPv4 network errors
  • UDP: For UDPv4 network traffic
  • SOCK6, IP6, EIP6, ICMP6, UDP6 : For IPv6
  • ALL: For all above mentioned information.
Example:

sar -n DEV 1 1
Linux 2.6.32-042stab090.4 (LinuxBox)    08/24/2014      _x86_64_        (16 CPU)

10:54:49 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
10:54:50 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:54:50 AM      eth0     87.00      4.00     17.30      0.48      0.00      0.00      4.00
10:54:50 AM      eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:54:50 AM      eth2     16.00     75.00      1.69     36.58      0.00      0.00      1.00
10:54:50 AM      eth3      2.00      0.00      0.18      0.00      0.00      0.00      2.00
10:54:50 AM     bond0    103.00     79.00     18.98     37.06      0.00      0.00      5.00
10:54:50 AM     bond1      2.00      0.00      0.18      0.00      0.00      0.00      2.00
10:54:50 AM       br0     18.00    125.00      0.93     25.24      0.00      0.00      8.00
10:54:50 AM       br1      2.00      0.00      0.15      0.00      0.00      0.00      2.00
10:54:50 AM    venet0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0     87.00      4.00     17.30      0.48      0.00      0.00      4.00
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth2     16.00     75.00      1.69     36.58      0.00      0.00      1.00
Average:         eth3      2.00      0.00      0.18      0.00      0.00      0.00      2.00
Average:        bond0    103.00     79.00     18.98     37.06      0.00      0.00      5.00
Average:        bond1      2.00      0.00      0.18      0.00      0.00      0.00      2.00
Average:          br0     18.00    125.00      0.93     25.24      0.00      0.00      8.00
Average:          br1      2.00      0.00      0.15      0.00      0.00      0.00      2.00
Average:       venet0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

That's all for this article. For more details about sar command, refer to the man page.

0 comments:

Post a Comment