Linux Performance Monitoring - iostat

Linux server performance monitoring through the ‘iostat’ command provides general information about disk operations and related statistics. ‘iostat’ stands for ‘Input/Output Statistics’ and it is a utility that monitors system input/output by observing the time the devices are active in relation to their average read and write rates. The output of the command ‘iostat’ helps to change system configuration to balance the input/output load between physical disks efficiently. IOPS or Input/Output Operations per second provides the system’s storage performance bottleneck. Such storages are Hard-disk drives (HDD), solid-state drives (SSD) or storage area networks (SAN).

Command Syntex:

$ iostat

Output

Refer to Figure 01, the iostat command output has three sections which are:

System Configuration: This is the first section displayed at the top. It provides the basic information about the system like version, server name, no. of CPUs etc.
CPU Utilization Statistics: It provides the CPU utilization summary in terms:
- %usr: It shows the percentage of CPU utilization that occurred while executing at the user level (application).
- %nice: It shows the percentage of CPU utilization that occurred while executing at the user level with nice priority.
- %sys: It shows the percentage of CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.
- %iowait: It shows the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
- %steal: It shows the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
- %idle: It shows the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.
Device Utilization Statistics: The third section displays the device utilization statistics. The device report provides statistics on a per physical device or partitions basis. The field description ar
- Device: The device/partition name is listed in the /dev directory.
- tps: The number of transfers per second that were issued to the device. Higher tps means the processor is busier. It is the sum of r/s and w/s get from -x option. TPS = r/s + w/s.
- kB_read/s: It shows the amount of data read from the device expressed in a number of blocks per second. MB_read/s is also one of the units.
- kB_wrtn/s: The amount of data written to the device described in a number of blocks per second. MB_wrtn/s is also one of the units.
- kB_read: It shows the total number of blocks read.
- kB_wrtn: It shows the total number of blocks written.

Options

-c: This option provides only CPU utilization statistics along with system configuration.

-d: This option provides only device utilization statistics along with system configuration.

-h: This option provides CPU and device utilization statistics along with system configuration in a human-readable format.

-k: This option provides the statistics in kB format (default).

-m: This option provides the statistics in MB format.

-N: This option displays the LVM2 statistics which is helpful in LVM2 format.

-t: This option adds the timestamp along with all the queried statistics.

-V: This option gives the version of sysstat

-x: Here, x implies ‘Extend’. This option provides more detailed information than the usual iostat command.

-o JSON: This option displays the statistics in JSON format.

[Delay Value]: The delay between the command output updates in seconds. If no delay is specified, only one output is printed with the average values of all the CPUs. In the below example, 5 seconds is a delay between command outputs.

[Count Value]: It represents the number of required outputs. In the absence of a count value, when the delay is defined, the default is infinite outputs until the command gets killed. In the below example, the count value is 2 and hence only 2 outputs are displayed at the interval of 10 seconds along with the Average.

These are the important options and keywords related to the iostat command.

How to capture iostat during a load test?

Before starting the load test you have to run the below command:

$ nohup iostat -x [Delay Value] [Count Value] > [File Name] &

Example: If you want to run a load test for 1 hour and need to capture the server statistics in the iostatLoadTest.dat file in every 10 seconds then the calculation of the Count Value will be:
=> 1 hour = 3600 seconds
=> 3600/10 = 360 // Count Value

And, the command will be:

$ nohup iostat -x 10 360 > iostatLoadTest.dat &

Refer iostatLoadTest.dat file, after the completion of the test and analyse the result.

How to analyse the iostat output?

For the analysis purpose, select the -x option with the iostat command. Refer to Figure 09.
Device TPS is the sum of r/s and w/s.
The way to calculate the efficiency of IOPS is to divide the kilobytes read (rkB/s) and written (wkB/s) per second by the reads per second (r/s) and writes per second (w/s) respectively. Refer to Figure 09 (use -x option) to get the statistics.
The avgqu-sz shows the number of operations that were either queued or being serviced. You have to look into this statistic when trouble occurs. Single-digit numbers with the occasional double-digit spike are safe(ish) values. Triple-digit numbers lead to a performance bottleneck to serve the request.
%util closer to 100% for a device is, the more saturated it is.
If the system does not have enough RAM to accommodate all requests, it must start to use the SWAP device. Just like file system I/O, writes to the SWAP device are just as costly. If the system is extremely deprived of RAM, it is possible that it will create a paging storm to the SWAP disk. If the SWAP device is on the same file system as the data trying to be accessed, the system will enter into contention for the I/O paths. This will cause a complete performance breakdown in the system. If pages can’t be read or written to disk, they will stay in RAM longer. If they stay in RAM longer, the kernel will need to free the RAM. The problem is that the I/O channels are so clogged that nothing can be done. This inevitably can lead to kernel panic and crash of the system.

Important Points

If the iostat command is run without specifying a time interval, the output indicates a summary of the system data since the last system reboot and not the current values.
iostat helps to understand the Device utilization and pattern of the Linux system during the performance testing.
Calculate the Count Value by providing some extra time for the load test, so that you can have Linux server monitoring stats for pre, post and during the test.
Any time the CPU is waiting on I/O, the disks are overloaded.
Calculate the IOPS as per the system and disk.
- IOPS = Bytes per second / Amount of data transferred in bytes
Determine whether your applications require random or sequential disk access.
Monitor slow disks by comparing wait times and service times.
Monitor the swap and file system partitions to make sure that virtual memory is not contending for filesystem I/O.