Storage is one of the slowest components in a server's system, but is still the most important component. Storage is mainly used as a persistence mechanism to store a large amount of processed/unprocessed data. A slow storage device generally results in heavy utilization of read write buffers and higher memory consumption. You will see higher CPU usage, but most of the CPU time is spent waiting for I/O requests.
The recent developments of the flash storage medium have vastly improved storage performance. Still, it's one of the slowest performing components and needs proper planning— I/O planning in the application code, plus enough main memory for read write buffers.
In this recipe, we will learn to monitor storage performance. The main focus will be on local storage devices rather than network storage.
As always, you will need sudo
access for some commands.
Some of the commands many not be available by default. Using them will prompt you if the command is not available, along with the process necessary to install the required package.
Install the sysstat
package as follows. We have already used it in previous recipes:
$ sudo apt get install sysstat
vmstat
. Using vmstat
without any option displays an io
column with two sub entries: bytes in (bi
) and bytes out (bo
). Bytes in represents the number of bytes read in per second from the disk and bytes out represents the bytes written to the disk:-d
and -D
, to get disk statistics. Flag -d
displays disk statistics and flag -D
displays a summary view of disk activity:-p
, that displays partition-specific disk statistics. Use the command lsblk
to get a list of available partitions and then use the vmstat -p
partition:dstat
, is a nice replacement for vmstat
, especially for disk statistics reporting. Use it with flag -d
to get disk read writes per seconds. If you have multiple disks, you can use dstat
to list their stats separately:$ dstat -d -D total,sda
iostat
. When used without any options, this command displays basic CPU utilization, along with read write statistics for each storage device:tps
specifies the I/O requests sent to a device per second, and kb_read/s
and kb_wrtn/s
specifies per second blocks read and blocks written respectively. kb_read
and kb_wrtn
shows the total number of blocks read and written.iostat
include –d
, that displays disk only statistics, -g
that displays statistics for a group of devices, flag -p
to display partition specific stats, and -x
to get extended statistics. Do not forget to check the manual entries for iostat
to get more details.iotop
, which is very similar to the top
command but it displays disk utilization and relevant processes.lsof
can display the list of all open files and respective processes using that file. Use lsof
with the process name to get files opened by that process:$ lsof -c sshd
$ lsof -p 1134.
Or, to get a list of files opened by a specific user, use the $ lsof -u ubuntu
command.All these commands provide details on the read write performance of a storage device. Another important detail to know is the availability of free space. To get details of space utilization, you can use command df -h
. This will list a partition-level summary of disk space utilization:
sar
command to track disk performance over a period of time. To get real-time disk activity, use sar
with the -d
option, as follows:$ sar -d 1
-F
to get details on file system utilization and flag -S
to display swap utilization. You can also enable sar logging and then extract details from those logs. Check the previous recipes in this chapter for how to enable sar logging. Also check manual entries for sar to get details of various options.