When we write application code, we tend to forget that there is a huge ecosystem of the operating system supporting the code. How we use these resources is key to how scalable the code is. Some common sources of these type of bottlenecks are as follows:

• Disk-related: Most disks are optimized for block access and sequential I/O and not random I/O. If your code is doing a lot of disk seeks, it will seriously slow down your code and cause non-intuitive bottlenecks. Also, multiple disks have different performance (IOPS) characteristics. As far as possible, we should use local SSD disks for high I/O use cases and reserve using networked-drives when data needs to be shipped remotely. For example, if you are writing a compiler that is supposed to write the output to a remove drive, you don't really need to store the intermediate files in the same location! The code can use different locations for the final and intermediate files (including logs), and thus achieve much better performance and scalability. Also important is how your code is using the OS cache for disk (buffer cache)— frequent unwarranted fsync is a sure-shot recipe for slow-downs
• Networking: Common source of bottlenecks are as follows:
  • Interrupt Request handler (IRQ) saturation: Soft interrupts taking up 100% CPU.
  • Non-optimal TCP buffers: TCP uses a slow start algorithm to avoid congesting clogged/low-bandwidth links. At any point in time, it uses a congestion window buffer size to determine how many packets it can send at one time. The larger the congestion window size, the higher the throughput. The slow start algorithm works by starting with a modest buffer size and with each instance of not finding congestion, the buffer size is increased. On most OSes, there is a tunable for the maximum congestion window size and it defines the amount of buffer space that the kernel allocates for each socket. Though there is a default, individual programs can override this value by making a system call before opening the socket. If the buffer size is too low, the sender will be throttled. If, on the other hand, the buffer size is too large, then the sender can overrun the receiver and cause congestion control to kick in. To achieve maximum throughput, it is critical to set the buffer sizes to the right values for the network link being used. A rule of thumb for setting the buffer size is double the value for delay times bandwidth (buffer size = 2 * bandwidth * delay). The ping utility is an easy way to get the round-trip-time (RTT) or twice the delay, so you can just multiply this with the bandwidth of the link to get the buffer size.
  • Helper services such as DNS lookups: If you are using an internal DNS server, you need to ensure that they are setup for the scale needed.
• File descriptor limits: in most modern systems such as Linux, nearly everything is a file descriptor. The OS caps these with some default values, and you need to ensure these are set up to the right levels to avoid process failures. In Linux, you can use the unlimit command to get and set the limits, as shown here:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 32767
max locked memory       (kbytes, -l) 32max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 50
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ulimit -u 100