Optimizing Nginx for (large) file delivery

Here are some useful tips if you want to optimize your Nginx for large file delivery. Use at your own risk!

1. Turn off sendfile. The Linux sendfile call is known to have throughput degradation when in high load. Disabling it helps to keep a higher throughput at high load. Also, when serving large files with sendfile, there are no ways to control read ahead.
2. Enable TCP nopush. TCP nopush fills the TCP packet to its maximum size before sending. This can help increase throughput if you’re serving large files.
3. Use Nginx’s directio to load file. Using directio can help improving performance by skipping a bunch of steps happened in the kernel when reading files, thus speed up the throughput.
4. Enable the use of libaio for optimal performance libaio allows asynchronous I/O to be done in kernel, which results in faster read and write speed. However, it needs libaio to be installed and re-compiling your Nginx in order to have it supported. I used the following flow to recompiling Nginx with aio support.

Please be aware that this last part is rather old. I do not advise to perform the next step, unless you do it with a more recent version of Nginx. I cannot guarantee if libaio will work at all nowadays. But fee free to try this for yourself. Leave a comment when you know more.

# Install libaio on RHEL/CentOS
yum install libaio libaio-devel -y

wget http://nginx.org/download/nginx-1.9.4.zip
unzip -q nginx-1.9.4.zip
cd nginx-1.9.4

# Configure Nginx according to your needs, but it should also include
# --with-file-aio in order to use libaio
./comnfigure --with-file-aio
make

The complete nginx.conf should look like this:

http {
    ...
    sendfile off;
    tcp_nopush on;
    aio on;
    directio 512; # required if you're using Linux and uses aio
    ...
}

There are also some lower-level tweaks like mounting your disks with noatime flags and use ext4/xfs when serving files.

Source for his article: https://licson.net/post/optimizing-nginx-for-large-file-delivery/.