Anatomy of NodeJS and Apache
The purpose of the article is to explain default configurations (which can cause bottleneck) at Node.js, Apache and OS level and how to change those configurations. Client request processing time can be effected by any configuration listed below in diagram.
In below example, Node.js server is running behind reverse proxy server like Apache. And Process Manager PM2 is also used to load balance Node.js applications.
NodeJS
Concurrent Requests
By default how many concurrent requests can a Node.js handle?
Since javascript execution in Node.js is single threaded, at a time one request will be executing but by doing asynchronous operations and using event loop, at a given point of time Node.js will handling many requests.
For every connection to/from node server needs a socket to be created by kernel. In unix based systems, for any I/O operation (socket, accept, read, write etc.) done by a process needs a file descriptor. By default each process have a limit on how many file descriptors it can create. Below arelimits on a Node.js server process.
As per above limits, node process can create 16384 file descriptors. To know how many file descriptors got created by each Node.js process running with PM2.
If this (16384) number is reached for a Node.js process then — when a Node.js process requests to create new file descriptor to kernel, kernel will through EMFILE (EMFILE The per-process limit on the number of open file descriptors has been reached.) and Node.js will throw
Error: EMFILE, too many open filesTo fix this we can increase the limit, add below line at the end of the file /etc/security/limits.conf
* - nofile 20000if you want to add based on user
Note: please note that the process (PM2/node) should start with below user.
username soft nofile 20000
username hard nofile 20000There is system wide limit on number of file descriptors to be created
➜ ~ more /proc/sys/fs/file-max
2367356To know currently system wide how many descriptors got created
➜ ~ more /proc/sys/fs/file-nr | awk '{print $1}'
1888To test various configurations on Node.js and Apache, below blocking and non-blocking request methods are used.
And you can also refer below Apache proxy configuration (httpd.conf) used to route requests to Node.js
Regarding OS, below experiment was done on CentOS.
Node.js Backlog
What is default backlog of node process?
If there is blocking thread executing in Node.js then how many requests can be kept in backlog?
In unix based systems, max number of connections that can be kept in backlog at a socket is based on below setting.
➜ ~ more /proc/sys/net/core/somaxconn
128For example — 131 (blocking) requests are sent to Node.js using below command.
➜ ~ seq 1 131 | xargs -P 131 -I % sh -c 'time curl localhost:8443/block; echo %'Using netstat command we can see that 131 requests came to 8443 port
# netstat -antp | grep "::1:8443" | grep -v curl | wc -l
131Out of those 1 got established with node
# netstat -antp | grep "::1:8443" | grep "ESTABLISHED.*node" | wc -l
1129 requests connections waiting to 8443 port to get connection to node and 1 request is on SYN state.
# netstat -antp | grep "::1:8443" | grep "ESTABLISHED -" | wc -l
129# netstat -antp | grep "::1:8443" | grep 'SYN' | grep -v curl | wc -l1
SYN retries are based on below settings
➜ ~ more /proc/sys/net/ipv4/tcp_synack_retries
5
➜ ~ more /proc/sys/net/ipv4/tcp_syn_retries
6SYN connections timeout can happen in ~100 seconds After retries the connection reset will be send by server
131st request got TCP RST which lead to Connection reset by peer after 102 seconds
curl: (56) Recv failure: Connection reset by peer
real 1m42.688s
user 0m0.003s
sys 0m0.003s
131To fix above issue, we can increase backlog on a socket
sysctl -w net.core.somaxconn=1024To make above setting permanent add below line to /etc/sysctl.conf
net.core.somaxconn=1024Run sysctl -p or restart to reflect above setting.
After doing above setting, if 1000 blocking requests are sent to node server
➜ ~ seq 1 1000 | xargs -P 1000 -I % sh -c 'time curl localhost:8443/block; echo %'Only 512 requests are in backlog
# netstat -antp | grep "::1:8443" | grep "ESTABLISHED -" | wc -l
512After 105 seconds, 513th request got Connection reset by peer
➜ ~ seq 1 1000 | xargs -P 1000 -I % sh -c 'time curl localhost:8443/block; echo %'curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
real 1m45.306s
user 0m0.005s
sys 0m0.011s
513
The reason why only 512 are in backlog is because of default backlog setting on node server is 511.
node default backlog can be increased by passing backlog parameter in listen method like below.
const server = app.listen(8443,"localhost",1024, () => console.log('app listening on port 8443'));After above setting, when 1000 blocking requests are made, 999 requests are kept in backlog.
# netstat -antp | grep "127.0.0.1:8443" | grep "ESTABLISHED -" | wc -l999
Connection timeout
If 131 non-blocking requests are sent to node
➜ ~ seq 1 131 | xargs -P 131 -I % sh -c 'time curl localhost:8443/non-block; echo %'131 requests came to 8443 port
# netstat -antp | grep "::1:8443" | grep -v curl | wc -l
131Out of those, all 131 got established with node
# netstat -antp | grep "::1:8443" | grep "ESTABLISHED.*node" | wc -l131
0 requests are waiting in SYN backlog & 0 requests are waiting on 8443 port to get connections
# netstat -antp | grep "::1:8443" | grep 'SYN' | grep -v curl | wc -l
0# netstat -antp | grep "::1:8443" | grep "ESTABLISHED -" | wc -l
0
After 2 minutes, 1st request got disconnected
curl: (52) Empty reply from server
real 2m0.093s
user 0m0.006s
sys 0m0.017s
1Because Node.js by default has 2 minutes timeout after that a connection will be terminated.
Above timeout can be increased by using setTimeout method.
server.setTimeout(5000000);Note: Timeout will happen only when the request is kept in event loop. If a (blocking) request is currently executing in Node.js will have no effect on this setting.
Apache
Proxying to node
Proxying requests to node from apache using mod_proxy
ProxyRequests on
ProxyPass / http://localhost:8443/ retry=0 acquire=1000 timeout=600 Keepalive=On
ProxyPassReverse / http://localhost:8443notice timeout setting — when apache forwards the request to node, it waits for 600 seconds (10 mins) before terminating.
Having fine tuned Node.js lets look at Apache configurations.
502 Proxy error
By default net.core.somaxconn set to 128. when 131 blocking requests are sent through httpd to Node.js using below command.
➜ ~ seq 1 131 | xargs -P 131 -I % sh -c 'time curl localhost/block; echo %'131 requests established with httpd on port 80
# netstat -antp | grep "::1:80" | grep "ESTABLISHED.*httpd" | wc -l
131131 requests forwarded to Node.js 8443
# netstat -antp | grep "::1:8443" | grep "ESTABLISHED.*httpd" | wc -l131
1 is SYN backlog & 129 requests connections waiting to 8443 port to get connections, 1 request is with node process.
131st request got TCP RST from Node.js, apache sent 502 Proxy Error to client.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request <em><a href="/block">GET /block</a></em>.<p>
Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>
real 1m50.354s
user 0m0.006s
sys 0m0.001s
131Note: Apache will send 502 Proxy Error when SIGTERM (kill -3) or SIGKILL (kill -9) or restart is done on Node.js.
Server limit
When 512 non blocking requests are sent to apache, only 256 requests are sent to Node.js.
Note: SYN connections timeout can be ~100 seconds. After certain retries the connection reset will be send by Apache server
➜ ~ seq 1 512 | xargs -P 512 -I % sh -c 'time curl localhost/non-block; echo %'
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
real 1m49.746s
user 0m0.007s
sys 0m0.002s
385why only 256 requests are sent to Node.js? by default the ServerLimit is 256, so Apache can create only 256 processes. (one process for each request).
This can be increased by adding below configuration in Apache httpd.conf
<IfModule mpm_prefork_module>
StartServers 100
MinSpareServers 50
MaxSpareServers 80
ServerLimit 1300
MaxClients 1300
MaxRequestsPerChild 0
</IfModule>Apache Backlog
After increasing the socket backlog by increasing net.core.somaxconn to 1024 When 1000 requests are sent to apache, apache only sends 512 requests.
Apache has by default have backlog of 511 requests. This can be increased by adding below setting in httpd.conf configuration
ListenBacklog 1023I hope this helps, to find any bottlenecks at Node.js, Apache and OS.
