Zonelimits State: The Silent Killer
Tuesday, December 30, 2008 12:29
You’ve got your FreeBSD web server up, server is running smoothly, and you’re getting a lot of hits. Great! Until your webserver stops responding, you log in, see no load, but all your httpd processes are running in zoneli state. You can’t kill them, the only way you can fix it is to reboot the server. Not so great. What happened?
The “zonelimits” state in FreeBSD appears when you have run out of “mbuf clusters”, groups of mbufs that carry information up and down the network stack. As the mbuf man page states, “An mbuf is a basic unit of memory management in the kernel IPC subsystem. Network packets and socket buffers are stored in mbufs.” So how do we see usage of these mbuf clusters?
$ netstat -m
You should see an output like this:
390/255/645 mbufs in use (current/cache/total)
389/129/518/25600 mbuf clusters in use (current/cache/total/max)
By default FreeBSD comes with 25600 mbuf clusters for network usage. This is fine for servers with small amonts of network connectivity, but too low for a system with any real network usage (e.g, a webserver).
So how do we get rid of this zoneli state? We increase the mbuf clusters of course!
# sysctl kern.ipc.nmbclusters=65535
kern.ipc.nmbclusters: 25600 -> 65535
This will allow more kernel memory to be allocated to the mbuf clusters, allowing more network data in memory, allowing more tcp connections. Too easy.
But wait, my server is still going into zoneli state!
Ok so we have increased mbuf clusters, but our server is still hitting the limit. Now we look at the buffers we have for tcp sending and receiving, and reduce as needed:
net.inet.tcp.sendspace: 32768
net.inet.tcp.recvspace: 32768
The higher the buffer, the more mbufs will be used for network connections. Lowering these buffers down will reduce your memory usage and save yourself some mbufs.
sysctl net.inet.tcp.sendspace=8192
sysctl net.inet.tcp.recvspace=8192
This should dramatically reduce your mbuf usage and get your server out of zonelimits once and for all.
No related posts.
(2 votes, average: 5.00 out of 5)
Loading ...
izo says:
December 30th, 2008 at 12:44 pm
The last command is missing an ’s’, should be –
sysctl net.inet.tcp.recvspace=8192
izo says:
December 30th, 2008 at 12:46 pm
Otherwise, great job. You Rock !!!
Chris says:
December 30th, 2008 at 12:52 pm
Thanks, I searched for a long time online to find the answer to this so thought it needed to be shared.
Mxx says:
January 12th, 2009 at 9:43 pm
but doesn’t decreasing recvspace and sendspace can limit maximum potential throughput on high bandwidth/high latency connections
(ala calibeach..)
Chris says:
January 13th, 2009 at 2:28 am
Thats true, moreso for LAN connections, I’ve found that an improved route on the WAN interface by far surpasses any work you can do with tcp buffers. Also this is optimizing for loads of connections, not just for a few connections having speed. It depends whether or not you want to secrifice a bit in speed for a big increase in network connections.
Jason Thomas says:
March 5th, 2009 at 5:05 pm
As noticed with one of our servers, reducing the send/receive window to such a low value will decrease the maximum transfer throughput of the server. This might be okay if you are serving small files, however when large files are in question it becomes an issue.
throughput <= Receive Window / Round Trip Time of Path
If your Round Trip Time of Path is equal to 90 ms, usually the case on a transatlantic link, the throughput value is calculated below:
throughput <= 8192 Bytes / .09 second
throughput <= 91 KBytes/second
The receive window value could remain low if you are not uploading large amount of content, however send window should be higher so that clients with higher latency get better speeds.