0
点赞
收藏
分享

微信扫一扫

Tomcat9.0.13 Bug引发的java.io.IOException:(打开的文件过多 Too many open files)导致服务假死

史值拥 2022-10-06 阅读 198

问题背景:

笔者所在的项目组最近把生产环境Tomcat迁移到Linux,算是顺利运行了一段时间,最近一个低概率密度的(too many open files)问题导致服务假死并停止响应客户端客户端请求。

进入服务器查看日志,发现tomcat凌晨6-7点的日志丢失,查看进程端口仍旧开放。

root@# lsof -i:8080
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 29511 root 67u IPv4 147648 0t0 TCP *:8080

 从存档的日志找到一些端倪,发现凌晨1点的日志就开始出现异常

23-Dec-2018 01:12:12.514 严重 [http-nio-83-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)

错误日志一直追到凌晨6-7点,这个错误仍旧比较高密度的出现

23-Dec-2018 07:35:06.932 严重 [http-nio-83-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)

23-Dec-2018 07:35:07.692 严重 [http-nio-81-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)

23-Dec-2018 07:35:08.532 严重 [http-nio-83-Acceptor-0] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: 打开的文件过多
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.lang.Thread.run(Thread.java:748)

故障排除步骤:

笔者最初怀疑自己在预安装生产环境的过程中忘记调优内核参数,于是按步骤查询了最大文件打开数:

root@# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031211
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1031211
virtual memory (kbytes, -v) unlimited
file

open files那一行就代表系统目前允许单个进程打开的最大句柄数,这里是4096。

按照这个参数想了一下,即使class文件没有打包,项目中也不会消耗这么多文件句柄的。

有打开内存参数确认了一下

vim /etc/security/limits.conf

#<domain>      <type>  <item>         <value>
#

#* soft core 0
#root hard core 100000
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#ftp - chroot /ftp
#@student - maxlogins 4

# End of file
*soft nofile 63335
*hard nofile 65535

这一切看上去都没有文件,文件句柄看上去足够tomcat使用。

查看tomcat进程

jps

或者

ps -ef | grep -i 'bootstrap.jar' | grep -v grep | awk '{print $2}'

查得tomcat进程

root@# ps -ef | grep -i 'bootstrap.jar' | grep -v grep | awk '{print $2}'
29511

 

笔者调大文件句柄最多打开数

然后用命令查看进程打开句柄数

lsof -p 29511| wc

root@# lsof -p 29511| wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs
Output information may be incomplete.
265

root@# lsof -p 29511| wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs
Output information may be incomplete.
268

通过以下命令查看进程打开的文件句柄详情:

root@# lsof -p 29511
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 29511 root cwd DIR 8,16 53248 160366607 /opt/tomcat-9.0.13/logs
java 29511 root rtd DIR 8,1 4096 2 /
java 29511 root txt REG 8,1 7734 15729394 /opt/jdk1.8.0_181/bin/java
java 29511 root mem REG 8,1 41080 23855198 /var/cache/fontconfig/945677eb7aeaf62f1d50efc3fb3ec7d8-le64.cache-6
java 29511 root mem REG 8,1 20584 23855180 /var/cache/fontconfig/2cd17615ca594fa2959ae173292e504c-le64.cache-6
java 29511 root mem REG 8,1 84432 23855174 /var/cache/fontconfig/04aabc0a78ac019cf9454389977116d2-le64.cache-6
java 29511 root mem REG 8,1 12392 23855184 /var/cache/fontconfig/385c0604a188198f04d133e54aba7fe7-le64.cache-6
java 29511 root mem REG 8,1 265387 15731055 /opt/jdk1.8.0_181/jre/lib/amd64/libjpeg.so
java 29511 root mem REG 8,1 493889 15731072 /opt/jdk1.8.0_181/jre/lib/amd64/libt2k.so
java 29511 root mem REG 8,1 525421 15731090 /opt/jdk1.8.0_181/jre/lib/amd64/libfontmanager.so
java 29511 root mem REG 8,1 3416 23855177 /var/cache/fontconfig/0d8c3b2ac0904cb8a57a757ad11a4a08-le64.cache-6
java 29511 root mem REG 8,1 3144 23855179 /var/cache/fontconfig/1ac9eb803944fde146138c791f5cc56a-le64.cache-6
java 29511 root mem REG 8,1 1632 23855211 /var/cache/fontconfig/dc05db6664285cc2f12bf69c139ae4c3-le64.cache-6
java 29511 root mem REG 8,1 8640 23855194 /var/cache/fontconfig/767a8244fc0220cfb567a839d0392e0b-le64.cache-6
java 29511 root mem REG 8,1 2320 23860602 /var/cache/fontconfig/4794a0821666d79190d59a36cb4f44b5-le64.cache-6
java 29511 root mem REG 8,1 17256 23855197 /var/cache/fontconfig/8801497958630a81b71ace7c5f9b32a8-le64.cache-6
java 29511 root mem REG 8,1 1800 23855204 /var/cache/fontconfig/bab58bb527bb656aaa9f116d68a48d89-le64.cache-6
java 29511 root mem REG 8,1 25168 23855181 /var/cache/fontconfig/3047814df9a2f067bd2d96a2b9c36e5a-le64.cache-6
java 29511 root mem REG 8,1 1816 23855188 /var/cache/fontconfig/56cf4f4769d0f4abc89a4895d7bd3ae1-le64.cache-6
java 29511 root mem REG 8,1 3184 23855203 /var/cache/fontconfig/b9d506c9ac06c20b433354fa67a72993-le64.cache-6
java 29511 root mem REG 8,1 21520 23855202 /var/cache/fontconfig/b47c4e1ecd0709278f4910c18777a504-le64.cache-6
java 29511 root mem REG 8,1 76688 23855208 /var/cache/fontconfig/d52a8644073d54c13679302ca1180695-le64.cache-6
java 29511 root mem REG 8,1 1816 23855187 /var/cache/fontconfig/551ecf3b0e8b0bca0f25c0944f561853-le64.cache-6
java 29511 root mem REG 8,1 15216 23855209 /var/cache/fontconfig/d589a48862398ed80a3d6066f4f56f4c-le64.cache-6
.....
java 29511 root 133r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 134r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 135r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 136r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 137r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 138r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 139r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 140r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 141r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 142r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 143r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 144r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 145r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 146r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 147r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 148w REG 8,16 1416 160368886 /opt/tomcat-9.0.13/logs/logs_tomcat.log

 

发现一直还在长,于是查看进程打开文件,多次对比查看打开的文件句柄,发现tomcat-users.xml只增不减

java    29511 root  171r      REG               8,16      2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 172r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 173r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 174r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 175r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 176r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 177r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 178r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 179r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 180r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 181r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 182r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 183r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 184r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 185r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 186r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 187r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 188r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 189r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 190r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 191r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 192r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 193r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 194r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 195r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 196r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 197r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 198r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 199r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 200r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 201r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 202r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 203r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml
java 29511 root 204r REG 8,16 2164 160366602 /opt/tomcat-9.0.13/conf/tomcat-users.xml

至此,问题单症结基本确定了,就是tomcat这里

通过Google查询:too many tomcat-users.xml open by tomcat

最终确定这是tomcat9.0.13的一个bug:​​Tomcat too many files open (tomcat-users.xml)​​

This was caused by a ​​bug in Tomcat 9.0.13​​ which has been fixed in Tomcat 9.0.14.

升级tomcat到9.0.14解决问题

 

笔者按:以笔者目前粗浅的win32知识触类旁通来分析这个问题,进程打开的文件句柄数达到设定的最大值,在向操作系统申请资源的时候一直处于等待状态,操作系统不再分拨资源

给他,于是进程还在,没死掉,只是资源被他耗尽,待客户端来访问的时候,他还会去申请文件句柄,一直就处于假死状态了。

参考:

​​Tomcat报java.io.IOException: 打开的文件过多​​

too many open files(打开的文件过多)解决方法

​​Error in tomcat “too many open files”​​

​​How to really fix the too many open files problem for Tomcat in Ubuntu​​

举报

相关推荐

0 条评论