java进程卡死问题

原文地址：
​​​http://stackoverflow.com/questions/28739600/jvm-hang-and-kill-3-jmap-failed​​

tomcat进程出现了如下异常，并且卡在这个状态几天时间了：

1. 所有线程都停止, 有一些连接停留在​​SYNC_RECV​​ 状态
2. ​​kill -3​​ 没有输出线程栈
3. ​​jmap​​​ 和 ​​jstack​​ 无法正常工作

sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap 19938
Attaching to process ID 19938, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jmap -heap -F 19938

Attaching to process ID 19938, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

1. ​​jstat -gc​​ 的输出一直没有变化, 甚至连 timestamp 字段都不变

sudo -u tomcat /usr/java/jdk1.6.0_45/bin/jstat -gc -t 19938 1000 5

Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301
       370651.7 33408.0 33536.0  0.0   32416.0 3078592.0 2424720.7 6291456.0   74894.4   262144.0 71831.7     77    8.268   1      0.033    8.301

1. CPU一直很空闲

环境信息：

1. linux

Linux xxxx 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

1. jdk

java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

1. JAVA_OPTS

-server -Xms9g -Xmx9g -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseParallelGC -XX:ParallelGCThreads=24 -XX:+UseParallelOldGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/tomcatdump  
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/tmp/tomcatlog.log -XX:NewSize=3g -XX:MaxNewSize=3g -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true

1. tomcat:​​7.0.54​​

更新

​​SYN_RECV​​​ 状态应该是LVS导致的，初步猜测是LVS检测到tomcat异常后将流量切走，造成三次握手最后的ACK包没有到达这台服务器，连接卡在​​SYN_RECV​​状态,但是这个应该对java进程没有影响才