前言
呵呵 最近在 rcp项目的中导入 spark 环境之后, 呵呵 gg 了, 项目也没有报错, 差不多是 30s 之后超时了
断点一下 看见代码抛出异常的地方是在 spark driver 暴露 tcp 服务的地方
呵呵 当时还是很令人费解的, 我还以为是 rcp 项目在创建 tcp 服务的时候有什么限制呢 ?
呵呵 这里记录一下
业务代码里面的 SpringContext.init
spark 的死锁
java12 中引入 spark 启动不了, 卡在了 TransportServer.init(main线程)
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();
另外一个线程 rpc-boss-3 在执行 如下任务的时候卡住了
register0:502, AbstractChannel$AbstractUnsafe
和 EquinoxClassLoader 有关系, 加载 io.netty.bootstrap.ServerBootstrap$1$1 的时候卡住了?
EclipseLazyStarter.postFindLocalClass 卡住了
呵呵 死锁, main 线程拥有了 stateChangeLock, 然后等待 rpc-boss-3 唤醒自己, 之后 rpc-boss-3 在等待 stateChangeLock
main 线程里面 start:468, Module (org.eclipse.osgi.container) 获取了 stateChangeLock 的锁
原因是 Activator 里面 start 里面初始化的 SpringContext, 导致了如上的一系列
spark 初始化, 卡在了如下位置
新的死锁
呵呵 今天尝试复现之前的这个问题, 呵呵 结果没有复现成功, 因为 这里似乎是变成了另外的一个死锁了 ..
大致的情况都是一样的, 这里就不去浪费时间去复现之前的 spark 的那个死锁了
这里的死锁主要是 Thread-4 和 Thread-5[main] 之间造成了死锁
Thread-5 持有 EquinoxReentrantLock@300c6244 在等待 ContextFinder@6ed3ccb2
Thread-4 持有 ContextFinder@6ed3ccb2 在等待 EquinoxReentrantLock@300c6244
Thread-4 申请 EquinoxReentrantLock@300c6244 有一个 30s 的超时设置, 因此 打印出了 如下面所看到的堆栈信息
!ENTRY org.eclipse.osgi 2 0 2021-01-16 22:24:43.423
!MESSAGE While loading class "org.springframework.cache.CacheManager", thread "Thread[Thread-4,5,main]" timed out waiting (73888ms) for thread "Thread[main,5,main]" to finish starting bundle "MyArp-client_1.0.0.qualifier [1]". To avoid deadlock, thread "Thread[Thread-4,5,main]" is proceeding but "org.springframework.cache.CacheManager" may not be fully initialized.
!STACK 0
org.osgi.framework.BundleException: Unable to acquire the state change lock for the module: osgi.identity; type="osgi.bundle"; version:Version="1.0.0.qualifier"; osgi.identity="MyArp-client"; singleton:="true" [id=1] STARTED [STARTED]
at org.eclipse.osgi.container.Module.lockStateChange(Module.java:350)
at org.eclipse.osgi.container.Module.start(Module.java:419)
at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:468)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.util.concurrent.TimeoutException: Timeout after waiting 30 seconds to acquire the lock.
at org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)
... 22 more
Caused by: org.eclipse.osgi.framework.util.ThreadInfoReport: Thread dump
ThreadId: 1 ThreadName: main ThreadState: BLOCKED
Blocked On: org.eclipse.osgi.internal.framework.ContextFinder@6ed3ccb2 LockOwnerId: 26 LockOwnerName: Thread-4
Synchronizers Locked:
java.util.concurrent.locks.ReentrantLock$NonfairSync@356da172
Monitors Locked:
java.lang.Object@51340208
Stack Trace:
java.base@15/java.lang.Class.forName0(Native Method)
java.base@15/java.lang.Class.forName(Class.java:468)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:214)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)
org.springframework.boot.autoconfigure.condition.OnClassCondition.resolveOutcomesThreaded(OnClassCondition.java:69)
org.springframework.boot.autoconfigure.condition.OnClassCondition.getOutcomes(OnClassCondition.java:53)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.match(FilteringSpringBootCondition.java:49)
org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$ConfigurationClassFilter.filter(AutoConfigurationImportSelector.java:370)
org.springframework.boot.autoconfigure.AutoConfigurationImportSelector.getAutoConfigurationEntry(AutoConfigurationImportSelector.java:128)
org.springframework.boot.autoconfigure.AutoConfigurationImportSelector$AutoConfigurationGroup.process(AutoConfigurationImportSelector.java:434)
org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGrouping.getImports(ConfigurationClassParser.java:879)
org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:809)
org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:780)
org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:193)
org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:319)
org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236)
org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280)
org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:96)
org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:89)
MyArpclient.context.SpringContext.init(SpringContext.java:52)
MyArpclient.Activator.start(Activator.java:24)
org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814)
org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1)
java.base@15/java.security.AccessController.executePrivileged(AccessController.java:784)
java.base@15/java.security.AccessController.doPrivileged(AccessController.java:554)
org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806)
org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763)
org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1011)
org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
org.eclipse.osgi.container.Module.doStart(Module.java:605)
org.eclipse.osgi.container.Module.start(Module.java:468)
org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)
org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)
org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)
org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)
org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)
java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:616)
org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.createFromBundle(ReflectionContributionFactory.java:89)
org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.doCreate(ReflectionContributionFactory.java:60)
org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.create(ReflectionContributionFactory.java:42)
org.eclipse.e4.ui.internal.workbench.swt.E4Application.lambda$0(E4Application.java:243)
org.eclipse.e4.ui.internal.workbench.swt.E4Application$$Lambda$103/0x0000000800d6ba80.accept(Unknown Source)
java.base@15/java.util.Optional.ifPresent(Optional.java:176)
org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:242)
org.eclipse.e4.ui.internal.workbench.swt.E4Application.start(E4Application.java:153)
org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
java.base@15/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base@15/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
java.base@15/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base@15/java.lang.reflect.Method.invoke(Method.java:564)
app//org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
app//org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
app//org.eclipse.equinox.launcher.Main.run(Main.java:1461)
app//org.eclipse.equinox.launcher.Main.main(Main.java:1434)
app//MyArpclient.Bootstrap.main(Bootstrap.java:16)
ThreadId: 26 ThreadName: Thread-4 ThreadState: RUNNABLE
Blocked On: org.eclipse.osgi.internal.container.EquinoxReentrantLock@300c6244[Locked by thread main]
Synchronizers Locked: none
Monitors Locked:
org.eclipse.osgi.internal.framework.ContextFinder@6ed3ccb2
Stack Trace:
java.management@15/sun.management.ThreadImpl.dumpThreads0(Native Method)
java.management@15/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
java.management@15/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
org.eclipse.osgi.framework.util.ThreadInfoReport.getThreadDump(ThreadInfoReport.java:30)
org.eclipse.osgi.framework.util.ThreadInfoReport.<init>(ThreadInfoReport.java:23)
org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)
org.eclipse.osgi.container.Module.start(Module.java:419)
org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)
org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)
org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)
org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)
org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)
java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)
java.base@15/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
java.base@15/java.lang.Class.forName0(Native Method)
java.base@15/java.lang.Class.forName(Class.java:468)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)
org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)
org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)
org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)
org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver$$Lambda$197/0x0000000800e0d7a8.run(Unknown Source)
java.base@15/java.lang.Thread.run(Thread.java:832)
... 23 more
Root exception:
java.util.concurrent.TimeoutException: Timeout after waiting 30 seconds to acquire the lock.
at org.eclipse.osgi.container.Module.lockStateChange(Module.java:347)
at org.eclipse.osgi.container.Module.start(Module.java:419)
at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:506)
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:117)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:572)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:346)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:398)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:477)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:171)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.eclipse.osgi.internal.framework.ContextFinder.loadClass(ContextFinder.java:142)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:468)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition.resolve(FilteringSpringBootCondition.java:108)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter.isPresent(FilteringSpringBootCondition.java:140)
at org.springframework.boot.autoconfigure.condition.FilteringSpringBootCondition$ClassNameFilter$2.matches(FilteringSpringBootCondition.java:128)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:227)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcome(OnClassCondition.java:211)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.getOutcomes(OnClassCondition.java:201)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$StandardOutcomesResolver.resolveOutcomes(OnClassCondition.java:190)
at org.springframework.boot.autoconfigure.condition.OnClassCondition$ThreadedOutcomesResolver.lambda$new$0(OnClassCondition.java:150)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.eclipse.osgi.framework.util.ThreadInfoReport: Thread dump
main 持有 EquinoxReentrantLock@300c6244
main 等待 ContextFinder@6ed3ccb2
Thread-4 持有 ContextFinder@6ed3ccb2
Thread-4 等待 EquinoxReentrantLock@300c6244
完
