When I use following threadpool with heavy load.
private final ScheduledExecutorService serverExecutor2 = Executors.newScheduledThreadPool(2000, Thread.ofVirtual().name("COAP-SERVER2-", 0).factory());
it seems to stuck:
serverExecutor2=@ScheduledThreadPoolExecutor[java.util.concurrent.ScheduledThreadPoolExecutor@327e5be5[Running, pool size = 2000, active threads = 0, queued tasks = 60000, completed tasks = 97847]],
All the correspond virtual thread is parking:
#75 "COAP-SERVER2-5" virtual
java.base/java.lang.VirtualThread.park(VirtualThread.java:592)
java.base/java.lang.System$2.parkVirtualThread(System.java:2639)
java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54)
java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1177)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base/java.lang.VirtualThread.run(VirtualThread.java:311)
And it seems to be caused by blocking of VirtualThread-unparker
while it has a tons of tasks waiting:
"VirtualThread-unparker" #92 [593799] daemon prio=5 os_prio=0 cpu=2238.28ms elapsed=373.27s tid=0x00007f1b3002fd50 nid=593799 waiting on condition [0x00007f1e6db43000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00001000177791b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
at java.lang.Thread.runWith([email protected]/Thread.java:1583)
at java.lang.Thread.run([email protected]/Thread.java:1570)
at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:186)
Locked ownable synchronizers:
- None
I used a debug tool to get the queue of unparker. it’s obvious that task at queue top should run a long time ago (it’s time is 84137649857516 while current time is 84707647860354):
[arthas@593599]$ ognl '@[email protected]()'
@String[java.util.concurrent.ScheduledThreadPoolExecutor@154b1210[Running, pool size = 1, active threads = 0, queued tasks = 30027, completed tasks = 413674]]
[arthas@593599]$ ognl '@System@nanoTime()'
@Long[84707647860354]
[arthas@593599]$ ognl '@[email protected]()'
@ScheduledFutureTask[
sequenceNumber=@Long[1227676],
time=**@Long[84137649857516]**,
period=@Long[0],
outerTask=@ScheduledFutureTask[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5d021eb6[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7d5f12fb[Wrapped task = java.lang.VirtualThread$$Lambda/0x00007f1bf357cc70@56d5ca23]]],
heapIndex=@Integer[0],
this$0=@ScheduledThreadPoolExecutor[java.util.concurrent.ScheduledThreadPoolExecutor@154b1210[Running, pool size = 1, active threads = 0, queued tasks = 30027, completed tasks = 413674]],
state=@Integer[0],
NEW=@Integer[0],
COMPLETING=@Integer[1],
NORMAL=@Integer[2],
EXCEPTIONAL=@Integer[3],
CANCELLED=@Integer[4],
INTERRUPTING=@Integer[5],
INTERRUPTED=@Integer[6],
callable=@RunnableAdapter[java.util.concurrent.Executors$RunnableAdapter@7d5f12fb[Wrapped task = java.lang.VirtualThread$$Lambda/0x00007f1bf357cc70@56d5ca23]],
outcome=null,
runner=null,
waiters=null,
STATE=@FieldInstanceReadWrite[VarHandle[varType=int, coord=[class java.util.concurrent.FutureTask]]],
RUNNER=@FieldInstanceReadWrite[VarHandle[varType=java.lang.Thread, coord=[class java.util.concurrent.FutureTask]]],
WAITERS=@FieldInstanceReadWrite[VarHandle[varType=java.util.concurrent.FutureTask$WaitNode, coord=[class java.util.concurrent.FutureTask]]],