I am using Spring MVC 5, java 11, mod_cluster/1.3.16. Final. Suddenly 1 node fails out of 2 and the below exception prints on the server console. I am out of solution and need help.
2024-06-19 02:22:08,245 INFO [org.infinispan.CLUSTER]
(thread-57,ejb,jvm1) ISPN100001: Node jvm3 left the cluster 2024-06-19
02:22:08,245 INFO [org.wildfly.clustering.ee.infinispan] (default
task-21) WFLYCLEEINF0001: Failed to cancel
WRQ0WOd-ioxzBQV6kiyhumj2RQBcyM73Wxyr05Wf on primary owner.:
java.util.concurrent.CancellationException
at [email protected]//org.wildfly.clustering.server.dispatcher.ServiceRequest.completeExceptionally(ServiceRequest.java:86)
at [email protected]//org.jgroups.blocks.UnicastRequest.viewChange(UnicastRequest.java:78)
at [email protected]//org.jgroups.blocks.RequestCorrelator.lambda$receiveView$1(RequestCorrelator.java:281)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3605)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at [email protected]//org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:281)
at [email protected]//org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:228)
at [email protected]//org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:569)
at [email protected]//org.jgroups.JChannel.up(JChannel.java:765)
at [email protected]//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911)
at [email protected]//org.jgroups.protocols.FORK.up(FORK.java:132)
at [email protected]//org.jgroups.protocols.FRAG3.up(FRAG3.java:151)
at [email protected]//org.jgroups.protocols.FlowControl.up(FlowControl.java:338)
at [email protected]//org.jgroups.protocols.pbcast.GMS.installView(GMS.java:707)
at [email protected]//org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65)
at [email protected]//org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:113)
at [email protected]//org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:330)
at [email protected]//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:297)
at [email protected]//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:450)
at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:952)
at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:886)
at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:860)
at [email protected]//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:682)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338)
at [email protected]//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338)
at [email protected]//org.jgroups.stack.Protocol.up(Protocol.java:338)
at [email protected]//org.jgroups.protocols.TP.passBatchUp(TP.java:1411)
at [email protected]//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284)
at [email protected]//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:136)
at [email protected]//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at [email protected]//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at [email protected]//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java. base/java.lang.Thread.run(Thread.java:829)
2 – web.xml contains <distributable/>
tag
3- The Widlfy 21 standalone configuration is below.
<subsystem xmlns="urn:jboss:domain:infinispan:11.0">
<cache-container name="web" default-cache="routing" module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="sso">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
</replicated-cache>
<replicated-cache name="routing"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="routing" module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="routing"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="hibernate" module="org.infinispan.hibernate-cache">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<heap-memory size="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="pending-puts">
<expiration max-idle="60000"/>
</local-cache>
<invalidation-cache name="entity">
<heap-memory size="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps"/>
</cache-container>
</subsystem>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp"/>
<socket-protocol type="MPING" socket-binding="jgroups-mping"/>
<protocol type="org.jgroups.protocols.TCPPING">
<property name="initial_hosts">10.20.30.101[7600],10.20.30.107[7600]</property>
<property name="port_range">0</property>
</protocol>
<protocol type="MERGE3"/>
<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="MFC"/>
<protocol type="FRAG3"/>
</stack>
<subsystem xmlns="urn:jboss:domain:modcluster:5.0">
<proxy name="default" advertise="true" advertise-security-key="######" advertise-socket="modcluster" balancer="cslt1" listener="ajp" node-timeout="120" sticky-session="true">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</proxy>
</subsystem>
4 – Apache configuration is below.
<VirtualHost 10.20.30.80:9080>
<Directory />
Require ip 10.20.30.101
Require ip 10.20.30.107
</Directory>
ServerAdvertise On
AdvertiseGroup 224.0.1.105:23367
EnableMCPMReceive on
ManagerBalancerName cslt1
AdvertiseSecurityKey ######
<Location /mod_cluster-manager>
SetHandler mod_cluster-manager
#Require ip 10.20.30.80
</Location>
</VirtualHost>