Issue :
We recently upgraded hazelcast version from 3.12.13 to 5.4.0.
Initially started with single pod.This single pod join the cluster(master).When we increased pod to 2 or more, Members couldn’t establish connection to each other when
deployed in Kubernetes cluster with Istio Envoy Proxy enabled
Versions:
Java 17
Spring 6.1.7
Hazelcast 5.4.0
Kubernetes
Client Version: 1.29.2
Server Version: 1.28.9-eks-036c24b
Istio 1.20.1
The problem is that one pod goes on 2/2 without problems , the other one continue to crash and print below error
Log Snippet:
com.hazelcast.core.HazelcastException: java.util.concurrent.TimeoutException: JoinMastershipClaimOp failed to complete within 9999991909 NANOSECONDS. Invocation{op=com.hazelcast.internal.cluster.impl.operations.JoinMastershipClaimOp{serviceName=’hz:core:clusterService’, identityHash=1854415116, partitionId=-1, replicaIndex=0, callId=2, invocationTime=1656088715788 (2022-06-24 16:38:35.788), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=1, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1656088715788, firstInvocationTime=’2022-06-24 16:38:35.788′, lastHeartbeatMillis=0, lastHeartbeatTime=’1970-01-01 00:00:00.000′, target=[172.16.70.174]:5701, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=null}
at com.hazelcast.internal.util.ExceptionUtil.lambda$static$0(ExceptionUtil.java:56)
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:134)
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:77)
at com.hazelcast.internal.util.ExceptionUtil.rethrow(ExceptionUtil.java:139)
at com.hazelcast.internal.util.FutureUtil$1.handleException(FutureUtil.java:53)
at com.hazelcast.internal.util.FutureUtil.returnWithDeadline(FutureUtil.java:275)
at com.hazelcast.internal.util.FutureUtil.returnWithDeadline(FutureUtil.java:240)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.claimMastership(TcpIpJoiner.java:214)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.tryClaimMastership(TcpIpJoiner.java:169)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:155)
at com.hazelcast.internal.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:96)
at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137)

Found below link related to the issue
https://github.com/hazelcast/hazelcast/issues/24630
https://github.com/hazelcast/hazelcast/issues/22256#issuecomment-1357440679
But solution mentioned in the link ,that we have tried using appProtocol but that didn’t work for us.

How can we prevent this ? how to make sure Hazelcast mark the connection properly between the pods ip and not the loopback?