Usage of Hazelcast (5.1.7) ICountDownLatch produces an OOM in an OLTP system despite I call destroy() method as the documentation recommends.

I have an Spring Boot application which already works in production. Currently it uses embedded Hazelcast version 3.12.13. The application is an OLTP system and uses ICountDownLatch for a certain purpose. It obtains an instance of ICountDownLatch for every transaction and then destroys it after some time.

Currently I obtain the ICountDownLatch in the following manner hazelcastInstance.getCountDownLatch("name")

And later destroy it like:

hazelcastInstance.getDistributedObjects()
                .stream()
                .filter(object -> type.isAssignableFrom(ICountDownLatch.class))
                .filter(object -> "name".equals(object.getName()))
                .forEach(object -> {
                    logger.debug("destroy data collector countDownLatch with name {}", object.getName());
                    object.destroy();
                });

I’m not fully sure that it is correct, but it works without known OOM issues under a quite high load.

For one reason I’m planning to upgrade Hazelcast version to 5.1.7 (this version is a transitive dependency of my current Spring Boot version).

In the documentation they say that I have to use ICountDownLatch.destroy() to destroy the latch which I don’t need any more. But they also say the following:

If you call the destroy() method on a CP data structure object, that data structure is terminated in the underlying CP group and cannot be reinitialized until the CP group is force-destroyed. For this reason, please make sure that you are completely done with a CP data structure before destroying it.

This was the first thing to confuse me. How can Hazelcast keep information about the destroyed object ‘forever’ without producing an OOM at some time?

So, I did an experiment. I wrote a simple Controller and tried to load it with JMeter:

    @GetMapping(value = {"/hz-test"})
    public ResponseEntity<?> hzTest() {
        try {
            final String latchId = "latch-" + UUID.randomUUID();

            ICountDownLatch latch = hazelcastInstance.getCPSubsystem().getCountDownLatch(latchId);

            latch.trySetCount(1);

            latch.countDown();

            latch.destroy();

            return ResponseEntity.ok("OK");
        } catch (Exception e) {
            logger.error("error", e);
            return ResponseEntity.internalServerError().body(e.getClass().getSimpleName() + ":" + e.getMessage());
        }
    }

As I expected, in VisualVM I see that heap usage grows gradually until it ends up with an OOM. Then probably Hazelcast’s OutOfMemoryHandler kills the HazelcastInstance and GC cleans all HC objects from the heap.

I didn’t do any particular configuration of CP subsystem of the embedded Hazelcast instance.

So, I doubt. Either I need some additional configuration of my CP subsystem? Or I somehow destroy my latch in the wrong manner? Or ICountDownLatch is not supposed to be used in the manner I use it?