A Java application (Spring Boot based) has a Kafka producer configured for secured remote connection, see the defined properties below:
<code>2024-07-01 15:04:24,238 [main] INFO c.i.t.a.TzKafkaMonitoringConfig:31 - Configuration of the ida.monitoring producer:
- bootstrap.servers = SASL_SSL://kafka-broker-blahblah1:9093,SASL_SSL://kafka-broker-blahblah2:9093,SASL_SSL://kafka-broker-blahblah3:9093
- buffer.memory = 32768000
- key.serializer = org.apache.kafka.common.serialization.StringSerializer
- max.in.flight.requests.per.connection = 1
- sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required username="ida" password="????" user_username="password";
- sasl.mechanism = SCRAM-SHA-256
- security.protocol = SASL_SSL
- ssl.endpoint.identification.algorithm = HTTPS
- ssl.truststore.location = our-sr-truststore.jks
- ssl.truststore.password = ???
- value.serializer = org.apache.kafka.common.serialization.StringSerializer
2024-07-01 15:04:24,243 [main] WARN o.a.k.c.a.AdminClientConfig:385 - These configurations '[batch.size, acks, buffer.memory, key.serializer, value.serializer, max.block.ms, max.in.flight.requests.per.connection]' were supplied but are not used yet.
2024-07-01 15:04:24,244 [main] INFO o.a.k.c.u.AppInfoParser:119 - Kafka version: 3.4.1
<code>2024-07-01 15:04:24,238 [main] INFO c.i.t.a.TzKafkaMonitoringConfig:31 - Configuration of the ida.monitoring producer:
- acks = 1
- batch.size = 2048000
- bootstrap.servers = SASL_SSL://kafka-broker-blahblah1:9093,SASL_SSL://kafka-broker-blahblah2:9093,SASL_SSL://kafka-broker-blahblah3:9093
- buffer.memory = 32768000
- key.serializer = org.apache.kafka.common.serialization.StringSerializer
- max.block.ms = 6000
- max.in.flight.requests.per.connection = 1
- retries = 2147483647
- sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required username="ida" password="????" user_username="password";
- sasl.mechanism = SCRAM-SHA-256
- security.protocol = SASL_SSL
- ssl.endpoint.identification.algorithm = HTTPS
- ssl.truststore.location = our-sr-truststore.jks
- ssl.truststore.password = ???
- value.serializer = org.apache.kafka.common.serialization.StringSerializer
2024-07-01 15:04:24,243 [main] WARN o.a.k.c.a.AdminClientConfig:385 - These configurations '[batch.size, acks, buffer.memory, key.serializer, value.serializer, max.block.ms, max.in.flight.requests.per.connection]' were supplied but are not used yet.
2024-07-01 15:04:24,244 [main] INFO o.a.k.c.u.AppInfoParser:119 - Kafka version: 3.4.1
</code>
2024-07-01 15:04:24,238 [main] INFO c.i.t.a.TzKafkaMonitoringConfig:31 - Configuration of the ida.monitoring producer:
- acks = 1
- batch.size = 2048000
- bootstrap.servers = SASL_SSL://kafka-broker-blahblah1:9093,SASL_SSL://kafka-broker-blahblah2:9093,SASL_SSL://kafka-broker-blahblah3:9093
- buffer.memory = 32768000
- key.serializer = org.apache.kafka.common.serialization.StringSerializer
- max.block.ms = 6000
- max.in.flight.requests.per.connection = 1
- retries = 2147483647
- sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required username="ida" password="????" user_username="password";
- sasl.mechanism = SCRAM-SHA-256
- security.protocol = SASL_SSL
- ssl.endpoint.identification.algorithm = HTTPS
- ssl.truststore.location = our-sr-truststore.jks
- ssl.truststore.password = ???
- value.serializer = org.apache.kafka.common.serialization.StringSerializer
2024-07-01 15:04:24,243 [main] WARN o.a.k.c.a.AdminClientConfig:385 - These configurations '[batch.size, acks, buffer.memory, key.serializer, value.serializer, max.block.ms, max.in.flight.requests.per.connection]' were supplied but are not used yet.
2024-07-01 15:04:24,244 [main] INFO o.a.k.c.u.AppInfoParser:119 - Kafka version: 3.4.1
On start the application reports in its logs an OutOfMemoryError-exception specifically for the producer. Inspite of this the application runs further and does all the tasks not related to the Kafka producer:
<code>2024-07-01 15:04:24,368 [main] INFO c.i.m.IdaMonitoringSink:27 - IIdaMonitoringSink for writing monitoring data has been initialized
2024-07-01 15:04:24,381 [main] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor:595 - Autowired annotation should only be used on methods with parameters: public boolean com.ida.metrics.IdaKafkaConsumerMetricsProvider.isMetricsSourceSet()
2024-07-01 15:04:24,559 [kafka-producer-network-thread | producer-1] ERROR o.a.k.c.u.KafkaThread:49 - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':
**java.lang.OutOfMemoryError: Java heap space**
at java.base/java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Unknown Source)
2024-07-01 15:04:24,631 [main] INFO c.i.m.MonitoringConfigurationService:282 - Connector to local JMX server established
2024-07-01 15:04:24,634 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-CONSUMER] from the JSON-configuration
2024-07-01 15:04:24,635 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-HEARTBEAT] from the JSON-configuration
2024-07-01 15:04:24,636 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-SYSTEM] from the JSON-configuration
2024-07-01 15:04:24,642 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-SYSTEM has been configured as: type=METRICS/description=System metrics for ISR full database mirror/initial delay 30000 ms/polling interval 1800000 ms
2024-07-01 15:04:24,649 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-SYSTEM every 1800000 ms, after initial delay of 30000
2024-07-01 15:04:24,652 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-CONSUMER has been configured as: type=METRICS/description=Kafka consumer metrics for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,668 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-CONSUMER every 60000 ms, after initial delay of 10000
2024-07-01 15:04:24,669 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-HEARTBEAT has been configured as: type=HEARTBEAT/description=Heartbeat for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,675 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-HEARTBEAT every 60000 ms, after initial delay of 10000
<code>2024-07-01 15:04:24,368 [main] INFO c.i.m.IdaMonitoringSink:27 - IIdaMonitoringSink for writing monitoring data has been initialized
2024-07-01 15:04:24,381 [main] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor:595 - Autowired annotation should only be used on methods with parameters: public boolean com.ida.metrics.IdaKafkaConsumerMetricsProvider.isMetricsSourceSet()
2024-07-01 15:04:24,559 [kafka-producer-network-thread | producer-1] ERROR o.a.k.c.u.KafkaThread:49 - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':
**java.lang.OutOfMemoryError: Java heap space**
at java.base/java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Unknown Source)
2024-07-01 15:04:24,631 [main] INFO c.i.m.MonitoringConfigurationService:282 - Connector to local JMX server established
2024-07-01 15:04:24,634 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-CONSUMER] from the JSON-configuration
2024-07-01 15:04:24,635 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-HEARTBEAT] from the JSON-configuration
2024-07-01 15:04:24,636 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-SYSTEM] from the JSON-configuration
2024-07-01 15:04:24,642 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-SYSTEM has been configured as: type=METRICS/description=System metrics for ISR full database mirror/initial delay 30000 ms/polling interval 1800000 ms
2024-07-01 15:04:24,649 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-SYSTEM every 1800000 ms, after initial delay of 30000
2024-07-01 15:04:24,652 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-CONSUMER has been configured as: type=METRICS/description=Kafka consumer metrics for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,668 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-CONSUMER every 60000 ms, after initial delay of 10000
2024-07-01 15:04:24,669 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-HEARTBEAT has been configured as: type=HEARTBEAT/description=Heartbeat for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,675 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-HEARTBEAT every 60000 ms, after initial delay of 10000
</code>
2024-07-01 15:04:24,368 [main] INFO c.i.m.IdaMonitoringSink:27 - IIdaMonitoringSink for writing monitoring data has been initialized
2024-07-01 15:04:24,381 [main] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor:595 - Autowired annotation should only be used on methods with parameters: public boolean com.ida.metrics.IdaKafkaConsumerMetricsProvider.isMetricsSourceSet()
2024-07-01 15:04:24,559 [kafka-producer-network-thread | producer-1] ERROR o.a.k.c.u.KafkaThread:49 - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':
**java.lang.OutOfMemoryError: Java heap space**
at java.base/java.nio.HeapByteBuffer.<init>(Unknown Source)
at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Unknown Source)
2024-07-01 15:04:24,631 [main] INFO c.i.m.MonitoringConfigurationService:282 - Connector to local JMX server established
2024-07-01 15:04:24,634 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-CONSUMER] from the JSON-configuration
2024-07-01 15:04:24,635 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-HEARTBEAT] from the JSON-configuration
2024-07-01 15:04:24,636 [main] INFO c.i.m.MonitoringConfigurationService:336 - Fetched monitor [ISR-DB-SYSTEM] from the JSON-configuration
2024-07-01 15:04:24,642 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-SYSTEM has been configured as: type=METRICS/description=System metrics for ISR full database mirror/initial delay 30000 ms/polling interval 1800000 ms
2024-07-01 15:04:24,649 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-SYSTEM every 1800000 ms, after initial delay of 30000
2024-07-01 15:04:24,652 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-CONSUMER has been configured as: type=METRICS/description=Kafka consumer metrics for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,668 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-CONSUMER every 60000 ms, after initial delay of 10000
2024-07-01 15:04:24,669 [main] INFO c.i.m.MonitoringConfigurationService:349 - Monitor ISR-DB-HEARTBEAT has been configured as: type=HEARTBEAT/description=Heartbeat for ISR full database mirror/initial delay 10000 ms/polling interval 60000 ms
2024-07-01 15:04:24,675 [main] INFO c.i.m.MonitoringConfigurationService:224 - Scheduled monitor ISR-DB-HEARTBEAT every 60000 ms, after initial delay of 10000
Question: what is wrong with the producer? Why the OutOfMemory is thrown?
I have tried to play with the properties “buffer.memory” and “batch.size” but I see no effect.