We are currently getting errors of this type in a Quarkus native app. The app works fine, and the errors are really sporadic (once a week, maybe?). Depending on usage, this app has a high consumption of RAM when generating big xlsx files with POI.
We understand that it’s memory related, but the error is not necessarily related to any execution (the last execution was about 30 minutes before the error, while the next one was 30 minutes later).
What we want to understand if this is a framework error (something that should be reported), a dev error (something we can fix/tweak), or if it’s just expected to happen.
Error:
<code>[io.mic.cor.ins.int.DefaultGauge] (executor-thread-348) Failed to apply the value function for the gauge 'jvm.memory.used'. Note that subsequent logs will be logged at debug level.: java.lang.IllegalArgumentException: committed = 1660640336 should be < max = 1659895808
at [email protected]/java.lang.management.MemoryUsage.<init>(MemoryUsage.java:166)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.AbstractMemoryPoolMXBean.memoryUsage(AbstractMemoryPoolMXBean.java:75)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.GenScavengeMemoryPoolMXBeans$EdenMemoryPoolMXBean.getUsage(GenScavengeMemoryPoolMXBeans.java:94)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsage(JvmMemory.java:79)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsageValue(JvmMemory.java:69)
at io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics.lambda$bindTo$0(JvmMemoryMetrics.java:82)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$5(PrometheusMeterRegistry.java:338)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:75)
at io.prometheus.client.Collector.collect(Collector.java:45)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71)
at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:154)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:149)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:120)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.doHandle(PrometheusHandler.java:63)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:43)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:18)
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at [email protected]/java.lang.Thread.run(Thread.java:840)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210)
<code>[io.mic.cor.ins.int.DefaultGauge] (executor-thread-348) Failed to apply the value function for the gauge 'jvm.memory.used'. Note that subsequent logs will be logged at debug level.: java.lang.IllegalArgumentException: committed = 1660640336 should be < max = 1659895808
at [email protected]/java.lang.management.MemoryUsage.<init>(MemoryUsage.java:166)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.AbstractMemoryPoolMXBean.memoryUsage(AbstractMemoryPoolMXBean.java:75)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.GenScavengeMemoryPoolMXBeans$EdenMemoryPoolMXBean.getUsage(GenScavengeMemoryPoolMXBeans.java:94)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsage(JvmMemory.java:79)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsageValue(JvmMemory.java:69)
at io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics.lambda$bindTo$0(JvmMemoryMetrics.java:82)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$5(PrometheusMeterRegistry.java:338)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:75)
at io.prometheus.client.Collector.collect(Collector.java:45)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71)
at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:154)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:149)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:120)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.doHandle(PrometheusHandler.java:63)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:43)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:18)
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at [email protected]/java.lang.Thread.run(Thread.java:840)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210)
</code>
[io.mic.cor.ins.int.DefaultGauge] (executor-thread-348) Failed to apply the value function for the gauge 'jvm.memory.used'. Note that subsequent logs will be logged at debug level.: java.lang.IllegalArgumentException: committed = 1660640336 should be < max = 1659895808
at [email protected]/java.lang.management.MemoryUsage.<init>(MemoryUsage.java:166)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.AbstractMemoryPoolMXBean.memoryUsage(AbstractMemoryPoolMXBean.java:75)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.genscavenge.GenScavengeMemoryPoolMXBeans$EdenMemoryPoolMXBean.getUsage(GenScavengeMemoryPoolMXBeans.java:94)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsage(JvmMemory.java:79)
at io.micrometer.core.instrument.binder.jvm.JvmMemory.getUsageValue(JvmMemory.java:69)
at io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics.lambda$bindTo$0(JvmMemoryMetrics.java:82)
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$5(PrometheusMeterRegistry.java:338)
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:75)
at io.prometheus.client.Collector.collect(Collector.java:45)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71)
at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:154)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:149)
at io.micrometer.prometheus.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:120)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.doHandle(PrometheusHandler.java:63)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:43)
at io.quarkus.micrometer.runtime.export.handlers.PrometheusHandler.handle(PrometheusHandler.java:18)
at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at [email protected]/java.lang.Thread.run(Thread.java:840)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210)
Monitoring tool at error time:
jvm_memory_used_bytes (eden space)
went from ~1.55 GB
to ~14.8 MB
jvm_memory_used_bytes (old generation space)
went from ~232.67 MB
to ~251.04 MB
Also found:
- https://github.com/oracle/graal/issues/6814
- https://github.com/quarkusio/quarkus/issues/35196