My team’s working on an app that is using Spring Cloud Gateway backed by Reactor Netty, and an IOException occurs periodically during an attempt to make a request to various upstream services backed by a gateway:

java.io.IOException: Error while acquiring from reactor.netty.internal.shaded.reactor.pool.SimpleDequePool@162aedc4
     at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:248)
     Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
     *__checkpoint ⇢ ...
Original Stack Trace:
          at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:248)
          at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
          at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
          at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
          at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
          at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
          at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          at java.base/java.lang.Thread.run(Thread.java:840)

and from what I can see in Reactor’s source code (namely reactor.netty.resources.DefaultPooledConnectionProvider.DisposableAcquire#run):

            // The connection might be closed after checking the eviction predicate
            if (!c.isActive()) {
                pooledRef.invalidate()
                         .subscribe(null, null, () -> {
                             if (log.isDebugEnabled()) {
                                 logPoolState(c, pool, "Channel closed");
                             }
                         });
                if (!retried) {
                    if (log.isDebugEnabled()) {
                        log.debug(format(c, "Immediately aborted pooled channel, re-acquiring new channel"));
                    }
                    pool.acquire(Duration.ofMillis(pendingAcquireTimeout))
                            .contextWrite(ctx -> ctx.put(CONTEXT_CALLER_EVENTLOOP, c.eventLoop()))
                            .subscribe(new DisposableAcquire(this));
                }
                else {
// Line 248:                sink.error(new IOException("Error while acquiring from " + pool));
                }

this happens when a pooled Channel’s not active and it was a re-acquiring attempt. Is this a normal situation and can it be avoided somehow (e.g. changing Provider’s settings)? Or maybe an algorithm can be modified to avoid an inactive connection in the first place?

The following premature closure exceptions also pop up from time to time and may be related:

reactor.netty.http.client.PrematureCloseException: Connection has been closed BEFORE response, while sending request body

reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation
     at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)
     Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
     *__checkpoint ⇢ ...
Original Stack Trace:
          at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)
          at reactor.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:317)
          at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
          at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
          at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
          at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
          at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
          at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
          at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
          at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
          at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
          at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:410)
          at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
          at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
          at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
          at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
          at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
          at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
          at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
          at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
          at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
          at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
          at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
          at java.base/java.lang.Thread.run(Thread.java:840)

reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
     Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Error has been observed at the following site(s):
     *__checkpoint ⇢ ...
Original Stack Trace:

An idle timeout/periodic eviction has been set up for a client’s Connection Provider, plus the HTTP timeouts were lowered, but the errors did not go away completely. Are there any additional measures that can be applied here to mitigate this?