I’m currently working on this demo project using Selenium Java to explore the functionalities that I usually use in e2e test automation projects and to serve as a reference for future projects. I’d like to know if there’s a way to intercept network requests and access their response body using the Selenium BiDi APIs.

I’ve already implemented network interception using the DevTools package, but it’s tied to a specific version and only works on Chromium browsers. To achieve the same functionality with more flexibility, I’m looking to use the BiDi APIs. This is what I’ve got so far:

A class for managing network interception and accessing request responses.

import org.openqa.selenium.devtools.DevTools;
import org.openqa.selenium.devtools.v125.network.Network;
import org.openqa.selenium.devtools.v125.network.model.Request;
import org.openqa.selenium.devtools.v125.network.model.RequestId;
import org.openqa.selenium.devtools.v125.network.model.Response;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.ConfigurableBeanFactory;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Service;

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

@Service
@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
public class NetworkInterceptor {
    @Autowired
    private DevTools devTools;
    private final CountDownLatch latch;
    private RequestId interceptedRequestId;

    @Autowired
    public NetworkInterceptor() {
        this.latch = new CountDownLatch(1);
    }

    /**
     * Intercepts a request with a specific URL and method, so that the test can wait for it to complete.
     * Be aware that you must call {@link NetworkInterceptor#waitForResponse()} after the request is triggered.
     * @param urlRegex The regex to match the URL of the request to intercept. For example: <code>".*articles/" + variable</code>
     * @param requestMethod The method of the request to intercept
     */
    public void interceptResponse(String urlRegex, String requestMethod) {
        final ConcurrentHashMap<String, Request> requestMap = new ConcurrentHashMap<>();
        devTools.addListener(Network.requestWillBeSent(), request -> requestMap.put(request.getRequestId().toString(), request.getRequest()));
        devTools.addListener(Network.responseReceived(), response -> {
            Response res = response.getResponse();
            if (res.getUrl().matches(urlRegex) && requestMap.containsKey(response.getRequestId().toString())) {
                Request interceptedRequest = requestMap.get(response.getRequestId().toString());
                if (interceptedRequest.getMethod().equals(requestMethod)) {
                    this.interceptedRequestId = response.getRequestId();
                    System.out.println("Intercepted response: " + res.getUrl() + " with status " + res.getStatus());
                    markRequestAsCompleted();
                }
            }
        });
    }

    /**
     * Waits for the intercepted request to complete and returns the response body.
     * @return The response body of the intercepted request
     */
    public String waitForResponse() {
        try {
            boolean completed = latch.await(10, TimeUnit.SECONDS);
            if (!completed) {
                throw new RuntimeException("The specific request did not complete within the timeout period");
            }
            return getResponseBody();
        } catch (InterruptedException e) {
            throw new RuntimeException("Interrupted while waiting for request to complete", e);
        }
    }

    private String getResponseBody() {
        Network.GetResponseBodyResponse bodyResponse = devTools.send(Network.getResponseBody(interceptedRequestId));
        return bodyResponse.getBody();
    }

    private void markRequestAsCompleted() {
        latch.countDown();
    }
}

Here’s an example test that demonstrates using these methods:

public void followAuthor() {
        // Arrange
        authorApi.unfollowAuthor(0);
        var authorName = authorDetailPage.visit(0);

        // Act
        networkInterceptor.interceptResponse(".*/profiles/" + authorName + "/follow", "POST");
        followAuthorButton.clickButton();
        boolean isFollowing =  JsonPath.parse(networkInterceptor.waitForResponse()).read("$.profile.following");


        // Assert
        assertThat(followAuthorButton.getButton().getText()).contains("Unfollow");
        assertThat(isFollowing).isTrue();
    }

I saw a similar solution on Stack Overflow, but it involved overly verbose code within the tests themselves. Additionally, I couldn’t find a way to access the response body. Ideally, I’d like to achieve the functionality demonstrated in the test:

• Intercept request providing a url regex and the request method, just using a simple method
• Do some action that triggers the request
• Wait for the request to complete and have access to its response body

BrowserMob proxy is not an option since I read that is no longer mantained since 2016