I am working on a Spring Boot application. I’m trying to send around 3.5k GET requests to an endpoint, however, when using the following code, it sends the java.io.IOException: too many concurrent streams
error.
public void scrapeEventsForDateRange(LocalDate startDate, LocalDate endDate) {
for (LocalDate date = endDate; !date.isBefore(startDate); date = date.minusDays(1)) {
fetchAndSaveEventsForDateAsync(date, false);
}
}
public void runStartup() {
LocalDate currentDate = LocalDate.now();
LocalDate startDate = currentDate.minusYears(10);
scrapeEventsForDateRange(startDate, currentDate);
}
private void fetchAndSaveEventsForDateAsync(LocalDate date, boolean live) {
String formattedDate = date.format(DateTimeFormatter.ISO_LOCAL_DATE);
String url = "HIDDEN ENDPOINT" + formattedDate;
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(url))
.build();
client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.thenAcceptAsync(response -> {
if (live) {
processEventsResponseLive(response);
} else {
processEventsResponse(response);
}
})
.exceptionally(ex -> {
ex.printStackTrace();
return null;
});
}
I tried many ways, such as Semaphore’s – but those are very slow. I would appreciate if anyone can help me with this.
New contributor
Timo is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.