I develop service which will use ML for create response. So we can wait for response for about several seconds. So it’s quite long time and I need not to block my frontend while waiting for response.
I develop my ped project and I’m newer in programming, so facing with such problem (wait for long response) for the first time. So how can I build architecture in my project, to solve this task?
One way I’ve found that I can send from frontend my request (it’ll be simple string) and with this I’ll send unique API endpoint (generate it with userId and queryId: http/…/query/{userId}/{queryId}) and the create endpoint with this url in frontend.
When response will be prepared my ApiGateway will send it to provided endpoint. So as a result, it turns out that the backend will send a ready response to the frontend, and this api will listen to the frontend.
I came to this approach, beacuse I have ApiGateway on the server and as I understood, I shouldn’t block this ApiGateway by opening websocket connection or something like this. Because when some client open connection with server another client won’t be able to connect to server.
As a result, I have the problem that I have a server that inherently (due to working with ML) will take a long time to process requests. At the same time, I need to somehow think about the moment when sending a request from the client (frontend) so that the client, while waiting for its long response, frees up the ability to connect to the server of other clients
I do programming unprofessionally and do not work anywhere, but now I’m just working on my project. Therefore, perhaps my question may sound stupid, but I hope that someone can help me, I’ll be grateful for any help
My current architecture looks like here:
- client (frontend on react) send request to backend (ApiGateway – Asp Net minimal API)
- ApiGateway check the query and then send it through kafka to ML
- One of services get the response from ML, handle it and send to ApiGateway through kafka
- ApiGateway needs to send this response to client — how to do that? – or maybe I need to restructure my logic?
So as you see ApiGateway just forwards requests, and so in theory can handle requests from multiple clients
The “correct” way to do this from a HTTP standpoint is to return a HTTP 102 response from the endpoint until the ML processing is complete and the response delivered to that endpoint. Websockets obviously could “solve” this, at the cost of a higher connectivity requirement.
- Client requests through API gateway
- API Gateway provisions an endpoint url and sends the request through kafka (passing in the endpoint so it knows how to retrieve the payload)
- API Gateway returns a “Continue” message to client (202) with the URL for the endpoint
3.1 Client begins polling endpoint, receiving a HTTP 102 response until it’s complete - Service gets response from ML, fulfills the endpoint (this will require some kind of storage, probably a cache like Redis or similar)
- API Gateway (or backing service) returns the fulfilled response and sends it when the client next polls