First, allow me to share some context: I have an Amazon Kinesis Data Stream, where data is continuously stored. I also have an Apache Flink application that will integrate with the Amazon Kinesis Data Stream and have access to the data in the Amazon Kinesis Data Stream.

I have to create a job in the Apache Flink that, based on two timestamp input parameters that will be sent to the job, called “from” and “to,” would retrieve all the data from the Amazon Kinesis Data Stream that fits in the interval mentioned by the two input parameters (the data records from the Kinesis Data Stream also have a ‘timestamp’ field), and then return the data.

I also have a NodeJS backend application that exposes a GET endpoint. When the respective GET endpoint is called, I need to trigger the job I mentioned in the Apache Flink application, and then, after the job finishes running/executing, I need to get all the data that was retrieved by the Apache Flink application from the KDS back into the NodeJS backend service, within the same NodeJS execution context, because in the end, I need to return the data retrieved from KDS, by the Flink job, as a response to the GET endpoint of the NodeJS application. Is achieving this functionality possible, and if yes, how can it be done?

From what I read in the Apache Flink’s APIs documentation: https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/rest_api/, I have the following idea:

1. Upload the ‘jar’ that retrieves the data from KDS to the Apache Flink application
2. When the GET endpoint of my NodeJS backend application is called, I call POST on the /jars/:jarid/run API from Apache Flink, which will run the job to get the data from KDS, and then I continuously call GET on the /jobs/:jobid/execution-result API from Apache Flink to find out when the job is completed. After the /jobs/:jobid/execution-result API returns ‘completed,’ my question is: how can I now send/transfer the data retrieved from KDS as part of the Apache Flink job execution to the NodeJS application within the same NodeJS execution context that was created by the initial call to the GET endpoint of the NodeJS backend application.

NOTE: In the Apache Flink job, after I retrieve the data from KDS, I must not output the data to a temporary storage solution and then retrieve it from the storage in the NodeJS application, the data has to make its way directly from the Apache Flink job to the NodeJS application, within the same context, so I must also not make a separate https call from the Apache Flink job to the NodeJS application to send the data. Making use of other AWS services should be ok, if they would help, as long as it is not for stoing the data anywhere.