I am working on a Laravel application that needs to offload some heavy computations to external services written in other programming languages (e.g., Python, Node.js, Go). The key requirements for my setup are:

Data Transfer: I need to pass potentially large datasets (several MBs or more) to these external services.

Same Machine: All scripts and services will run on the same physical/virtual machine as my Laravel application.

Performance: The solution should be efficient in terms of speed and memory usage, given the potential size of the data.

Scalability: While the services will run on a single machine for now, I’d like a solution that can scale to multiple services or machines in the future if needed.
What I’ve considered:

HTTP REST API: I could expose the external scripts via a REST API, but I’m concerned about the overhead of serializing large amounts of data in JSON format and potential performance bottlenecks.
CLI calls: Using exec() in PHP to directly run scripts, but this approach doesn’t seem robust or flexible for large data transfers or future scalability.
Queueing system: Setting up a queue (e.g., Redis, RabbitMQ) to pass data to external services. However, I’m not sure if this is the most efficient method for large data sets on the same machine.

My question:
What is the best way to organize calls to external scripts or services (Python, Node.js, Go, etc.) from a Laravel application, especially when handling large volumes of data, all running on the same server? Are there specific patterns, libraries, or communication protocols (e.g., gRPC, WebSockets, message queues) that would work best for this use case?

Any advice or examples would be greatly appreciated!

The goal:
I receive data from a third-party API and save it to the database. Then, I apply user data and generate a preliminary response in Laravel. After that, I want to pass part of the data for further processing to an external script. This script is located on the same machine. It will not duplicate or implement any server logic from Laravel or interact with third-party APIs. The goal is to obtain processed data that has already gone through server-side logic and perform post-processing on it, which will then be returned back to PHP (Laravel) to provide to the client.