I have a producer that generates a batch job that consists of multiple operations (approx. 100 – 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of the operations involves making several API calls to a 3rd party service which is pretty slow and has quite aggressive rate limits. The limits are bounded to a particular API key that is provided with the job and shared across the operations from the same batch. The same API key might be used in multiple different batch jobs.
Once the rate limit is hit, all the operations using the same API key need to wait for approx. 1 hour.
I’m trying to wrap my head around this and find an elegant solution that would fulfill the following criteria:
- Multiple operations can be executed in parallel, but they should not exceed MAX_PARALLEL_OPERATIONS constant, otherwise the 3rd party API will start failing with 500 errors
- Rate limiting an API key should not prevent operations using other API keys from being executed
- The number of different API keys can be dynamic
I’m struggling designing an elegant solution for this. I’ve considered using a single SQS queue and putting each operation in a separate message. However after receiving first rate limit error I would have to mark the API key as rate limited, skip all the messages that use the same API key and persist them somewhere else (queue / db / scheduler). This would increase the costs on the queue.
The second thing I’ve considered was putting multiple operations in a single message, but I don’t like the idea much. It limits the ability to execute the operations by multiple machines and it is not even viable, since I would hit the max size of item limit on SQS.
The best solution I could think of would involve putting the operations in a DB instead of a queue. Each row would represent each operation and contain a timestamp when it can be processed. Hitting a rate limit would involve doing an update on all the items that were not processed, belongs to the same API key and increasing their timestamp.
Then there would have to be some polling mechanism that would check for unprocessed item with due timestamp.
Is there any better approach for this? I have a feeling that I could be overengineering / reinventing the wheel.
Marian Galik is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I assume there is no penalty for a request that is rejected other than the failure. For example you’d hope that rejected requests don’t count towards rate limiting.
Create one queue for each key. Then execute requests and remove them from their queue unless you get an error. At the time of the error figure out at which point you can make further requests successfully and store that time with the queue.
So if you have 10 users with ten keys making lots of requests, and you have rate limiting after 100 requests for a key, you can perform 1,000 requests per hour.
If you are free to use these keys for different users, you could use any keys that havent been rate limited, but the server you are using might not be happy with that.
On macOS/iOS you wouldn’t do any polling but determine the earliest time for further requests and set a timer triggering at that time, at zero runtime cost.