I’m trying to scrape some data off web, but I’m getting HTTP error 429. I know that I can only send 20 requests per minute, so I wanted to solve this by using Celery in Python to create something like a token bucket. Simplified example:
import celery
from datetime import datetime
@celery.Task(rate_limit="10/m")
def tsk():
print(f"Time is{datetime.now()}")
for _ in range(100):
tsk()
But I’m getting an error that celery.Task()
isn’t taking arguments. I want to run function tsk()
10 times, then take a break until the next minute starts, then run it 10 times again, and so on for 100 times in total. I’m trying to do this the simplest way possible, but most answers/examples on SO and on Celery documentation are either incomplete or use unncessesary classes. How can I fix this?