Can processing of request/grequest responses be run concurrently in Python2?
I’m writing a Python2 program that pulls a large amount of JSON data from a remote CouchDB database. The data is indexed by timestamp: I pass a startkey and an endkey, and the database returns all rows in-between. I was able to slice the request into multiple partial requests (creating multiple param
dicts, stored in threadedParamsList
), then send them concurrently (using grequests
) which offered a significant performance boost. However, I then need to unpack the responses and return all rows, which seems to be adding quite a bit of overhead: