Relative Content

Tag Archive for python-2.7python-requestscouchdbconcurrent.futuresgrequests

Can processing of request/grequest responses be run concurrently in Python2?

I’m writing a Python2 program that pulls a large amount of JSON data from a remote CouchDB database. The data is indexed by timestamp: I pass a startkey and an endkey, and the database returns all rows in-between. I was able to slice the request into multiple partial requests (creating multiple param dicts, stored in threadedParamsList), then send them concurrently (using grequests) which offered a significant performance boost. However, I then need to unpack the responses and return all rows, which seems to be adding quite a bit of overhead: