Im using gpt 3.5 LLM model along with necessary prompts to get different levels of product description. My dataset contains 35K rows. I’m able to send 90 rows in one api call else I get rate limit error. But if im sending 35k in batches like this. Its consuming lot of time and I’m again getting same error.
This is my code and error attached. Please let me know how to handle for large csv dataset if anyone how I can use Batch API ( https://platform.openai.com/docs/api-reference/batch ) from Open AI here ?

This is my error
[

RateLimitError Traceback (most recent call last)
in <cell line: 7>()
5
6 # Call the OpenAI API for Level 1 categories
—-> 7 levels_1_response = process_descriptions_in_parallel(prompt_template_levels_1, item_descriptions)
8
9 # Call the OpenAI API for Level 2 product types without brand names
13 frames
/usr/local/lib/python3.10/dist-packages/openai/_base_client.py in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
1018
1019 log.debug(“Re-raising status error”)
-> 1020 raise self._make_status_error_from_response(err.response) from None
1021
1022 return self._process_response(
RateLimitError: Error code: 429 – {‘error’: {‘message’: ‘Rate limit reached for gpt-3.5-turbo-16k in organization org-7fREFFIkR6K6GISy1X3ZezF2 on tokens per min (TPM): Limit 80000, Used 79546, Requested 1142. Please try again in 516ms. Visit https://platform.openai.com/account/rate-limits to learn more.’, ‘type’: ‘tokens’, ‘param’: None, ‘code’: ‘rate_limit_exceeded’}}
]

Hi everyone,

Im using gpt 3.5 LLM model along with necessary prompts to get different levels of product description. My dataset contains 35K rows. I’m able to send 90 rows in one api call else I get rate limit error. But if im sending 35k in batches like this. Its consuming lot of time and I’m again getting same error.
This is my code and error attached. Please let me know how to handle for large csv dataset if anyone how I can use Batch API ( https://platform.openai.com/docs/api-reference/batch ) from Open AI here ?