I am using Airbyte to hit an API and retrieve details for a specific trading pair. I’m utilizing the builder feature of Airbyte to construct and send the API request. Below is the request I am firing:
{
"url": "https://api.binance.com/api/v3/klines?startTime=1721964898&interval=1m&symbol=ETHBTC",
"headers": {
"User-Agent": "python-requests/2.32.3",
"Accept-Encoding": "gzip, deflate",
"Accept": "*/*",
"Connection": "keep-alive"
},
"http_method": "GET",
"body": ""
}
I receive the following response (shortened version for brevity):
{
"status": 200,
"body": [
[
1500004800000,
"0.08000000",
"0.08000000",
"0.08000000",
"0.08000000",
"0.04300000",
1500004859999,
"0.00344000",
1,
"0.00000000",
"0.00000000",
"0"
],
[
1500004860000,
"0.08000000",
"0.08000000",
"0.08000000",
"0.08000000",
"0.00000000",
1500004919999,
"0.00000000",
0,
"0.00000000",
"0.00000000",
"0"
],
[
1500004920000,
"0.08000000",
"0.08000000",
"0.08000000",
"0.08000000",
"0.30600000",
1500004979999,
"0.02448000",
2,
"0.00000000",
"0.00000000",
"0"
]
]
}
However, I am unable to extract the response data. I believe the issue is due to the untyped array of arrays called body in the response. Below is the exception I am encountering:
ERROR
Something went wrong in the connector. See the logs for more details. - Traceback (most recent call last):
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 133, in read
yield from self._read_stream(
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 232, in _read_stream
for record_data_or_message in record_iterator:
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/streams/core.py", line 190, in read
for record_data_or_message in records:
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 136, in read_records
yield from self.retriever.read_records(self.get_json_schema(), stream_slice)
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 375, in read_records
for stream_data in self._read_pages(record_generator, self.state, _slice):
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 299, in _read_pages
yield from records_generator_fn(response)
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 445, in _parse_records
yield from self._parse_response(
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 245, in _parse_response
for record in record_generator:
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 67, in select_records
for data in normalized_data:
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/extractors/record_selector.py", line 76, in _normalize_by_schema
normalized_record = dict(record)
TypeError: cannot convert dictionary update sequence element #0 to a sequence
ERROR
None - Traceback (most recent call last):
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 302, in _read_stream
yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, state)
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 166, in read
for message in self.source.read(self.logger, config, catalog, state):
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 167, in read
yield from super().read(logger, config, catalog, state)
File "/home/airbyte/.pyenv/versions/3.9.19/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 186, in read
raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None
Has anyone encountered a similar issue or can provide guidance on how to properly extract the response data in these scenarios using Airbyte Builder? I believe the issue is because of the untyped array of arrays in the response key : body, Any suggestions or insights would be greatly appreciated.