I am building a DBT data transformation pipeline which needs to read parquet data from s3 location and write the output again to another S3 location.
What are the configuration changes which will be needed in –
- profiles.yml
- How do I read the parquet in models sql file and store the output in s3 again.
*note: dont want to install duckdb or any other integration.
Expecting a seemless integration which pulls data from S3, processes it and stores the processed output back to another S3 location.
New contributor
Hrishikesh Khurpe is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.