I am exporting data from PostgreSQL to another database using the following query:
INSERT INTO db_name.public.example_table (id, name, content, boolean, timestamp) SELECT id, name, content, CAST(boolean AS tinyint), CAST(timestamp AS timestamp(0)) FROM postgresql.public.example_table;
However, the transfer speed is always limited to around 50 MB per second. Here are the details of my setup:
Ping between hosts: 0.6 ms
Network bandwidth: 1 Gbps
PostgreSQL server specs: Intel Cascade Lake, 8 CPUs, 8 GB RAM
Target database server specs: Similar to PostgreSQL server.
Trino server: Intel Cascade Lake 16 CPU 32 GB RAM.
I have set the following configurations in Trino:
SET SESSION task_writer_count = 128;
SET SESSION task_concurrency = 128;
SET SESSION query_max_memory = '10GB';
SET SESSION query_max_total_memory = '10GB';
SET SESSION query_max_memory_per_node = '4GB';
Despite these settings, the transfer speed does not improve. I am 100% sure the issue is not with the target database, as I have tried multiple different setups. Additionally, during the export process, resource utilization never exceeds 50%. What could be causing this bottleneck and how can I optimize the data transfer speed?
I want to improve export speed.
Panteley is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.