val sql = s””” INSERT OVERWRITE TABLE dynamic_member PARTITION(tenant_id = $tenantId, list_id = $listId) select customer_id from $view “”” sparkSession.sql(sql) Sometimes, when multiple job concurrent write data to different partition, I get this exception: FileNotFoundException File hdfs://XXX/cdp/dynamic_member/_temporary/0 does not exist And I found spark3 will use InsertIntoHadoopFsRelationCommand to write data to hive table partition, and […]