I’ve got a Spark Structured Streaming job that reads data from Kafka and writes them to S3 (NetApp StorageGRID appliance, on-prem) as an Apache Iceberg table (via Nessie catalog).
Afterwards I access the table via Dremio which worked in the beginning, but now it seems that some Parquet files are corrupt – or at least Dremio is not able to read them.