I’m trying to do a spark.read() on a table using

readDataFrame = sparkSession.read()
                        .format("jdbc")
                        .option("url", dbUrl)
                        .option("dbtable", table)
                        .option("driver", driver)
                        .option("fetchsize", fetchSize)
                        .option("oracle.net.wallet_location","(SOURCE=(METHOD=file)(METHOD_DATA=(DIRECTORY=/tmp/wallet_location")))
                        .load();

I am using an internal API which fetches me the wallet files at runtime and I store them in /tmp/wallet_location folder. However, the issue is the path isn’t available on all executors and I’m getting a FileNotFound Error.

Is there a way I can make the files and folder accessible across all executors?

The only way I can think is by creating them before running the spark-submit command and passing it in --files. However, I would need to add another process to make this possible as these files need to be fetched via API. So, I am looking for alternatives first.