Hi,
I'm trying to save a dataframe to csv in using spark with the code below :
output.write
.format("csv")
.mode("overwrite")
.option("path","C:/Users/Hp/Desktop/DE/Spark/code/op/")
.partitionBy("age")
.save()
but i am getting “Py4JJavaError: An error occurred while calling o132.save.” this error
Py4JJavaError: An error occurred while calling o61.save. : java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)' at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1249) at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1454) at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:601) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:761) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334) at
and i am using these versions ( spark – 3.5.1 python- 3.8.10 java – 22.0.1 )
New contributor
Ria is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.