I have just started working on big data and I’m struggling with Apache Spark and Delta Lake.

I’ve installed apache spark on my local machine, and now when I am tring to create a delta lake table it shows error. I can’t really understand why is it happening when I’ve exactly installed everything as per the instructions mentioned in delta lake documentation.

I am just using this code as an example from delta lake docs to create delta lake table delta lake table

import pyspark
from delta import *

builder = pyspark.sql.SparkSession.builder.appName("Testing") 
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") 
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")

spark = configure_spark_with_delta_pip(builder).getOrCreate()

data = spark.range(0, 5)

data.write.format("delta").save("testing")

but it shows error, and I don’t understand why is it happening. Below is the error

Py4JJavaError: An error occurred while calling o42.save.
: com.google.common.util.concurrent.ExecutionError: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

Also I see this when I run spark-shell command in terminal. Here is what I see in terminal

C:Usersuser>spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at {ip address}
Spark context available as 'sc' (master = local[*], app id = local-1722176431791).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _ / _ / _ `/ __/  '_/
   /___/ .__/_,_/_/ /_/_   version 3.5.1
      /_/

Using Scala version 2.12.18 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_421)
Type in expressions to have them evaluated.
Type :help for more information.

I think this line ‘NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable’ might be causing issue to create delta lake table. I am not sure though. Can anyone help me with this?

I was expecting a delta table