Windows 11
Spark -> C:spark-3.0.0-bin-hadoop2.7
Winutils -> C:winutils
Java -> C:Program FilesJavajdk-22.0.1
I downloaded the above versions, created env variables,
HADOOP_HOME =’c:spark-3.0.0-bin-hadoop2.7′ ,
JAVA_HOME = ‘c:winutils’,
SPARK_HOME = ‘c:Program FilesJavajdk-22.0.1’
and added these paths to both system and user variables
I have checked the env and sure they are available in path (both system and user). I am facing following issue while running spark shell:
C:spark-3.0.0-bin-hadoop2.7bin>spark-shell
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:54)
at org.apache.spark.internal.config.package$.<init>(package.scala:1006)
at org.apache.spark.internal.config.package$.<clinit>(package.scala) at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$loadEnvironmentArguments$3(SparkSubmitArguments.scala:157)
at scala.Option.orElse(Option.scala:447)
at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:157)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:115)
at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:990)
at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:990)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:62)
... 13 more
Caused by: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at java.base/java.lang.Class.getConstructor0(Class.java:3784)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2955)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:55)
... 13 more
New contributor
sai krishna ragula is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.