I am new to python world of coding . While I try to run existing pyspark job in my location windows maching using PyCharm community editioin IDE I am getting error “Py4JJavaError” as show below.

I am using python 3.10 version with pyspark 3.5.1 and spark-3.3.0-bin-hadoop3

My python script which is a pyspark code which reads as S3 data. But when run my job localling in IDE I am facing below error. Not sure what went wrong and how to fix it.

Below is the error trace :

  File "D:SoftwarespythonPython310libsite-packagespy4jprotocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o33.get.
: java.util.NoSuchElementException: spark.sql.execution.pythonUDF.arrow.enabled
        at org.apache.spark.sql.errors.QueryExecutionErrors$.noSuchElementExceptionError(QueryExecutionErrors.scala:1660)
        at org.apache.spark.sql.internal.SQLConf.$anonfun$getConfString$3(SQLConf.scala:4546)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:4546)
        at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:72)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.base/java.lang.Thread.run(Thread.java:1623)


Process finished with exit code 1