I am using Windows 11 doing a test with pytest for a simple PySpark job.
Python used is 3.10.11 (also happens with 3.10.12 and 3.10.14).
PySpark used is 3.5.0. Java is 21.0.2.
It seems just a warning but on the pipeline it is treated as an error.

The error also happens on a Ubuntu container on Azure Agents and the pipeline never finishes:

24/07/24 14:46:04 WARN SparkEnv: Exception while deleting Spark temp dir: C:UsersmyuserAppDataLocalTempsparkspark-d5847c32userFiles-eff55070
java.io.IOException: Failed to delete: C:UsersmyuserAppDataLocalTempsparkspark-d5847c32userFiles-eff55070org.antlr_antlr4-runtime-4.9.3.jar
    at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:146)
    at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:117)
    at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:129)
    at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:117)
    at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:90)
    at org.apache.spark.util.SparkFileUtils.deleteRecursively(SparkFileUtils.scala:121)
    at org.apache.spark.util.SparkFileUtils.deleteRecursively$(SparkFileUtils.scala:120)
    at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1126)
    at org.apache.spark.SparkEnv.stop(SparkEnv.scala:108)
    at org.apache.spark.SparkContext.$anonfun$stop$25(SparkContext.scala:2310)
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1375)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:2310)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:2216)
    at org.apache.spark.api.java.JavaSparkContext.stop(JavaSparkContext.scala:550)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:75)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:52)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:1583)

the previous folders have all permissions.