I have some problem, which occurs when executing PySpark on yarn.
The following error is reported when processing rdd: WARN scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, node03, executor 3): java.io.IOException: Cannot run program “-home-user-anaconda3-bin-python3”: error=13, Permission denied
I have configured anaconda on all nodes and changed the anaconda directory to 777. I searched the entire web for solutions, but none of them worked.
This is my .bashrc file
if [ -f "/home/user/anaconda3/etc/profile.d/conda.sh" ]; then
. "/home/user/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/home/user/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop
export SPARK_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export PYSPARK_PYTHON=/home/user/anaconda3/bin/python3
export PYSPARK_DRIVER_PYTHON=/home/user/anaconda3/bin/python3
export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH
# JAVA
export JAVA_HOME=/usr/java/default
export PATH=$PATH:$JAVA_HOME/bin
# added by Anaconda3 installer
export PATH="/home/zhenyu/anaconda3/bin:$PATH"
#export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
#export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
this is my warn:
enter image description here
A code sheep is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1