【This is my code:】
from pyspark import SparkConf, SparkContext
if name == ‘main‘:
conf = SparkConf().setMaster(“local[*]”).setAppName(“WordCountHelloWorld”)
sc = SparkContext(conf=conf)
file_rdd = sc.textFile(“hdfs://node1:8020/input/words.txt”)
words_rdd = file_rdd.flatMap(lambda line: line.split(” “))
words_with_one_rdd = words_rdd.map(lambda x:(x,1))
result_rdd = words_with_one_rdd.reduceByKey(lambda a, b: a+b)
print(result_rdd.collect())`
【And this is error imformation:】
ssh://root@node1:22/export/server/anaconda3/envs/pyspark/bin/python -u /tmp/
pycharm_project_111/00_exampl/HelloWorld.py
/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyspark/bin/spark-class: line 71:
export/server/jdk/bin/java: No such file or directory
/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyspark/bin/spark-class: line 96: CMD:
bad array subscript
Traceback (most recent call last):
File “/tmp/pycharm_project_111/00_exampl/HelloWorld.py”, line 6, in
sc = SparkContext(conf=conf)
File “/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyspark/context.py”, line 144,
in init
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyspark/context.py", line 339,
in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/export/server/anaconda3/envs/pyspark/lib/python3.8/site-packages/pyspark/java_gateway.py", line
108, in launch_gateway
raise RuntimeError("Java gateway process exited before sending its port number ")
RuntimeError: Java gateway process exited before sending its port number
In fact, when I write Python code in PyCharm, if I need to import the SparkContext package, it throws this error. When I write simple code, such as ‘Hello World’, it runs fine.
Similarly, when I directly submit this Python file to the Spark cluster, it runs smoothly.[
Please help me. How can I solve it?
I expect to be able to output the result of the ‘wordcount’ code in PyCharm instead of encountering an error.
jiax zhang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.