I am a starter for big data. Now I’m trying to connect R to Spark using Sparklyr (1.8.6), and have the following error message I haven’t found answers in stackoverflow.
sc <- spark_connect(master = “local”,
-
version = "2.1",
-
config = conf)
Error in system2(file.path(spark_home, “bin”, “spark-submit”), “–version”, :
‘”C:UsersdellAppDataLocal/spark/spark-2.1.0-bin-hadoop2.7/bin/spark-submit”‘ not found
sc <- spark_connect(master = “local”)
- Using Spark: 2.4.3
Error in system2(file.path(spark_home, “bin”, “spark-submit”), “–version”, :
‘”C:UsersdellAppDataLocal/spark/spark-2.4.3-bin-hadoop2.7/bin/spark-submit”‘ not found
Any suggestions?
Notice that I only need to deal with local big data (2-3 GB) without overwhelming R memory, so solutions with EMR/YARN might be too complicated for me.
user25734083 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.