I am attempting to use pyspark on sagemaker notebooks as described here:
https://github.com/aws/sagemaker-spark/tree/master/sagemaker-pyspark-sdk#local-spark-on-sagemaker-notebook-instances
The page reads:
Local Spark on SageMaker Notebook Instances
Create a notebook using the conda_python2 or conda_python3 Kernels. Then you can initialize a spark context the same way it is described in the QuickStart section:
I attempted to use either of the two kernels using the Data Science 3.0 image as well as others. None had the conda_python2 or conda_python3 Kernels described, and using the Python 3 Kernel gave a Java Home not found error, as I expected since I’m not connected to an EMR cluster
nww is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.