I would like to install the Spark 4.0 preview using the jupyter/pyspark-notebook as the base.

The docker stacks documentation suggests that I can override the Spark version. I’ve tried this in my Docker file but it’s still installing Spark 3.5.

FROM quay.io/jupyter/pyspark-notebook
ARG spark_version=4.0.0-preview1
ARG spark_download_url="https://archive.apache.org/dist/spark/"

Looking for some guidance on how to achieve what I want to do.