The end of Python 2 is near. (The end might be a bit later though.) The latest release of Apache Spark (2.4.4) still uses Python 2 by default. Configuring PySpark to run with Python 3 and IPython is pretty straightforward.
export PYSPARK_PYTHON=python3 export PYSPARK_DRIVER_PYTHON=ipython3 bin/pyspark
and everything should be set up.