The end of Python 2 is near. (The end might be a bit later though.) The latest release of Apache Spark (2.4.4) still uses Python 2 by default. Configuring PySpark to run with Python 3 and IPython is pretty straightforward.

Run

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=ipython3 
bin/pyspark

and everything should be set up.