Setup local Spark Cluster on Mac and explore UI for Master, Workers, and Jobs(Python using Jupyter notebook)

What is covered

  • In this blog, I would describe my experiments about setting up a local (standalone) Spark cluster on Mac M1 machine.
  • Would start jupyter notebook using anaconda navigator
  • Submit jobs using Python code executing in the notebook with jobs submitted to the master created above.
  • Explore the UI for:
    • Master
    • Worker
    • Jobs (transformation and action operations, Directed Acyclic Graph)

Why local (standalone) server

  • local server is a simple deployment model and is possible to run the daemons in a single node
  • Jobs submitted using pyspark are explored using the same UI in previous blog
  • Jobs submitted using python programs are explored in this blog

Start the master and worker

  • After installation of pyspark (using homebrew), scripts for starting the master and worker were available at the following location:
cd /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/sbin
  • start the server (note that would be specific to the machine)
% ./ 
starting org.apache.spark.deploy.master.Master, logging to /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/<your-local-logfile>.out
  • check the log file for the spark master server details ( is obtained from the previous step>
% tail -10 /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/<your-local-logfile>.out
22/05/18 07:41:57 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
22/05/18 07:41:57 INFO Master: Starting Spark master at spark://<your-spark-master>:7077
22/05/18 07:41:57 INFO Master: Running Spark version 3.2.1
22/05/18 07:41:57 INFO Utils: Successfully started service 'MasterUI' on port 8080.
  • check the master UI
  • notice that workers are zero, since, we have not yet started any

Screen Shot 2022-05-18 at 7.44.38 AM.png

  • start the worker by selecting the number of cores and memory based on your system configuration.
  • please check logfile in the previous step and replace with the appropriate value for
 % ./ --cores 2 --memory 2G spark://<your-spark-master>:7077
starting org.apache.spark.deploy.worker.Worker, logging to /opt/homebrew/Cellar/apache-spark/3.2.1/libexec/logs/<your-local-worker-logfile>.out
  • check the master UI for worker information (now it shows 1 worker)

Screen Shot 2022-05-18 at 7.48.16 AM.png

  • also, check the worker UI for the cores and memory allocated

Screen Shot 2022-05-18 at 7.50.01 AM.png

submit jobs using python in jupyter notebook and observe jobs UI

  • started jupyter notebook using anaconda navigator
  • write python code to submit jobs to the master setup above
  • operations are performed on RDDs (Resilient Distributed Dataset)
  • spark has two kinds of operations:
    • Transformation → operations such as map, filter, join or union that are performed on an RDD that yields a new RDD containing the result
    • Action → operations such as reduce, first, count that return a value after running a computation on an RDD
  • Actions could be displayed as jobs in the UI and transformations could be observed while exploring the DAG (Directed Acyclyic Graph) output

  • started jupyter notebook from anaconda navigator

Screen Shot 2022-05-27 at 10.51.46 PM.png

  • create Spark Application in notebook (replace with your-master)
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("<your-master>") \
                    .appName('SparkTestPython') \

print("First SparkContext:");
print("APP Name :"+spark.sparkContext.appName);
print("Master :"+spark.sparkContext.master);

Screen Shot 2022-05-27 at 11.50.44 PM.png

  • Now observe the master and worker that the Application is listed

Screen Shot 2022-05-27 at 11.51.11 PM.png

Screen Shot 2022-05-27 at 11.55.46 PM.png

  • we will write a simple code in the notebook to do one transformation (parallelize) and one action (count) operation

Screen Shot 2022-05-27 at 11.54.09 PM.png

  • From the master, click on the application to check the application UI:
  • In the application UI, click on "Application Detail UI" to get details on the jobs

Screen Shot 2022-05-27 at 11.54.59 PM.png

  • to see DAG (Directed Acyclic Graph) visualization click on the job (under the Description column)
  • this is a simple DAG with only one transformation and one operation

Screen Shot 2022-05-27 at 11.56.52 PM.png

  • we will write another simple code in the notebook to do two transformations (read from file and map) and one action (collect) operation
x=spark.sparkContext.textFile('/tmp/sparktest.txt') n: n.upper())

Screen Shot 2022-05-28 at 12.04.02 AM.png

  • Check the jobs UI

Screen Shot 2022-05-28 at 12.06.04 AM.png

Next steps

  • try out additional spark operations, and actions, and check additional details like storage, etc. in the UI
  • using data frames