Description
Spark Architecture was one of the toughest elements to grasp when initially learning about Spark. I think one of the main reasons is that there is a vast amount of information out there, but nothing…
Summary
- Apache Spark — Multi-part Series: Driver Node and Worker Node Architecture (created by Luke Thorp) Worker nodes are able to communicate and pass data between each other but in regards to work and tasks, the Driver node is solely responsible for providing Workers with jobs to complete.
- Cluster Task Assignment (created by Luke Thorp) Once this is done and resources have been allocated, tasks are distributed to the worker nodes (executors) who have free time, and the driver program monitors the progress.
- For example, there are tabs for Jobs and Stages.