Threaded Tasks in PySpark Jobs

Threaded Tasks in PySpark Jobs

There are circumstances when tasks (Spark action, e.g. save, count, etc) in a PySpark job can be spawned on separate threads. Doing so, optimizes distribution of tasks on executor cores.

1 Like