According to python's GIL we cannot use threading in CPU bound processes so my question is how does Apache Spark utilize python in multi-core environment?
Add a comment
|
1 Answer
Multi-threading python issues are separated from Apache Spark internals. Parallelism on Spark is dealt with inside the JVM.
And the reason is that in the Python driver program, SparkContext uses Py4J to launch a JVM and create a JavaSparkContext.
Py4J is only used on the driver for local communication between the Python and Java SparkContext objects; large data transfers are performed through a different mechanism.
RDD transformations in Python are mapped to transformations on PythonRDD objects in Java. On remote worker machines, PythonRDD objects launch Python sub-processes and communicate with them using pipes, sending the user's code and the data to be processed.
PS: I'm not sure if this actually answers your question completely.
9 Comments
zero323
It think that the main point here is that PySpark doesn't use multi-threading so GIL is simply not an issue.
Vahid Hashemi
@zero323 can you elaborate your comment?
zero323
There is not much to elaborate. Excluding tests there are only a few places where PySpark is using threads to perform some secondary tasks like starting external process. Everything else it just a good old single threaded processing.
eliasah
I concur with @zero323 that's why I said all the parallel processing is dealt with inside the JVM.
zero323
@eliasah To be fair JVM part is not that heavy on multithreading either, don't you think? There are multiple threads required for housekeeping and JVM executors use threads but in practice it is not really required to achieve parallelism in Spark. One could start equivalent number of workers on each machine and get the same parallelism although at a higher price.
|