Creating sparkContext on Google Colab gives: `RuntimeError: Java gateway process exited before sending its port number`

Question

Following are the dependencies, which got installed successfully.

!apt-get install openjdk-8-jre
!apt-get install scala
!pip install py4j
!wget -q https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz
!tar xf spark-2.4.8-bin-hadoop2.7.tgz
!pip install -q findspark

Now to create the spark context:

# Setting up environment variables
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.8-bin-hadoop2.7"
# export PYSPARK_SUBMIT_ARGS ="--master local[2]"

# Importing and initating spark
import findspark
findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("Test Setup").getOrCreate()
sc = spark.sparkContext

I'm getting this error:

RuntimeError: Java gateway process exited before sending its port number

Please note that this is a colab notebook. Any kind of help would be great.

Does this answer your question? Exception: Java gateway process exited before sending the driver its port number while creating a Spark Session in Python — user17173390
– user17173390, Commented Jan 1, 2022 at 11:34
@CodingLive this issue is resolved on local machine, whereas I am using pyspark on Google Colab. — Hira Tanveer
– Hira Tanveer, Commented Jan 1, 2022 at 11:43

blackbishop · Accepted Answer · 2022-01-01 14:36:55Z

4

You can install Pyspark using PyPI as an alternative:

For Python users, PySpark also provides pip installation from PyPI. This is usually for local usage or as a client to connect to a cluster instead of setting up a cluster itself.

Install pyspark + openjdk

%pip install pyspark==2.4.8
!apt-get install openjdk-8-jdk-headless -qq > /dev/null

Create spark session

from pyspark.sql import SparkSession

spark = SparkSession.builder\
        .master("local[*]")\
        .appName("Test Setup")\
        .getOrCreate()

Tested in Google Colab Notebook:

answered Jan 1, 2022 at 14:36

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Creating sparkContext on Google Colab gives: `RuntimeError: Java gateway process exited before sending its port number`

1 Answer 1

Install pyspark + openjdk

Create spark session

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Install pyspark + openjdk

Create spark session

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related