Spark can't find Python module

Question

I'm trying to run the following Python script locally, using spark-submit command:

import sys
sys.path.insert(0, '.')
from pyspark import SparkContext, SparkConf
from commons.Utils import Utils

def splitComma(line):
    splits = Utils.COMMA_DELIMITER.split(line)
    return "{}, {}".format(splits[1], splits[2])

if __name__ == "__main__":
    conf = SparkConf().setAppName("airports").setMaster("local[2]")
    sc = SparkContext(conf = conf)

    airports = sc.textFile("in/airports.text")
    airportsInUSA = airports\
    .filter(lambda line : Utils.COMMA_DELIMITER.split(line)[3] == "\"United States\"")

    airportsNameAndCityNames = airportsInUSA.map(splitComma)
    airportsNameAndCityNames.saveAsTextFile("out/airports_in_usa.text")

The command used (while inside the project directory):

spark-submit rdd/AirportsInUsaSolution.py

I keep getting this error:

Traceback (most recent call last): File "/home/gustavo/Documentos/TCC/python_spark_yt/python-spark-tutorial/rdd/AirportsInUsaSolution.py", line 4, in from commons.Utils import Utils ImportError: No module named commons.Utils

Even though there is a commons.Utils with a Utils class.

It seems that the only imports it accepts are the ones from Spark, because this error persists when I try to import any other class or file from my project.

pvy4917 · Accepted Answer · 2018-10-17 18:08:23Z

3

from pyspark import SparkContext, SparkConf

def splitComma(line):
    splits = Utils.COMMA_DELIMITER.split(line)
    return "{}, {}".format(splits[1], splits[2])

if __name__ == "__main__":
    conf = SparkConf().setAppName("airports").setMaster("local[2]")
    sc = SparkContext(conf = conf)

    sc.addPyFile('.../pathto commons.zip')
    from commons import Utils

    airports = sc.textFile("in/airports.text")
    airportsInUSA = airports\
    .filter(lambda line : Utils.COMMA_DELIMITER.split(line)[3] == "\"United States\"")

    airportsNameAndCityNames = airportsInUSA.map(splitComma)
    airportsNameAndCityNames.saveAsTextFile("out/airports_in_usa.text")

Yes, it only accepts the ones from the Spark. You can zip the required files (Utils, numpy) etc and specify the parameter --py-files in the spark-submit.

spark-submit  --py-files rdd/file.zip rdd/AirportsInUsaSolution.py

edited Oct 17, 2018 at 18:08

answered Oct 16, 2018 at 19:32

pvy4917

1,83820 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Anonymous · Accepted Answer · 2019-08-08 23:28:58Z

0

for python to consider a directory as package you need to create __init__.py in that directory. The __init__.py file doesn't need to contain anything.

In this case once you create __init__.py in the commons directory you will be able to access that package.

answered Aug 8, 2019 at 23:28

Anonymous

1

Comments

Andrey Ershov · Accepted Answer · 2024-11-05 08:25:39Z

0

I think problem is in SRARK configuration. Add, pls, PYSPARK_PYTHON environment variable in your ~/.bashrc. In my case, it looks like : export PYSPARK_PYTHON =/home/comrade/environments/spark/bin/python3, where PYSPARK_PYTHON is path to my python executable in "spark" environment.1

Hope, it helps)

edited Nov 5, 2024 at 8:25

answered Nov 5, 2024 at 8:25

Andrey Ershov

12 bronze badges

Comments

A. Traoré · Accepted Answer · 2019-11-05 13:18:17Z

-1

Create a python script named: Utils.py which will contain:

import re

class Utils():

    COMMA_DELIMITER = re.compile(''',(?=(?:[^"]*"[^"]*")*[^"]*$)''')

Put this Utils.py python script on a commons folder and put this folder in your working directory (type pwd to know it). You can then import the Utils class:

from commons.Utils import Utils

Hope it will help you.

edited Nov 5, 2019 at 13:18

answered Nov 5, 2019 at 13:10

A. Traoré

767 bronze badges

Collectives™ on Stack Overflow

Spark can't find Python module

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related