How to install pyspark.pandas in Apache Spark?

Question

I downloaded Apache Spark 3.3.0 bundle which contains pyspark

$ pyspark

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.3.0
      /_/

Using Python version 3.7.10 (default, Jun  3 2021 00:02:01)
Spark context Web UI available at http://XXX-XXX-XXX-XXXX.compute.internal:4041
Spark context available as 'sc' (master = local[*], app id = local-1669908157343).
SparkSession available as 'spark'.
**>>> import pyspark.pandas as ps**
Traceback (most recent call last):
  File "/home/ec2-user/bin/spark/latest/python/pyspark/sql/pandas/utils.py", line 27, in require_minimum_pandas_version
    import pandas
ModuleNotFoundError: No module named 'pandas'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/bin/spark/latest/python/pyspark/pandas/__init__.py", line 31, in <module>
    require_minimum_pandas_version()
  File "/home/ec2-user/bin/spark/latest/python/pyspark/sql/pandas/utils.py", line 36, in require_minimum_pandas_version
    ) from raised_error
ImportError: Pandas >= 1.0.5 must be installed; however, it was not found.

How do I import python packages inside Apache-Spark in custom directory like /home/ec2-user/bin/spark/latest/python/pyspark?

I also tried: $ pip install pandas -bash: pip: command not found

If I try to install pip, how can ensure the libraries are compatible with the Python version 3.7.20 in Spark?

Bartosz Gajda · Accepted Answer · 2022-12-01 17:16:31Z

1

Have you tried installing Pandas in the following way:

pip install pyspark[pandas_on_spark]

If the pip is not discoverable by bash, maybe try to active your Python environment first (whether virtualenv, conda or anything else).

answered Dec 1, 2022 at 17:16

Bartosz Gajda

1,1877 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to install pyspark.pandas in Apache Spark?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related