AWS Spark EMR Numpy Import Error

Question

I'm trying to submit a Python script on AWS EMR that imports numpy but I get

ImportError: No module named numpy

I tried using one of the answers here: No module named numpy when spark-submitting. I created a bootstrap_actions.sh script that includes

 sudo yum install python-numpy python-scipy -y

and I run the script when I create the cluster but still get the import error. Any solution on how can I get import numpy to work?

John Hanley · Accepted Answer · 2017-12-10 21:12:38Z

2

For Amazon EMR you need to use bootstrap actions. Installing from the console only changes the master node and not the task nodes.

runners:
  emr:
    bootstrap:
    - sudo yum install -y python27-numpy

I am assuming that you will be using Python 2.7. If you are using Python 3.x, the link below has examples installing with PIP in the bootstrap. I am also assuming that you are using a recent EMR AMI.

EMR Bootstrapping Cookbook

answered Dec 10, 2017 at 21:12

John Hanley

81.8k7 gold badges115 silver badges181 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jakko Over a year ago

It works! I just changed my bootstrap script to include the line you had: sudo yum install -y python27-numpy - Thanks!

Collectives™ on Stack Overflow

AWS Spark EMR Numpy Import Error

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related