Import Error Loading Module using Bash Script on EC2 Instance (AWS)

Question

I am trying to run a bash script through the user_data argument of the spot_fleet_request method. The script executes all the installations and connects to the filesystem, but when it runs the python file I need to run it gives me the following error:

Traceback (most recent call last):
  File "/home/ec2-user/efs/dir/create_lvl_output.py", line 8, in <module>
    from modeling import generators
ImportError: No module named modeling

Where modeling is a folder with a script I am importing "modeling/generators.py". I have attempted to assign the directory to the PYTHONPATH environmental variable as suggested in this post, here is my code:

#!/bin/bash
export PYTHONPATH=/home/ec2-user/efs/Day-Trader/day_trader/
echo $PYTHONPATH

...mount efs
...installs

python /home/ec2-user/efs/dir/create_lvl_output.py > levelset.txt

but it continues to fail with the same error. I tried this solution with each of the following variations and they all failed to solve the import issue.

echo "export PYTHONPATH=/home/ec2-user/efs/dir/" >> ~/.bashrc
. ~/.bashrc      #--- failed to import, set PYTHONPATH variable correctly     

echo "export PYTHONPATH=/home/ec2-user/efs/dir/" >> /home/ec2-user/.bashrc
. /home/ec2-user/.bashrc   #--- failed to import, set PYTHONPATH variable correctly but got following error as well: /home/ec2-user/.bashrc: line 2: /root/.dlamirc: No such file or directory

I also made sure that the script actually worked. If I copy and paste the script directly into the linux terminal when I've logged in a ec2-user (i.e. $HOME=/home/ec2-user/) and run perfectly and produced all the right output. So I thought that if I changed the user at the top of the bash script as suggested in this post it might work but the following commands still ran everything that followed as root:

su -l "ec2-user", su - ec2-user

and I tried running the python script using

su ec2-user -c 'python /home/ec2-user/efs/dir/create_lvl_output.py'

but it didn't work either. I'm out of ideas. Please help.

joshua.software.dev · Accepted Answer · 2020-07-11 05:21:03Z

3

I think the recommended method is of course to try to package your code as a proper module and install it prior with pip, either at the user/system level or a pythonenv style environment. This of course, can be highly difficult and time consuming depending on how much code you have to change to accommodate this, and its recommended that you try to do this at the beginning of making projects to avoid this headache.

Short of the doing the nontrivial amount of work that would be though, I actually have encountered this specific error you're having on Amazon's EC2 LTS ubuntu. No idea what causes it, but its like python just side steps the normal environment settings, and doesn't work as you expect. In my case, I just went for the dirty fix of:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('my/addition/to/pythonpath', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints my/addition/to/pythonpath:existing/things/on/pythonpath

so for your usecase, try:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('/home/ec2-user/efs/Day-Trader/day_trader/', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints /home/ec2-user/efs/Day-Trader/day_trader/:

you can of course do some pretty nasty things with this, such as:

python -c "import os; os.environ['PYTHONPATH'] = '/my/addition/to/pythonpath'; import mymodule; mymodule.doThing()'

if for some reason you dislike everyone who has the displeasure of working with this. Jokes aside, I really did use things like this in production, if you every figure out what the hell causes these env issues, I'd be interested to know even though I no longer work on the machines that had these issues.

answered Jul 11, 2020 at 5:21

joshua.software.dev

3071 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

seeiespi Over a year ago

Thanks, I made this post because I was trying everything I could to avoid making it a pip package. Fortunately, I have structured my project as a package so it shouldn't be to bad. Its comforting to know that its the wisest course of action.

seeiespi Over a year ago

Also, I laughed out loud at your joke. Using that solution was tempting but literally made me cringe. It did give me some ideas of some last things to try though

seeiespi Over a year ago

By the way I tried the dirty fix you suggested and it didn't work. I even tried changing the PATH variable and it didn't work.

joshua.software.dev Over a year ago

Darn, sorry to hear that. Not sure what could be causing it to reject proper path setups like this, if the dirty fix of fixing the paths within the program's environment doesn't work, then I'm not sure what will. I hope packaging your program works out.

Collectives™ on Stack Overflow

Import Error Loading Module using Bash Script on EC2 Instance (AWS)

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related