How to have Python find files in a subfolder without hard-coding path names?

Question

I use Git VC for my Python codes, mostly scripts with scientific calculations. To parse raw data (which are also part of the repository) into Python variables, I do:

import numpy as np
t_x_q_obs = np.genfromtxt('MeasuredAlgebrProductionRate_30min_18h.csv', delimiter=';')

Meanwhile, the repo has grown to contain > 100 files. So I would like to group certain kind of files, especially *.csv files with raw data, in a subfolder. No problem with Git, but Python would not find them anymore. In Matlab, I would just tell people to add the whole folder+subfolders to their pathdef.m, but since Python programmers use a lot of different editors/IDEs, there is no universal how-to. I would like my scripts to be executable OOTB by anyone cloning my repo and running the scripts from the command line or his/her favorite IDE. If I added those subfolders to the PYTHONPATH variable, this would not be universal either. Also, I would not like to hard-code the relative paths in all scripts.

Is there an elegant way to have Python look for external files which are part of the same folder structure?

André Müller · Accepted Answer · 2019-09-20 08:34:40Z

1

You could improve your code by using paths relative to your main python file like this:

import os
base_path = os.path.dirname(os.path.abspath(__file__))
my_file = os.path.join(base_path, 'my_data_file.csv')
# now my_file is referring to a path relative to your python script

answered Sep 20, 2019 at 8:34

André Müller

1077 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

winkmal Over a year ago

While this would work, it would coerce me to rewrite all parse commands in all of my scripts! I was hoping to avoid that, if possible. If not, I will probably go down your route.

André Müller Over a year ago

What if you would provide a bash script in your repository which sets the Python working path relative to the checkout position of the git repo before calling? Alternatively using os.chdir(path) might be a possibility for setting the current working director. Sure - if you have a bunch of absolute paths in your code... then you really need to change that!

winkmal Over a year ago

Well, I don't have "absolute paths" - I just don't have pathnames at all, just filenames! And in Matlab, this works even when the files are located in subfolders, due to the pathdef.m. (Sometimes, I check their existence by if exist(parametersFilename, 'file').) Also, wouldn't "bash script" imply that people used Linux/Mac OS? This will probably not work for the average Windows user.

Collectives™ on Stack Overflow

How to have Python find files in a subfolder without hard-coding path names?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related