How to select specific csv files from a folder in python?

Question

this is a 2 part question. I have a folder (existing in the same directory as the python script) with a lot of csv files and I want to read only specific csv files from the folder into python in order to later merge it into one data frame.

I do not want to read all the csv files from the folder, just the csv files that have the substring "BANK_NIFTY_5MINs*". Can someone please help on how I can do that?

The files I want to read are named as:

BANK_NIFTY_5MINs_2020-01-01.csv, BANK_NIFTY_5MINs_2020-01-02.csv, BANK_NIFTY_5MINs_2020-01-03.csv and so on.

I have tried with the following code but it does not work. 'files' variable does not store anything. (The code works if the folder consists of only the files I want to work with - But that is not what I want):

import glob
import pandas as pd

path = 'C:/Users/User1/Desktop/Test/2020/'
files = sorted(glob.glob(path + "/BANK_NIFTY_5MINs*.csv"),reverse=True)

#Code to merge the files
df = pd.merge(
    read_csv(files.pop()),
    read_csv(files.pop()),
    left_index=True, 
    right_index=True
)
while files:
    df = pd.merge(
        df, 
        read_csv(files.pop()), 
        left_index=True, 
        right_index=True,
        how='outer'
    )

Also, is there a way I can access the folder without specifying the absolute path of the directory? So that it runs on another system without having to change the path.

Any help would be appreciated. Thank you in advance.

Maybe first use print() (and print(type(...)), print(len(...)), etc.) to see which part of code is executed and what you really have in variables. It is called "print debuging" and it helps to see what code is really doing. — furas
– furas, Commented Oct 26, 2022 at 0:29
maybe first check os.listdir("'C:/Users/User1/Desktop/Test/2020/") to see what you really have in folder. Maybe files have little different name. OR maybe you should use r"C:\...\..." instead of "C:/.../...". BTW: when you add ".../" + "/..." then you get ...//... and this can also makes problem. — furas
– furas, Commented Oct 26, 2022 at 0:30

handyman4624 · Accepted Answer · 2022-10-25 23:32:54Z

1

If you want to use relative path:

import os

currentDir = os.path.dirname(__file__) 
absolutePath = os.path.join(currentDir, "the/relative/path")

For the file name, RegEx should work fine, but maybe a simple solution like this will work better:

for file in os.listdir(currentDir):
    if filename.startswith("BANK_NIFTY_5MINs") and filename.endswith(".csv"):
        # do the thing that you want here

answered Oct 25, 2022 at 23:32

handyman4624

913 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to select specific csv files from a folder in python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related