Create multiple dataframes from files in a folder

Question

I have a folder with a lot of files. I want to create a loop or function where I can read all the files and create corresponding data frames for each one of the files with the prefix format 'filename'+_df.

I have this code that reads everything and creates a large data frame from all the files in the folder, but I'm not sure how to modify it to create multiple data frames and use the filename as a prefix:

import os
import pandas as pd

path = os.path.join(os.getcwd(),'folder')

files = [os.path.join(path,i) for i in os.listdir(path) if os.path.isfile(os.path.join(path,i))]

df = pd.DataFrame()

for file in files:
    _df = pd.read_csv(file)
    df = df.append(_df)

If there's a better way to do this, that would be great as well.

you can store all the DataFrames into a list. Then concat all dataframes into one — Joe Ferndz
– Joe Ferndz, Commented Feb 16, 2021 at 6:15
I don't want to create a single data frame. I want to create a data frame per file — mlenthusiast
– mlenthusiast, Commented Feb 16, 2021 at 6:17

jezrael · Accepted Answer · 2021-02-16 06:26:31Z

1

You can create dictionary of DataFrames in dictionary comprehension:

dfs = {f'{file}_df': pd.read_csv(file) for file in files}

Then for each DataFrame select by key, e.g. file='myfile':

dfs['myfile_df']

Thank you @Abdul Niyas P M for idea:

from pathlib import Path
dfs = {f'{Path(file).stem}_df': pd.read_csv(file) for file in files}

Or:

import os
dfs = {f'{os.path.basename(file)}_df': pd.read_csv(file) for file in files}

edited Feb 16, 2021 at 6:26

answered Feb 16, 2021 at 6:17

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user459872 Over a year ago

In the OP the files seems like a list of abs paths. May be you can use Pathlib to get only the filename. dfs = {f'{Path(file).stem}_df': pd.read_csv(file) for file in files}

Collectives™ on Stack Overflow

Create multiple dataframes from files in a folder

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related