How to load many CSV files in a folder using Python?

Question

I am new to python code and started to notice this pickle function in python. I am trying to load all (50) csv files in a folder and save it as pickle files. The csv files might contain same or different column names too. Any suggestions on how to approach that.

Welcome to StackOverflow! Please always provide a minimal reproducible example and be sure to read stackoverflow.com/help/how-to-ask — Pitto
– Pitto, Commented Jan 16, 2020 at 14:18
You need to attempt it yourself, and then ask questions about specific problems you are having. At the moment you are saying "I don't know where to start". If this is really true, you need to read a tutorial. — Open AI - Opting Out
– Open AI - Opting Out, Commented Jan 16, 2020 at 14:20
where are the files currently? If as I understand you want to move all csv files into one directory, then look into os.walk, os.mkdir, and shutil. — Ahmet
– Ahmet, Commented Jan 16, 2020 at 14:21
@Pitto "Not all questions benefit from including code." Source People seem to forget to read SO's help center. We are here to help, no? — Raphael
– Raphael, Commented Jan 16, 2020 at 14:23
While this is true @Raphael don't forget that SO is not a place where people do your homeworks or do your job in your place. Writing code may not be mandatory but it will lead you easily to the downvote / question close path (like you see it is already happening for this question). Please read this, for example: idownvotedbecau.se/noattempt and also read this: meta.stackoverflow.com/questions/334822/… — Pitto
– Pitto, Commented Jan 16, 2020 at 14:34

AnarchistGeek · Accepted Answer · 2020-01-16 14:29:39Z

1

You can try something like this:

import glob, os
import pandas as pd
import pickle

os.chdir(r"path/to/folder")
df_list = []
for file in glob.glob("*.csv"):
    df = pd.read_csv(file)
    df_list.append(df)

with open(r'\df_list.pickle', 'wb') as handle:
    pickle.dump(df_list, handle, protocol=pickle.HIGHEST_PROTOCOL)

edited Jan 16, 2020 at 14:29

AnarchistGeek

3,1987 gold badges35 silver badges47 bronze badges

answered Jan 16, 2020 at 14:24

gbruenjes

4,2251 gold badge18 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to load many CSV files in a folder using Python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related