My machine was laggy while trying to read a 4GB of csv in jupyter notebook with chunksize option:
raw = pd.read_csv(csv_path, chunksize=10**6)
data = pd.concat(raw, ignore_index=True)
This takes forever to run and also freeze my machine (Ubuntu 16.04 with 16GB of RAM). What is the right way to do this?
Add a comment
|
1 Answer
The point of using chunk is that you don't need the whole dataset in memory at one time and you can process each chunk when you read the file. Assuming you don't need the whole dataset in memory at one time, you can do
chunksize = 10 ** 6
for chunk in pd.read_csv(filename, chunksize=chunksize):
do_something(chunk)
