create dataframe from read many path files

Question

thanks for your time.

I need to read several file paths, which are divided into months and days (/mm/dd/*.json)

I've been trying to traverse the path associated with days, but my loop always sticks with the last read:

for i_dia in range(1, 9):
  df_json = spark.read.json('/mnt/datalake/'+Year+'/'+ Month +'/'+ str(0) + str(i_dia) +'/'+ '*', mode="PERMISSIVE",multiLine = "true")
  return df_json
 
display(df_json)

How should the correct reading be done? I want to read all files in only one big dataframe please.

From already thank you very much.

Regards

but my loop always sticks with the last read Can you clarify this part? What's going wrong? PS: Python range is not inclusive, so if you do range(1, 9) you will get 1 through 8. This may be the cause of your problem. — Nick ODell
– Nick ODell, Commented Mar 21, 2022 at 18:46
you are returning after the first file the way you indented this piece of code. So you only read one JSON file. That's clearly not what you want, but what do you want? Read all files and then? Append them all to one big dataframe? Please provide more information. — Mushroomator
– Mushroomator, Commented Mar 21, 2022 at 18:50
Thanks you for respond. I want to read all files in only one big dataframe please. — Gonza
– Gonza, Commented Mar 21, 2022 at 19:03

Y U · Accepted Answer · 2022-03-21 19:31:25Z

2

import pandas as pd
df_json=pd.DataFrame()
for i_dia in range(1, 9):
        df_json= pd.concat([df_json,pd.read_json(i_dia )])

answered Mar 21, 2022 at 19:31

Y U

1043 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gonza Over a year ago

Sorry for the late response, I was traveling. I managed to understand your logic and applied it to my need, thank you very much!!

Collectives™ on Stack Overflow

create dataframe from read many path files

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related