How to write a JSONL file in Python

Question

I am trying to write a .jsonl file that needs to look like this:

{"file_name": "0001.png", "text": "This is a golden retriever playing with a ball"}
{"file_name": "0002.png", "text": "A german shepherd"}
{"file_name": "0003.png", "text": "One chihuahua"}

This is my attempt:

import json
import pandas as pd

dt = pd.read_csv('data.csv')
df = pd.DataFrame(dt)

file_name = df['image']
file_caption = df['text']

data = []

for i in range(len(file_name)):
    entry = {"file_name": file_name[i], "text": file_caption[i]}
    data.append(entry)

json_object = json.dumps(data, indent=4)

# Writing to sample.json
with open("metadata.jsonl", "w") as outfile:
    outfile.write(json_object)

But this is the output I get:

[
    {
        "file_name": "images/image_0.jpg",
        "text": "Fattoush Salad with Roasted Potatoes"
    },
    {
        "file_name": "images/image_1.jpg",
        "text": "an analysis of self portrayal in novels by virginia woolf A room of one's own study guide contains a biography of virginia woolf, literature essays, quiz questions, major themes, characters, and a full summary and analysis about a room of one's own a room of one's own summary."
    },
    {
        "file_name": "images/image_2.jpg",
        "text": "Christmas Comes Early to U.K. Weekly Home Entertainment Chart"
    },
    {
        "file_name": "images/image_3.jpg",
        "text": "Amy Garcia Wikipedia a legacy of reform: dorothea dix (1802\u20131887) | states of"
    },
    {
        "file_name": "images/image_4.jpg",
        "text": "3D Metal Cornish Harbour Painting"
    },
    {
        "file_name": "images/image_5.jpg",
        "text": "\"In this undated photo provided by the New York City Ballet, Robert Fairchild performs in \"\"In Creases\"\" by choreographer Justin Peck which is being performed by the New York City Ballet in New York. (AP Photo/New York City Ballet, Paul Kolnik)\""
    },
...
]

I know that its because I am dumping a list so I know where I'm going wrong but how do I create a .jsonl file like the format above?

Mark Tolonen · Accepted Answer · 2022-10-30 18:43:20Z

4

Don't indent the generated JSON and don't append it to a list. Just write out each line to the file:

import json
import pandas as pd

df = pd.DataFrame([['0001.png', "This is a golden retriever playing with a ball"],
                   ['0002.png', "A german shepherd"],
                   ['0003.png', "One chihuahua"]], columns=['filename','text'])

with open("metadata.jsonl", "w") as outfile:
    for file, caption in zip(df['filename'], df['text']):
        entry = {"file_name": file, "text": caption}
        print(json.dumps(entry), file=outfile)

Output:

{"file_name": "0001.png", "text": "This is a golden retriever playing with a ball"}
{"file_name": "0002.png", "text": "A german shepherd"}
{"file_name": "0003.png", "text": "One chihuahua"}

edited Oct 30, 2022 at 18:43

answered Oct 30, 2022 at 18:36

Mark Tolonen

181k26 gold badges182 silver badges278 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

mchd Over a year ago

I got way more than 3 lines. Its about a few thousand so I can't just write all of them out.

Mark Tolonen Over a year ago

@oo92 Sure you can, if they are already in the dataframe. I typed them as an example since you didn't provide any sample input file.

Mark Tolonen Over a year ago

@oo92 You can also use df.to_json('metadata.jsonl', orient='records', lines=True) to do it in one step.

Collectives™ on Stack Overflow

How to write a JSONL file in Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related