sqlite3.DataError: query string is too large

Question

I'm trying to analyze this wiktionary dump with is 5GB, but I'm getting the above error message in the title. Here is my code:

    import sqlite3
    conn = sqlite3.connect(':memory:')
    c = conn.cursor()
    sql_file = open(f'wiktionary_categories.sql',encoding = "ISO-8859-1")
    sql_as_string = sql_file.read()
    c.executescript(sql_as_string)

Is there some way I can use a with statement? Such as:

    with open(f'wiktionary_categories.sql',encoding = "ISO-8859-1") as sql_file:
        sql_as_string = sql_file.read()
        c.executescript(sql_as_string)

I just want to run through the file and pick out all of the words that belong to a certain category.

so you dont necessarily need to run all the statements but just get some words? — Bending Rodriguez
– Bending Rodriguez, Commented May 7, 2024 at 7:39
maybe this will help you? stackoverflow.com/questions/78039596/… had a similar problem in the past — Bending Rodriguez
– Bending Rodriguez, Commented May 7, 2024 at 7:40
sqlparse module might help to split into smaller queries pypi.org/project/sqlparse — yotheguitou
– yotheguitou, Commented May 7, 2024 at 14:07
Avoid calling read() on large files since you read all at once into a single string. Usually, you want to iterate across lines. See I/O docs. — Parfait
– Parfait, Commented May 7, 2024 at 22:54

bobsmith76 · Accepted Answer · 2024-05-07 21:44:11Z

0

It took me a while to figure out but the sql file is basically just a text file. You loop over the lines and then you execute the sql statements that you need. You can just open it with a with statement.

answered May 7, 2024 at 21:44

bobsmith76

3042 gold badges14 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

sqlite3.DataError: query string is too large

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related