Duckdb release memory in python

Question

I've started to experiment with duckdb but have struggled to figure out, how we can release memory again. If I have a loop like below, I would have imagined that memory gets either freed after process is over or after con.close() was called. But on my system neither seems to be the case, if I look at the memory used by the process, there is still memory occupied (very likely by duckdb, as this is essentially all I'm doing. Unless deltalake or pyarrow have issues)

import duckdb
import deltalake
for src in ["a","b", "c"]:
        def process():
                con = duckdb.connect(":memory:")
                dt = deltalake.DeltaTable(f"s3a://{src}", storage_options=deltalake_storage_options)
                pa = dt.to_pyarrow_table()
                r1 = con.from_arrow(pa)
                duckdb.sql("select * from r1").write_parquet("/tmp/test.parquet")
        process()

As some of the tables I'm dealing with are very large, I cannot keep all of them in memory at the same time. So how would I be able to release all of the memory, that is allocated to duckdb?

Mause · Accepted Answer · 2023-11-24 13:07:17Z

0

You might have luck using

con.from_arrow(pa).write_parquet("/tmp/test.parquet")

As that doesn't use the "default" connection, as the methods directly on the module (eg, duckdb.sql) do

answered Nov 24, 2023 at 13:07

Mause

6481 gold badge5 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ian Gow · Accepted Answer · 2023-11-26 15:00:17Z

0

Would con.close() at the end of process() help?

answered Nov 26, 2023 at 15:00

Ian Gow

3,5552 gold badges28 silver badges34 bronze badges

2 Comments

Syrius Over a year ago

Unfortuantely not. I think I have the same problem that someone had reported here: github.com/duckdb/duckdb/issues/9033

Ian Gow Over a year ago

I had a similar issue and managed to get code that emitted a pyarrow.RecordBatchReader of smaller pyarrow tables that could be written to parquet one at a time. This dramatically reduced the memory footprint for me. Perhaps something similar could work for you. gist.github.com/iangow/2d8f7be06fea688ec9b84bc45c6c473a

Collectives™ on Stack Overflow

Duckdb release memory in python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related