Creating Datafusion's Dataframe from Vec<Struct> in Rust?

Question

I am trying to do something similar to this question here but instead of using the polars library, I will like to use the Datafusion library

The idea is to go from a vec of struct like this:

#[derive(Serialize)]
struct Test {
    id:u32,
    amount:u32
}

and save to Parquet files, just like in the question I referenced.

While it was possible using polars, as seen in the accepted answer to achieve this by going from the Struct, serialise to JSON and then build the Dataframe from that, I could not find similar approach using Datafusion.

Any suggestions will be appreciated.

user2237975 · Accepted Answer · 2023-01-15 12:29:09Z

1

I think the parquet_derive is designed exactly for the usecase of writing Rust structs to/from Parquet files. DataFusion would be useful if you wanted to process the resulting data, for example filtering or aggregating it with SQL

Here is an example in the docs: https://docs.rs/parquet_derive/30.0.1/parquet_derive/derive.ParquetRecordWriter.html

answered Jan 15, 2023 at 12:29

user2237975

262 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dade Over a year ago

So basically use the correct tool to generate the parquet files…use data fusion to query

Lan · Accepted Answer · 2025-07-29 05:28:09Z

-1

user2237975 pointed to an example in the docs for writing a vector to a parquet file with parquet_derive. The version is outdated, but it's the same example. The latest link is here: https://docs.rs/parquet_derive/latest/parquet_derive/derive.ParquetRecordWriter.html

However, it does not work as is. I opened an issue and a PR to fix that here: https://github.com/apache/arrow-rs/issues/7732

To quickly test this example, you can also run:

git clone https://github.com/LanHikari22/rs_repro.git
cd rs_repro
cargo run --features "repro001"

Note that this is still limited and does not provide enum support for example. Only structs at the moment.

Optional context: tech journal.

edited Jul 29 at 5:28

answered Jun 21 at 18:22

Lan

134 bronze badges

Collectives™ on Stack Overflow

Creating Datafusion's Dataframe from Vec<Struct> in Rust?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related