create parquet file with Clickhouse and read with DuckDB

Question

Following this guide https://clickhouse.com/docs/knowledgebase/mysql-to-parquet-csv-json I've exported from a MySQL server some tables to parquet.

But I'm not able to read these parquet files with DuckDB.

I can inspect the structure:

DESCRIBE SELECT * FROM 'mytable.parquet';

but if I try to read:

select ID from mytable.parquet;
Error: Invalid Error: Unsupported compression codec "7". Supported options are uncompressed, gzip, snappy or zstd

I guess that clickhouse is writing LZ4 compressed parquet files, and duckdb doesn't support them. Can I change the compression format in clickhouse-local?

Pavel Kruglov · Accepted Answer · 2023-11-21 18:55:14Z

5

To change Parquet compression method in ClickHouse, use setting output_format_parquet_compression_method (see all Parquet settings in https://clickhouse.com/docs/en/sql-reference/formats#parquet-format-settings).

For example:

select ... format Parquet settings output_format_parquet_compression_method='snappy'

answered Nov 21, 2023 at 18:55

Pavel Kruglov

1762 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Denny Crane · Accepted Answer · 2023-11-21 18:59:21Z

1

--output_format_parquet_compression_method Compression method for Parquet output format. Supported codecs: snappy, lz4, brotli, zstd, gzip, none (uncompressed)

try output_format_parquet_compression_method='snappy'

clickhouse-client -q "select * from numbers(1e6)  settings
 output_format_parquet_compression_method='snappy' format Parquet " > test.parquet

answered Nov 21, 2023 at 18:59

Denny Crane

13.7k2 gold badges25 silver badges45 bronze badges

Collectives™ on Stack Overflow

create parquet file with Clickhouse and read with DuckDB

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related