Why Iceberg load is creating many folders in s3?

Question

I am writing data into s3 and table format is Iceberg in Glue Catalog. I see the /data and /metadata folders are getting created. However when I am writing data, it's creating 001/002 kind of folders. Is there any way I can keep the table/partition

Here is the code and folder strcture.

result_df.writeTo(catalog_table_name) \
                .tableProperty("write.format.default", "parquet") \
                .tableProperty("write.distribution-mode", "hash") \
                .tableProperty("format-version", "2") \
                .tableProperty("write.merge.mode", "copy-on-write") \
                .tableProperty("write.object-storage.path-style.enabled", "true")\
                .partitionedBy("srce", "regon", "coury", "datte") \
                .overwritePartitions()

HariUserX · Accepted Answer · 2025-07-24 00:43:21Z

0

Why do you want to keep it like table/partition? In Iceberg table format, the folder structure doesn't matter.

The write config write.object-storage.enabled is responsible for adding hash to file paths. See: https://iceberg.apache.org/docs/latest/configuration/#write-properties. It is actually helpful in S3 as S3 throttles the requests based on the prefixes. You can set it to false if you really don't need prefixes for some reason. The default is false but the cloud provider might be setting it to true.

In your case, I see

.tableProperty("write.object-storage.path-style.enabled", "true")\

which might be a provider specific iceberg config as it is not public.

answered Jul 24 at 0:43

HariUserX

1,3341 gold badge10 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why Iceberg load is creating many folders in s3?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related