Failing to create a table using pyiceberg and Aws Glue

Question

I'm trying to create new Iceberg table in AWS Glue:

glue_database_name = "alex_iceberg_test_db"
glue_catalog_uri = "s3://alex-iceberg-test-storage"
my_namespace = 'alex_db'


# Instantiate glue catalog
catalog = load_catalog("glue", **{"type": "glue", 
                                  "s3.region": "region", 
                                  "profile": "ALEX",
                                  "s3.access-key-id": "XXX",
                                  "s3.secret-access-key": "YYY"})


# Define the Iceberg schema
schema = Schema(
    NestedField(field_id=1, name="data", field_type=StringType(), required=False)        
)


## Create the Iceberg table using the Iceberg catalog
table_name = 'test_1'
catalog.create_table(
    identifier=f'{my_namespace}.{table_name}',
    location=glue_catalog_uri,
    schema=schema
)

and get the following error

OSError: When getting information for key 'metadata/00000-5df640cc-b47c-4b39-b578-07113565dab5.metadata.json' in bucket 'alex-iceberg-test-storage': AWS Error ACCESS_DENIED during HeadObject operation: No response body.

Inside alex-iceberg-test-storage I do have metadata but the 00000-5df640cc-b47c-4b39-b578-07113565dab5.metadata.json is not there.

Why does it searching for that file and who should create it?

are you using Boto3?

Bogdan
– Bogdan

2024-05-02 19:52:40 +00:00
Commented May 2, 2024 at 19:52 — Bogdan
– Bogdan, Commented May 2, 2024 at 19:52

Alex L · Accepted Answer · 2024-05-05 11:18:54Z

0

thanks for all the helpers! eventually what worked is to set the correct credentials in ~/.aws/credentials and then

catalog = load_catalog("glue", **{"type": "glue", 
                                  "s3.region": "us-west-2", 
                                  "profile": "my_profile
                                  })

is working correctly (even without the s3.access-key-id and s3.secret-access-key)

answered May 5, 2024 at 11:18

Alex L

1,0876 gold badges20 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Failing to create a table using pyiceberg and Aws Glue

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related