Redshift copy command recursive scan

Question

Is it possible to copy all files under root directory/bucket

Example folder structure:

/2016/01/file.json
/2016/02/file.json
/2016/03/file.json
...

I've tried with the following command:

copy mytable
FROM 's3://mybucket/2016/*'
CREDENTIALS 'aws_access_key_id=<>;aws_secret_access_key=<>'
json 's3://mybucket/jsonpaths.json'

Ismael Rodrigues · Accepted Answer · 2023-03-07 23:20:01Z

6

Specify a prefix for the load, and all Amazon S3 objects with that prefix will be loaded (in parallel) into Amazon Redshift.

Examples:

copy mytable
FROM 's3://mybucket/2016/'

will load all objets stored in: mybucket/2016/*

copy mytable
FROM 's3://mybucket/2016/02'

will load all objets stored in: mybucket/2016/02/*

copy mytable
FROM 's3://mybucket/2016/1'

will load all objets stored in: mybucket/2016/1* (eg 10, 11, 12)

Basically, it just makes sure the object starts with the given string (including the full path).

This also means that if you have something like mybucket/wallet and a mybucket/walletiventory it can apply the rule also, so be careful with names when using the COPY command from S3.

edited Mar 7, 2023 at 23:20

Ismael Rodrigues

54 bronze badges

answered Apr 13, 2016 at 21:15

John Rotenstein

273k28 gold badges456 silver badges541 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

drtf · Accepted Answer · 2016-04-13 14:44:53Z

2

Apparently this is a simple as changing the source url to s3://mybucket/2016/, no wildcards required.

answered Apr 13, 2016 at 14:44

drtf

1,9762 gold badges22 silver badges20 bronze badges

Collectives™ on Stack Overflow

Redshift copy command recursive scan

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related