26

Using C# and amazon .Net SDK, able to list all the files with in a amazon S3 folder as below:

ListObjectsRequest request = new ListObjectsRequest();           
request.BucketName = _bucketName; //Amazon Bucket Name
request.Prefix = _sourceKey; //Amazon S3 Folder path           
do
{
    ListObjectsResponse response = _client.ListObjects(request);//_client - AmazonS3Client

Output

Folder
Folder/file1.pdf
Folder/file2.pdf
Folder/file3.pdf

But i wanted to achieve something like this:

Desired Output

file1.pdf
file2.pdf
file3.pdf

Thanks in advance

0

7 Answers 7

33

Also you can use the following c# code to retrieve the files information.

var bucketName = "your bucket";
var s3Client = new AmazonS3Client("your access key", "your secret key");
var dir = new S3DirectoryInfo(s3Client, bucketName, "your AmazonS3 folder name");
foreach (IS3FileSystemInfo file in dir.GetFileSystemInfos())
{
    Console.WriteLine(file.Name);
    Console.WriteLine(file.Extension);
    Console.WriteLine(file.LastWriteTime);
}

I am using amazon AWSSDK.Core and AWSSDK.S3 version 3.1.0.0 for .net 3.5. I hope it can help you

Sign up to request clarification or add additional context in comments.

4 Comments

S3DirectoryInfo is under the Amazon.S3.IO namespace.
Namespace Amazon.S3.IO is not supported on .NET Core.
This code is also returning Folder/Directory. But I need only files from specific folder.Is there any way to get files only (excluding Folder) ?
This returns nothing for me.
19
AmazonS3Client s3Client = new AmazonS3Client(S3_ACCESS_KEY, S3_SECRET_ACCESS_KEY, S3_REGIAO);
var lista = s3Client.ListObjectsAsync(S3_BUCKET, $"{S3_SUBDIRETORIO}/{produto}/{produto}.").Result;
var files = lista.S3Objects.Select(x => x.Key);
var arquivos = files.Select(x => Path.GetFileName(x)).ToList();

3 Comments

Welcome to StackOverflow. Please read about giving a good answer here
if we have 45k odd files, then this approach is giving only 1K file only.
@VineetAgarwal Using the AWS SDK, you can only pull 1000 objects at a time anyway.
14

Let's make it clear, simple, and with a little description.
As we all know, in S3 there is no concept of directories (folders). Ah, what? So everything inside S3 is nothing but objects.
Let's consider the below example s3 bucket - the bucket name is testBucket, the directory name is testDirectory and the directory contains two files testImage.jpg and testUserData.txt.

  • testBucket
    • testDirectory
      • testImage.jpg
      • testUserData.txt
AmazonS3Client s3Client = new AmazonS3Client();
string bucketName = "testBucket";
// to list out all the elements inside of a director [To know more][1]
string prefix = "testDirectory/";

ListObjectsRequest request = new ListObjectsRequest
                {
                    BucketName = bucketName,
                    Prefix = prefix,
                    // Use this to exclude your directory name
                    StartAfter = prefix,
                };
ListObjectsResponse response =  s3Client.ListObjects(request);

foreach (S3Object itemsInsideDirectory in response .S3Objects)
                {
                    Console.WriteLine(itemsInsideDirectory.Key);
                }

output:
If you use 'StartAfter', then output will be
testDirectory/testImage.jpg
testDirectory/testUserData.txt*

If you donot use 'StartAfter', then output will be
testDirectory/
testDirectory/testImage.jpg
testDirectory/testUserData.txt

2 Comments

I think those commands were updated :ListObjectsV2Request and ListObjectsV2Async are new names
Thank you for the information. I will check and update it accordingly
9
    AmazonS3Client client = new AmazonS3Client();
    ListObjectsRequest listRequest = new ListObjectsRequest
    {
        BucketName = "your-bucket-name",
        Prefix = "example/path"
    };

    ListObjectsResponse listResponse;
    do
    {
        listResponse = client.ListObjects(listRequest);
        foreach (S3Object obj in listResponse.S3Objects)
        {
            Console.WriteLine(obj.Key);
            Console.WriteLine(" Size - " + obj.Size);
        }

        listRequest.Marker = listResponse.NextMarker;
    } while (listResponse.IsTruncated);

https://docs.aws.amazon.com/sdkfornet1/latest/apidocs/html/M_Amazon_S3_AmazonS3_ListObjects.htm

Comments

2
string bucketName = your bucket name;
S3DirectoryInfo dir = new S3DirectoryInfo(client, bucketName, "folder name");
foreach (IS3FileSystemInfo file in dir.GetFiles())
{
    Console.WriteLine(file.Name);
    Console.WriteLine(file.Extension);
    Console.WriteLine(file.LastWriteTime);
}

Comments

1

When there are large number of objects you may need to use ListObjectsV2Request to read the objects in batches, similar to shown below

    private readonly IAmazonS3 _amazonS3; // Make sure you initialize this

    public async IAsyncEnumerable<IEnumerable<string>> ListObjects(string bucketName, string prefix)
    {
        ListObjectsV2Request objectsV2Request = new ListObjectsV2Request
        {
            BucketName = bucketName,
            Prefix = prefix
        };
        ListObjectsV2Response listObjectsV2Response;
        do
        {
            listObjectsV2Response = await _amazonS3.ListObjectsV2Async(objectsV2Request);
            objectsV2Request.ContinuationToken = listObjectsV2Response.NextContinuationToken;
            yield return listObjectsV2Response.S3Objects.Select(s => s.Key);
        } while (listObjectsV2Response.IsTruncated);
    }

Comments

0
_client = new AmazonS3Client(_awsAccessKeyId, _awsSecretAccessKey, _regionEndpoint);
var req = new ListObjectsV2Request
{
                BucketName = _bucketName,
                Prefix = "upper_folder_name/" + 
                "lower_folder_name/",
                MaxKeys = 100,
                Delimiter = "/",
                FetchOwner = false
            };
            var response = await _client.ListObjectsV2Async(req);
            foreach (var s3Object in response.S3Objects)
            {
                Console.WriteLine(s3Object.Key);
            }
 }

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.