list S3 files in Pyspark

What if you use the SparkSession and SparkContext to read the files at once and then loop through thes s3 directory by using wholeTextFiles method. You can utilize the s3a connector in the url which allows to read from s3 through Hadoop.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('S3Example').getOrCreate()

s3_bucket = 'your-bucket'
s3_path = f's3a://{s3_bucket}/my-directory/'

# List files S3
file_list = spark.sparkContext.wholeTextFiles(s3_path).map(lambda x: x[0]).collect()

for file_path in file_list:
    print(file_path)

Please note, above I've only retrieved the file paths. If you want both, you can avoid only extracting the file path (x[0] in the lambda), and get both.

file_tuple = spark.sparkContext.wholeTextFiles(s3_path)

You can use s3fs - pip install s3fs

And can try below code -

import s3fs

fs = s3fs.S3FileSystem(anon=True)
fs.ls('my-bucket')

Also see this doc.

Another way is using hadoopFile:

SparkContext.hadoopFile("s3a://bucket_name/prefix)

refer to this doc.

In addition to all the other answers;

You can still use boto3 lib to get the list of objects with full path like below:

response = boto3.client('s3').list_objects_v2(Bucket=bucket, Prefix=prefix)
full_paths_list = list()
for obj in response["Contents"]:
    full_path = f's3://full_paths_list bucket}/{obj["Key"]}'
    full_paths_list.append(full_path)

Now, you have files list so you can read them with pyspark like below:

df = spark.read.parquet(*full_paths_list) 

You can use boto3 with pyspark. List objects in an S3 bucket and read them into a PySpark DataFrame.

1. Iterate over the list of objects and read each file into a PySpark DataFrame using the spark.read method.

2. Append each DataFrame to a list and then union all dataframes into one using the reduce function.

import boto3
import pyspark.sql.functions as F
from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("Read from S3").getOrCreate()

# Set up a boto3 client
s3 = boto3.client('s3')

# List all objects in the S3 bucket
bucket_name = "your-bucket-name"
prefix = "path-to-folder-within-bucket/"  # optional: if you want to read only files in a particular folder
objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)

# Read each file into a PySpark DataFrame
dataframes = []
for obj in objects['Contents']:
    file_name = obj['Key']
    s3_path = f's3://{bucket_name}/{file_name}'
    df = spark.read.option("header", True).csv(s3_path)
    dataframes.append(df)

# Union all dataframes into one
final_df = reduce(lambda a, b: a.union(b), dataframes)

# Show the final dataframe
final_df.show()