Creating a single Spark Dataframe from multiple sources?

Question

A dataframe in Spark can be created from an external file. Is there a way to use multiple files to create a single object?

EX: If I have say a csv files or parquet files that record the same data (say the counts of an item purchased on a day of the week). Is there a way to initialize a single dataframe from each day's file to get a single dataframe that represents the data of a week's purchases?

Community · Accepted Answer · 2017-05-23 12:22:33Z

2

The way spark reads input from a file depends on the underlying Hadoop API's. This means quite often that they extend the same usage, including being able to handle compressed files, or multiple files.

In your case, you would just provide a file input using a wildcard or separated individually by comma. See How to read multiple text files into a single RDD?

edited May 23, 2017 at 12:22

CommunityBot

11 silver badge

answered Jul 18, 2016 at 23:47

YoYo

9,4958 gold badges62 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

knowads Over a year ago

If I provided a folder name with one type of file, would Spark be able to take in all the data in said folder?

YoYo Over a year ago

Yes, assuming you can all parse them the same way, using following path specifier: "/my/own/path/*"

Collectives™ on Stack Overflow

Creating a single Spark Dataframe from multiple sources?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related