how to access multiple json files using dataframe from S3

Question

I am using apapche spark. I want to access multiple json files from spark on date basis. How can i pick multiple files i.e. i want to provide range that files ending with 1034.json up to files ending with 1434.json. I am trying this.

DataFrame df = sql.read().json("s3://..../..../.....-.....[1034*-1434*]");

But i am getting the following error

   at java.util.regex.Pattern.error(Pattern.java:1924)
    at java.util.regex.Pattern.range(Pattern.java:2594)
    at java.util.regex.Pattern.clazz(Pattern.java:2507)
    at java.util.regex.Pattern.sequence(Pattern.java:2030)
    at java.util.regex.Pattern.expr(Pattern.java:1964)
    at java.util.regex.Pattern.compile(Pattern.java:1665)
    at java.util.regex.Pattern.<init>(Pattern.java:1337)
    at java.util.regex.Pattern.compile(Pattern.java:1022)
    at org.apache.hadoop.fs.GlobPattern.set(GlobPattern.java:156)
    at org.apache.hadoop.fs.GlobPattern.<init>(GlobPattern.java:42)
    at org.apache.hadoop.fs.GlobFilter.init(GlobFilter.java:67)

Please specify a way out.

Shankar · Accepted Answer · 2016-11-07 14:38:59Z

3

You can read something like this.

sqlContext.read().json("s3n://bucket/filepath/*.json")

Also, you can use wildcards in the file path.

For example:

sqlContext.read().json("s3n://*/*/*-*[1034*-1434*]")

edited Nov 7, 2016 at 14:38

answered Nov 7, 2016 at 12:54

Shankar

9,02926 gold badges100 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

hitttt Over a year ago

but how can i provide range in this as specified in the problem. files within a specific range

Shankar Over a year ago

@Hitesh Goyal: what do you mean by specific range?

Shankar Over a year ago

@HiteshGoyal: Spark file path supports wildcards.. whats your file path structure?

hitttt Over a year ago

By specific range , i mean that if my json file name is like (s3://bucket_name/201611091207.json). Now consider the last 4 digits of this file and upnext file name is (s3://bucket_name/201611091210.json). so if i need to provide a range for last 4 digits so that i am able to fetch the json files existing in between this range. @Shankar

ChuckCottrill Over a year ago

Can you point to a specification of these wildcards?

|

Collectives™ on Stack Overflow

how to access multiple json files using dataframe from S3

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related