How to write dataframe (obtained from hive table) into hadoop SequenceFile and RCFile?

Question

I am able to write it into

ORC
PARQUET

directly and
TEXTFILE
AVRO

using additional dependencies from databricks.

    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-csv_2.10</artifactId>
        <version>1.5.0</version>
    </dependency>
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-avro_2.10</artifactId>
        <version>2.0.1</version>
    </dependency>

Sample code:

    SparkContext sc = new SparkContext(conf);
    HiveContext hc = new HiveContext(sc);
    DataFrame df = hc.table(hiveTableName);
    df.printSchema();
    DataFrameWriter writer = df.repartition(1).write();

    if ("ORC".equalsIgnoreCase(hdfsFileFormat)) {
        writer.orc(outputHdfsFile);

    } else if ("PARQUET".equalsIgnoreCase(hdfsFileFormat)) {
        writer.parquet(outputHdfsFile);

    } else if ("TEXTFILE".equalsIgnoreCase(hdfsFileFormat)) {
        writer.format("com.databricks.spark.csv").option("header", "true").save(outputHdfsFile);

    } else if ("AVRO".equalsIgnoreCase(hdfsFileFormat)) {
        writer.format("com.databricks.spark.avro").save(outputHdfsFile);
    }

Is there any way to write dataframe into hadoop SequenceFile and RCFile?

what kind of sequence file ? csv , json ?

Gaurav Shah
– Gaurav Shah

2016-10-17 01:18:22 +00:00
Commented Oct 17, 2016 at 1:18 — Gaurav Shah
– Gaurav Shah, Commented Oct 17, 2016 at 1:18

Dev · Accepted Answer · 2016-10-17 11:12:05Z

2

+50

You can use void saveAsObjectFile(String path) to save a RDD as a SequenceFile of serialized objects. So in your case you have to to retrieve the RDD from the DataFrame:

JavaRDD<Row> rdd = df.javaRDD;
rdd.saveAsObjectFile(outputHdfsFile);

edited Oct 17, 2016 at 11:12

Dev

13.8k24 gold badges90 silver badges189 bronze badges

answered Oct 16, 2016 at 23:17

nicoring

7236 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

nicoring Over a year ago

I am not completely sure, but I don't think Spark supports writing into RCFiles out of the box, after skimming through the documentation. I suppose you have to use something like Parquet.

nicoring Over a year ago

@dev ツ Could you mark this as an answer if it answered your question?

Collectives™ on Stack Overflow

How to write dataframe (obtained from hive table) into hadoop SequenceFile and RCFile?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related