Save rdd into mongo database using java

Question

I am attempting to save tweets in MongoDB using Java, This is what I have;

JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(
            3000));
    JavaDStream<Status> tweets = TwitterUtils.createStream(ssc);

    JavaDStream<String> statuses = tweets
            .map(new Function<Status, String>() {
                public String call(Status status) {
                    return status.getUser().getName() + ":"
                            + status.getText();
                }
            });

    JavaDStream<String> users = tweets.map(new Function<Status, String>() {
        public String call(Status status) {
            return status.getUser().getName();
        }
    });


    users.foreachRDD(new Function<JavaRDD<String>, Void>() {
        public Void call(JavaRDD<String> rdd) throws Exception {
            if (rdd.count() > 0)
                rdd.saveAsTextFile("storage/users/test" + rdd.id()
                        + "_.txt");
            return null;
        }
    });

Like you see I can store users in text file using rdd.saveAsTextFile but what I need is a way to save this rdd to a data base (MongoDB).

llovett · Accepted Answer · 2015-02-11 01:41:20Z

1

You can use the MongoDB Hadoop Connector to store save an RDD to MongoDB using com.mongodb.hadoop.MongoOutputFormat:

Configuration config = new Configuration();
config.set("mongo.output.format", "com.mongodb.hadoop.MongoOutputFormat");
config.set("mongo.output.uri", "mongodb://host:port/database.collection");
rdd.saveAsNewAPIHadoopFile("file://this-is-not-used",
                           <keyClass>,
                           <valueClass>,
                           MongoOutputFormat.class,
                           config);

It may also be helpful to look at an example project for how to do this.

edited Feb 11, 2015 at 1:41

answered Feb 10, 2015 at 18:06

llovett

1,4691 gold badge12 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Save rdd into mongo database using java

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related