How to send the data from a mongodb collection whenever it gets updated to spark streaming context. I have seen socketTextStream to write the data to mongodb. But is there any way to read the stream when the collection gets updated? Is there any way to avoid to implement Custom Receiver for mongodb ? Or If anyone has already implemented, then it will be nice, if someone can share.
Thanking you for any input on this.
One way to achieve this is using the CDC pattern where on one side you have the reactive mongo library that can hook onto a collection and receive events like update, insert, delete from the mongodb and on the other side you have a spark streaming application that will listen on this events.
The transport engine of the events can be any publish/subscriber system (e.g Kafka)
The transport engine of the events can be any publish/subscriber system (e.g Kafka) . I think this is more suitable in my case. Thanks