Apache Spark - Parallel Processing of messages from Kafka - Java

Question

JavaPairReceiverInputDStream<String, byte[]> messages = KafkaUtils.createStream(...);
JavaPairDStream<String, byte[]> filteredMessages = filterValidMessages(messages);

JavaDStream<String> useCase1 = calculateUseCase1(filteredMessages);
JavaDStream<String> useCase2 = calculateUseCase2(filteredMessages);
JavaDStream<String> useCase3 = calculateUseCase3(filteredMessages);
JavaDStream<String> useCase4 = calculateUseCase4(filteredMessages);

...

I retrieve messages from Kafka, filter that and use the same messages for mutiple use-cases. Here useCase1 to 4 are independent of each other and can be calculated parallely. However, when i look at the logs, i see that calculations are happening sequentially. How can i make them to run parallely. Any suggestion would be helpful.

Jeff L · Accepted Answer · 2015-08-17 18:14:29Z

1

Try creating creating Kafka topics for each of your 4 use cases. Then try creating 4 different Kafka DStreams.

answered Aug 17, 2015 at 18:14

Jeff L

1411 gold badge1 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mohan Over a year ago

Thank you. But, it is the same message which will be consumed differently by the use cases.

Jeff L Over a year ago

I would double-check your solution because different messages will go to different partitions depending on the message's key.

Mohan Over a year ago

In this case, i have the same message, first use case strips country out of it and aggreagtes, second use case strips out trans type and aggregates and so on.

Mohan · Accepted Answer · 2015-08-18 03:30:45Z

1

I moved all code inside a for loop and iterated by the number of partitions in the kafka topic and i see an improvement.

for(i=0;i<numOfPartitions;i++)
{
JavaPairReceiverInputDStream<String, byte[]> messages =
KafkaUtils.createStream(...);
JavaPairDStream<String, byte[]> filteredMessages =
filterValidMessages(messages);

JavaDStream<String> useCase1 = calculateUseCase1(filteredMessages);
JavaDStream<String> useCase2 = calculateUseCase2(filteredMessages);
JavaDStream<String> useCase3 = calculateUseCase3(filteredMessages);
JavaDStream<String> useCase4 = calculateUseCase4(filteredMessages);
}

Reference : http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/

answered Aug 18, 2015 at 3:30

Mohan

2923 silver badges16 bronze badges

Collectives™ on Stack Overflow

Apache Spark - Parallel Processing of messages from Kafka - Java

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related