Parse XML data in Apache Spark

Question

I need to know how to how to parse XML file in Spark. I am receiving streaming data from kafka and then need to parse that streamed data.

Here is my Spark code to receive data:

directKafkaStream.foreachRDD(rdd ->{
            rdd.foreach(s ->{
                System.out.println("&&&&&&&&&&&&&&&&&" +s._2 );
            });

And results:

<root>
<student>
<name>john</name>
<marks>90</marks>
</student>
</root>

How to pass these XML elements?

Have you searched for previous questions on this? Such as: stackoverflow.com/questions/33078221/xml-processing-in-spark — Binary Nerd
– Binary Nerd, Commented Sep 26, 2016 at 7:11
@Binary Nerd, Thanks for response. My spark application is reading data line by line. So i need to parse line by line without using start element and/or end element. — user6325753
– user6325753, Commented Sep 26, 2016 at 8:43

Binary Nerd · Accepted Answer · 2016-09-26 14:10:24Z

3

Thanks guys.. Problem Solved. Here is the solution.

String xml = "<name>xyz</name>";
DOMParser parser = new DOMParser();
try {
    parser.parse(new InputSource(new java.io.StringReader(xml)));
    Document doc = parser.getDocument();
    String message = doc.getDocumentElement().getTextContent();
    System.out.println(message);
} catch (Exception e) {
    // handle SAXException 
}

edited Sep 26, 2016 at 14:10

Binary Nerd

13.9k4 gold badges46 silver badges46 bronze badges

answered Sep 26, 2016 at 13:13

user6325753

5774 gold badges11 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user6325753 Over a year ago

@MasudRahman, please look at the mentioned link stackoverflow.com/questions/33078221/xml-processing-in-spark/…

Amit Kulkarni · Accepted Answer · 2016-09-26 08:18:00Z

2

As you are processing streaming data, it would be helpful to use databricks's spark-xml lib for xml data processing.

Reference: https://github.com/databricks/spark-xml

answered Sep 26, 2016 at 8:18

Amit Kulkarni

7145 silver badges8 bronze badges

2 Comments

user6325753 Over a year ago

Thanks for response. My spark application is reading data line by line. So i need to parse line by line without using start element and/or end element.

Masud Rahman Over a year ago

I spent couple of hours with this, and then I found it does not read self-closing rows.

Collectives™ on Stack Overflow

Parse XML data in Apache Spark

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related