corrupt record while reading xml file using pyspark

Question

I am trying to read an xml file in dataframe in pyspark.

Code : df_xml=spark.read.format("com.databricks.spark.xml").option("rootTag","dataset").option("rowTag","AUTHOR").load(FilePath)

when i display the dataframe, it shows a single column corrupt_records :

below is the xml file content

<?xml version='1.0' encoding='UTF-8'?>

<dataset>
 
 <AUTHOR AUTHOR_UID = 1>
    <FIRST_NAME>Fiona</FIRST_NAME>
    <MIDDLE_NAME/>
    <LAST_NAME>Macdonald</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = 2>
    <FIRST_NAME>Gian</FIRST_NAME>
    <MIDDLE_NAME>Paolo</MIDDLE_NAME>
    <LAST_NAME>Faleschini</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = 3>
    <FIRST_NAME>Laura</FIRST_NAME>
    <MIDDLE_NAME>K</MIDDLE_NAME>
    <LAST_NAME>Egendorf</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = 4>
    <FIRST_NAME>Jan</FIRST_NAME>
    <MIDDLE_NAME/>
    <LAST_NAME>Grover</LAST_NAME>
 </AUTHOR>

Luiz Viola · Accepted Answer · 2022-07-18 10:36:14Z

1

That XML is not valid:

The AUTHOR_UID must be defined in quotes
The dataset tag is not closed

This example below is a valid one:

<?xml version='1.0' encoding='UTF-8'?>

<dataset>
 
 <AUTHOR AUTHOR_UID = '1'>
    <FIRST_NAME>Fiona</FIRST_NAME>
    <MIDDLE_NAME/>
    <LAST_NAME>Macdonald</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = '2'>
    <FIRST_NAME>Gian</FIRST_NAME>
    <MIDDLE_NAME>Paolo</MIDDLE_NAME>
    <LAST_NAME>Faleschini</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = '3'>
    <FIRST_NAME>Laura</FIRST_NAME>
    <MIDDLE_NAME>K</MIDDLE_NAME>
    <LAST_NAME>Egendorf</LAST_NAME>
 </AUTHOR>
 <AUTHOR AUTHOR_UID = '4'>
    <FIRST_NAME>Jan</FIRST_NAME>
    <MIDDLE_NAME/>
    <LAST_NAME>Grover</LAST_NAME>
 </AUTHOR>
 
 </dataset>

answered Jul 18, 2022 at 10:36

Luiz Viola

2,4642 gold badges17 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ankit Tyagi Over a year ago

thanks Luiz, it seems Author_ID value not being in the quote was causing the problem. Dataset tag was closed in the file, though i only shared a few records, so it was missing in the example.

Collectives™ on Stack Overflow

corrupt record while reading xml file using pyspark

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related