PySpark how to read file having string with multiple encoding

Question

I am writing a python spark utility to read files and do some transformation. File has large amount of data ( upto 12GB ). I use sc.textFile to create a RDD and logic is to pass each line from RDD to a map function which in turn split's the line by "," and run some data transformation( changing fields value based on a mapping ).

Sample line from the file. 0014164,02,031270,09,1,,0,0,0000000000,134314,Mobile,ce87862158eb0dff3023e16850f0417a-cs31,584e2cd63057b7ed,Privé,Gossip

Due to values "Privé" I get UnicodeDecodeError. I tried to following to parse this value:

if isinstance(v[12],basestring):
            v[12] = v[12].encode('utf8')
        else:
            v[12] = unicode(v[12]).encode('utf8')

but when I write data back to file this field gets translated as 'Privï¿½'. on Linux source file type is shown as "ISO-8859 text, with very long lines, with CRLF line terminators".

Could someone let me know right way in Spark to read/write files with mixed encoding please.

zero323 · Accepted Answer · 2019-02-06 01:59:56Z

3

You can set use_unicode to False when calling textFile. It will give you RDD of str objects (Python 2.x) or bytes objects (Python 3.x) which can further processed using desired encoding, for example

sc.textFile(path, use_unicode=False).map(lambda x: x.decode("iso-8859-1"))

If that's not sufficient data can be loaded as-is using binaryFiles

sc.binaryFiles(path).values().flatMap(lambda x: x.decode("iso-8859-1").splitlines())

edited Feb 6, 2019 at 1:59

answered Sep 1, 2016 at 20:15

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

PySpark how to read file having string with multiple encoding

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related