JSON Recognized as Invalid Through Stream When Using UTF-8

Question

I am trying to parse a large JSON file using a stream. If the file is encoded in ANSI I have no trouble, but if the file is UTF-8 formatted it throws an error:

stream.js:74
throw er; // Unhandled stream error in pipe.
^
Error: Invalid JSON (Unexpected "ï" at position 0 in state STOP)

the portion of my code where it fails looks like this:

fileStream = fs.createReadStream(jsonfile, { encoding: 'utf8' });
                        fileStream.pipe(JSONStream.parse('*'))
                            .pipe(es.through(function (data) { .....

Any Ideas why it doesn't recognize the formatting?

LukasKroess · Accepted Answer · 2016-03-10 08:56:24Z

1

This looks like it's heaving trouble with the Byte Order Mark "ï»¿" in the beginning of a UTF-8 File. ( see https://en.wikipedia.org/wiki/Byte_order_mark )

you could try the solution to set the start pointer position from 0 to 3. I reckon this is quite hacky, but it is one way of doing it.

see https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options for how to do it and further info.

edited Mar 10, 2016 at 8:56

answered Mar 10, 2016 at 8:41

LukasKroess

4111 gold badge4 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Justin Levine Over a year ago

Awesome that works for me for now. I wasnt aware of the Byte Order Mark. Thanks for the tip.

LukasKroess Over a year ago

You're welcome @JustinLevine. If this solved your problem, consider marking the answer as accepted. Thank you.

Collectives™ on Stack Overflow

JSON Recognized as Invalid Through Stream When Using UTF-8

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related