Node.js request turns umlauts to �

Question

I'm doing the following request (using request/request) against a web service:

return request.postAsync({
    url,
    charset: 'Cp1252', // I also tried utf-8
    encoding: null, //
    // I also tried Cp1252 -> unknown encoding,
    // I also tried utf-8 and nothing at all
    headers: {
         "Accept": "application/octet-stream, text, text/plain, text/xml",
         "Accept-Encoding": "UTF-8",
         'Content-Type': "text/plain; charset=Cp1252;", // also tried utf-8, ISO-8859-1
         "User-Agent": "me"
    }
}).spread((res, body) => {
    body = body.toString();  // I also tried without toString();
    let ws = fs.createWriteStream('hhh.csv');
    ws.write(body);
    ws.end();

Whatever I do, umlauts are turned into �.

Those are the headers the web service sends back:

'content-type': 'text; charset=Cp1252',
'content-length': '1895980',
vary: 'Accept-Encoding,User-Agent'

I'm trying this for days with no luck at all. What am I doing wrong?

Here's a list of question/answers that didn't solve my problem so far:

Can it be that one of the following causes my input string to not be UTF-8?

let hash = crypto.createHmac("sha256", this.options.Signer);
this.query['Signature'] = hash.update(stringToSign).digest("base64");

signer is a string containing 0-9, a-z, A-Z, +, and /.

this.query['Signature'] is part of the URL.

What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? — zerkms
– zerkms, Commented Oct 5, 2015 at 21:53
"node js request turns umlauts to �" "Whatever I do, umlauts are turned into �." --- what these phrases mean then? — zerkms
– zerkms, Commented Oct 5, 2015 at 21:58
What are the exact characters you are sending? How are they currently encoded and what is the current character set for them? — zerkms
– zerkms, Commented Oct 5, 2015 at 22:01
Have you tried changing the defaultEncoding of your writeStream? i.e. fs.createWriteStream('hhh.csv', {defaultEncoding: 'utf8'}); — Yan Foto
– Yan Foto, Commented Oct 5, 2015 at 22:03

baao · Accepted Answer · 2015-10-05 23:42:20Z

5

I finally solved it, using iconv-lite and setting request's encoding¹ to null, making it return the body as a Buffer. Here is my now working configuration:

return request.getAsync({
        url,
        encoding: null,
        headers: {
            "Accept": "text, text/plain, text/xml",
            "Accept-Encoding": "UTF-8",
            'Content-Type': "text/plain; charset=utf-8;",
            "User-Agent": "me"
        }
    }).spread((res, body) => {
        let a = iconv.decode(new Buffer(body), 'cp1252');
        // now a is holding a string with correct Umlauts and ß

edited Oct 5, 2015 at 23:42

answered Oct 5, 2015 at 23:16

baao

73.5k18 gold badges152 silver badges209 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

D.Tate Over a year ago

related: stackoverflow.com/questions/12040643/…

Collectives™ on Stack Overflow

Node.js request turns umlauts to �

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related