Correct encoding for body from Request NodeJS

How do I find out the correct encoding of a response?

I went to the website you are attempting to scrape, and found this:

And another character encoding declaration here:

This website defined two different charater encodings! Which do I use?

Well, this doesn't apply to you. When reading an HTML file from a local machine, then the charset or content-type defined in the meta tags will be used for encoding.

Since you are retrieving this document, over HTTP, the files will be encoded according to the response header.

Here's the reponse header I received after visiting the website.

As you can see, they don't have a defined character set. It should be located in the Content-Type property. Like this:

Since they don't have any indicated charset in the response header, then, according to this post, it should use the meta declaration.

But wait, there was two meta charset declarations.

Since the compiler reads the file top to bottom, the second declared charset should be used.

Conclusion: They use UTF-8

Also, I don't think you need the conversion. I may be wrong, but you should just be able to access the response.

request({
    headers: {
        'Content-Length': contentLength,
        'Content-Type': 'application/x-www-form-urlencoded'
    },
    uri: 'xxxxxx.php',
    body: formData,
    method: 'POST'
}, function (err, res, body) {
    console.log(body);
    writer.write(body);
});

Edit: I don't believe the error is on their side. I believe it's on your side. Give this a try:

Remove the writer:

var writer = fs.createWriteStream('outputBodyutf8String.html');

And in the request callback, replace everything with this:

function (err, res, body) {
    console.log(body);
    fs.writeFile('outputBodyutf8String.html', body, 'utf8', function(error) {
        if(error)
            console.log('Error Occured', error);
    );
}

All the code should look like this:

var querystring = require('querystring');
var iconv = require('iconv-lite')
var request = require('request');
var fs = require('fs');

var form = {
    id:'2974',
    opt1:'',
    opt2:'30',
    ref:'A1',
    tid:'157',
    tid2:'',
    fnum:'2'
};

var formData = querystring.stringify(form);
var contentLength = formData.length;

request({
    headers: {
        'Content-Length': contentLength,
        'Content-Type': 'application/x-www-form-urlencoded'
    },
    uri: 'xxxxxxx.php',
    body: formData,
    method: 'POST'
}, function (err, res, body) {
    console.log(body);
    fs.writeFile('outputBodyutf8String.html', body, 'utf8', function(error) {
        if(error)
            console.log('Error Occured', error);
    );
}