Nodejs: convert string to buffer

Question

I'm trying to write a string to a socket (socket is called "response"). Here is the code I have sofar (I'm trying to implement a byte caching proxy...):

var http = require('http');
var sys=require('sys');

var localHash={};

http.createServer(function(request, response) {
    var proxy = http.createClient(80, request.headers['host'])
    var proxy_request = proxy.request(request.method, request.url, request.headers);
    proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(x) {
        var responseData=x.toString();
        var f=50;
        var toTransmit="";
        var p=0;        

        var N=responseData.length;
        if(N>f){
            p=Math.floor(N/f);

            var hash="";
            var chunk="";
            for(var i=0;i<p;i++){
                chunk=responseData.substr(f*i,f);
                hash=DJBHash(chunk);
                if(localHash[hash]==undefined){
                    localHash[hash]=chunk;
                    toTransmit=toTransmit+chunk;
                }else{
                    sys.puts("***hit"+chunk);
                    toTransmit=toTransmit+chunk;//"***EOH"+hash;
                }
            }
            //remainder:
            chunk=responseData.substr(f*p);
            hash=DJBHash(chunk);
            if(localHash[hash]==undefined){
                localHash[hash]=chunk;
                toTransmit=toTransmit+chunk;
            }else{
                toTransmit=toTransmit+chunk;//"***EOH"+hash;
            }
        }else{
            toTransmit=responseData;
        }
        response.write(new Buffer(toTransmit));   /*error occurs here */
    });
    proxy_response.addListener('end', function() {
        response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
    });
    request.addListener('data', function(chunk) {
        sys.puts(chunk);
        proxy_request.write(chunk, 'binary');
    });
    request.addListener('end', function() {
        proxy_request.end();
    });
}).listen(8080);



function DJBHash(str) {
    var hash = 5381;
    for(var i = 0; i < str.length; i++) {
        hash = (((hash << 5) + hash) + str.charCodeAt(i)) & 0xffffffff;
    }
    if(hash<-1){
        hash=hash*-1;
    }
    return hash;
}

The trouble is, I keep getting a "content encoding error" in Firefox. It's as if the gizipped content isn't being transmitted properly. I've ensured that "toTransmit" is the same as "x" via console.log(x) and console.log(toTransmit).

It's worth noting that if I replace response.write(new Buffer(toTransmit)) with simply response.write(x), the proxy works as expected, but I need to do some payload analysis and then pass "toTransmit", not "x".

I've also tried to response.write(toTransmit) (i.e. without the conversion to buffer) and I keep getting the same content encoding error.

I'm really stuck. I thought I had this problem fixed by converting the string to a buffer as per another thread (http://stackoverflow.com/questions/7090510/nodejs-content-encoding-error), but I've re-opened a new thread to discuss this new problem I'm experiencing.

I should add that if I open a page via the proxy in Opera, I get gobblydeegook - it's as if the gzipped data gets corrupted.

Any insight greatly appreciated.

Many thanks in advance,

Buffer.byteLength

Muhammad Umer
– Muhammad Umer

2020-11-01 17:40:58 +00:00
Commented Nov 1, 2020 at 17:40 — Muhammad Umer
– Muhammad Umer, Commented Nov 1, 2020 at 17:40

Kirko · Accepted Answer · 2020-02-13 13:35:37Z

43

How about this?

var responseData = Buffer.from(x, 'utf8');

from: Convert string to buffer Node

edited Feb 13, 2020 at 13:35

Kirko

644 bronze badges

answered Dec 10, 2016 at 17:03

user1527225

1,08910 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ezabaw Over a year ago

This is the right way to do it without deprecated functions

Andris · Accepted Answer · 2011-08-17 14:59:49Z

22

Without digging very deep into your code, it seems to me that you might want to change

var responseData=x.toString();

to

var responseData=x.toString("binary");

and finally

response.write(new Buffer(toTransmit, "binary"));

answered Aug 17, 2011 at 14:59

Andris

27.9k4 gold badges37 silver badges38 bronze badges

8 Comments

thejh Over a year ago

"binary" encoding is generally a bad idea.

Eamorr Over a year ago

@Andris: Legend, here's what I did: var responseData=x.toString("binary"); and response.write(toTransmit,"binary"); It's all working now! Thanks so much for pointing me in the right direction!!!

thejh Over a year ago

@Eamorr, seriously, don't do that! The "binary" encoding is deprecated and will get removed from node! I cite the docs: "'binary' - A way of encoding raw binary data into strings by using only the first 8 bits of each character. This encoding method is deprecated and should be avoided in favor of Buffer objects where possible. This encoding will be removed in future versions of Node."

thejh Over a year ago

@Eamorr, could you update your original post with the code you're using now?

Andris Over a year ago

'binary' has been available for several years by now and it seems it is not going anywhere. Even several node.js core apis return 'binary' by default.

|

thejh · Accepted Answer · 2011-08-17 15:07:02Z

From the docs:

Pure Javascript is Unicode friendly but not nice to binary data. When dealing with TCP streams or the file system, it's necessary to handle octet streams. Node has several strategies for manipulating, creating, and consuming octet streams.

Raw data is stored in instances of the Buffer class. A Buffer is similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap. A Buffer cannot be resized.

So, don't use strings for handling binary data.

Change proxy_request.write(chunk, 'binary'); to proxy_request.write(chunk);.

Omit var responseData=x.toString();, that's a bad idea.

Instead of doing substr on a string, use slice on a buffer.

Instead of doing + with strings, use the "concat" method from the buffertools.

int0x80 · Accepted Answer · 2023-07-13 08:09:05Z

2

Actually, now new Buffer() is deprecated since Node.js v10+, so instead of new Buffer() better to use Buffer.from(you_string,enecoding_format) Change this

 response.write(new Buffer(toTransmit));

to this:

 response.write(Buffer.from(toTransmit,'binary'));

edited Jul 13, 2023 at 8:09

answered Apr 19, 2022 at 20:35

int0x80

313 bronze badges

Collectives™ on Stack Overflow

Nodejs: convert string to buffer

4 Answers 4

1 Comment

8 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

8 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related