Response Buffer data is truncated in Node JS

Question

I'm trying to figure out how to use the request function of the http and https built-in modules in Node JS (I am aware of third-party wrappers but I'm trying to figure it out myself). I'm running into an issue where the Buffer data from the Response is partially cut off at the end. The issue does not occur when testing with cURL.

Here is the code I've been using:

const { request: httpRequest } = require("http");
const { request: httpsRequest } = require("https");
const parseURLData = (url, init) => {
  const { hostname, protocol, port: somePort, pathname, search } = new URL(url);
  const port = +somePort || (protocol === "https:" ? 443 : 80);
  const options = { hostname, port, path: pathname + search, ...init };
  return [protocol, options];
};
const makeRequest = (url, init = { method: "GET" }) => {
  const [protocol, options] = argumentsToOptions(url, init);
  const request = protocol === "http:" ? httpRequest : httpsRequest;
  return new Promise((resolve, reject) =>
    request(options, (res) => {
      res.on("error", reject);
      resolve(res);
    })
      .on("error", reject)
      .end()
  );
};
// not using `async/await` while testing
makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) =>
      res.on("data", (buffer) => {
        resolve(buffer.toString("utf8")); // part of data is cut off
        // resolve(JSON.parse(buffer.toString()));
      })
    )
  )
  .then(console.log)
  .catch(console.error);

Here is the expected output (from cURL):

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "[email protected]",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
  }
}

And here is the actual output, which for some reason is slightly different each time I run the code:

{
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "[email protected]",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "p

What is the correct solution to this problem? How can I get the all of the data from the request?

Gamedroit · Accepted Answer · 2022-01-21 21:16:17Z

5

The buffer should only be processed when the end event fires, otherwise you may be processing an incomplete buffer.

makeRequest("https://jsonplaceholder.typicode.com/users/1/")
  .then((res) => 
    new Promise((resolve) => {
        let totalBuffer = "";

        res.on("data", (buffer) => {
            totalBuffer += buffer.toString("utf8");    
        });

        res.on("end", () => resolve(totalBuffer));
    })
  )
  .then(console.log)
  .catch(console.error);

The response is almost always truncated into several pieces when the file exceeds 1mb, so it is necessary to use the end event which indicates that all available data has been handled by the stream.

https://nodejs.org/api/http.html Look for "JSON fetching example"

answered Jan 21, 2022 at 21:16

Gamedroit

3852 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Response Buffer data is truncated in Node JS

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related