Node JS upload file streams over HTTP

Question

I'm switching one of my projects from request over to something a bit more light-weight (such as got, axios, or fetch). Everything is going smoothly, however, I'm having an issue when attempting to upload a file stream (PUT and POST). It works fine with the request package, but any of the other three return a 500 from the server.

I know that a 500 generally means an issue on the server's end, but it is consistent only with the HTTP packages that I'm testing out. When I revert my code to use request, it works fine.

Here is my current Request code:

Request.put(`http://endpoint.com`, {
  headers: {
    Authorization: `Bearer ${account.token.access_token}`
  },
  formData: {
    content: fs.createReadStream(localPath)
  }
}, (err, response, body) => {
  if (err) {
    return callback(err);
  }

  return callback(null, body);
});

And here is one of the attempts using another package (in this case, got):

got.put(`http://endpoint.com`, {
  headers: {
    'Content-Type': 'multipart/form-data',
    Authorization: `Bearer ${account.token.access_token}`,
  },
  body: {
    content: fs.createReadStream(localPath)
  }
})
  .then(response => {
    return callback(null, response.body);
  })
  .catch(err => {
    return callback(err);
  });

Per the got documentation, I've also tried using the form-data package in conjunction with it according to its example and I still get the same issue.

The only difference between these 2 I can gather is with got I do have to manually specify the Content-Type header otherwise the endpoint does give me a proper error on that. Otherwise, I'm not sure how the 2 packages are constructing the body with the stream, but as I said, fetch and axios are also producing the exact same error as got.

If you want any of the snippets using fetch or axios I'd be happy to post them as well.

Not sure if that helps, but according to the got documentation you need to pass a FormData as the body of the object - github.com/sindresorhus/got#form-data — Vasil Dininski
– Vasil Dininski, Commented Jun 24, 2016 at 12:23
I did try this as well. Using the form-data package, I called append with 'content' and the stream info but I still get the same issue. I will edit my question to reflect this. — alex-phillips
– alex-phillips, Commented Jun 24, 2016 at 12:24

SuperSephy · Accepted Answer · 2020-09-22 01:58:22Z

11

I know this question was asked a while ago, but I too am missing the simple pipe support from the request package

const request = require('request');

request
  .get("https://res.cloudinary.com/demo/image/upload/sample.jpg")
  .pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))


// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
  .pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))

and had to do some experimenting to find similar features from current libraries.

Unfortunately, I haven't worked with "got" but I hope the following 2 examples help someone else that are interested in working with the Native http/https libraries or the popular axios library

HTTP/HTTPS

Supports piping!

const http = require('http');
const https = require('https');

console.log("[i] Test pass-through: http/https");

// Note: http/https must match URL protocol
https.get(
  "https://res.cloudinary.com/demo/image/upload/sample.jpg",
  (imageStream) => {
    console.log("    [i] Received stream");

    imageStream.pipe(
      http.request("http://localhost:8000/api/upload/stream/", {
        method: "POST",
        headers: {
          "Content-Type": imageStream.headers["content-type"],
        },
      })
    );
  }
);

// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
  .pipe(
    http.request("http://localhost:8000/api/upload/stream/", {
      method: "POST",
      headers: {
        "Content-Type": imageStream.headers["content-type"],
      },
    })
  )

Axios

Note the usage of imageStream.data and that it's being attached to data in the Axios config.

const axios = require('axios');

(async function selfInvokingFunction() {
  console.log("[i] Test pass-through: axios");

  const imageStream = await axios.get(
    "https://res.cloudinary.com/demo/image/upload/sample.jpg",
    {
      responseType: "stream", // Important to ensure axios provides stream
    }
  );

  console.log("  [i] Received stream");

  const upload = await axios({
    method: "post",
    url: "http://127.0.0.1:8000/api/upload/stream/",
    data: imageStream.data,
    headers: {
      "Content-Type": imageStream.headers["content-type"],
    },
  });

  console.log("Upload response", upload.data);
})();

answered Sep 22, 2020 at 1:58

SuperSephy

1452 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dmitry Zlykh Over a year ago

Thank you for the Axios solution. Response type 'stream' works fine when used as multipart POST

Adam Marsh Over a year ago

And thank you for the "HTTP/HTTPS Supports piping!" example - that implementation was the closest to my request implementation. Been trying to shift away from request since it was depreciated.

alex-phillips · Accepted Answer · 2016-06-25 00:52:13Z

2

Looks like this was a headers issue. If I use the headers directly from FormData (i.e., headers: form.getHeaders()) and just add in my additional headers afterwards (Authorization), then this ends up working just fine.

answered Jun 25, 2016 at 0:52

alex-phillips

8684 gold badges15 silver badges24 bronze badges

Comments

Marlon Klagenberg · Accepted Answer · 2021-06-30 20:05:09Z

0

For me just works when I added other parameters on FormData.

before

const form = new FormData();
form.append('file', fileStream);

after

const form = new FormData();
form.append('file', fileStream, 'my-whatever-file-name.mp4');

So that way I can send stream from my backend to another backend in node, waiting a file in multipart/form-data called 'file'

edited Jun 30, 2021 at 20:05

answered Jun 30, 2021 at 19:26

Marlon Klagenberg

114 bronze badges

Collectives™ on Stack Overflow

Node JS upload file streams over HTTP

3 Answers 3

HTTP/HTTPS

Axios

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

HTTP/HTTPS

Axios

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related