Problem with async when downloading a series of files with nodejs

Question

I'm trying to download a bunch of files. Let's say 1.jpg, 2.jpg, 3.jpg and so on. If 1.jpg exist, then I want to try and download 2.jpg. And if that exist I will try the next, and so on.

But the current "getFile" returns a promise, so I can't loop through it. I thought I had solved it by adding await in front of the http.get method. But it looks like it doesn't wait for the callback method to finish. Is there a more elegant way to solve this than to wrap the whole thing in a new async method?

// this returns a promise
var result = getFile(url, fileToDownload);    

const getFile = async (url, saveName) => {
        try {
                const file = fs.createWriteStream(saveName);
                const request = await http.get(url, function(response) {
                        const { statusCode } = response;

                        if (statusCode === 200) {
                                response.pipe(file);
                                return true;
                        }
                        else
                                return false;
                });

        } catch (e) {
                console.log(e);
                return false;
        }
}

Ashish Modi · Accepted Answer · 2020-01-07 13:22:06Z

1

I don't think your getFile method is returning promise and also there is no point of awaiting a callback. You should split functionality in to two parts - get file - which gets the file - saving file which saves the file if get file returns something.

try the code like this

const getFile = url => {
  return new Promise((resolve, reject) => {
    http.get(url, response => {
      const {statusCode} = response;
      if (statusCode === 200) {
        resolve(response);
      }
      reject(null);
    });
  });
};

async function save(saveName) {
  const result = await getFile(url);
  if (result) {
    const file = fs.createWriteStream(saveName);
    response.pipe(file);
  }
}

answered Jan 7, 2020 at 13:22

Ashish Modi

7,8002 gold badges23 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Gaurav Varshney · Accepted Answer · 2020-01-07 14:38:49Z

0

What you are trying to do is getting / requesting images in some sync fashion.

Possible solutions :

You know the exact number of images you want to get, then go ahead with "request" or "http" module and use promoise chain.
You do not how the exact number of images, but will stop at image no. N-1 if N not found. then go ahed with sync-request module.

answered Jan 7, 2020 at 14:38

Gaurav Varshney

5105 silver badges17 bronze badges

Comments

Max · Accepted Answer · 2020-01-07 14:41:43Z

your getFile does return a promise, but only because it has async keyword before it, and it's not a kind of promise you want. http.get uses old callback style handling, luckily it's easy to convert it to Promise to suit your needs

const tryToGetFile = (url, saveName) => {
  return new Promise((resolve) => {
    http.get(url, response => {
      if (response.statusCode === 200) {
        const stream = fs.createWriteStream(saveName)
        response.pipe(stream)
        resolve(true);
      } else {
        // usually it is better to reject promise and propagate errors further
        // but the function is called tryToGetFile as it expects that some file will not be available
        // and this is not an error. Simply resolve to false
        resolve(false);
      }
    })
  })
}


const fileUrls = [
  'somesite.file1.jpg',
  'somesite.file2.jpg',
  'somesite.file3.jpg',
  'somesite.file4.jpg',
]

const downloadInSequence = async () => {
  // using for..of instead of forEach to be able to pause
  // downloadInSequence function execution while getting file
  // can also use classic for
  for (const fileUrl of fileUrls) {
    const success = await tryToGetFile('http://' + fileUrl, fileUrl)
    if (!success) {
      // file with this name wasn't found
      return;
    }
  }
}

This is a basic setup to show how to wrap http.get in a Promise and run it in sequence. Add error handling wherever you want. Also it's worth noting that it will proceed to the next file as soon as it has received a 200 status code and started downloading it rather than waiting for a full download before proceeding

Collectives™ on Stack Overflow

Problem with async when downloading a series of files with nodejs

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related