How to receive multiple requests with Express.JS

Question

I'm writing an Angular 6 + Express.JS app and now I stuck with the following problem: when there are some multiple requests made at the same time, sometimes (especially when there are more than 4 requests) all of them response with 404 or even get cancelled. Is there any problem with the way I handle requests in Express or I should add some tweaks for concurrent requests?

Requests:

let requests = [];
files.forEach((file) => {
    if (file.type.toLowerCase().includes('zip')) {
        requests.push(this.imagesService.uploadArchive(file).pipe(first()));
    } else {
        requests.push(this.imagesService.saveImage(file).pipe(first()));
    }
});

forkJoin(requests).subscribe(
    (res) => res.forEach(response => {
        this.onSave.emit(response);
    }), 
    (error) => {
        console.error(error);
    }, 
    () => {
        this.close.emit();
    }
);

Express handling routes:

router.post('/images',
    formidable({
        encoding: 'utf-8',
        uploadDir: path.resolve(__dirname, '..', '..', 'uploads'),
        multiples: true,
        keepExtensions: true
    }),
    (req, res, next) => {
        const image = req.fields;
        const data = req.files;
        image.path = data.image.path;

        const file = fs.createReadStream(image.path);

        saveImage(image).then(
            result => {
                if (result) {
                    res.status(200).send(result);
                } else {
                    console.error("Cannot save image");
                    res.status(400).send("Cannot save image");
                }
        }).catch(e => console.error(e.stack));
});

Responses:

UPDATE

router.post('/archives',
    formidable({
        encoding: 'utf-8',
        uploadDir: path.resolve(__dirname, '..', '..', 'uploads'),
        multiples: true,
        keepExtensions: true
    }),
    (req, res, next) => {
        const data = req.files;

        let promises = [];

        fs.readFile(data.archive.path, async (err, archive) => {
            if (err) throw err;

            await extractImagesFromZip(archive, data.archive.path).then((images) =>
                images.forEach((image) => {
                    promises.push(
                        saveImage(image).then(
                            result => {
                                if (result) {
                                    result.path = result.path.split('/').pop();
                                    return result;
                                } else {
                                    console.error("Cannot save image " + image.name);
                                    fs.unlink(image.path, () => {});
                                }
                        }).catch(e => {
                            fs.unlink(image.path, () => {});
                            console.error(e.stack)
                        })
                    );
                })
            );

            Promise.all(promises)
            .then((result) => {
                if (result.length > 0) {
                    res.status(200).send(result)
                } else {
                    res.status(400).send("None images were saved")
                }
            }).catch((error) => {
                console.log(error.stack);
                res.status(400).send("None images were saved")
            });
        });
    }
);

export const extractImagesFromZip = (file, link) => {
    let promises = [];

    var zip = new JSZip();
    return zip.loadAsync(file)
    .then((archive) => {
        Object.values(archive.files).filter(
            f => 
                ['.jpg', '.jpeg', '.png'].some((suffix) => f.name.toLowerCase().endsWith(suffix))
                && ![...f.name.toLowerCase().split('/')].pop().startsWith('.')
                && !f.dir 
        ).forEach(f => promises.push(zip.file(f.name).async('nodebuffer').then((content) => {
            const ext = f.name.split('.').pop().toLowerCase();
            var dest = path.resolve(__dirname, '..', '..') + '/uploads/upload_'
                + crypto.randomBytes(Math.ceil(1322)).toString('hex').slice(0, 32).toLowerCase() 
                + '.' + ext;

            return new Promise((res, rej) => { 
                fs.writeFile(dest, content, (err) => { 
                    if (err) rej(err); 

                    res(new Promise((resolve, reject) => {
                        fs.readFile(dest, (erro, data) => {
                            if (erro) reject(erro);
                            if (data) resolve({ 
                                name: f.name, 
                                type: 'image/' + (ext === 'jpg' ? 'jpeg' : ext), 
                                path: dest 
                            });
                        });
                    }));
                });
            });
        })));

        fs.unlink(link, () => {});

        return Promise.all(promises);
    });
}

export const saveImage = (image) => {
    return database.raw(
        "INSERT INTO images (name, type, path) " +
        "VALUES (?, ?, ?) " +
        "RETURNING name, type, path, id",
        [image.name, image.type, image.path]
    ).then(data => data.rows[0]).catch(e => console.error(e.stack));
};

UPDATE 2

Everything works fine if user and server are on localhost (regardless server running with nginx or without it), but problem appears when server is remote

Where your saveImage api? are you requesting same method different parameter into one api method? — Shohel
– Shohel, Commented Feb 14, 2019 at 12:41
Here is the good architecture of express js just see and try to build your project look like this. github.com/mdshohelrana/mean-stack/tree/master/server — Shohel
– Shohel, Commented Feb 14, 2019 at 12:54
@Shohel I updated the question, pls, check it. Regarding architecture: mostly my app looks like this, yes. Any single request works well, but when I try to make some of them at once, I get stuck with this problem — Nikita Popov
– Nikita Popov, Commented Feb 14, 2019 at 13:11
@Shohel the main problem is that I can't understand what's going wrong. Logging shows no errors at all and I don't know where should I start to look for solution. None of guides I found specify work with multiple requests at once (I guess, that shouldn't be a problem) and haven't found any examples of the same problem I have — Nikita Popov
– Nikita Popov, Commented Feb 14, 2019 at 13:26
OK, bro, I am trying to find something. I will notify If I get. — Shohel
– Shohel, Commented Feb 14, 2019 at 15:52

Nikita Popov · Accepted Answer · 2019-05-02 18:09:09Z

1

Such code worked

public async uploadFiles(files: File[]) {
    of(files)
        .pipe(
            concatMap(files =>
                files.map(file => {
                    return this.imagesService
                        .saveImage(file)
                        .pipe(
                            map(),
                            catchError((error, caught) => {
                                console.error(error);

                                return empty();
                            })
                        );
                })
            ),
            concatAll(),
            toArray(),
            map(res => console.log)
        )
        .subscribe();
}

answered May 2, 2019 at 18:09

Nikita Popov

1039 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to receive multiple requests with Express.JS

UPDATE

UPDATE 2

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

UPDATE

UPDATE 2

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related