node.js async/await or generic-pool causes infinite loop?

Question

I was trying to create an automation script for work, it is supposed to use multiple puppeteer instances to process input strings simultaneously. the task queue and number of puppeteer instances are controlled by the package generic-pool, strangely, when i run the script on ubuntu or debian, it seems that it fells into an infinite loop. tries to run infinite number of puppeteer instances. while when run on windows, the output was normal.

const puppeteer = require('puppeteer');
const genericPool = require('generic-pool');
const faker = require('faker');
let options = require('./options');
let i = 0;
let proxies = [...options.proxy];

const pool = genericPool.createPool({
    create: async () => {
        i++;
        console.log(`create instance ${i}`);
        if (!proxies.length) {
            proxies = [...options.proxy];
        }
        let {control = null, proxy} = proxies.pop();
        let instance = await puppeteer.launch({
            headless: true,
            args: [
                `--proxy-server=${proxy}`,
            ]
        });
        instance._own = {
            proxy,
            tor: control,
            numInstance: i,
        };
        return instance;
    },
    destroy: async instance => {
        console.log('destroy instance', instance._own.numInstance);
        await instance.close()
    },
}, {
    max: 3, 
    min: 1, 
});

async function run(emails = []) {
    console.log('Processing', emails.length);
    const promises = emails.map(email => {
        console.log('Processing', email)
        pool.acquire()
            .then(browser => {
                console.log(`${email} handled`)
                pool.destroy(browser);})
    })
    await Promise.all(promises)
    await pool.drain();
    await pool.clear();
}

let emails = [a,b,c,d,e,];
run(emails)

Output

create instance 1
Processing 10
Processing Stacey_Haley52
Processing Polly.Block
create instance 2
Processing Shanny_Hudson59
Processing Vivianne36
Processing Jayda_Ullrich
Processing Cheyenne_Quitzon
Processing Katheryn20
Processing Jamarcus74
Processing Lenore.Osinski
Processing Hobart75
create instance 3
create instance 4
create instance 5
create instance 6
create instance 7
create instance 8
create instance 9

is it because of my async functions? How can I fix it? Appreciate your help!

Edit 1. modified according to @James suggested

You might want to have a look at the library puppeteer-cluster (disclaimer: I'm the author) which handles a pool of puppeteer instances for you (works on windows and linux) and also takes care of errors and browser restarts. Might be easier to use than generic pool. — Thomas Dondorf
– Thomas Dondorf, Commented Oct 14, 2018 at 12:30
@ThomasDondorf Thank you for your suggestion! I have already found your repository, and also submitted an issue at your github page :). the problem to me is i need to use different proxies for different puppeteer instances, which I could not figure out how to do that with puppeteer-cluster. did i miss something? — a_rabbit
– a_rabbit, Commented Oct 14, 2018 at 13:02
Sorry, just saw the question. That's currently not supported, but could be introduced in one of the future versions (not any time soon though). — Thomas Dondorf
– Thomas Dondorf, Commented Oct 14, 2018 at 13:46

Md. Abu Taher · Accepted Answer · 2018-10-14 21:26:16Z

The main problem you are trying to solve,

It is supposed to use multiple puppeteer instances to process input strings simultaneously.

Promise Queue

You can use a rather simple solution that involves a simple promise queue. We can use p-queue package to limit the concurrency as we wish. I used this on multiple scraping projects to always test things out.

Here is how you can use it.

// emails to handle
let emails = [a, b, c, d, e, ];

// create a promise queue
const PQueue = require('p-queue');

// create queue with concurrency, ie: how many instances we want to run at once
const queue = new PQueue({
    concurrency: 1
});

// single task processor
const createInstance = async (email) => {
    let instance = await puppeteer.launch({
        headless: true,
        args: [
            `--proxy-server=${proxy}`,
        ]
    });
    instance._own = {
        proxy,
        tor: control,
        numInstance: i,
    };
    console.log('email:', email)
    return instance;
}

// add tasks to queue
for (let email of emails) {
    queue.add(async () => createInstance(email))
}

Generic Pool Infinite Loop Problem

I removed all kind of puppeteer related code from your sample code and saw how it was still producing the infinite output to console.

create instance 70326
create instance 70327
create instance 70328
create instance 70329
create instance 70330
create instance 70331
...

Now, if you test few times, you will see it will throw the loop only if you something on your code is crashing. The culprit is this pool.acquire() promise, which is just re queuing on error.

To find what is causing the crash, use the following events,

pool.on("factoryCreateError", function(err) {
  console.log('factoryCreateError',err);
});

pool.on("factoryDestroyError", function(err) {
  console.log('factoryDestroyError',err);
});

There are some issues related to this:

acquire() never resolves/rejects if factory always rejects, here.
About the acquire function in pool.js, here.
.acquire() doesn't reject when resource creation fails, here.

Good luck!

James · Accepted Answer · 2018-10-14 12:33:45Z

0

You want to return from your map rather than await, also don't await inside the destroy call, return the result and you can chain these e.g.

const promises = emails.map(e => pool.acquire().then(pool.destroy));

Or alternatively, you could just get rid of destroy completely e.g.

pool.acquire().then(b => b.close())

answered Oct 14, 2018 at 12:33

James

82.4k19 gold badges172 silver badges245 bronze badges

4 Comments

a_rabbit Over a year ago

I modified according to you suggested, but it still does not work on debian and ubuntu.

James Over a year ago

@a_rabbit nope, you still aren't returning the promise from your map....if you use {} then you cant implicitly return from a function (as per my example), you need to explicitly return the result. Also, you haven't adjusted the destroy call.

a_rabbit Over a year ago

sorry i did not know that:), but after i changed the code to const promises = emails.map( email => pool.acquire().then(), it still instantly prints the same lines as showed in my question. am i still doing it wrong?

James Over a year ago

@a_rabbit Does the code actually do what you expect? Seems to me that you're getting caught up in the log output, different platforms handle writing to streams differently hence why you might find different results. The log output to me is irrelevant given the tasks are scheduled to run simultaneously so there is no "correct" log output as you cant guarentee it.

Collectives™ on Stack Overflow

node.js async/await or generic-pool causes infinite loop?

2 Answers 2

Promise Queue

Generic Pool Infinite Loop Problem

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Promise Queue

Generic Pool Infinite Loop Problem

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related