The "await" property of "async" function sleeps after an instance - Javascript

Question

I am working on a scraper . I am using Phantom JS along with Node JS. Phantom JS loads the page with async function, just like : var status = await page.open(url). Sometimes, because of the slow internet the page takes longer to load and after a time the page status is not returned, to check while its loaded or not. And the page.open() sleeps, which doesn't return anything at all, and all the execution is waiting.

So, my basic question is; is there any way to keep this page.open(url) alive, as the execution of the rest of the code waits until the page is loaded.

My Code is

const phantom = require('phantom');

ph_instance = await phantom.create();
ph_page = await ph_instance.createPage();

var status = await ph_page.open("https://www.cscscholarship.org/");

if (status == 'success') {
  console.log("Page is loaded successfully !");
  //do more stuff
}

Can you please edit your title into something more sensical? — Bergi
– Bergi, Commented Dec 10, 2018 at 11:15

Joshua Terrill · Accepted Answer · 2018-12-10 09:20:41Z

1

From your comment, it seems like it might be timing out (because of slow internet sometimes)... you can validate this by adding the onResourceTimeout method to your code (link: http://phantomjs.org/api/webpage/handler/on-resource-timeout.html)

It would look something like this:

ph_instance.onResourceTimeout = (request) => {
    console.log('Timeout caught:' + JSON.stringify(request));
};

And if that ends up being true, you can increase the default resource timeout settings (link: http://phantomjs.org/api/webpage/property/settings.html) like this:

ph_instance.settings.resourceTimeout = 60000 // 60 seconds

Edit: I know the question is about phantom, but I wanted to also mention another framework I've used for scraping projects before called Puppeteer (link: https://pptr.dev/) I personally found that their API's are easier to understand and code in, and it's currently a maintained project unlike Phantom JS which is not maintained anymore (their last release was two years ago).

edited Dec 10, 2018 at 9:20

answered Dec 10, 2018 at 9:18

Joshua Terrill

2,0175 gold badges23 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nadeem Ahmad Over a year ago

Yes, you are right. Few days back I got aware of that framework. But my Scrapper is almost developed, I will change it to Puppeteer when I am done with it.

Collectives™ on Stack Overflow

The "await" property of "async" function sleeps after an instance - Javascript

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related