Using Multiple page.open in Single Script

Question

My goal is to execute PhantomJS by using:

// adding $op and $er for debugging purposes
exec('phantomjs script.js', $op, $er);
print_r($op);
echo $er;

And then inside script.js, I plan to use multiple page.open() to capture screenshots of different pages such as:

var url = 'some dynamic url goes here';
page = require('webpage').create();
page.open(url, function (status) {
    console.log('opening page 1');  
    page.render('./slide1.png');            
});

page = require('webpage').create();
page.open(url, function (status) {
    console.log('opening page 2');  
    page.render('./slide2.png');        
});

page = require('webpage').create();
page.open(url, function (status) {
    console.log('opening page 3');  
    page.render('./slide3.png');        
    phantom.exit(); //<-- Exiting phantomJS only after opening all 3 pages
});

On running exec, I get the following output on page:

Array ( [0] => opening page 3 ) 0

As a result I only get the screenshot of the 3rd page. I'm not sure why PhantomJS is skipping the first and second blocks of code (evident from the missing console.log() messages that were supposed to be output from 1st and 2nd block) and only executing the third block of code.

I found this github.com/ariya/phantomjs/blob/master/examples/… But I was looking to implement it in a simple manner as the one I'm using currently. — asprin
– asprin, Commented Jun 8, 2013 at 7:58
Yes but from your question it's not really clear where you hit the roadblock doing that. You basically only write that "it does not work" and that you "think it should work somehow". That's pretty in-descriptive. So even someone who might know that might not be able to decipher your question in a manner knowing to have an answer for it. code-examples add good context, but you also should pin-point what exactly you expected to work with your code example. — hakre
– hakre, Commented Jun 8, 2013 at 8:01
have you double-checked the first two of the three functions are executed at all? — hakre
– hakre, Commented Jun 8, 2013 at 8:35
Doesn't console.log prove that the first two functions are not running?. I also tried running only the 1st code by commenting out the other two and that worked fine. — asprin
– asprin, Commented Jun 8, 2013 at 8:38

user663031 · Accepted Answer · 2013-06-08 15:17:58Z

44

The problem is that the second page.open is being invoked before the first one finishes, which can cause multiple problems. You want logic roughly like the following (assuming the filenames are given as command line arguments):

function handle_page(file){
    page.open(file,function(){
        ...
        page.evaluate(function(){
            ...do stuff...
        });
        page.render(...);
        setTimeout(next_page,100);
    });
}
function next_page(){
    var file=args.shift();
    if(!file){phantom.exit(0);}
    handle_page(file);
}
next_page();

Right, it's recursive. This ensures that the processing of the function passed to page.open finishes, with a little 100ms grace period, before you go to the next file.

By the way, you don't need to keep repeating

page = require('webpage').create();

answered Jun 8, 2013 at 15:17

user663031

Sign up to request clarification or add additional context in comments.

5 Comments

asprin Over a year ago

That makes sense. Let me try it out and get back to you on this.

Justin Geeslin Over a year ago

Helpful and simple! Thank you!

user941238 Over a year ago

You have a source for the fact that having multiple page.open being invoked at the same time causes problems for PhantomJS (or just from experience)? I'm running into a related issue and looking for a workaround.

George Mauer Over a year ago

I came to this page because we're having a similar problem. The odd thing is that it just started happening after the software being in operation opening as many as 70 urls (all to file://) at the same time. So its absolutely possible and will work to open multiple urls but - as evidenced by our situation, it might have arbitrary problems.

MKoosej Over a year ago

I tried this with a large number of pages and got a segmentation fault on ubuntu. It seems the recursive calls is doing something nasty with the memory. Is there a more scalable solution? eg. not recursive and horizontally scalable.

Community · Accepted Answer · 2020-06-20 09:12:55Z

8

I've tried the accepted answer suggestions, but it doesn't work (at least not for v2.1.1).

To be accurate the accepted answer worked some of the time, but I still experienced sporadic failed page.open() calls, about 90% of the time on specific data sets.

The simplest answer I found is to instantiate a new page module for each url.

// first page
var urlA = "http://first/url"
var pageA = require('webpage').create()

pageA.open(urlA, function(status){
    if (status){
        setTimeout(openPageB, 100) // open second page call
    } else{
        phantom.exit(1)
    }
})

// second page
var urlB = "http://second/url"
var pageB = require('webpage').create()

function openPageB(){
    pageB.open(urlB, function(){
        // ... 
        // ...
    })
}

The following from the page module api documentation on the close method says:

close() {void}

Close the page and releases the memory heap associated with it. Do not use the page instance after calling this.

Due to some technical limitations, the web page object might not be completely garbage collected. This is often encountered when the same object is used over and over again. Calling this function may stop the increasing heap allocation.

Basically after I tested the close() method I decided using the same web page instance for different open() calls is too unreliable and it needed to be said.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Mar 15, 2016 at 10:23

Jim

3,1263 gold badges31 silver badges37 bronze badges

1 Comment

Dan Mercer Over a year ago

That setTimeout(openPageB(), ...) line shouldn't have the parens after openPageB. As is, you're just calling openPageB right there and passing the return value into setTimeout.

sidanmor · Accepted Answer · 2016-11-14 12:20:52Z

You can use recursion:

var page = require('webpage').create();

// the urls to navigate to
var urls = [
    'http://phantomjs.org/',
    'https://twitter.com/sidanmor',
    'https://github.com/sidanmor'
];

var i = 0;

// the recursion function
var genericCallback = function () {
    return function (status) {
        console.log("URL: " + urls[i]);
        console.log("Status: " + status);
        // exit if there was a problem with the navigation
        if (!status || status === 'fail') phantom.exit();

        i++;

        if (status === "success") {

            //-- YOUR STUFF HERE ---------------------- 
            // do your stuff here... I'm taking a picture of the page
            page.render('example' + i + '.png');
            //-----------------------------------------

            if (i < urls.length) {
                // navigate to the next url and the callback is this function (recursion)
                page.open(urls[i], genericCallback());
            } else {
                // try navigate to the next url (it is undefined because it is the last element) so the callback is exit
                page.open(urls[i], function () {
                    phantom.exit();
                });
            }
        }
    };
};

// start from the first url
page.open(urls[i], genericCallback());

froilanq · Accepted Answer · 2017-02-22 21:17:44Z

Using Queued Processes, sample:

var page = require('webpage').create();

// Queue Class Helper
var Queue = function() {
    this._tasks = [];
};
Queue.prototype.add = function(fn, scope) {
    this._tasks.push({fn: fn,scope: scope});
    return this;
};
Queue.prototype.process = function() {
    var proxy, self = this;
    task = this._tasks.shift();
    if(!task) {return;}
    proxy = {end: function() {self.process();}};
    task.fn.call(task.scope, proxy);
    return this;        
};
Queue.prototype.clear = function() {
    this._tasks = []; return this;
};

// Init pages .....  
var q = new Queue();       

q.add(function(proxy) {
  page.open(url1, function() {
    // page.evaluate
    proxy.end();
  });            
});

q.add(function(proxy) {
  page.open(url2, function() {
    // page.evaluate
    proxy.end();
  });            
});


q.add(function(proxy) {
  page.open(urln, function() {
    // page.evaluate
    proxy.end();
  });            
});

// .....

q.add(function(proxy) {
  phantom.exit()
  proxy.end();
});

q.process();

I hope this is useful, regards.

Collectives™ on Stack Overflow

Using Multiple page.open in Single Script

4 Answers 4

5 Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related