How to select html elements using jQuery in Phantomjs?

Question

I am going to scrape some contents from a website that use javascript to load dynamic content. Before, I have used request and cheerio to scrape and they worked just fine. But I just find out that request and cheerio cannot scrape dynamic content. After do a research, I found phantomjs that can get all the content after the page has loaded. I have a problem with it now, I cannot use jQuery selector like I used to use in cheerio. This is my sample code but the selector is return nothing.

var page = require('webpage').create();
var url = 'http://angkorauto.com/vehicle';
page.open(url, function (status) {
    if (status !== 'success') {
        console.log('Unable to load the address!');
        phantom.exit();
    } else {
        window.setTimeout(function () {
            // console.log(page.content);
            page.includeJs('https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min.js', function(){

                page.evaluate(function(){
                    console.log($('.divTitle').find('a').attr('href'));
                });
            });

            phantom.exit();
        }, 1500);
    }
});

Could you help me with this problem? I really get stuck now.

Thanks for ur time to help.

You want to scrape from only this website (ankkorauto.com) or some other websites too? — Mert Metin
– Mert Metin, Commented Dec 10, 2016 at 15:44

Mert Metin · Accepted Answer · 2016-12-10 15:56:32Z

2

The website you want to scrape has jQuery already (like many other websites) so you don't have load it again.

This works fine:

var page = require('webpage').create();
var url = 'http://angkorauto.com/vehicle';
page.open(url, function(status) {

    var href = page.evaluate(function(){
        return jQuery('.divTitle').find('a').attr('href');
    });

    console.log(href);
});

answered Dec 10, 2016 at 15:56

Mert Metin

4111 gold badge8 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Doo Doo Over a year ago

I will try this

Artjom B. Over a year ago

This code solves two problems without clearly stating why. 1. Loading a another jQuery version tends to break all jQuery functionality on the site. So if it's already available, it should not be loaded. 2. console.log calls inside of the page context (inside page.evaluate) are not printed to the console by default. They would need the page.onConsoleMessage event handler. (@DooDoo)

Collectives™ on Stack Overflow

How to select html elements using jQuery in Phantomjs?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related