Puppeteer: Cannot find element loaded by Javascript

Question

I've been using Puppeteer to scrape some websites, and it works well when the element I need is in the DOM; however I can't get it to work when the element is loaded via Javascript. E.g. please see my code below. More specifically, the page.waitForSelector always triggers a timeout error. I've tried a page.screenshot and the resulting image does show a fully loaded page, which contains this .evTextFont element.

How can I modify this code to successfully retrieve the .evTextFont element?

I've tried both Puppeteer versions 1.11 and 1.17, but am getting the same problem for both

Thanks a lot

Adapted from here

const puppeteer = require('puppeteer');
const URL = 'https://www.paintbar.com.au/events-1/moments-in-moonlight';

puppeteer.launch({ headless: true, args: ['--no-sandbox', '--disable-setuid-sandbox'] }).then(async browser => {
    const page = await browser.newPage();
    await page.setViewport({width: 1200, height: 600})
    await page.goto(URL, {waitUntil: 'networkidle0'});
    await page.waitForSelector('.evTextFont');
    await page.addScriptTag({url: 'https://code.jquery.com/jquery-3.2.1.min.js'});
    // await page.screenshot({ path: './image.jpg', type: 'jpeg' });

    const result = await page.evaluate(() => {
        try {
            var data = [];

            $('.evTextFont').each(function() {
                const title = $(this).text();
                data.push({
                    'title' : title,
                });
            });
            return data;

        } catch(err) {
            console.log(err.toString());
        }
    });

    await browser.close();

    for(var i = 0; i < result.length; i++) {
        console.log('Data: ' + result[i].title);
    }

    process.exit();
}).catch(function(error) {
    console.error(error);
    process.exit();
});

Vaviloff · Accepted Answer · 2019-05-28 09:46:06Z

1

It happens because the event you're looking for is shown inside of an iframe element, from another site, so you need to find that iframe first and then do operation on it.

await page.goto(URL, {waitUntil: 'networkidle0'});

// Looking for the iframe with the event
const frame = (await page.frames()).find(f => f.url().includes("events.wix.com"));

// Then do work as before, but on that frame
await frame.waitForSelector('.evTextFont');
await frame.addScriptTag({url: 'https://code.jquery.com/jquery-3.2.1.min.js'});

const result = await frame.evaluate(() => {...})

answered May 28, 2019 at 9:46

Vaviloff

16.9k6 gold badges55 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user6122500 Over a year ago

Thanks @Vaviloff! That works perfectly and I learned something new

Collectives™ on Stack Overflow

Puppeteer: Cannot find element loaded by Javascript

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related