Interception xhr requests after pressing the load more button with puppeteer

Question

Initially two json files arrive on the site - the rest arrive after clicking the ‘load more’ button. I'm trying to intercept all requests. The example shows how I am waiting for the button to appear and click it accordingly. But so far I'm capturing only two requests - although there should be three, because I've already pressed the button once.

How can I wait for the rest of the xhr to intercept after pressing the ‘load more’ button?

const grab = async(url) => { 
  const browser = await puppeteer.launch({ headless: false});
  const page = await browser.newPage();

  await page.setRequestInterception(true);
  page.on("request", async (request) => {
    const current = await request.url();
    if (current.includes('betty-variants')) {
     console.log(request.url())
    }
  //This clicking not working here, the page does not see this selector
  // await page.waitForSelector('.well-sm.well p.text-primary.pull-left span', { timeout: 5000})
  // await page.click('.well-sm.well p.text-primary.pull-left span')


// Allow the request to be sent
    await request.continue();
  });

  await page.goto(url, { waitUntil: 'networkidle2' });

  await page.waitForSelector('.well-sm.well p.text-primary.pull-left span');      
  await page.click('.well-sm.well p.text-primary.pull-left span')

  await browser.close(); 
};

I tried getting the data after clicking on the button, but I only get the standard two json responses from the site.

Can you share the site you're automating, so the problem is reproducible? Thanks! — ggorlen
– ggorlen, Commented Apr 26 at 1:12

Sagar Galave · Accepted Answer · 2025-04-25 13:11:40Z

You're on the right track, but the issue you're facing is due to closing the browser immediately after clicking the button, which doesn’t give Puppeteer enough time to wait for and capture the additional XHR/fetch requests that fire after pressing "Load More".

const puppeteer = require('puppeteer');

const grab = async (url) => {
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();

  await page.setRequestInterception(true);

  const interceptedRequests = [];

  page.on("request", (request) => {
    const reqUrl = request.url();
    if (reqUrl.includes('betty-variants')) {
      console.log('Captured request:', reqUrl);
      interceptedRequests.push(reqUrl);
    }
    request.continue();
  });

  await page.goto(url, { waitUntil: 'networkidle2' });

  await page.waitForSelector('.well-sm.well p.text-primary.pull-left span', { timeout: 5000 });
  await page.click('.well-sm.well p.text-primary.pull-left span');

  await page.waitForResponse(response =>
    response.url().includes('betty-variants') && response.status() === 200,
    { timeout: 10000 }
  );

  await page.waitForTimeout(2000);

  console.log(`Total intercepted betty-variants requests: ${interceptedRequests.length}`);
  await browser.close();
};

grab('https://your-target-site.com');

Collectives™ on Stack Overflow

Interception xhr requests after pressing the load more button with puppeteer

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related