Parsing a dynamic table with Selenium and Java

Question

The goal of this program is to visit https://www.cia.gov/the-world-factbook/countries/ and grab the links contained in the table for all 262 countries. I am using Selenium to accomplish this but I am running into issues grabbing more than the first page of results.

My test code is the following:

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;

import java.util.ArrayList;

public static void main(String[] args){
    WebDriver driver = new HtmlUnitDriver();
    driver.get("https://www.cia.gov/the-world-factbook/countries/");
    ArrayList<WebElement> links = (ArrayList<WebElement>) driver.findElements(By.className("inline-link"));
    System.out.println(links);
    WebElement button = driver.findElement(By.cssSelector("span[class='pagination__arrow-right']"));
    button.click();
    links = (ArrayList<WebElement>) driver.findElements(By.className("inline-link"));
    System.out.println(links);
    driver.close();
}

The driver successfully finds the button but after it is clicked and I grab the next set of links, they are still the same instead of being updated. I'm new to using Selenium and am not sure why the driver does not pick up the new table changes. The first screen shot is the button on the page. The second screenshot is the table. When looking at it, Croatia is the inline-link. When the next button is pressed, that value changes to salvadore.

Prophet · Accepted Answer · 2022-11-20 19:19:59Z

0

The main issue you asking about here caused by the following fact:
You are collecting the links on a "new" page immediately after clicking on the pagination button, so that page is not got changed yet.
What you should do:

Introduce the WebDriverWait ExpectedConditions to wait for collected elements.
Both on the first page and on all the other pages.
Add a small delay after clicking the pagination button so that this will wait for really new elements on a new page.

So, your existing code can be changed as following:

public static void main(String[] args){
    WebDriver driver = new HtmlUnitDriver();
    WebDriverWait wait = new WebDriverWait(driver, 30);
    driver.get("https://www.cia.gov/the-world-factbook/countries/");
    ArrayList<WebElement> links = wait.until(ExpectedConditions.visibilityOfAllElementLocated(By.className("inline-link")));
    System.out.println(links);
    driver.findElement(By.cssSelector("span[class='pagination__arrow-right']")).click();
    Thread.sleep(400);
    links = wait.until(ExpectedConditions.visibilityOfAllElementLocated(By.className("inline-link")));
    System.out.println(links);
    driver.close();
}

The code above should give you another values in list after clicking the pagination.

edited Nov 20, 2022 at 19:19

answered Nov 20, 2022 at 19:08

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

OCEAN OESTREICHER Over a year ago

I implemented this code however it still does not return more than the 'initial' page. I say it this way because the main html page never changes. Instead only the displayed rows update on the page itself once the button is clicked.

Prophet Over a year ago

This is exactly what happens when you clicking the pagination button there. The page itself and the URL are not changing, only the presented countries list is updated. This is also what I wrote in the answer: the collected links content will change.

Collectives™ on Stack Overflow

Parsing a dynamic table with Selenium and Java

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related