How to get new page source after navigating Python Selenium

Question

I am facing an issue. I navigate on the page via Selenium Chrome. I have timeouts and WebDriverWait as I need a full page to get JSON out of it. Then I click the navigation button with

driver.execute_script("arguments[0].click();", element)

as normal click never worked. And it is navigating OK, I see Selenium is surfing normally. No problem. But the driver.page_source remains for the first page that I got via 'get' method

All timeouts are the same as for the first page. And I see those new pages normally, but the page_source never updates.

What am I doing wrong?

How can you be so sure that you have timeouts and WebDriverWait as I need a full page to get JSON out of it? — undetected Selenium
– undetected Selenium, Commented Jun 26, 2020 at 9:53
Because first page works fine and it took some time to adjust timeouts and WebDriverWait to make sure JSON loads completely — Victor Semenov
– Victor Semenov, Commented Jun 26, 2020 at 10:01
The problem point was that I was getting the same values as many times as many pages there were. After every click Selenium was giving me the same first source — Victor Semenov
– Victor Semenov, Commented Jun 26, 2020 at 10:05

Jonas · Accepted Answer · 2020-06-26 10:03:25Z

1

After navigating to the new Page, you need to get the current URL by:

url = driver.current_url()

and then:

driver.get(url)
driver.getPageSource()

edited Jun 26, 2020 at 10:03

answered Jun 26, 2020 at 9:50

Jonas

1,7691 gold badge12 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Victor Semenov Over a year ago

Thank you. For python it would be url = driver.current_url. It worked! Is there any other way, alternative? Because after 'get' new url page a little bit changes. Which is not exactly what I want

Jonas Over a year ago

Ah yeah, thanks for the correction. What do you mean by it "changes a little bit" and when does this change happen?

Victor Semenov Over a year ago

It changes right upon loading. Driver is going on direct link in pagination and (I have no idea why) the search list changes a bit. It is like getting duplicated values from previous page. It is like there was 150 items found and each time after direct link it is getting 151, 154 , etc. All those are duplicates. No problem to eliminate them, but I feel like site protection might spot me because of direct link usage

Victor Semenov Over a year ago

I didn't know about this 'current_url', must have skipped while reading docs, thanks again. I was trying refreshing, searching elements, timeouts, everything to make the page "arrive" to selenium. I tried executing JS to get InnerHtml, but it was returning file without JSON and selenium tools to get InnerHtml again were returning the same first page, Nothing worked. For now only your approach is working and I and wondering if there is another one,

Jonas Over a year ago

That seems like a loading issue to me. So i would highly recommend to let you programm sleep for a few seconds after getting to a new page so it can load propably. Getting the current URL with current_url() is the only approach I know.

Collectives™ on Stack Overflow

How to get new page source after navigating Python Selenium

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related