Get JSON data from driver console in selenium python

Question

I need to get JSON data from console, which can be seen in this image.

So far I have used the following codes:

j = driver.execute_script(" return document.getElementsByClassName('form-group text required assessment_questions_choices_text')")

I tried to save the data to a list, that didn't work as well.

list = []
list = driver.execute_script(" return document.getElementsByClassName('form-group text required assessment_questions_choices_text')")

Also, I came across a method online which can be found here, using it I am able to get only the index of the data to an external file.

{ "0": {}, "1": {}, "2": {}, "3": {}, "4": {} }

Looking for a way to import all data, with each attribute directly to my python code.

Thanks in advance.

The JS returns an HTMLCollection object, which python might have trouble fetching in a format you want. Try converting to an array before returning: return Array.from(document.get...) If that doesn't work, might have to fetch the data you want from the Collection individually. — Shivashriganesh Mahato
– Shivashriganesh Mahato, Commented Apr 16, 2020 at 0:30
@ShivashriganeshMahato This code returns just returns an empty list. — Syed Azeem Javed
– Syed Azeem Javed, Commented Apr 16, 2020 at 10:22

supputuri · Accepted Answer · 2020-04-16 12:57:35Z

1

Here is the snippet that worked without any issue.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())
url = "https://stackoverflow.com/questions/61240545/get-json-data-from-driver-console-in-selenium-python"
driver.get(url)
elems = driver.execute_script("return document.getElementsByClassName('post-tag js-gps-track')")
for elem in elems:
    print(elem.text)
driver.quit()

output:

javascript
python
json
selenium
console.log

Process finished with exit code 0

Edit 1:

If you want to get the specific matching element then please use the below.

value = driver.execute_script("return document.getElementsByClassName('name_of_class')[0].innerHTML")
# or you can also use the below if you are interested in first item only
value = driver.execute_script("return document.querySelector('.single_class')[0].innerHTML")
# query selector with multiple classes (class name with whitespaces)
value = driver.execute_script("return document.querySelector('.class1.class2')[0].innerHTML")

edited Apr 16, 2020 at 12:57

answered Apr 16, 2020 at 2:24

supputuri

14.2k2 gold badges26 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Syed Azeem Javed Over a year ago

Your code returns an empty list, however it did guide me in the proper direction.

driver.execute_script("x = document.getElementsByClassName('name_of_class).item(0).innerHTML")  elems = driver.execute_script("return x")

This seems to be working perfectly for me. Thank you for your time and consideration.

supputuri Over a year ago

Glad that you are able to resolve the issue. Please check the updated answer with simple solutions. Make sure you accept the answer if you feel this answer resolved/helped you by clicking on accept button on the left hand side in my answer.

Syed Azeem Javed Over a year ago

The answer above by @supputuri was the most help, however I was unable to get the required values by returning them directly as pointed out by him. A simple tweak by first saving everything to a variable and then returning it did the trick for me.

Collectives™ on Stack Overflow

Get JSON data from driver console in selenium python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related