Python get web page contents that have javascripts - maybe Selenium

Question

I need to analyse web page contents. Page has javascrips. Can you advice on better way than using Selenium?

If not: page when loaded in browser has elements:

<div class="js-container">    <table class="zebra" style="width: 100%;">
        <tbody><tr>
            <th>A</th>
            <th>B</th>
            <th>C</th>
        </tr>
            <tr>
                <td>A1</td>
                <td>A2</td>
                <td>
                    <a href="http://X" style="color: black">T1</a>
                </td>
            </tr>
            <tr>
                ....
            </tr>
....

I need to read a table, element by element. I run for example:

myList = myDriver.find_elements_by_class_name("js-container").

Then how do I get inner elements of "js-container" object?

The only element resulting myList has is: print (myList[0]):

<selenium.webdriver.remote.webelement.WebElement (session="61238", element="{71293}")>

Alex Martian · Accepted Answer · 2016-07-16 14:45:46Z

2

Maybe you need BeautifulSoup - feeding to it Selenium driver.page_source. It is a python tool and it can build a tree based on the web page. BeautifulSoup document

edited Jul 16, 2016 at 14:45

Alex Martian

3,9207 gold badges44 silver badges76 bronze badges

answered Jul 16, 2016 at 11:33

Ben Lee

1021 gold badge1 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ben Lee Over a year ago

Is it the fact that, you want to fetch a page, which will be changed on loading, and you need the result page?

Ben Lee Over a year ago

When I try to use Selenium, I find the question stackoverflow.com/a/30103931/5359105 .Try browser.page_source to get page to convey it to BS.

Alex Martian Over a year ago

@Ben Lee, looks like browser.page_source does it. Thank you. I wander why it's not documented as one of main features of selenium.

JeffC · Accepted Answer · 2016-07-16 13:23:52Z

2

Selenium can do this just fine.

tableDescendants = myDriver.find_elements_by_css_selector("table.zebra *")
for tableDescendant in tableDescendants
    outer = tableDescendant.get_attribute("outerHTML")
    inner = tableDescendant.get_attribute("innerHTML")
    print outer[:outer.find(inner)]

This code grabs all descendants of the TABLE tag, removes everything after the start of the innerHTML string and prints the result. outerHTML contains the element itself and all descendants and innerHTML contains only the descendants. So, to get only the HTML of the element itself, we need to remove innerHTML from outerHTML.

answered Jul 16, 2016 at 13:23

JeffC

26.5k5 gold badges35 silver badges57 bronze badges

4 Comments

Alex Martian Over a year ago

Thank you. How to specify table name with spaces?

JeffC Over a year ago

It sounds like you are asking a new question. If not, please clarify what you are asking.

Alex Martian Over a year ago

I mean if not class="zebra", but e.g. class="zebra ver2"

JeffC Over a year ago

CSS Selectors is the way to go there. The basic format is <tagname>.<classname1>.<classname2>, e.g. table.zebra.ver2. Check out these resources: CSS Selector Reference and CSS Selector Tips.

Collectives™ on Stack Overflow

Python get web page contents that have javascripts - maybe Selenium

2 Answers 2

3 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related