Scraping Angular data with Selenium

Question

Below there is some html that I can extract Text with Selenium driver.

<td colspan="2"><strong>Owner</strong>
                <div ng-class="{'owner-overflow' : property.ownersInfo.length > 4}">
                    <!-- ngRepeat: owner in property.ownersInfo --><div ng-repeat="owner in property.ownersInfo" class="ng-scope">
                        <div class="ng-binding">ERROL P BROWN LLC 
                            <!-- &nbsp;&nbsp; <span ng-if="owner.shortDescription != null && owner.shortDescription.length > 0">({{owner.shortDescription}})</span> -->
                        </div>
                    </div><!-- end ngRepeat: owner in property.ownersInfo -->
                </div>
            </td>    

<td colspan="2" class="pi_mailing_address"><strong>Mailing Address</strong>
                <div>
                    <span class="ng-binding">1784 NE 163 ST </span>
                    <span ng-show="property.mailingAddress.address2" class="ng-binding ng-hide"></span>
                    <span ng-show="property.mailingAddress.address3" class="ng-binding ng-hide"></span>
                    <span ng-show="property.mailingAddress.city" ng-class="{'inline':property.mailingAddress.city}" class="ng-binding inline">NORTH MIAMI,</span>
                    <span class="inline ng-binding">FL</span>
                    <span class="inline ng-binding">33162</span>
                    <span ng-hide="isCountryUSA(property.mailingAddress.country)" class="ng-binding ng-hide">USA</span>
                </div>
            </td>

When I run the code manually all the fields get picked up no issue. How ever if I run the script in a loop to extract this data These elements are blank. I am collecting other fields as well they are not coming up blank. There is no error in processing. Its just that when I save the data to a database these values are coming up empty. Is there some work around to have this NOT happen?

These are the lines of code:

Owner = driver.find_element(By.XPATH, "//strong[text()='Owner']//following::div[1]").text
SubDivision = driver.find_element(By.XPATH, "//strong[text()='Sub-Division:']//following::div[1]").text
Address1 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[1]").text
Address2 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[2]").text
Address3 = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[3]").text
city = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[4]").text.replace(",", "")
state = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[5]").text
zipcode = driver.find_element(By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[6]").text

You must have made a mistake, somewhere, in the code you didn't show us... — John Gordon
– John Gordon, Commented Jul 13, 2023 at 0:57
John Like I said the code works manually and data is inserted into database only when done manually. IF ran by the script other fields are stored to DB just fine but these are not. There is something else going on here that I am not aware of. — Leo Torres
– Leo Torres, Commented Jul 13, 2023 at 1:01
You have to show us the ACTUAL CODE that isn't working. Telling us "I ran this code in a loop" isn't enough. — John Gordon
– John Gordon, Commented Jul 13, 2023 at 1:03
Also, your code is looking for //strong, but I don't see any <strong> tags in that html at all... — John Gordon
– John Gordon, Commented Jul 13, 2023 at 1:05
Again I am not having issue pulling the data its Security mechanism in place. that does not allow me to collect data. If I run it manually it works. I added the strong tags but i didnt think it was relevant for me example. — Leo Torres
– Leo Torres, Commented Jul 13, 2023 at 1:34

undetected Selenium · Accepted Answer · 2023-07-13 21:38:31Z

Incase you are able to extract the required texts in standalone execution but not in a loop that may be due to race conditions that occur between the browser and the user's instructions.

Solution

As the elements are angular elements, so to extract the texts ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies:

Owner = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Sub-Division:']//following::div[1]"))).text
SubDivision = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Sub-Division:']//following::div[1]"))).text
Address1 = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[1]"))).text
Address2 = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[2]"))).text
Address3 = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[3]"))).text
city = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[4]"))).text.replace(",", "")
state = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[5]"))).text
zipcode = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Mailing Address']//following::div[1]//following::span[6]"))).text

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

Collectives™ on Stack Overflow

Scraping Angular data with Selenium

1 Answer 1

Solution

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Solution

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related