How do I find the elements in a dynamic webpage with Selenium in VBA?

Question

I am trying to navigate this webpage using Selenium in VBA: https://www.familysearch.org/search/collection/1401638/. I am using Chrome as the browser and inspect to see the elements in the page. I am not familiar with javascript, but am expert in VBA. I am able to navigate the elements that appear to be static on the page, such clicking the sign in button in the upper right. But I haven't figured out how to interact with the form in the center that has the tag Search Collection.

Here is the code I have so far:

Sub WebScraper()
    Dim FSBrowser As New WebDriver
    Dim FSWebElements As WebElements

    FSBrowser.Start "chrome", "https://www.familysearch.org/search"
    FSBrowser.Get "/collection/1401638/"        ' United States Census, 1850
    
    FSBrowser.FindElementById("signInLink").Click
    
    'FSBrowser.FindElementById("userName").SendKeys ("xxxx")
    'FSBrowser.FindElementById("password").SendKeys ("xxxx")
    'FSBrowser.FindElementById("login").Click

    FSBrowser.Quit
End Sub

I would like to enter "Virginia" in the Birth Place text box and click Search.

The complete HTML code as rendered in Inspect is really long, since there are quite a lot of form elements. Here is the form information from the javascript and the two elements:

<fs-search-form mobile-breakpoint="500">
<form id="form" method="get" class="search-form " action="/search/record/results">
<input type="text" class="fs-field-group__input" name="birthLikePlace" id="birthLikePlace">
<button type="submit" class="fs-button fs-button--recommended">Search</button>

This seems like a simple task, but I am clearly missing something. Could someone show me what that is? It looks like finding the element by name or Id of birthLikePlace would work, but it doesn't.

Edit: It seems that shadow-root has a big part to play in this issue. It is marked as open, but I suspect there is more to it that I am not seeing. Here is the complete section for the birthLikePlace:

<sf-search-field-group field-list="[&quot;birthLikePlace&quot;, &quot;birthLikeDate&quot;]" trigger-id="birth">
   #shadow-root (open)
      <style> ... </style>
      <div class="field-section" visible="">
         <h2 class="section-header fs-h5">KEY: null (en)</h2>
         <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
         <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
         <div class="trigger-group shown active flexible-container" for="birth">
            <legend role="button" class="closer " for="birth" aria-label="Hide search fields for birth"> </legend>
            <slot></slot> </div> </div>
   <sf-search-field field-id="birthLikePlace" field-type="place">
      #shadow-root (open)
         <style> ... </style>
         <span class="field" visible="" field-id="birthLikePlace">
            <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
            <span class="event-place flex-item">
               <label for="birthLikePlace_place">Birthplace</label>
               <span class="fs-field-group">
                  <input type="text" class="fs-field-group__input" name="birthLikePlace" id="birthLikePlace">
                  <span class="fs-field-group__attachment fs-tooltip">
                     <span class="fs-tooltip__trigger">
                        <input type="checkbox" name="birthLikePlace.exact" id="birthLikePlaceexact">
                        <label aria-label="Match birthplace exactly" for="birthLikePlaceexact"></label> </span>
                     <span class="fs-tooltip__body" role="tooltip">::before Match birthplace exactly ::after</span> </span> </span> </span>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if>
               <dom-if style="display: none;"><template is="dom-if"></template></dom-if> 
         </span>
   </sf-search-field>
   <sf-search-field field-id="birthLikeDate" field-type="year"> ... </sf-search-field>
</sf-search-field-group>

It seems at first glance that you can log in to that site using xmlhttp requests method issuing post requests. Did you try that? — SIM
– SIM, Commented Feb 8, 2021 at 19:20
I thought to try using Selenium first, since I want to use Chrome as the browser and not Internet Explorer. Would xmlhttp be a better way to complete this task? — Alex Campbell
– Alex Campbell, Commented Feb 8, 2021 at 19:24
Selenium is definitely a better choice than IE. Xmlhttp requests method is always the best way to go if you can handle it in the right way and the content you wish to grab are not heavily dynamic meaning javascript encrypted. — SIM
– SIM, Commented Feb 8, 2021 at 19:26
The request here is just the starting place, so I can understand where I am missing something. I plan on a lot more once I figure out the basics. The website is essentially all written in javascript and fully dynamic. — Alex Campbell
– Alex Campbell, Commented Feb 8, 2021 at 19:30
If you wanna enter the text within birthplace text box after logging in, you wanna define an explicit timeout for the script to wait for that box to be available like Set obirthPlace = FSBrowser.FindElementById("birthLikePlace", timeout:=10000): obirthPlace.SendKeys "virginia" — SIM
– SIM, Commented Feb 8, 2021 at 19:42

QHarr · Accepted Answer · 2021-02-08 19:35:36Z

1

Assuming HTML is as you show then SendKeys to input using id for birthplace then target adjacent button by class

FSBrowser.FindElementById("birthLikePlace").SendKeys "Virginia"
FSBrowser.FindElementByCss("#birthLikePlace + fs-button").Click

If the HTML is not flat in the way shown in your provided sample you may need to go with a more distinctive class e.g.

FSBrowser.FindElementByCss(".fs-button--recommended").Click

Or perhaps an attribute

FSBrowser.FindElementByCss("[type=submit]").Click

Depends really on where the first matches to those are the required ones. Cannot tell from small html sample which is why I went initially with a selector anchored to an id (hopefully unique id).

answered Feb 8, 2021 at 19:35

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Alex Campbell Over a year ago

Neither of those lines worked. I did copy the elements straight out of the Inspect window. but I'm guessing there is something that I didn't know to include.

QHarr Over a year ago

Please describe what didn't work? Error messages? Is there a parent iframe/frame to negotiate? Does a longer wait help?

Alex Campbell Over a year ago

For everything that has been suggested for the text box, I get the error NoSuchElementError, Element not found for Id =birthLikePlace. Essentially the same error for the button, except for the last suggestion. FSBrowser.FindElementByCss("[type=submit]").Click gets an error that says ElementNotVisibleError, element not interactable. It seems that the css has found something, but maybe not the element I am looking for because the search box I need is visible.

QHarr Over a year ago

Are these elements within a parent iframe/frame?

Alex Campbell Over a year ago

I don't think there is an iframe or frame. I have updated the question to give the full set of code for one section.

Collectives™ on Stack Overflow

How do I find the elements in a dynamic webpage with Selenium in VBA?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related