Trying to select element by xpath with Selenium but getting error "Unable to locate element"

Question

I am trying to scrape the list of followings for a given instagram user. This requires using Selenium to navigate to the user's Instagram page and then clicking "following". However, I cannot seem to click the "following" button with Selenium.

driver = webdriver.Chrome()
url = 'https://www.instagram.com/beforeeesunrise/'
driver.get(url)
driver.find_element_by_xpath('//*[@id="react-root"]/section/main/article/header/div[2]/ul/li[3]/a').click()

However, this results in a NoSuchElementException. I copied the xpath from the html, tried using the class name, partial link and full link and cannot seem to get this to work! I've also made sure that the above xpath include the element with a "click" event listener.

UPDATE: By logging in I was able to get the above information. However (!), now I cannot get the resulting list of "followings". When I click on the button with the driver, the html does not include the information in the pop up dialog that you see on Instagram. My goal is to get all of the users that the given username is following.

In Chrome, if you inspect the element and use the "Copy XPath" feature, the XPath it gives is this, which is different from the one in your question: //*[@id="react-root"]/section/main/article/header/div[2]/div[1]/span/button Does that work? — Random Davis
– Random Davis, Commented Dec 16, 2016 at 21:27
@RandomDavis This is the not the button for the list of followings, this is the "Follow" button. What I wrote above is the result of using "Copy XPath" on the "Followings" button. — Melanie
– Melanie, Commented Dec 16, 2016 at 21:34
Are you actually logged into instagram in your selenium browser session? — alecxe
– alecxe, Commented Dec 16, 2016 at 21:43

Community · Accepted Answer · 2017-05-23 10:30:42Z

1

Make sure you are using the correct X Path.

Use the following link to get perfect X Paths to access web elements and then try.

Selenium Command

Hope this helps to solve the problem!

edited May 23, 2017 at 10:30

CommunityBot

11 silver badge

answered Dec 17, 2016 at 21:10

Mohsin Awan

1,1722 gold badges12 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

JeffC · Accepted Answer · 2018-12-18 14:26:26Z

1

Try a different XPath. I've verified this is unique on the page.

driver.find_element_by_xpath("//a[contains(.,'following')]")

edited Dec 18, 2018 at 14:26

answered Dec 17, 2016 at 20:00

JeffC

26.4k5 gold badges35 silver badges57 bronze badges

2 Comments

Jitendra Over a year ago

I tried as xpath specified //a[contains(.,'following')] using the chrome console. A parentheses less from the one that is mentioned by @jeffc. The link having the text "following" is highlighted in the Chrome Console.

JeffC Over a year ago

@stupidnetizen Yep, that extra set of parens was a mistake. Fixed, thanks!

GustavoIP · Accepted Answer · 2016-12-17 20:39:20Z

0

It's not the main goal of selenium to provide rich functionalities, from a web-scraping perspective, to find elements on the page, so the better option is to delegate this task to a specific tool, like BeautifulSoup. After we find what we're looking for, then, we can ask for selenium to interact with the element.

The bridge between selenium and BeautifulSoup will be this amazing function below that I found here. The function gets a single BeautifulSoup element and generates a unique XPATH that we can use on selenium.

import os
import re
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import itertools

def xpath_soup(element):
    """
    Generate xpath of soup element
    :param element: bs4 text or node
    :return: xpath as string
    """
    components = []
    child = element if element.name else element.parent
    for parent in child.parents:
        """
        @type parent: bs4.element.Tag
        """
        previous = itertools.islice(parent.children, 0, parent.contents.index(child))
        xpath_tag = child.name
        xpath_index = sum(1 for i in previous if i.name == xpath_tag) + 1
        components.append(xpath_tag if xpath_index == 1 else '%s[%d]' % (xpath_tag, xpath_index))
        child = parent
    components.reverse()
    return '/%s' % '/'.join(components)


driver = webdriver.Chrome(executable_path=YOUR_CHROMEDRIVER_PATH)
driver.get(url = 'https://www.instagram.com/beforeeesunrise/')
source = driver.page_source

soup = bs(source, 'html.parser')
button = soup.find('button', text=re.compile(r'Follow'))
xpath_for_the_button = xpath_soup(button)

elm = driver.find_element_by_xpath(xpath_for_the_button)
elm.click()

...and works!

( but you need writing some code to log in with an account)

edited Dec 17, 2016 at 20:39

answered Dec 16, 2016 at 22:25

GustavoIP

9332 gold badges8 silver badges26 bronze badges

3 Comments

JeffC Over a year ago

This answer makes no sense... "It's not the main goal of selenium to provide rich functionalities to find elements on the page". Sure it is... Selenium was built to automate the browser. Beautiful Soup was created to scrape HTML. You wrote 20 lines of code for BS to prove that writing one line using Selenium was inefficient?!?

GustavoIP Over a year ago

You are absolutely wrong. The Selenium's functionalities to search for elements on the page will not be enough in any realistic scenario when you are web scraping a complex page like Instagram or Facebook. The purpose of Selenium is not to be a web scraping tool. If you want a script to interact with another people's website, you will need an HTML parser.

GustavoIP Over a year ago

...otherwise, you are just bad guessing.

Collectives™ on Stack Overflow

Trying to select element by xpath with Selenium but getting error "Unable to locate element"

3 Answers 3

Comments

2 Comments

...and works!

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

...and works!

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related