Why isn't python's selenium button.click() working on a twitter follow button?

Question

So I know I'm locating the correct element. I've tried it with xpath and css selectors, both are correctly retrieving the correct button.

Then when I call .click() on the button, it does not follow the account.

button = self.driver.find_element(By.CSS_SELECTOR, ".user-actions-follow-button.js-follow-btn.follow-button.btn")
print(button.text)
button.click()
print('should have followed')

Does anyone know why it's behaving like this?

Edit: Here's the whole class code:

class Scraper:
    def __init__(self):
        self.driver = webdriver.PhantomJS(executable_path='phantomjs')
        self.loggedIn = False;


    def login(self, url, username, password):
        self.driver.get(url)

        try:
            usernameTextField = self.driver.find_element_by_css_selector(".js-username-field.email-input.js-initial-focus")

            passwordTextField = self.driver.find_element_by_css_selector('.js-password-field')

            usernameTextField.send_keys(username)
            passwordTextField.send_keys(password + '\n')
        except:
            print('Already logged in')

        try:
            WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.ID, 'dashboard-profile-prompt'))
            )

        except:
            self.loggedIn = False

        finally:
            print('succesfully logged in')
            self.loggedIn = True

    def followByUrl(self, url):
        if self.loggedIn:
            self.driver.get(url)

            actions = ActionChains(self.driver)
            wait = WebDriverWait(self.driver, 10)
            follow = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".user-actions-follow-button.js-follow-btn.follow-button.btn")))


            print(follow.text)

            actions.move_to_element(follow).click().perform()

            print('should have followed')

And here's a picture of the element

What error are you getting? An Element not visible error suggests the element isn't loaded yet, and is common on dynamically-loaded pages. — alphazwest
– alphazwest, Commented Dec 15, 2016 at 19:35
@Frank Not getting any errors. It's also correctly printing out the button.text — joe
– joe, Commented Dec 15, 2016 at 19:36
Usually when I run into that sort of issue, it means the wrong element is receiving the action. — alphazwest
– alphazwest, Commented Dec 15, 2016 at 19:46
@Frank I don't see this being wrong? The same method has worked for me on other scrapers also it's printing out the text correctly. imgur.com/a/bio5K — joe
– joe, Commented Dec 15, 2016 at 19:51

Community · Accepted Answer · 2017-05-23 12:09:49Z

1

First of all, since you are using PhantomJS, pretending not to be PhantomJS might solve the problem, see this post with a working Python/Selenium sample:

Set phantomjs user-agent string

And, make sure you are clicking the same button you are checking to be clicked. There can be multiple "Follow" buttons on a page.

Also, add an Explicit Wait to wait for the button to become clickable:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

follow = wait.until(element_to_be_clickable((By.CSS_SELECTOR, "button.follow-button")))
follow.click()

You may also try moving to the element and then clicking via "ActionChains":

from selenium.webdriver import ActionChains

actions = ActionChains(driver)
actions.move_to_element(follow).click().perform()

Or, you can click "via JavaScript":

driver.execute_script("arguments[0].click();", follow)

And don't forget to stay on the legal side, follow the Terms of Use, be a good web-scraping citizen.

edited May 23, 2017 at 12:09

CommunityBot

11 silver badge

answered Dec 15, 2016 at 19:39

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

joe Over a year ago

Tried everything, still nothing is working. Do you think twitter may be detecting the request is coming from a bot?

alecxe Over a year ago

@joe doubt so, though you have to follow the rules to stay on the legal side of things. What browser do you use? Could you show the complete code you have so far? Thanks.

joe Over a year ago

I'm using PhantomJS, going to edit with my whole class so far

alecxe Over a year ago

@joe ah, that might be important. Try changing the user-agent (coderwall.com/p/9jgaeq/set-phantomjs-user-agent-string).

alphazwest Over a year ago

Another means of diagnosis may be to try to use chromedriver or the firefox in your webdriver instantiation to see if that resolves the issue. I have had occasions where what worked in chrome driver didn't work in phantomjs. @alecxe 's user agent suggestion would likely resolve that issue.

|

joe · Accepted Answer · 2016-12-15 20:32:35Z

0

I got it to work by switching from the PhantomJS driver to chromedriver.

answered Dec 15, 2016 at 20:32

joe

1,7334 gold badges18 silver badges37 bronze badges

Collectives™ on Stack Overflow

Why isn't python's selenium button.click() working on a twitter follow button?

2 Answers 2

10 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related