1

I'm trying to scrape the promotion information of each product from a website by clicking on the product and go to its detailed page. When the spider clicks on the product, the web will ask it to log in, and I tried the following code:

    def __init__(self):
        self.driver = webdriver.Chrome(executable_path = '/usr/bin/chromedriver')
...
    def start_scraping(self, response):
        self.driver.get(response.url)    
        self.driver.find_element_by_id('fm-login-id').send_keys('iamgooglepenn')
        self.driver.find_element_by_id('fm-login-password').send_keys('HelloWorld1_')
        self.driver.find_element_by_class_name('fm-button fm-submit password-login').click()
        ...

However, there is NoSuchElementException when I run it.

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="fm-login-id"]"}
'spider_exceptions/NoSuchElementException': 14,

The HTML of the login page is as follows:

<div class='input-plain-wrap input-wrap-loginid'>
    <input id='fm-login-id' class='fm-text' name='fm-login-id'...>
    event
</div>

So, I'm pretty sure the id should be 'fm-login-id'. The reason I could think of that might cause this issue is that this login page is a popup. login page

Basically, it pops up in the middle of the main page. Looking at the HTML of the site, I can see that the login type seems to be a new HTML window

<!DOCTYPE html>
<html>event
....
<\html>

I'm not sure if this is the issue, and if so, how to fix it? Also, is there other reasons that might've caused the issue?

Improve this question
1

6 Answers 6

Reset to default
2

The popup will have an ID. You might have to add f'#{popup_id}' to the end of response.url. Like this URL: https://stackoverflow.com/questions/62906380/nosuchelementexception-when-using-selenium-python/62906409#62906409. It contains #62906409 because 62906409 is the ID of an element in the page.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

https://detail.tmall.com/item.htm?spm=a220m.1000858.1000725.6.2d375757n6RzBN&id=35550626143&skuId=4372921049418&user_id=1669409267&cat_id=2&is_b=1&rn=e69905ae0d51cc9e426ec93a52d95bcc This is the link I copied from browser, what would be the ID for the popup?
You have to inspect the popup and find it's ID.
Something like https://detail.tmall.com/item.htm?spm=a220m.1000858.1000725.6.2d375757n6RzBN&id=35550626143&skuId=4372921049418&user_id=1669409267&cat_id=2&is_b=1&rn=e69905ae0d51cc9e426ec93a52d95bcc#login-form might work. I just inspected the page. The ID of the form is login-form. Also, it looks like the popup doesn't have an ID.
I'm sure the popup is what's causing the error. For some reason, Selenium doesn't count the popup as an element in the page.
So, the e6605ae0d51.. part would be the form id? Should I do something like self.driver.get(response.url#{e6605ae0d51.})?
|
1

The login page inside a frame, you need switch it first:

#switch it first
self.driver.switch_to.frame(driver.find_element_by_id('J_loginIframe'))
self.driver.find_element_by_id('fm-login-id').send_keys('iamgooglepenn')
self.driver.find_element_by_id('fm-login-password').send_keys('HelloWorld1_')

And for login button you can't use .find_element_by_class_name, this method just for single class name. This element having multiple class name, so use .find_element_by_css_selector like bellow:

#submit button
self.driver.find_element_by_css_selector('.fm-button.fm-submit.password-login').click()
Improve this answer

4 Comments

Now it's saying selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="J_loginIframe"]"} Where did you locate the id of the frame?
@TianheXie Based on comment in answer other user, actually I thinking this issue refer to this link login page : https://login.tmall.com/?spm=a220o.7142085.a2226mz.1.6c997e96viyscY&redirectURL=https%3A%2F%2Fdetail.tmall.com%2Fitem.htm%3F. Is possible to share the real url?
Not sure what you meant by real URL? https://detail.tmall.com/item.htm?spm=a220m.1000858.1000725.6.2d375757n6RzBN&id=35550626143&skuId=4372921049418&user_id=1669409267&cat_id=2&is_b=1&rn=e69905ae0d51cc9e426ec93a52d95bcc is the URL I copied from the browser when the log in page poped up
The starting URL for the spider is 'https://list.tmall.com/search_product.htm?q=iPad' Then the spider will click on each individual product fore more detailed information. Once the spider clicks, the login page will pop up, which is the url I copied above
1

The login content seems to be nested in an iFrame element (if you trace it all the way to the top, you should find an iFrame with id="sufei-dialog-content"), which means you need to switch to that iFrame for that nested html before selecting your desired element, otherwise it will not work.

First you will need to use driver.switch_to.frame("sufei-dialog-content"), and then select your element with driver.find_element_by_name() or whatever you had.

A similar issue can be found here: Selenium and iframe in html

Improve this answer

Comments

0

Just a simple mistake:

<div class='input-plain-wrap input-wrap-loginid'>
    <input id='fm-login-id class='fm-text' name='fm-login-id'...>
    event
</div>

is actually supposed to be:

<div class='input-plain-wrap input-wrap-loginid'>
    <input id='fm-login-id' class='fm-text' name='fm-login-id'...>
    event
</div>

You forgot a single-quote.

Improve this answer

4 Comments

Sorry I made a typo when posting the question. This HTML is from the website I want to scrape information from. The code I'm having trouble with is the first part of posted
Is self.driver a web browser object like Chrome() from selenium.webdriver?
Yes, I added the part where I defined self.driver to the question
I'm not sure what you mean by popup ID? Any ideas where I might find the pop ID?
0

Have you tried driver.find_element_by_name('fm-login-id')?

Improve this answer

2 Comments

Yes, and it's still NoSuchElementException
Can you look at the page again? To me it seems that the pop up is created in an iFrame element with id id="sufei-dialog-content". If so you might need to switch to that with driver.switch_to.iframe(self,frame reference) before trying to get the element.
0

You should try finding the elements by their XPaths. You just have to inspect the element, right-click on it and copy its XPath. The XPath of the first <input ... is //*[@id="fm-login-id"].

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.