Load URLs to python through text file

Question

I want my for loop to take URLs one by one and hit and do some operations like, [clicking button and extracting text] first time when the URL is hit it needs uname, pwd and login from next time onwards no need credentials, it can directly just click button and extract URL.

Operational code : for clicking and extracting is working fine. only for loop has some issue

 def lookup(driver,uname,pwd):
    with open("A.txt") as f:
        for line in f:
            url = line.strip()
            print("Getting %s" % url)
            driver.get(url)
            box = driver.wait.until(EC.presence_of_element_located((By.NAME, "j_username")))
            box.send_keys(uname)
            box = driver.wait.until(EC.presence_of_element_located((By.NAME, "j_password")))
            box.send_keys(pwd)
            button = driver.wait.until(EC.element_to_be_clickable((By.NAME, "login")))
            button.click()
            button = driver.wait.until(EC.presence_of_element_located((By.ID, "fetchCandidateChanges_0")))
            button.click()
            driver.wait = WebDriverWait(driver, 15)
            time.sleep(15)
            labels = driver.find_elements_by_xpath('//label[@class="pagebanner"]')
            for label in labels:
                print (label.text)

Never used selenium but I suppose you need to get rid of the double quotes in driver.get("content"). It should be driver.get(content). And you missed a loop here as well, if you want to read all the URLs. — Nitheesh A S
– Nitheesh A S, Commented Jun 29, 2017 at 7:00

Thierry Lathuille · Accepted Answer · 2017-06-29 07:04:58Z

1

If each line contains an URL:

def lookup(driver,uname,pwd):
    with open("A.txt") as f:
        for line in f:
            url = line.strip()  # to remove the trailing \n
            print("Getting %s" % url)
            driver.get(url)

A for loop will iterate on the lines of the file.

edited Jun 29, 2017 at 7:04

answered Jun 29, 2017 at 7:00

Thierry Lathuille

24.4k10 gold badges49 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Nathan.Eilisha Shiraini Over a year ago

You forgot the indentation of the for loop ^^

Thierry Lathuille Over a year ago

Yes, and stripping the trailing \n :/, I just noticed, thanks!

Abdul Salam Over a year ago

@ThierryLathuille Thanks for the answer.. when tried with 3 URLs. driver.get(url) is loading the 3rd URL alone

Thierry Lathuille Over a year ago

If all of your URLs are valid, maybe there's something going wrong in another part of your code...

DeepSpace · Accepted Answer · 2017-06-29 07:06:19Z

1

read() reads the whole file as a single string, so content is the string "url1\nurl2\url3" etc
driver.get("content") You assign the content of the file to content but passing the string "content" to get. content and "content" are not the same thing.

Instead, you should read the file line by line and pass each url to get:

def lookup(driver, uname, pwd):
    with open("A.txt") as f:
        urls = f.readlines()
    for url in urls:
        print(url)
        driver.get(url)

or

def lookup(driver, uname, pwd):
    with open("A.txt") as f:
        for url in f:
            print(url)
            driver.get(url)

I usually tend to go with the first style as it doesn't keep the file opened unnecessarily.

edited Jun 29, 2017 at 7:06

answered Jun 29, 2017 at 7:01

DeepSpace

82.2k12 gold badges119 silver badges166 bronze badges

7 Comments

Abdul Salam Over a year ago

Thanks @DeepSpace. am able to read file now, but I wants to read the URLs one by one and for each URL I need to perform set of operations.. when tried with 3 URLs. driver.get(url) is loading the 3rd URL alone. please help

BoboDarph Over a year ago

Print an example of your textfile. If the urls arent separated by newlines then they will get mangleled when you iterate over them.

Abdul Salam Over a year ago

sample file : ops.stg2.ddd.com/… ops.stg2.ddd.com/… ops.stg2.ddd.com/…

Abdul Salam Over a year ago

ops.stg2.ddd.com/… ops.stg2.ddd.com/… ops.stg2.ddd.com/…

BoboDarph Over a year ago

This comments editor strips out newlines. That's why I said "print" not paste :D. If you are using DeepSpace's suggested implementation, note that the get actions might be executed in very fast succession in your browser and you won't be able to physically see the page change. Add a delay or some selenium logging to see which URLs are opened.

|

Collectives™ on Stack Overflow

Load URLs to python through text file

2 Answers 2

4 Comments

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related