My script is writing the last item to csv [python and selenium]

Question

I am trying to automate (selenium) the process of writing the contents of my transcript to a CSV file with python. So far I have fine-tuned the script to click all the divs on the first page and extract data. But when I've tried to output the scraped data to CSV; It only writes the last item data.

here is my code:

i = 1
while i < 10:
    myxpath = "/html/body/div[1]/div[3]/div/div[1]/leads-list/div/ul/li["+str(i)+"]"
    print(myxpath)
    time.sleep(5)
    item = (By.XPATH, '//*[@id="description"]')
    WebDriverWait(driver, 50).until(EC.element_to_be_clickable(item))
    Itemlinks = driver.find_elements_by_xpath(myxpath)
    print('total items displayed in current page', len(Itemlinks))
    for items in Itemlinks:
        items.click()
        print("item clicked")
        time.sleep(10)
        sections = driver.find_elements_by_xpath('//*[@id="longDescriptionHTML"]')
        with open('zbuyer6.csv', 'w') as file:
            writer = csv.writer(file)
            for section in sections:
                name = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/div[2]/h1").text
                email = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/ul[1]/li[1]/div[1]/ul/li[2]").text
                print(name, email)
            writer.writerow((name, email))
            driver.find_element_by_class_name("back-link.pull-left.col-md-3").click()
            driver.execute_script("window.scrollTo(0, 50)")
            time.sleep(5)
    i += 1
    continue

Result:

I expect a CSV file with rows and separate columns for every field. Currently, I only get the last item data. Thanks.

Kaushal panchal · Accepted Answer · 2021-08-24 15:39:40Z

You are opening file in write mode, so in every iteration it clears the data and write the output. Try this:

i = 1
with open('zbuyer6.csv', 'w') as file:
    writer = csv.writer(file)
    while i < 10:
        myxpath = "/html/body/div[1]/div[3]/div/div[1]/leads-list/div/ul/li["+str(i)+"]"
        print(myxpath)
        time.sleep(5)
        item = (By.XPATH, '//*[@id="description"]')
        WebDriverWait(driver, 50).until(EC.element_to_be_clickable(item))
        Itemlinks = driver.find_elements_by_xpath(myxpath)
        print('total items displayed in current page', len(Itemlinks))
        for items in Itemlinks:
            items.click()
            print("item clicked")
            time.sleep(10)
            sections = driver.find_elements_by_xpath('//*[@id="longDescriptionHTML"]')
            for section in sections:
                name = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/div[2]/h1").text
                email = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/ul[1]/li[1]/div[1]/ul/li[2]").text
                print(name, email)
            writer.writerow((name, email))
            driver.find_element_by_class_name("back-link.pull-left.col-md-3").click()
            driver.execute_script("window.scrollTo(0, 50)")
            time.sleep(5)
        i += 1
        continue

Jeremy Savage · Accepted Answer · 2021-08-24 15:43:49Z

Your code has a couple of small mistakes in it.

Firstly you need to open your file in append mode or it will overwrite the file each time.

Secondly you need to include your writer.writerow((name, email)) in the for loop so it writes every name and email and not just a single one which it is currently doing.

Finally you should probably remove the selenium components from your file writing as you should close the file as soon as possible so it only contains necessary code.

        with open('zbuyer6.csv', 'a') as file:
            writer = csv.writer(file)
            for section in sections:
                name = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/div[2]/h1").text
                email = section.find_element_by_xpath(
                    "/html/body/div[1]/div[3]/div/div[2]/lead-detail/div[4]/div[1]/ul[1]/li[1]/div[1]/ul/li[2]").text
                print(name, email)
                writer.writerow((name, email))
        driver.find_element_by_class_name("back-link.pull-left.col-md-3").click()
        driver.execute_script("window.scrollTo(0, 50)")
        time.sleep(5)

These may not be the only issues as I cannot see your site, but this should help.

Collectives™ on Stack Overflow

My script is writing the last item to csv [python and selenium]

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related