Web Scraper - getting duplicates in output

Question

I am completely new to Python and just trying my coding skills for developing a few programs

I have coded the following program in Python 2.7 to fetch the profile URLs from the dir - http://www.uschirodirectory.com/entire-directory/list/alpha/a.html

However, I am noticing a lot of duplicate entries in the list of URLs fetched. Could someone please review the code and tell me if there's something that I am doing here or is there a way this code could be optimized further.

Many thanks

import requests
from bs4 import BeautifulSoup

def web_crawler(max_pages):
p = '?site='
page = 1
alpha = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
while page <= max_pages:
    for i in alpha:
        url = 'http://www.uschirodirectory.com/entire-directory/list/alpha/' + str(i) + '.html' + p + str(page)
        code = requests.get(url)
        text = code.text
        soup = BeautifulSoup(text)
        for link in soup.findAll('a',{'class':'btn'}):
            href = 'http://www.uschirodirectory.com' + link.get('href')
            print(href)
    page += 1
i += alpha[0 + 1]

#Run the crawler
web_crawler

Igor Savinkin · Accepted Answer · 2015-05-04 09:17:59Z

2

Basically your code is ok. You might get lots of duplicate links cause the directory results are designed to issue results not just for 1-st letter in doctor name but also for 1-st letter in company title or other important db field.

answered May 4, 2015 at 9:17

Igor Savinkin

6,36910 gold badges39 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

pb_ng Over a year ago

Thanks for reviewing this for me. I am a Python novice learner and feedback from pro Python programmers like you is immensely valuable to me.

Ranvijay sachan · Accepted Answer · 2015-05-04 09:51:14Z

2

You can store the data in a list and also you can remove the duplicate url using this code :

parsedData = []

data = {}

if not any(d['url'] == data['url'] for d in data):

   parsedData.append(data)

edited May 4, 2015 at 9:51

answered May 4, 2015 at 9:46

Ranvijay sachan

2,4424 gold badges32 silver badges53 bronze badges

Collectives™ on Stack Overflow

Web Scraper - getting duplicates in output

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related