Remove duplicate URLs in a list in python

Question

This code giving me duplicate URLs, how do I filter them

sg = []
for url in soup.find_all('a', attrs={'href': re.compile("^https://www.somewebsite")}):
    print(url['href'])
    sg.append(url['href'])
print(sg)

thanx it did work

sebhat GG
– sebhat GG

2019-05-08 13:27:12 +00:00
Commented May 8, 2019 at 13:27 — sebhat GG
– sebhat GG, Commented May 8, 2019 at 13:27

Kafels · Accepted Answer · 2019-05-08 12:33:24Z

1

You can check if url is already inserted on list

sg = []
for url in soup.find_all('a', attrs={'href': re.compile("^https://www.somewebsite")}):
    href = url['href'])
    print(href)
    if href not in sg:
        sg.append(href)
print(sg)

answered May 8, 2019 at 12:33

Kafels

4,0891 gold badge18 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nevermore · Accepted Answer · 2019-05-08 12:32:55Z

0

You can use a set instead of a list

sg = set()
for url in soup.find_all('a', attrs={'href': re.compile("^https://www.somewebsite")}):
    print(url['href'])
    sg.add(url['href'])
print(sg)

answered May 8, 2019 at 12:32

Nevermore

7,4797 gold badges48 silver badges68 bronze badges

Comments

Manu mathew · Accepted Answer · 2019-05-08 12:34:51Z

0

Instead of a list, using a set would solve the issue.

sg = set()
for url in soup.find_all('a', attrs={'href': re.compile("^https://www.somewebsite")}):
    print(url['href'])
    sg.add(url['href'])
print(sg)

answered May 8, 2019 at 12:34

Manu mathew

9998 silver badges26 bronze badges

1 Comment

sebhat GG Over a year ago

I think the webpage has two separate NAV, one for desktop another for mobile. Beautifulsoup grabbing both, I guess.

Collectives™ on Stack Overflow

Remove duplicate URLs in a list in python

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related