Error while crawling web python

Question

when I try to run the code below this error was returned. I'd be much appreciated if someone can help to point out where I did wrong. Thank you.

Traceback (most recent call last):
  File "web_crawler.py", line 26, in <module>
    links = get_all_links(page)
  File "web_crawler.py", line 14, in get_all_links
    url, endpos = get_next_target(page)
  File "web_crawler.py", line 2, in get_next_target
    start_link = page.find("<a href=")
TypeError: a bytes-like object is required, not 'str'

def get_next_target(page):
    start_link = page.find("<a href=")
    if start_link == -1:
        return None, 0
    start_quote = page.find('"',start_link)
    end_quote = page.find('"',start_quote+1)
    url = page[start_quote+1:end_quote]
    print(url)
    return url, end_quote

def get_all_links(page):
    links = []
    while True:
        url, endpos = get_next_target(page)
        if url:
            links.append(url)
            page = page[endpos:]
        else:
            break
    return links

import requests
url='https://en.wikipedia.org/wiki/Moon'
r = requests.get(url)
page = r.content
links = get_all_links(page)

For the difference between r.content and r.text (when r is a response object returned from requests.get), see the documentation of requests — Odysseas
– Odysseas, Commented Apr 13, 2018 at 18:41

Azsgy · Accepted Answer · 2018-04-13 18:38:36Z

3

response.content is the raw contents of the request. They are not decoded it or anything, it's just the raw bytes.

What you want to use instead is the response.text attribute, which contains the decoded content as a string.

(You also probably want to use an html parsing library like BeautifulSoup instead of your current page.find approach)

answered Apr 13, 2018 at 18:38

Azsgy

3,3052 gold badges31 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error while crawling web python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related