How to avoid AttributeError: 'NoneType' object has no attribute 'text' webscraping subreddits?

Question

Trying to make a program that gets a random subreddit, displays the title to the terminal, asks the user if they want to see it and if so opens a browser with that subreddit.

I'm running into an issue here whilst doing so. I am very new with python/beautiful soup just started using it a few weeks ago for school any help will be very appreciated.

import requests
from bs4 import BeautifulSoup
import webbrowser

while True:
    url = requests.get("https://www.reddit.com/r/random")
    soup = BeautifulSoup(url.content, "html.parser")
    title = soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ").text ## this is supposed to get the title of the subreddit and this is where my error is occurring

    print(f"{title} \nSelect this subreddit? (Y/N)")
    ans = input("").lower()

    if ans == "y":
        url = "https://www.reddit.com/%s" % title ## Some issue, not sure what
        webbrowser.open(url)
        break
    elif ans == "n":
        print("Try again!")
        continue
    else:
        print("Wrong choice!")
        break

Sharim09 · Accepted Answer · 2022-09-09 19:08:45Z

1

soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ") returns None if there is no element with the given class, That's why you got this error

Try to catch the error using try-except and print something if soup not found any title.

Try this code.

from bs4 import BeautifulSoup

import webbrowser

while True:
    url = requests.get("https://www.reddit.com/r/random")
    soup = BeautifulSoup(url.content, "html.parser")
    try:
        title = soup.find(class_="_2yYPPW47QxD4lFQTKpfpLQ").text
    ## this is supposed to get the title of the subreddit and this is where my error is occurring
    except AttributeError:
        print("Title Not Found")

    else:
        print(f"{title} \nSelect this subreddit? (Y/N)")
        ans = input("").lower()

        if ans == "y":
            url = "https://www.reddit.com/%s" % title ## Some issue, not sure what
            webbrowser.open(url)
            break
        elif ans == "n":
            print("Try again!")
            continue
        else:
            print("Wrong choice!")
            break

answered Sep 9, 2022 at 19:08

Sharim09

6,2262 gold badges10 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Trick Over a year ago

This did work! But unfortunately that would render my program useless. Is there an alternative so I can ensure I am getting the title of the subreddit somehow?

Sharim09 Over a year ago

I will edit the answer if I got some solution

HedgeHog · Accepted Answer · 2022-09-09 20:17:47Z

1

Element you try to find() is not available in your soup and calling a methode on None raises this error, so:

Always and at first check your response, does it include the expected information - You will be detected as bot, so add some headers:
```
url = requests.get('https://www.reddit.com/r/random', headers = {'User-Agent': 'Mozilla/5.0'})
```
Avoid to select your elements by dynamic looking classes, instead us static identifier and HTML structure, also select the right elements:
```
soup.h2.get_text(strip=True)
```

Example

from bs4 import BeautifulSoup
import requests, webbrowser

while True:
    url = requests.get('https://www.reddit.com/r/random', headers = {'User-Agent': 'Mozilla/5.0'})
    soup = BeautifulSoup(url.content)
    title = soup.h2.get_text(strip=True)
    print(f"{title} \nSelect this subreddit? (Y/N)")
    ans = input("").lower()

    if ans == "y":
        url = f"https://www.reddit.com/{title}"
        webbrowser.open(url)
        break
    elif ans == "n":
        print("Try again!")
        continue
    else:
        print("Wrong choice!")
        break

edited Sep 9, 2022 at 20:17

answered Sep 9, 2022 at 20:11

HedgeHog

25.4k5 gold badges18 silver badges43 bronze badges

3 Comments

Trick Over a year ago

Thank you for this information it is extremely helpful. Though I believe the issue of not getting a title still persists. the terminal says "title not defined" Any work around to get a title no matter what of the random sub reddit?

HedgeHog Over a year ago

There is no such message in my example - What is your response or soup telling you if you print it? Is the element available?

Trick Over a year ago

I just realized my error, you changed code in the if statement too. it works perfectly now thank you SO much!!

Collectives™ on Stack Overflow

How to avoid AttributeError: 'NoneType' object has no attribute 'text' webscraping subreddits?

2 Answers 2

2 Comments

Example

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Example

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related