I'm trying to scrape the news articles from prnewswire.com. Each article is stored in a div called "row".
The problem for me is that some article previews have an image beside their title and description. Therefor under the "row"-classes it's either the class name "card" (with image) or "col-sm-12 card" (without image):
My current code is the following:
import requests
from bs4 import BeautifulSoup
import pandas
headers = {
'User-Agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko)' +
'Version/14.0.1 Safari/605.1.15'
}
articlelist = []
def getarticles(page):
url = 'https://www.prnewswire.com/news-releases/news-releases-list/?page=' + str(page) + '&pagesize=100'
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.text, 'html.parser')
prnewswire_articles = soup.find_all('div', {'class': 'col-sm-12 card'})
for item in prnewswire_articles:
prnewswire_article = {
'page': page,
'article_title': item.find('h3').text,
'article_link': 'https://www.prnewswire.com/' +
item.find('a')['href'],
'article_description': item.find('p').text,
}
articlelist.append(prnewswire_article)
return
for x in range(1, 3):
getarticles(x)
df = pandas.DataFrame(articlelist)
print(df.head())
print(len(df))
df.to_excel('PRNewsWire.xlsx', index=False)
print('Finished.')
I have discovered the following: In the line where I declare "prnewswire_articles" and look for a div with a certain class name, I get the results I want with the class "col-sm-12 card". But "card" or "row" doesn't work.
I noticed that the html structure of "card" classes is different to "col-sm-12 card" classes, but they both contain one "h3" element (the article's title), one "a href" and one "p" element 
This is the error message I get when using "row" or "card" as class name:
Traceback (most recent call last):
File "/Users/myname/PycharmProjects/projectname/prnewswire.py", line 33, in <module>
getarticles(x)
File "/Users/myname/PycharmProjects/projectname/prnewswire.py", line 24, in getarticles
'article_title': item.find('h3').text,
AttributeError: 'NoneType' object has no attribute 'text'
I've searched a whole day and didn't find anything. Just recently started learning Python, so I'm sorry if this is a stupid mistake, but I am at the end of finding an answer. Would really appreciate help a lot! :)
