Python BeautifulSoup Extracting Data From Header

Question

This is a follow-up from another question. Thanks for the help so far.

I've got some code to loop through a page and create a dataframe. I'm trying to add a third piece of information but it is contained within the header so it's just returning blank. The level information contained in the td and h3 part of the code. It returns the error "AttributeError: 'NoneType' object has no attribute 'text'" If I change level.h3.text to level.h3 it will run but then it will have the full tags in the data frame, instead of just the number.

import urllib
import bs4 as bs
import pandas as pd
#import csv as csv

sauce = urllib.request.urlopen('https://us.diablo3.com/en/item/helm/').read()
soup = bs.BeautifulSoup(sauce, 'lxml')

item_details =  soup.find('tbody')

names = item_details.find_all('div', class_='item-details')
types = item_details.find_all('ul', class_='item-type')
#levels = item_details.find_all('h3', class_='subheader-3')
levels = item_details.find_all('td', class_='column-level align-center')
print(levels)

mytable = []



for name, type, level in zip(names, types, levels):
    mytable.append((name.h3.a.text, type.span.text, level.h3.text))



export = pd.DataFrame(mytable, columns=('Item', 'Type','Level'))

Andersson · Accepted Answer · 2019-01-17 16:56:39Z

2

Try to modify your code as below:

for name, type, level in zip(names, types, levels):
    mytable.append((name.h3.a.text, type.span.text, level.h3.text if level.h3 else "No level"))

Now "No level" (you can use "N/A", None or whatever you like the most) will be added as third value in case there is no level (no header)

answered Jan 17, 2019 at 16:56

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lucas Over a year ago

Thanks! Much appreciated.

Collectives™ on Stack Overflow

Python BeautifulSoup Extracting Data From Header

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related