AttributeError when scraping data from URL via Python

Question

I am using the code below to try an extract the data from the table in this URL. I asked the same question here and got an Answer for it. However, despite the code from the Answer working at that time I've now come to realize that data in the code hasn't been defined and hence the code below results in the error AttributeError: 'NoneType' object has no attribute 'text' in the line final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]. I have asked the user what data is, but I haven't been able to figure it out myself and was therefore hoping for some help in this regard.

from bs4 import BeautifulSoup
import requests
from itertools import islice
import pandas as pd

r = requests.get(
    "http://www.federalreserve.gov/econresdata/researchdata/feds200628_1.html")
soup = BeautifulSoup(r.content)

h = data.find_all("th", scope="col") #Issue
final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]
headers = [th.text for th in h]

headers.insert(0,"dates")
df = pd.DataFrame(final,columns=headers)

print df

Vikas Ojha · Accepted Answer · 2015-06-16 21:43:29Z

1

from bs4 import BeautifulSoup
import requests
from itertools import islice

r = requests.get(
    "http://www.federalreserve.gov/econresdata/researchdata/feds200628_1.html")
soup = BeautifulSoup(r.content)

data = soup.find('table', attrs={'rules': 'all'})
h = data.find_all("th", scope="col") #Issue
final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr")]
headers = [th.text for th in h]

answered Jun 16, 2015 at 21:43

Vikas Ojha

7,0106 gold badges24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user131983 Over a year ago

Thank You. However, I still get the Error AttributeError: 'NoneType' object has no attribute 'text' in the line final = [[t.th.text] + [ele.text for ele in t.find_all("td")] for t in h[-1].find_all_next("tr"). Do you know why this might be

Vikas Ojha Over a year ago

Are you sure you copied it properly? I just tried it and its working perfectly fine.

user131983 Over a year ago

Yes, its exactly the same. Do you think the version of beautifulsoup or Python could be having an impact?

Vikas Ojha Over a year ago

Try using htmllib. First install it as pip install html5lib. Then use it to generate the soup - soup = BeautifulSoup(r.content, 'html5lib')

Collectives™ on Stack Overflow

AttributeError when scraping data from URL via Python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related