Beautiful soup looping through array of URLs

Question

I'm trying to loop through an array of URLs and scrape board members from a list of companies. There seems to be a problem with my loop below, where it's only running the first element in the array and duplicating results. Any help with this would be appreciated. Code:

from bs4 import BeautifulSoup
import requests

#array of URLs to loop through, will be larger once I get the loop working correctly
tickers = ['http://www.reuters.com/finance/stocks/companyOfficers?symbol=AAPL.O', 'http://www.reuters.com/finance/stocks/companyOfficers?symbol=GOOG.O']

board_members = []
output = []
soup = BeautifulSoup(html, "html.parser")

for t in tickers:
    html = requests.get(t).text
    officer_table = soup.find('table', {"class" : "dataTable"})
    for row in officer_table.find_all('tr'):
        cols = row.find_all('td')
        if len(cols) == 4:
            board_members.append((t, cols[0].text.strip(), cols[1].text.strip(), cols[2].text.strip(), cols[3].text.strip()))

        for t, name, age, year_joined, position in board_members:
            output.append(('{} {:35} {} {} {}'.format(t, name, age, year_joined, position)))

This code shouldn't run. BeautifulSoup(html will error that html is not defined — OneCricketeer
– OneCricketeer, Commented Dec 1, 2016 at 5:54
Anyways, consider storing just the symbols in the list. Format the URL later when you make the request — OneCricketeer
– OneCricketeer, Commented Dec 1, 2016 at 5:55

宏杰李 · Accepted Answer · 2016-12-03 08:19:46Z

1

soup = BeautifulSoup(html, "html.parser")

for t in tickers:
    html = requests.get(t).text
    officer_table = soup.find('table', {"class" : "dataTable"})

you put soup out of the for loop, this will cause a error, because the 'html' dose not exist when you use BeautifulSoup(html, "html.parser") just put it in the loop after html is assigned.

for t in tickers:
    html = requests.get(t).text
    soup = BeautifulSoup(html, "html.parser")
    officer_table = soup.find('table', {"class" : "dataTable"})

answered Dec 3, 2016 at 8:19

宏杰李

12.2k2 gold badges32 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Beautiful soup looping through array of URLs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related