Web Scraping using BeautifulSoup on Python

Question

I am trying to scrape the Name of each API and Category from this website https://www.programmableweb.com/apis/directory and print them out in this format

name: Google Maps

Category: Mapping

For some reason my code only prints the first row .

My Code

from bs4 import BeautifulSoup as bs

import requests

url = 'https://www.programmableweb.com/apis/directory'

response = requests.get(url)

data = response.text

soup = bs(data, 'html.parser')  

info = soup.find_all('table',{'class':'views-table cols-4 table'})



for i in info:

    name = soup.find('td',{'class':'views-field views-field-title col-md-3'}).text
    category = soup.find('td',{'class':'views-field views-field-field-article-primary-category'}).text
    print('name:',name, '\nCategory:', category)

If you can even help me further what I'm trying to do is:

API name
API URL
API category
API description when you clink in the link
scrape next pages until no remaining pages
use pandas to make it into a DataFrame and then put it into a csv file

chitown88 · Accepted Answer · 2019-10-07 14:55:02Z

You're not iterating through the rows of the table. You find_all for <table> tags (which there is only 1, and then try to iterate through those tags. What you want to do is find all the <tr> tags, within the <table> tag, and then iterate through the <tr> tags. you are also only grabing the first element out of the soup object, not the info object.

Easier solution, since it is the <table> tag you are after, use pandas to grab that (it actually uses beautifulsoup under the hood). But it'll do all that hard work for you:

import pandas as pd

url = 'https://www.programmableweb.com/apis/directory'
table = pd.read_html(url)[0]

Output:

print (table.to_string())
                      API Name                                        Description              Category   Submitted
0                  Google Maps  [This API is no longer available. Google Maps'...               Mapping  12.05.2005
1                      Twitter  [This API is no longer available. It has been ...                Social  12.08.2006
2                      YouTube  The Data API allows users to integrate their p...                 Video  02.08.2006
3                       Flickr  The Flickr API can be used to retrieve photos ...                Photos  09.04.2005
4                     Facebook  [This API is no longer available. Its function...                Social  08.16.2006
5   Amazon Product Advertising  What was formerly the ECS - eCommerce Service ...             eCommerce  12.02.2005
6                       Twilio  Twilio provides a simple hosted API and markup...             Telephony  01.09.2009
7                      Last.fm  The Last.fm API gives users the ability to bui...                 Music  10.30.2005
8                   Twilio SMS  Twilio provides a simple hosted API and markup...             Messaging  02.19.2010
9          Microsoft Bing Maps  Bing Maps API and Interactive SDK features an ...               Mapping  12.02.2005
10                 del.icio.us  From their site: del.icio.us is a social bookm...             Bookmarks  10.30.2005
11           Google App Engine  [This API is no longer available. Its function...                 Tools  12.05.2008
12                  Foursquare  The Foursquare Places API provides location ba...                Social  09.10.2009
13             Google Homepage  From their site: The Google Gadgets API provid...               Widgets  12.14.2005
14         DocuSign Enterprise  DocuSign is a Cloud based legally compliant eS...  Electronic Signature  03.29.2008
15                   Amazon S3  Since 2006 Amazon Web Services has been offeri...               Storage  03.14.2006
16              Google AdSense  The Google AdSense API is ideal for developers...           Advertising  06.01.2006
17                    GeoNames  Geonames is a geographical database with web s...             Reference  01.12.2006
18                   Wikipedia  The unofficial Wikipedia API. Because Wikipedi...             Reference  09.05.2008
19                         Box  Box is a modern content management platform th...               Content  03.07.2006
20                  Amazon EC2  The Amazon Elastic Compute Cloud (Amazon EC2) ...                 Cloud  08.25.2006
21                        Bing  [The Bing API is now the Bing Web Search API. ...                Search  06.04.2009
22                    LinkedIn  LinkedIn is the world's largest business socia...                Social  12.10.2007
23             Instagram Graph  Instagram is a photo sharing iPhone app and se...                Photos  12.15.2010
24                 Yelp Fusion  The Yelp Fusion APIs are RESTful APIs and user...       Recommendations  08.03.2007

And if you click on next page, you see https://www.programmableweb.com/apis/directory?page=1, so it's just a matter of iterating through that in a for loop until the end, and append to your dataframe after each iteration.

Collectives™ on Stack Overflow

Web Scraping using BeautifulSoup on Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related