Using Python requests module to collect data tables from drop-down list from a webpage

Question

I am trying to scrape information from this webpage- https://www.dmr.nd.gov/oilgas/bakkenwells.asp

I want to collect all data tables from dropdown menu and use Beautiful Soup and requests module to pull the information.

import requests

from bs4 import BeautifulSoup
page = requests.get('https://www.dmr.nd.gov/oilgas/bakkenwells.asp')
soup = BeautifulSoup(page.text, 'html.parser')

Jules Civel · Accepted Answer · 2024-02-29 12:22:07Z

0

I am afraid you can't easily do it with Beautifoul Soup only.

Since this website is a SPA (single page application), you need to perform clicks in order to get all the needed informations, you can't just requests the page, and BS4 does not offer that.

To do it, you can use Selenium module. Here is an interesting answer on how to use Selenium to perform clicks on dropdown values that you can try. selenium iterate through options in dropdown select

Have a nice day

edited Feb 29, 2024 at 12:22

answered Feb 29, 2024 at 10:38

Jules Civel

6416 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sahil Over a year ago

Hey, is it possible to collect data table separately for each formation using just beautiful soup

Jules Civel Over a year ago

stackoverflow.com/questions/57062946/…

Collectives™ on Stack Overflow

Using Python requests module to collect data tables from drop-down list from a webpage

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related