Web Scraping: How to scrape event details from a dynamic website with Python?

Question

I'm trying to scrape event details (event name, date, time, and tags) from the Central Park events calendar at https://www.centralparknyc.org/calendar. The website is dynamic, and it seems that the event details are not loading while scraping.

I've attempted to use requests and BeautifulSoup to scrape the content, but I'm encountering a 403 Forbidden error, which I suspect is due to the website's bot protection measures.

Could someone guide me on how to properly scrape this dynamic content using Python? Any advice on handling dynamic content and bot detection would be greatly appreciated.

import requests
from bs4 import BeautifulSoup

url = 'https://www.centralparknyc.org/calendar'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
}

response = requests.get(url, headers=headers)

if response.status_code == 200:
    # Process the page
    soup = BeautifulSoup(response.content, 'html.parser')
    # ... scraping logic here ...
else:
    print(f'Failed to retrieve the webpage: {response.status_code}')

Prayson W. Daniel · Accepted Answer · 2023-11-02 07:52:42Z

1

You can directly get data from the API that populates the web page. That would remove the need of BeautifulSoup. If you open the developer tools on your browser, you will see that when you click to load the next events, the page makes an API call that returns JSON data you are looking for.

We can use that to get data and tweak to get more per call, for example getting data at page 1 but with 25 locations.

import requests

URL = "https://www.centralparknyc.org/calendar.json"
params = {"page":1, "elementsPerPage":25}

r = requests.get(URL, params=params)

print(r.json())

Developer Tool:

Note: Remember to read their Terms of Use.

edited Nov 2, 2023 at 7:52

answered Nov 2, 2023 at 6:21

Prayson W. Daniel

15.8k6 gold badges57 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Harshit soni Over a year ago

Thank you so much! I appreciate it! Is there any way I can scrape this website too: events.newyorkfamily.com I learning how to scrape websites and collect data, so looking at different ways to do this

Collectives™ on Stack Overflow

Web Scraping: How to scrape event details from a dynamic website with Python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related