Python Webscrape (using BeautifulSoup) question

Question

I am trying to webscrape this site https://www.edgeprop.sg/condo-apartment/aquarius-by-the-park to get the Land Size (sqm) in the overview table. Result should give me 40,608

However, I am unable to get the result I want. Here is my code:

#[Python] test webscrape on edgeprop
import gspread
import json
from oauth2client.service_account import ServiceAccountCredentials
from openpyxl.worksheet import worksheet
from requests.api import request
import requests
import time
from requests.models import Response
import scrapy
from bs4 import BeautifulSoup
from six import add_metaclass, class_types


query_string='https://www.edgeprop.sg/condo-apartment/aquarius-by-the-park'  
resp = requests.get(query_string)   
soup = BeautifulSoup(resp.content,'html.parser')
print("soup is: ", query_string)

try:
    landsize = soup.find_all("h4",class_="detail-title__text")
    print("Landsize is: ", landsize)

except IndexError:
    pass

dimay · Accepted Answer · 2022-02-19 17:02:38Z

1

Try this:

import json
import requests
from bs4 import BeautifulSoup

query_string='https://www.edgeprop.sg/condo-apartment/aquarius-by-the-park'  

resp = requests.get(query_string) 
  
soup = BeautifulSoup(resp.content,'html.parser')

# get data with all info
data = soup.find("script", id="__NEXT_DATA__").text

# convert string to python dict
json_data = json.loads(data)

# get land_size from dict
print(json_data["props"]["pageProps"]["projectInfo"]["data"]["land_size"])

In html response you can find json which includes all information.

edited Feb 19, 2022 at 17:02

answered Nov 27, 2021 at 15:52

dimay

2,8341 gold badge17 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Chris lee Over a year ago

Thank you so much! To be honest i have no idea how this works, but i'll try to figure it out!! Appreciate it so much

dimay Over a year ago

@Chrislee if you want better understand what happened try save your response like a file and look inside

Chris lee Over a year ago

Dimay, could you share with me how did u get this result from the webpage? print(json_data["props"]["pageProps"]["projectInfo"]["data"]["land_size"])

Collectives™ on Stack Overflow

Python Webscrape (using BeautifulSoup) question

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related