Can't scrape all the titles from the map on the webpage using the requests module

Question

I'm trying to create a script in Python to scrape all available titles that show up when clicking on the black-colored area on the map in this website.

For example, when I click on a certain area on the map, I see specific geometry details in the developer tools for that property, and I can retrieve the corresponding title using the parameters in the script. How can I get all the titles from the map using the requests module?

import re
import json
import urllib3
import requests
from pprint import pprint
from bs4 import BeautifulSoup

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

link = 'https://www.bcassessment.ca/Property/Info/SjAwMDAwQzRZMQ=='

params = {
    'f': 'json',
    'where': '',
    'geometry': json.dumps({'x':-13741429.991596092,'y':6199196.035742312,'spatialReference':{'wkid':102100,'latestWkid':3857}}),
    'returnGeometry': 'true',
    'geometryType': 'esriGeometryPoint',
    'inSR': '102100',
    'outFields': 'AFP_OID,UNIT_NUMBER,TOTAL_ASSESSED,ROLL,AREA_EVBC,JUR,TOTAL_LAND,TOTAL_BUILDING,ADDRESS,DESCRIPTION,STREET_NUMBER,STREET_NAME,OID_EVBC,SHORT_ADDRESS,IS_STRATA,FARM_FLAG,UTILITY_FLAG,MAJ_INDUSTRY_FLAG,MANAGED_FOREST_FLAG',
    'orderByFields': 'STREET_NAME,STREET_NUMBER,UNIT_NUMBER,SHORT_ADDRESS',
    'outSR': '102100'
}
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-language': 'en-US,en;q=0.9',
    'host': 'www.bcassessment.ca',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36'
}

headers_ano = {
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9',
    'host': 'arcgis.bcassessment.ca',
    'origin': 'https://www.bcassessment.ca',
    'referer': 'https://www.bcassessment.ca/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36'
}

with requests.Session() as session:
    resp = session.get(link,headers=headers,verify=False)

    params['token'] = re.findall(r"gistoken[^']+\'(.*?)\';",resp.text)[0]
    query_url  = re.findall(r"mapserverUrl[^']+\'(.*?)\';",resp.text)[0] + "/0/query"
    res = session.get(query_url,params=params,headers=headers_ano,verify=False)
    print(res.status_code)
    for item in res.json()['features']:
        print(item['attributes']['ADDRESS'])

The output it produces: 262 DURRANCE RD VICTORIA

what did you get with your code? If you get error then show it in question. If you get wrong result then also show it in question. Don't expect that we will run code to see details. You could also add screenshot to show what "area" and "details" you mean. — furas
– furas, Commented Jul 20 at 22:41
Looking at the code (on the site) you would need to brute force every combination of x and y over the area your looking at, then trim duplicate or empty results. You are aware that the site explicitly forbids data scraping right? — NickSlash
– NickSlash, Commented Jul 20 at 22:45

koal44 · Accepted Answer · 2025-07-29 07:44:29Z

You're trying to uncover all the hidden parcels (polygons) on an ArcGIS map. Click anywhere, and the site gives you back the geometry + attributes for the parcel under your cursor and not much more.

The real problem: How do you systematically discover every polygonal region, given only this point-and-click interface?

What you get on each click (simplified):

{
  "geometryType": "esriGeometryPolygon",
  "features": [{
    "attributes": { "ADDRESS": "..." },
    "geometry": { "rings": [ [[x1, y1], [x2, y2], ..., [xN, yN]] ] }
  }]
}
(rings form a loop so [x1,y1] == [xN, yN]

Each probe gives you the entire geometry (the ring) of a parcel, as an ArcGIS' Polygon type. Coords are Web Mercator (not lat/lon), so units are big, but you don’t need to brute-force every possible point.

Set a reasonable stride, maybe half the smallest parcel size, and walk the map. Every time you hit a new parcel, save its geometry and skip future probes that land inside it. CPU cycles are cheap; spamming server requests is not.

Here's a toy demo using a simple sweep method: We step through the grid, probe each point, and color new parcels as they're found. Real-world ArcGIS geometries (with rings, holes, etc.) are trickier, but you get the idea.

function createRandomMap(width, height, N, svg) {
  svg.innerHTML = "";
  const points = Array.from({
    length: N
  }, () => [
    Math.random() * width,
    Math.random() * height,
  ]);
  const delaunay = d3.Delaunay.from(points);
  const voronoi = delaunay.voronoi([0, 0, width, height]);
  const polygons = [];
  const svgPolys = [];
  for (let i = 0; i < N; ++i) {
    const poly = voronoi.cellPolygon(i);
    polygons.push(poly);
    const el = document.createElementNS('http://www.w3.org/2000/svg', 'polygon');
    el.setAttribute('points', poly.map(([x, y]) => `${x},${y}`).join(' '));
    el.setAttribute('fill', '#fff');
    el.setAttribute('stroke', '#222');
    el.setAttribute('stroke-width', 1);
    svg.appendChild(el);
    svgPolys.push(el);
  }
  return [polygons, svgPolys];
}

// https://en.wikipedia.org/wiki/Point_in_polygon#Ray_casting_algorithm
function pointInPolygon(polygon, [x, y]) {
  let inside = false;
  for (let i = 0, j = polygon.length - 1; i < polygon.length; j = i++) {
    const [xi, yi] = polygon[i];
    const [xj, yj] = polygon[j];
    if (
      ((yi > y) !== (yj > y)) &&
      (x < ((xj - xi) * (y - yi)) / (yj - yi) + xi)
    ) inside = !inside;
  }
  return inside;
}

async function discoverParcels(polygons, svgPolys, width, height) {
  const discovered = new Set();
  const paletteGreens = t => `hsl(${100 + 30 * t}, 60%, ${40 + 25 * t}%)`;
  for (let y = 0; y < height; ++y) {
    for (let x = 0; x < width; ++x) {
      for (let i = 0; i < polygons.length; ++i) {
        if (!discovered.has(i) && pointInPolygon(polygons[i], [x + 0.5, y + 0.5])) {
          discovered.add(i);
          svgPolys[i].setAttribute('fill', paletteGreens(i / polygons.length));
          await new Promise(r => setTimeout(r, 100));
          break;
        }
      }
    }
  }
}

const width = 150,
  height = 150,
  N = 115;
const svg = document.getElementById('voronoi');
async function autoRunLoop() {
  while (true) {
    let polygons, svgPolys;
    [polygons, svgPolys] = createRandomMap(width, height, N, svg);
    await discoverParcels(polygons, svgPolys, width, height);
    await new Promise(r => setTimeout(r, 2000));
  }
}
autoRunLoop();

<!DOCTYPE html>
<html lang="en">

<head>
  <script src="https://cdn.jsdelivr.net/npm/d3-delaunay@6"></script>
  <style>
    body {
      background: white;
    }
  </style>
</head>

<body>
  <svg id="voronoi" width="150" height="150"></svg>
</body>

</html>

Adel Alaa · Accepted Answer · 2025-07-29 05:01:00Z

-1

Scraping dynamically loaded elements, such as interactive maps, cannot be achieved through a "get-all-at-once" method.

This is because the data is retrieved based on specific inputs, typically geographic coordinates.

To extract all the data, you need to implement a loop that iterates over all available coordinates.

For each coordinate or coordinate set, your script should trigger the necessary network requests and capture the returned data individually.

While alternative approaches such as simulated dragging or viewport shifting can help explore the map, they still rely on a looping mechanism.

Ultimately, the data must be collected incrementally, input by input not in bulk.

answered Jul 29 at 5:01

Adel Alaa

4972 silver badges15 bronze badges

Collectives™ on Stack Overflow

Can't scrape all the titles from the map on the webpage using the requests module

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related