I have a complex issue with regards to scalability.
I am confused between putting load on the database, the backend server, or the frontend server.
I have 923K hexagons of levels 1-8 for France, which will be the highest number of hexagons for any country. Each hexagon has associated data with it, such as region stats, country name, country code, and most importantly, its GeoJSON.
I am using react-leaflet to plot the hexagons through their GeoJSON. Initially, I had everything precomputed and existing on the database, but that was taking ages to fetch from the database.
I now have only country name, code and hexagon ID on the database. The backend server fetches all the polygons for a country, regardless of level. My client is fine with 30 seconds' loading time, as long as after it's loaded, panning and zooming runs smoothly.
I also computed the full details of each hexagon (centroid coordinates, and GeoJSON) on the backend server, so that if someone requests the full details of France again, it won't have to do all the lengthy computations and can return data in 10-15 seconds instead of 30+ seconds. But this is also at risk with the increased max number of polygons (I'm deploying on Vercel so it might not even have that much memory).
The problem is that previously he had incomplete data of total 290K polygons. Upon investigation, we discovered that the actual data is 923K polygons, which is giving "network failed" errors when I send it back to frontend.
What can I do if I want to maintain smooth zooming and panning? I don't want to send a new request to the backend every time the user zooms or pans. That would lead to a very bad UX. If you have any suggestions, please shoot away.
I tried caching. I tried splitting based on the country selected. To improve frontend performance, I only render the polygons that are within the map view, so that instead of rendering 923 K polygons, we only render 20-50 depending on zoom level.
But I need the data to be readily available in the React app for that, as a polygon bank (hash). I have done that. The other issues are still coming up.
If it helps, the data will stay the same for every user, and not be editable at all.
If I shift the load to backend server or frontend server, then those will become slow. I'm not sure where to place the load and burden of computing everything and storing the results.
