How to partially initialize an array with huge load of data from API response

Question

I use Axios to get a response from an endpoint and then I input that response into a variable in the form of an object. The response I get from the server has more than 10000 data.

My question is :

How to process the data so it doesn't take up memory and can insert it into a table?
Is there a request handler that can process that response better?
Or is there a method to partially download the response so that it can be partially consumed by the user from the frontend?

It's 10000 columns or 10000 rows? If it's columns maybe narrow down the columns to only the ones you need initially, and then get the others when they're needed. If it's rows, maybe use pagination — yaakov
– yaakov, Commented May 18, 2022 at 4:37
Change the endpoint so you can get smaller batches of data at a time. Or, show us exactly what the data looks like as it comes in so we can offer ideas on whether it can be streamed and processed as it comes in rather than collecting it all before processing any. — jfriend00
– jfriend00, Commented May 18, 2022 at 4:56

DharmanBot · Accepted Answer · 2022-05-18 13:25:34Z

1

There are many things to look at here:

Every site has a memory limit ( Assigned by the browsers ) and the site crashes if the memory exceeds.
The short and ugly fix would be... taking all those and storing only some part of it (Ugly fix).
The right and correct fix would be making a kind of pagination and limiting the number of data per request. in this case, You'll have proper data and a good user experience with the help of Tabs/pages with limited scroll. And most important, the site's memory will be optimized.

edited May 18, 2022 at 13:25

DharmanBot

1,0742 gold badges7 silver badges10 bronze badges

answered May 18, 2022 at 7:06

sachin

1,1371 gold badge9 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to partially initialize an array with huge load of data from API response

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related