I'm looking to build a service that interacts with an existing web application's database. The service would retrieve data from an api for which the web application users own an account and warehouse/refresh this data for them. I'd like to make sure we can handle potentially large amounts of data for each user. Currently using Azure for both the user-facing web interface and the future data ingestion/loading service.
I don't have a lot of experience with containers and their orchestration, but initially I thought that maybe I could automatically deploy a container to run the ingestion/loading for each new user. Is this a viable use for containers? Are there better, less complicated approaches? Should I even worry about scalability of this kind (handling more and more ingestion/loading) with only a few potential users to start?
Thanks for any ideas or leads!
