If I have a server written in golang that processes requests containing user queries, is there any viable method to track and limit the memory used by any given user request?
The server process does not have a simple way to predict the memory required to service a given user query. Memory demands may depend on the shape and quantity of the data the query needs to process and is not knowable before execution begins. So actual memory use must be tracked, allocated to specific subtasks relating to a particular request. The memory accounting could then be used for over quota tasks to cooperatively abort.
To do this some mechanism is required to efficiently track which tasks particular allocations are associated with, query and aggregate them.
For example, say my server is limited to 1 GiB of RAM in a containerised context. Some user queries may only need 50MiB of RAN to execute. Others may need 500, others would need 3000. My goal is to efficiently determine when (a) any given user query has allocated more than X MiB in private memory associated specifically with its task and (b) detect when all tasks are collectively approaching a cumulative resource limit.
The goal is to prevent OOMs and maintain service robustness without having to grossly over-allocate memory for the worst possible case.
GOMEMLIMIT is helpful but insufficient as it is a soft limit and is process wide.
I'm looking for things like:
- Tagging objects with a owner with transitive propagation of the tag
- Allocation of objects into dedicated memory pools or arenas while still using the GC (not implementing my own bad malloc in giant byte arrays)
- Heirachical contextual allocators where all allocations done by a particular thread/goroutine count toward a particular memory bucket
A lazy, cooperative approach is fine. Memory stats may be somewhat delayed (e.g. since last GC sweep) and somewhat inaccurate. If I needed hard limits I would probably split the process into one shot workers for each query. Run each in its own short lived control group. But this is excessively expensive and inefficient especially with a runtime like go.
As far as I can tell nothing like this exists for go, nor does a program have any efficient way to determine an allocation's real size or to be notified when the object is GC'd. So there doesn't seem to be anything like this.
As far as I can see a go program can't really track objects memory manually without runtime assistance. In addition to the prohibitive overhead and concurrency impacts of attempting it, the program would need to impose the same allocation management logic on every library it used. And it can't always see the real memory usage of its objects due to interning, backing arrays of slices, etc. It's be impossibly slow and clumsy.
(My immediate use case is helping Thanos and Prometheus run in lower memory caps by letting them soft limit query memory use, allowing them to abort queries that use took much memory instead of OOMing. Coarse drained limits like series and sample counts are ineffective and require significant over allocation for worst case input data).
