27 questions
2
votes
0
answers
68
views
How can I dynamically trace and visualize memory allocation patterns of nested Python coroutines in real-time?
I'm trying to understand how memory is used in my async Python application. I have multiple coroutines running, and I want to see how much memory each one is using, especially when they are nested or ...
- 218
0
votes
0
answers
131
views
RuntimeWarning: Enable tracemalloc to get the object allocation traceback?
I am building a Telegram bot. This code is working:
import telebot
bot = telebot.TeleBot("TOKEN")
@bot.message_handler(func=lambda newMessage: True)
def main(newMessage):
print(...
- 13
1
vote
0
answers
264
views
Why does my python app use far more memory than tracemalloc suggests? [duplicate]
I trying to debug a memory leak in my python ML app. When I run it locally, the memory usage on my local machine increases gradually by 700mb. So I put a breakpoint in when memory use is around its ...
- 45
2
votes
0
answers
225
views
Understanding Python's memory allocation
I'm trying to track down a suspected memory leak in a Python application which uses numpy and pandas as two of the main libraries. I can see that the application uses more and more memory over time (...
- 8,252
1
vote
1
answer
1k
views
Tracking Memory Usage in Python
Python code:
def my_function():
my_list = [i for i in range(1_000_000)]
return my_list
my_list = my_function()
Memory usage reported by different libraries:
memory_profiler: 82MB
tracemalloc:...
- 139
0
votes
1
answer
2k
views
Is it correct that tracemalloc does not show the actual memory usage in python?
The official document says, "The tracemalloc module is a debug tool to trace memory blocks allocated by Python."
Python uses memory management as Arena and stores through pools and blocks.
...
- 169
2
votes
1
answer
597
views
ResourceWarning: unclosed socket <zmq.Socket(zmq.PUSH) when Threading
So I have an issue with the import of a driver, which apparently leads to almost all threads that I open to end in a Resourcewarning. (python version 3.9.16 on Windows in Anaconda environment; nidaqmx ...
- 21
1
vote
1
answer
707
views
python3 -- RAM usage (htop and tracemalloc give different values)
I have a code where I need to save RAM usage so I've been tracing RAM usage through tracemalloc.get_traced_memory. However, I have found that what tracemalloc.get_traced_memory gives is very different ...
- 11
1
vote
0
answers
430
views
Can I use tracemalloc to line-by-line profile a subprocess?
I am attempting to write a generic,python script that memory profiles a target script, but I can't figure out how to get line-by-line memory stats from the script being run by subprocess.
# ...
- 11
0
votes
0
answers
512
views
Precision about tracemalloc and memory consumption with python
I don't exactly understand how python use memory, and what Python gives with the tracemalloc builtins.
Let's take this exemple:
import tracemalloc
import gc
L=[1,2,3]
tracemalloc.start()
gc.collect()
...
- 1
0
votes
0
answers
431
views
sort tracemalloc statistics by size
I have this code:
def print_statistics(s: Snapshot, s2: Snapshot):
stats = s2.compare_to(s, 'lineno')
for stat in stats[:20]:
print(stat)
def some_func()
tracemalloc.start(50)
...
- 437
-1
votes
1
answer
2k
views
How to fix memory leak of flask application with tracemalloc
I have a flask application that makes requests to retrieve data and then exports the data as an excel file with openpyxl.
After ~ 50 exported excel files the flask application exceeds the RAM of 8GB ...
- 41
1
vote
1
answer
1k
views
long-running python program ram usage
I am currently working on a project where a python program is supposed to be running for several days, essentially in an endless loop until an user intervenes.
I have observed that the ram usage (as ...
- 31
6
votes
1
answer
1k
views
Does Python not reuse memory here? What does tracemalloc's output mean?
I create a list of a million int objects, then replace each with its negated value. tracemalloc reports 28 MB extra memory (28 bytes per new int object). Why? Does Python not reuse the memory of the ...
- 28k
0
votes
0
answers
380
views
How to trace memory allocations in Apache httpd server?
I am running apache benchmark (ab) with server [httpd2.4.52] running locally. I want to track how many memory allocations and what size allocations does the server make.
I run 'valgrind --trace-malloc=...
- 31
4
votes
1
answer
26k
views
Measuring the allocated memory with tracemalloc
The Python module tracemalloc offers a detailed insight into the allocated memory of the program. For example, one use case is to record the current and peak memory usage:
import tracemalloc
...
- 566
1
vote
0
answers
463
views
Understanding tracemalloc output for small lists
I'm running the following simple script to get some familiarity with tracemalloc:
import tracemalloc
import linecache
def display_custom(snapshot):
... # skipped for shorter code
tracemalloc....
- 43
1
vote
0
answers
48
views
what kind of memory is allocated by tracemalloc module
what kind of memory is allocated by tracemalloc module during program execution? is RAM memory?
According to the documentation: they are allocated memory blocks.
But is it RAM memory?
- 11
1
vote
0
answers
1k
views
memory leak that persists after colab cell executes
I'm encountering a subtle memory leak, and unable to determine the source using tracemalloc. I run the following code in google colab, which is meant to optimize hyperparameters for a custom ppo agent....
0
votes
0
answers
268
views
Find out Python interpreter memory consumption
How can I find out Python interpreter memory consumption?
I tried memory-profiler but it shows running code memory consumption.
PS. It seems tracemalloc does what I want
- 1,039
2
votes
1
answer
2k
views
Python tracemalloc, what is size and count?
I am using the tracemalloc library to pinpoint memory concerns in my application.
This is the code that I am using.
tracemalloc.start()
snapshot = tracemalloc.take_snapshot()
...
- 149
2
votes
0
answers
311
views
tracemalloc doesn't reflect memory usage of flask server?
I have a memory leak in my flask server (it's a server that serve a tensorflow and a pytorch machine learning model). It has a dread memory leak.
So I use tracemalloc to track the memory. The problem ...
- 307
1
vote
1
answer
987
views
Python tracemalloc's "compare_to" function delivers always "StatisticDiff" objects with len(traceback)=1
Using Python's 3.5 tracemalloc module as follows
tracemalloc.start(25) # (I also tried PYTHONTRACEMALLOC=25)
snapshot_start = tracemalloc.take_snapshot()
... # my code is running
snapshot_stop = ...
4
votes
3
answers
4k
views
How big is the overhead of tracemalloc?
I would like to log in a production system the current memory usage of a Python script. AWS has Container Insights, but they are extremely well-hidden and I'm not sure how to use them properly within ...
- 139k
1
vote
1
answer
4k
views
ResourceWarning for a file that is unclosed but UnitTest is throwing it
I have a function like this:
def init_cars(self, directory=''):
#some_code
cars = set([line.rstrip('\n') for line in open(directory + "../myfolder/all_cars.txt")])
#some_more_code
I am ...
- 1,013
1
vote
0
answers
409
views
Memory Leak in python undetected by tracemalloc
I have a python(3.6.5) package that creates multiple objects. The main process seems to have a memory leak based on what I see in task scheduler as well as a bail out criteria that I have put in place ...
- 19
-1
votes
1
answer
453
views
Serialize to JSON string tracemalloc in Python 3.7 [closed]
I have to serialize to JSON string the result of tracemalloc.
current_mem, peak_mem = tracemalloc.get_traced_memory()
overhead = tracemalloc.get_tracemalloc_memory()
stats = tracemalloc.take_snapshot(...
- 16.3k