Is it normal that running python under valgrind shows many errors with memory?

Question

I've tried to debug memory crash in my Python C extension and tried to run script under valgrind. I found there is too much "noise" in the valgrind output, even if I've ran simple command as:

valgrind python -c ""

Valgrind output full of repeated info like this:

==12317== Invalid read of size 4
==12317==    at 0x409CF59: PyObject_Free (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x405C7C7: PyGrammar_RemoveAccelerators (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x410A1EC: Py_Finalize (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x4114FD1: Py_Main (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x8048591: main (in /usr/bin/python2.5)
==12317==  Address 0x43CD010 is 7,016 bytes inside a block of size 8,208 free'd
==12317==    at 0x4022F6C: free (in /usr/lib/valgrind/x86-linux/vgpreload_memcheck.so)
==12317==    by 0x4107ACC: PyArena_Free (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x41095D7: PyRun_StringFlags (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40DF262: (within /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x4099569: PyCFunction_Call (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40E76CC: PyEval_EvalFrameEx (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40E70F3: PyEval_EvalFrameEx (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40E896A: PyEval_EvalCodeEx (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40E8AC2: PyEval_EvalCode (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40FD99C: PyImport_ExecCodeModuleEx (in /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x40FFC93: (within /usr/lib/libpython2.5.so.1.0)
==12317==    by 0x41002B0: (within /usr/lib/libpython2.5.so.1.0)

Python 2.5.2 on Slackware 12.2.

Is it normal behavior? If so then valgrind maybe is inappropriate tool for debugging memory errors in Python?

Nick Craig-Wood · Accepted Answer · 2009-10-05 11:45:32Z

22

You could try using the suppression file that comes with the python source

Reading the Python Valgrind README is a good idea too!

answered Oct 5, 2009 at 11:45

Nick Craig-Wood

54.4k13 gold badges131 silver badges138 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Falaina Over a year ago

As a high level note: In general Valgrind needs some help with custom allocators as it's not able to comprehend the behavior of a custom allocator as it could a standard implementaion.

black_puppydog Over a year ago

so, if I read the valgrind readme correctly, I cannot really use valgrind to debug a python c extension without compiling my own python distribution?!

liwt31 · Accepted Answer · 2018-11-17 09:18:14Z

5

Following links given by Nick I was able to find some updates on README.valgrind. In one word, for Python > 3.6, you can set PYTHONMALLOC=malloc environment variable to effectively disable the warnings. For example, in my machine:

export PYTHONMALLOC=malloc
valgrind python my_script.py

doesn't produce any error related to python.

answered Nov 17, 2018 at 9:18

liwt31

98810 silver badges10 bronze badges

Comments

unwind · Accepted Answer · 2009-10-05 11:29:16Z

2

This is quite common, in any largish system. You can use Valgrind's suppression system to explicitly suppress warnings that you're not interested in.

answered Oct 5, 2009 at 11:29

unwind

402k64 gold badges492 silver badges620 bronze badges

Comments

Andrew Grimm · Accepted Answer · 2012-12-20 23:46:20Z

1

The most correct option is to tell Valgrind that it should intercept Python's allocation functions. You should patch valgrind/coregrind/m_replacemalloc/vg_replace_malloc.c adding the new interceptors for PyObject_Malloc, PyObject_Free, PyObject_Realloc, e.g.:

ALLOC_or_NULL(NONE,                  PyObject_Malloc,      malloc);

(note that the soname for users allocation functions should be NONE)

edited Dec 20, 2012 at 23:46

Andrew Grimm

82.3k58 gold badges210 silver badges355 bronze badges

answered Dec 18, 2009 at 9:22

Glider

3933 silver badges5 bronze badges

Comments

Ned Batchelder · Accepted Answer · 2009-10-05 11:23:35Z

0

Yes, this is typical. Large systems often leave memory un-freed, which is fine so long as it is a constant amount, and not proportional to the running history of the system. The Python interpreter falls into this category.

Perhaps you can filter the valgrind output to focus only on allocations made in your C extension?

answered Oct 5, 2009 at 11:23

Ned Batchelder

378k77 gold badges583 silver badges675 bronze badges

Comments

bialix · Accepted Answer · 2009-12-02 12:20:27Z

0

There is another option I found. James Henstridge has custom build of python which can detect the fact that python running under valgrind and in this case pymalloc allocator is disabled, with PyObject_Malloc/PyObject_Free passing through to normal malloc/free, which valgrind knows how to track.

Package available here: https://launchpad.net/~jamesh/+archive/python

answered Dec 2, 2009 at 12:20

bialix

22k10 gold badges51 silver badges63 bronze badges

Comments

NicolasYL · Accepted Answer · 2022-08-16 13:05:14Z

0

This is quite NORMAL If you want to use Valgrind more effectively and catch even more memory leaks, you will need to configure python --without-pymalloc. https://svn.python.org/projects/python/trunk/Misc/README.valgrind

answered Aug 16, 2022 at 13:05

NicolasYL

111 bronze badge

1 Comment

DavidW Over a year ago

At least 3 other answers link to this readme so I don't think it adds a lot. They also link to GitHub rather than a 12-year out-of-date SVN repository.

Collectives™ on Stack Overflow

Is it normal that running python under valgrind shows many errors with memory?

7 Answers 7

2 Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

2 Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related