Python : Get subprocess STDOUT while logging the memory

Question

Here is my problem :

I am trying to write a code to log the memory taken by a subprocess job and get the stdout of this job at the same time.

To do this I call the python script with subprocess :

res = Popen(args, shell=False, stderr=STDOUT, stdout=PIPE)

To get the memory, I get the pid of the job with res.pid. Then I get the memory with psutil :

p = psutil.Process(pid)

mem_usage = p.memory_full_info()

I would like to combine it in the following way :

Run the code
every second, log the memory in mem_usage
Log the stderr in a list of strings

I tried this :

while res.poll() is None:

    (total_rss_Mo, total_vms_Mo, total_uss_Mo) = get_mem(pid)
    print "MEM ",total_rss_Mo, total_vms_Mo, total_uss_Mo

    # Wait a bit before next memory scam
    time.sleep(1)
stderr, stdout = res.communicate()

The result is not what I expect. The first 20s, it works but then the job goes to sleep and never ends:

MEM  7.77734375 152.52734375 4.30078125
MEM  290.49609375 1299.66015625 284.265625
MEM  436.53515625 1372.29296875 427.77734375
MEM  443.61328125 1633.8125 434.921875
MEM  544.26171875 1732.52734375 535.40625
MEM  548.59765625 1737.015625 539.89453125
MEM  552.765625 1741.01953125 543.8984375
MEM  552.51953125 1741.01953125 543.8984375
MEM  552.5703125 1741.01953125 543.8984375
MEM  552.55078125 1741.01953125 543.8984375
MEM  552.7265625 1741.01953125 543.8984375
MEM  455.63671875 1643.8828125 446.76171875
MEM  455.63671875 1643.8828125 446.76171875
MEM  455.63671875 1643.8828125 446.76171875
MEM  455.63671875 1643.8828125 446.76171875
MEM  455.63671875 1643.8828125 446.76171875

It seems that the stderr buffer is full and put the job in hold. How can I get the stderr / stdout and log my memory at the same time ? If I try to read the stdout in a loop, the loop continues until the end of the job and I cannot log the memory during the job execution anymore :

while True:
    line = res.stdout.readline()
    if not line or line == "":
        break
    print line

Thanks for your help,

Akira

Why not put the output reading loop inside the memory loop? i.e. in each iteration, read the output until an empty line then check the memory. — FamousJameous
– FamousJameous, Commented Apr 13, 2017 at 22:06
What job are you running that is getting stuck when the stderr buffer is full? — unddoch
– unddoch, Commented Apr 13, 2017 at 23:52
To lolopop : The job is an image simulator for a space telescope. It spits out quite a large number of lines describing what it is doing. To FamousJameous: Wherever I put the output reading loop, it keeps reading and never get out of the loop until the job is finished. So I do not get any memory log. \n What I would like to do is read only the output lines that have been produced up to that point and then exit the loop. I could then log the memory and resume checking the output afterward. — Akira Doe
– Akira Doe, Commented Apr 14, 2017 at 17:42

Akira Doe · Accepted Answer · 2017-04-16 08:23:53Z

0

I found a solution by putting the memory logging in a separate thread. It does not collide anymore with the loop to retrieve the stdout/stderr of tbhe subprocess.

answered Apr 16, 2017 at 8:23

Akira Doe

1

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python : Get subprocess STDOUT while logging the memory

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related