live plotting in matplotlib while performing a measurement that takes time

Question

I would like to perform a measurement and plot a graph while the measurement is running. This measurements takes quite some time in python (it has to retrieve data over a slow connection). The problem is that the graph freezes when measuring. The measurement consists of setting a center wavelength, and then measuring some signal.

My program looks something like this:

# this is just some arbitrary library that has the functions set_wavelength and
# perform_measurement
from measurement_module import set_wavelength, perform_measurement
from pylab import *

xdata = np.linspace(600,1000,30) # this will be the x axis
ydata = np.zeros(len(xdata)) # this will be the y data. It will
for i in range(len(xdata)):
  # this call takes approx 1 s
  set_wavelength(xdata[i])
  # this takes approx 10 s
  ydata[i] = perform_measurement(xdata)
  # now I would like to plot the measured data
  plot(xdata,ydata)
  draw()

This will work when it is run in IPython with the -pylab module switched on, but while the measurement is running the figure will freeze. How can modify the behaviour to have an interactive plot while measuring?

You cannot simply use pylab.ion(), because python is busy while performing the measurements.

regards,

Dirk

deinonychusaur · Accepted Answer · 2012-03-19 13:46:01Z

1

You can, though maybe a bit awkward, run the data-gathering as a serparate process. I find Popen in the subprocess module quite handy. Then let that data-gathering script save what it does to disk somewhere and you use

Popen.poll()

To check if it has completed.

It ought to work.

answered Mar 19, 2012 at 13:46

deinonychusaur

7,3543 gold badges32 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dirklinux Over a year ago

Thanks for the response, it helped! I will write a measurement program that can communicate with the plotting program using Popen, and send the measurement data back using <code>Popen.communicate()</code>. In that way I can rewrite the measurement program for a different program, and still use the same plotting program. Might even use a socket, so I can measure on one computer and plot the results on the other one.

lukecampbell · Accepted Answer · 2012-03-19 18:18:18Z

1

I recommend buffering the data in large chunks and render/re-render when the buffer fills up. If you want it to be nonblocking look at greenlets.

from gevent.greenlet import Greenlet
import copy
def render(buffer):
    '''
    do rendering stuff
    '''
    pass

buff = ''
while not_finished:
    buff = connection.read()
    g = Greenlet(render, copy.deepcopy(buff))
    g.start()

answered Mar 19, 2012 at 18:18

lukecampbell

15.3k4 gold badges37 silver badges32 bronze badges

Comments

tom10 · Accepted Answer · 2012-03-21 15:46:04Z

0

Slow input and output is the perfect time to use threads and queues in Python. Threads have there limitations, but this is the case where they work easily and effectively.

Outline of how to do this:
Generally the GUI (e.g., the matplotlib window) needs to be in the main thread, so do the data collection in a second thread. In the data thread, check for new data coming in (and if you do this in some type of infinite polling loop, put in a short time.sleep to release the thread occasionally). Then, whenever needed, let the main thread know that there's some new data to be processed/displayed. Exactly how to do this depends on details of your program and your GUI, etc. You could just use a flag in the data thread that you check for from the main thread, or a theading.Event, or, e.g., if you have a wx backend for matplotlib wx.CallAfter is easy. I recommend looking through one of the many Python threading tutorials to get a sense of it, and also threading with a GUI usually has a few issues too so just do a quick google on threading with your particular backend. This sounds cumbersome as I explain it so briefly, but it's really pretty easy and powerful, and will be smoother than, e.g., reading and writing to the same file from different processes.

edited Mar 21, 2012 at 15:46

answered Mar 20, 2012 at 16:37

tom10

69.5k11 gold badges133 silver badges143 bronze badges

3 Comments

Dirklinux Over a year ago

Ok, that was my initial thought, but how would you do that? If the plotting is in a separate thread, it won't help, because the program will be busy in the main thread. Could you give me an example?

tom10 Over a year ago

I gave a more detailed outline in my answer. To answer your question here though: about "the program will be busy in the main thread", since your data collection is hardware limited, your program isn't really "busy" in the data collection thread, it's mostly just waiting for data. At least in my experience, it's almost always the case that threading works very well in data collection programs.

Dirklinux Over a year ago

Thanks a lot! Will try it, and keep you posted.

Collectives™ on Stack Overflow

live plotting in matplotlib while performing a measurement that takes time

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related