Matplotlib slow with large data sets, how to enable decimation?

Question

I use matplotlib for a signal processing application and I noticed that it chokes on large data sets. This is something that I really need to improve to make it a usable application.

What I'm looking for is a way to let matplotlib decimate my data. Is there a setting, property or other simple way to enable that? Any suggestion of how to implement this are welcome.

Some code:

import numpy as np
import matplotlib.pyplot as plt

n=100000 # more then 100000 points makes it unusable slow
plt.plot(np.random.random_sample(n))
plt.show()

Some background information

I used to work on a large C++ application where we needed to plot large datasets and to solve this problem we used to take advantage of the structure of the data as follows:

In most cases, if we want a line plot then the data is ordered and often even equidistantial. If it is equidistantial, then you can calculate the start and end index in the data array directly from the zoom rectangle and the inverse axis transformation. If it is ordered but not equidistantial a binary search can be used.

Next the zoomed slice is decimated, and because the data is ordered we can simply iterate a block of points that fall inside one pixel. And for each block the mean, maximum and minimum is calculated. Instead of one pixel, we then draw a bar in the plot.

For example: if the x axis is ordered, a vertical line will be drawn for each block, possibly the mean with a different color.

To avoid aliasing the plot is oversampled with a factor of two.

In case it is a scatter plot, the data can be made ordered by sorting, because the sequence of plotting is not important.

The nice thing of this simple recipe is that the more you zoom in the faster it becomes. In my experience, as long as the data fits in memory the plots stays very responsive. For instance, 20 plots of timehistory data with 10 million points should be no problem.

Could you implement such a decimation algorithm outside of matplotlib rendering, just updating the data to be displayed upon zooming event? — Joël
– Joël, Commented Dec 12, 2013 at 14:54
Possibly of interest here: How can I subsample an array according to its density? — ImportanceOfBeingErnest
– ImportanceOfBeingErnest, Commented Dec 14, 2018 at 11:19

Chris Flesher · Accepted Answer · 2013-12-13 03:40:59Z

1

It seems like you just need to decimate the data before you plot it

import numpy as np
import matplotlib.pyplot as plt

n=100000 # more then 100000 points makes it unusable slow
X=np.random.random_sample(n)
i=10*array(range(n/10))
plt.plot(X[i])
plt.show()

edited Dec 13, 2013 at 3:40

answered Dec 13, 2013 at 2:54

Chris Flesher

1,0371 gold badge11 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

trampas · Accepted Answer · 2015-02-02 19:26:33Z

0

Decimation is not best for example if you decimate sparse data it might all appear as zeros.

The decimation has to be smart such that each LCD horizontal pixel is plotted with the min and the max of the data between decimation points. Then as you zoom in you see more an more detail.

With zooming this can not be done easy outside matplotlib and thus is better to handle internally.

answered Feb 2, 2015 at 19:26

trampas

1

Collectives™ on Stack Overflow

Matplotlib slow with large data sets, how to enable decimation?

Some code:

Some background information

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Some code:

Some background information

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related