matplotlib-devel Mailing List for matplotlib

On Fri, 2005-04-15 at 12:27 +0100, I wrote:
> My suggestion for improved performance would be to allow the Image class
> to work directly on the buffer passed to the generator function - I've
> started implementing this in c++.  Going with this approach should
> improve speed further and again conserve memory for very large images -
> and (without making other changes to the Image class) would require that
> rgba rather than rgb or intensity was used as an input.

Having tried this (patch attached) I get the following results on
running a slightly modified version of John Hunters test script
(attached).  In the original script it turned out that a significant
amount of time was being taken up with creating some of the test
environment - hence a smaller improvement was being shown.

Running with 1024x1024:
Starting array:
  Array set up: resident stack size:  39716, size:  10914
  Tests done: resident stack size:  43836, size:  11938
  Elapsed: 9.363
Starting buffer:
  Buffer set up: resident stack size:  15192, size:   4823
  Tests done: resident stack size:  15200, size:   4824
  Elapsed: 0.690
Fractional improvement: 12.577


Running with 2048x2048:
Starting array:
  Array set up: resident stack size: 146276, size:  37592
  Tests done: resident stack size: 158572, size:  40664
  Elapsed: 38.544
Starting buffer:
  Buffer set up: resident stack size:  39784, size:  10968
  Tests done: resident stack size:  39784, size:  10967
  Elapsed: 2.044
Fractional improvement: 17.855


Running with 4096x4096:
Starting array:
  Array set up: resident stack size: 564076, size: 142041
  Tests done: resident stack size: 613252, size: 154329
  Elapsed: 170.495
Starting buffer:
  Buffer set up: resident stack size:  67100, size:  35544
  Tests done: resident stack size: 133060, size:  35544
  Elapsed: 8.474
Fractional improvement: 19.120


As you can see - in both cases a big improvement.  In the case of large
data sets its a change from a noticeable delay 3.4s per plot to the very
usable 0.17s per plot (none of these plots required swapping - although
the set functions did).  If you don't have you data in the form of a
writable python buffer it's necessary to wrap the input buffer in a
Python array.array to get this performance (a read-only buffer is still
accepted but will be slightly slower).  Performance using a
non-modifiable buffer is slightly lower even with the overhead of an
additional Python function call (I guess it implements some more
intelligent buffer creation strategy).

> I made some alterations to these functions - but they are currently
> limited to my own situation.  I will make them general once I've played
> around with writable buffers a bit - this will be fairly easily but I
> don't want to spend time writing wrapper code until I'm happy with what
> I'm wrapping.

Changes in the patch I've attached (and were even simpler than I
imagined).

Nich

Hi,
I'm using code like this:
...
pylab.plot(..., label="something")
pylab.plot(..., label="something")
...
pylab.legend()

and i'd like to not include some plotted lines in legend.
I can specify it explicitly by legend(lines, labels), 
but it's more convenient for me to use:
pylab.plot(..., label=None)

I made a simple change in axes.py (attached patch).
I also have some questions about legend() function, that I put 
into the same patch.

Cheers,
Marcin

-- 
Marcin Wojdyr  |  http://www.unipress.waw.pl/~wojdyr/

On Thu, 2005-04-14 at 16:00 -0500, John Hunter wrote: 
> Thanks for the suggestions and patch.  I incorporated frombuffer and
> have been testing it.  I've been testing the performance of frombuffer
> vs fromarray, and have seen some 2-3x speedups but nothing like the
> numbers you are reporting.  [Also, I don't see any detectable memory
> leaks so I don't think you have any worries there]

That kind of speed up is probably more realistic - I probably made a
greater number of optimisations to the python buffer code than to the
numerix code (which I only remembered after posting my first message).
Performance do gains seem greater where memory is limited though as the
reduced memory consumption means less swapping, in cases where the
reduced memory consumption avoids swapping altogether they are, of
course, huge.

> Here is the test script I am using - does this look like a fair test?

It seems to be a fair test - I'd have created the string buffer outside
of the timing (as in reality you wouldn't go through that step) but as
it should be a fairly quick conversion it shouldn't matter too much.

> Another suggestion for Nicholas -- perhaps you want to support MxN,
> MxNx3 and MxNx4 arrays in your frombuffer function?

I could do - the main reason I don't particularly want to is that a
compiler should be able to more easily optimise a simple for(i,i<j,i++)
loop than a series of nested loops and other instructions.  As this code
is only really of use where performance is particularly important I'd
rather keep performance as high as possible - it's easy to generate
buffers in whatever format is necessary.

My suggestion for improved performance would be to allow the Image class
to work directly on the buffer passed to the generator function - I've
started implementing this in c++.  Going with this approach should
improve speed further and again conserve memory for very large images -
and (without making other changes to the Image class) would require that
rgba rather than rgb or intensity was used as an input.

> And a final question -- how are you getting your function into the
> matplotlib image pipeline.  Did you alter the image.py
> AxesImage.set_data function to test whether A is a buffer object?  If
> so, you might want to post these changes to the codebase as well.

I made some alterations to these functions - but they are currently
limited to my own situation.  I will make them general once I've played
around with writable buffers a bit - this will be fairly easily but I
don't want to spend time writing wrapper code until I'm happy with what
I'm wrapping.

Nicholas