matplotlib-devel Mailing List for matplotlib

John Hunter wrote:
[...]
I am in the process of changing your shiny new kwdocd machinery 
slightly.  I added artist.kwdoc so that

> artist.kwdocd['Line2D'] = '\n'.join(artist.ArtistInspector(Line2D).pprint_setters(leadingspace=12))
> 

becomes

artist.kwdocd['Line2D'] = artist.kwdoc(Line2D)

> Then in any function accepting Line2D passthrough kwargs, eg
> matplotlib.axes.Axes.plot
> 
>     def plot(self, *args, **kwargs):
>         """
> 	Some stuff omitted
> 
>         The kwargs are Line2D properties:
> %(Line2D)s

and the line above is changed to be indented the same as the rest of the 
string (improves readability to my eye)

> 
>         kwargs scalex and scaley, if defined, are passed on
>         to autoscale_view to determine whether the x and y axes are
>         autoscaled; default True.  See Axes.autoscale_view for more
>         information
>         """
> 	pass

because this

>     plot.__doc__ = plot.__doc__ % artist.kwdocd

becomes
     plot.__doc__ = dedent(plot.__doc__) % artist.kwdocd

where dedent is now in cbook.py

I can continue making the necessary changes if that is OK with you, but 
I don't want our versions to get tangled up if you are still working on 
this aspect, or if you don't like the strategy modification outlined 
above.  I committed dedent but not the other changes.

Eric

>>>>> "John" == John Hunter <jdh...@ac...> writes:
    John> I think we could lose python2.2 support if there is a good
    John> reason.  Is anyone still using it?  I'll also post to the
    John> user's list.  Perhaps in the next release we should issue
    John> deprecation warnings for 2.2 and encourage people to post to
    John> the list if they have an objection.  Typically with these
    John> things you don't see complaints until you take something
    John> away.



Well, I was just about to post to the user list to see if anyone
objected to remove 2.2 support, and wondered, do we really support 2.2
currently?  I just tried to compile under 2.2 and got

peds-pc311:~/mpl> sudo python2.2 setup.py install
Traceback (most recent call last):
  File "setup.py", line 63, in ?
    from setupext import build_agg, build_gtkagg, build_tkagg,
    build_wxagg,\
  File "setupext.py", line 120, in ?
    win32_compiler = get_win32_compiler()
  File "setupext.py", line 117, in get_win32_compiler
    if 'mingw32' in v:
TypeError: 'in <string>' requires character as left operand

And this has probably been in there for a while.  So we haven't
supported 2.2 for sometime and noone is complaining so let's make 2.3
the official target rather than fix this.  At least 2.3 compiles and
runs.

I'll update the docs and website when I get a minute.

JDH

>>>>> "Eric" == Eric Firing <ef...@ha...> writes:

    Eric> Nice, simple, elegant, but it doesn't quite match what I had
    Eric> in mind for axes.  I was thinking that the place to put the
    Eric> hook call is inside the draw() method, so it can take
    Eric> advantage of the intial work done there: getting a cached
    Eric> renderer, checking for visibility.  Maybe this doesn't
    Eric> matter.

Well for these two pieces, perhaps the method should look like

    def draw(self, renderer=None, **kwargs):
        if renderer is None:
            renderer = self._cachedRenderer

        if renderer is None:
            raise RuntimeError('No renderer defined')
        if not self.get_visible(): return


        for func in self.before_draw_hook.ordered_values():
            func(self, renderer)
        self._draw(renderer, **kwargs)
        for func in self.after_draw_hook.ordered_values():
            func(self, renderer)
        

and perhaps the renderer caching should be done at the figure level

       if renderer is None:
            renderer = self.figure._cachedRenderer


    Eric> Also, looking at backend_bases, I can't figure out where
    Eric> this goes. There is a RendererBase class, but it doesn't
    Eric> have a draw method.  And there is a FigureCanvasBase, but
    Eric> its draw method is overridden by each backend.  What am I
    Eric> missing?

I meant artist.Artist.draw -- sorry for the bad info.


    Eric> and have def register_before_draw(self, name, func): if
    Eric> self.before_draw_hook is None: self.before_draw_hook =
    Eric> OrderedDictionary() self.before_draw_hook.append(name, func)

We also need a connectionid for connecting and disconnecting or
something like it.


    Eric> Looking at the axes draw method, I am wondering whether a
    Eric> more general application of an OrderedDictionary (or
    Eric> ListDict, or whatever) wouldn't be a better approach.  Why
    Eric> should there be before_draw_hook, draw, and after_draw_hook,
    Eric> with a big list generation and sorting operation at draw
    Eric> time?  Could everything be done by having artists register
    Eric> all draw-time functions in a single ListDict, with
    Eric> sufficient ordering information to keep everything straight?
    Eric> That's pretty vague, and there is a lot I don't
    Eric> understand--but it just seems like there is an opportunity
    Eric> here for simplifying the overall structure with improved
    Eric> flexibility and no loss of speed.  (Again, some kind of
    Eric> result-caching and need-for-recomputation-signalling would
    Eric> be needed.)

This is interesting -- the devil will be in the details.  Eg when one
artist changes it's zorder, the whole connection hierarchy will need
to be redone, which is currently accomplished by doing the sort w/
every draw.  Your way would be faster since we would only need to do
the ordering when the zscore changes, or when new artists are added,
and not with every draw.  But it might become difficult to manage
resorting if users have added their own hooks.  And then it wouldn't
really be a before draw hook but a draw hook....

Actually, if we go this route, we may need to have one registry for
user hooks, and another registry from internal calls.  So any artist
could support drawing other artists, eg Lines and Labels register with
Ticks register which registers with an Axis which registers with an
Axes which registers with a Figure.  Of course at this point we are
talking about a major architectural change and so there would need to
be some clear utility, and I'm not sure what it is.  The current
design assumes an artist knows how to draw it's child artist, the one
you are proposing it seems assumes the child artist knows how to
insert itself into the parent.

JDH

>>>>> "Eric" == Eric Firing <ef...@ha...> writes:

    >> * Are your changes python2.3 compatible?  We are still trying
    >> to support 2.3, so avoid 2.4 only features like decorators
    >> until we remove 2.3 support

    Eric> Good, I thought we were still restricted to 2.2!

I was wondering how long it would take for someone to notice that
<wink>

I think we could lose python2.2 support if there is a good reason.  Is
anyone still using it?  I'll also post to the user's list.  Perhaps in
the next release we should issue deprecation warnings for 2.2 and
encourage people to post to the list if they have an objection.
Typically with these things you don't see complaints until you take
something away.

We haven't been compiling 2.2 for windows for sometime now.

JDH

>   * Are your changes python2.3 compatible?  We are still trying to
>     support 2.3, so avoid 2.4 only features like decorators until we
>     remove 2.3 support

Good, I thought we were still restricted to 2.2!

Does this mean boilerplate.py should be updated?  (Or maybe it isn't 
worth fiddling with it.)
# wrap the plot commands defined in axes.  The code generated by this
# file is pasted into pylab.py.  We did try to do this the smart way,
# with callable functions and new.function, but could never get the
# docstrings right for python2.2.  See
http://groups-beta.google.com/group/comp.lang.python/browse_thread/thread/dcd63ec13096a0f6/17739e70ac6f710c?lnk=gst&q=dcd63ec13096a0f6&rnum=1#17739e70ac6f710c

Eric

John Hunter wrote:
> Hey Eric, I'm CC-ing the devel list because some of this has
> potentially far reaching consequences.
> 
>>>>>> "Eric" == Eric Firing <ef...@ha...> writes:
> 
>     Eric> In connection with the colorbar sizing problem I have been
>     Eric> thinking about adding a function list to axes, with the
>     Eric> functions to be executed early in the draw() method, to
>     Eric> handle this sort of thing.  Using this mechanism to handle
> 
> I think this is a very nice idea -- I saw a talk at pycon one year about
> the utility of adding hooks, eg before_somefunc_hook and after_somefunc_hook
> that are customizable *for every function in your API*.  The questions are
> 
>   1) what is the right data structure -- a single callable, a
>      sequence, or a dictionary registry with connect/disconnect
>      semantics (eg the backend event callback handler)
> 
>   2) what is the signature (see below)
> 
>   3) which artists should support it
> 
> Most elegant, but most expensive, is something like a ordered
> dictionary which is supported for every artist.  By renaming all the
> artist draw methods _draw, we can do this quite simply in
> backend_bases.Renderer with
> 
> def draw(self, renderer):
>     for func in self.before_draw_hook.ordered_values():
>         func(self, renderer)
> 
>     self._draw(renderer)
> 
>     for func in self.after_draw_hook.ordered_values():
>         func(self, renderer)
> 

Nice, simple, elegant, but it doesn't quite match what I had in mind for 
axes.  I was thinking that the place to put the hook call is inside the 
draw() method, so it can take advantage of the intial work done there: 
getting a cached renderer, checking for visibility.  Maybe this doesn't 
matter.

Also, looking at backend_bases, I can't figure out where this goes. 
There is a RendererBase class, but it doesn't have a draw method.  And 
there is a FigureCanvasBase, but its draw method is overridden by each 
backend.  What am I missing?

> 
> I think this could be really useful and would help handle problems
> that crop up all the time -- how to handle physical layout problems
> where we need the renderer to get size information like you are
> experiencing with colorbar.  This problem also crops up in a lot of
> text layout problems.  Unfortunately, draw time layout creates other
> problems (sometimes users want this info at the API level), and order
> effects becomes very important and potentially complicated.  But that
> said, given the current architecture, something like this would be a
> definite improvement.
> 
> I'm inclined to use an ordered dictionary because one wants the
> connect/disconnect semantics that keys make easy, and because ordering
> matters.  There is a python cookbook recipe for an ordered dictionary,
> and we could easily roll our own.  Because the calls are potentially
> expensive, we might want an optimization like
> 
I agree, an ordered dictionary is ideal if one needs the option of 
multiple functions being executed.  I would probably have it inherit 
from list and tack on the dictionary; I think this might make for the 
fastest cycling through the list, and that is the place where speed is 
needed.

> class MyOrderedDictionary:
>     def __init__(blah):
>         self.isempty = True
> 
> 
> so we can do
> 
> def draw(renderer):
>     if not self.before_draw_hook.isempty:
>         for func in self.before_draw_hook.ordered_values():
>             func(self, renderer)
> 
>     self._draw(renderer)
> 
>     if not self.after_draw_hook.isempty:
>         for func in self.after_draw_hook.ordered_values():
>             func(self, renderer)
> 
> Since 99% of the time I guess these will be empty and then we are just
> adding a few attribute lookups and a boolean check for each draw.

Detail:
Maybe it would be simpler and faster to initialize like this:

     self.before_draw_hook=None

and have
     def register_before_draw(self, name, func):
         if self.before_draw_hook is None:
             self.before_draw_hook = OrderedDictionary()
         self.before_draw_hook.append(name, func)

It saves one attribute lookup at draw time.	

Larger picture:

Looking at the axes draw method, I am wondering whether a more general 
application of an OrderedDictionary (or ListDict, or whatever) wouldn't 
be a better approach.  Why should there be before_draw_hook, draw, and 
after_draw_hook, with a big list generation and sorting operation at 
draw time?  Could everything be done by having artists register all 
draw-time functions in a single ListDict, with sufficient ordering 
information to keep everything straight?  That's pretty vague, and there 
is a lot I don't understand--but it just seems like there is an 
opportunity here for simplifying the overall structure with improved 
flexibility and no loss of speed.  (Again, some kind of result-caching 
and need-for-recomputation-signalling would be needed.)

Eric

Eric Firing had the excellent idea of making a CODING_GUIDE which
summarizes the conventions used in matplotlib development, and I've
added this to svn.  Feel free to make changes, additions and
comments.  I think we could add a lot here, including an overview of
the API.  Here  is the document I just committed::

Devs, feel free to edit this document.  This is meant to be a guide to
developers on the mpl coding practices and standards


== Committing Changes ==

When committing changes to matplotlib, there are a few things to bear
in mind.  

  * if your changes are nontrivial, please make an entry in the
    CHANGELOG

  * if you change the API, please document it in API_CHANGES, and
    consider posing to mpl-devel

  * Are your changes python2.3 compatible?  We are still trying to
    support 2.3, so avoid 2.4 only features like decorators until we
    remove 2.3 support

  * Are your changes Numeric, numarray and numpy compatible?  Try
    running simple_plot.py or image_demo.py with --Numeric, --numarray
    and --numpy (Note, someone should add examples to
    backend_driver.py which explicitly require numpy, numarray and
    Numeric so we can automatically catch these)

  * Can you pass examples/backend_driver.py?  This is our poor man's
    unit test.

  * If you have altered extension code, do you pass
    unit/memleak_hawaii.py?


== Naming conventions ==

  functions and class methods  : lower or lower_underscore_separated

  attributes and variables     : lower or lowerUpper

  classes                      : Upper or MixedCase

Personally, I prefer the shortest names that are still readable.

== kwargs processing == 

Matplotlib makes extensive use of **kwargs for pass through
customizations from one function to another, eg the pylab plot ->
Axes.plot pass through.  As a general rule, the use of **kwargs should
be reserved for pass-through keyword arguments, eg

  def somefunc(x, k1='something', **kwargs):
      # do some thing with x, k1
      return some_other_func(..., **kwargs)

If I intend for all the keyword args to be used in somefunc alone, I
just use the key/value keyword args in the function definition rather
than the **kwargs idiom.  In some cases I want to consume some keys
and pass through the others, in which case I pop the ones I want to
use locally and pass on the rest, eg I pop scalex and scaley in
Axes.plot and assume the rest are Line2D keyword arguments.  Whenever
you mutate a kwargs dictionary (eg by popping it), you must first copy
it since the user may be explitly passing in a dictionary which is
used across many function calls.  As an example of a copy, pop,
passthrough usage, see Axes.plot:

    def plot(self, *args, **kwargs):
        kwargs = kwargs.copy()
        scalex = popd(kwargs, 'scalex', True)
        scaley = popd(kwargs, 'scaley', True)
        if not self._hold: self.cla()
        lines = []
        for line in self._get_lines(*args, **kwargs):
            self.add_line(line)
            lines.append(line)

popd is a matplotlib.cbook function to pop an item from a dictionary
with a default value if the item doesn't exist

Note there is a use case when kwargs are meant to be used locally in
the function (not passed on), but you still need the **kwargs idiom.
That is when you want to use *args to allow variable numbers of
non-keyword args.  In this case, python will not allow you to use
named keyword args after the *args usage, so you will be forced to use
**kwargs.  An example is matplotlib.contour.ContourLabeler.clabel

    def clabel(self, *args, **kwargs):
        fontsize = kwargs.get('fontsize', None)
        inline = kwargs.get('inline', 1)
        self.fmt = kwargs.get('fmt', '%1.3f')
        colors = kwargs.get('colors', None)
        if len(args) == 0:
            levels = self.levels
            indices = range(len(self.levels))
        elif len(args) == 1:
	   ...etc...



== class documentation ==

matplotlib uses artist instrospection of docstrings to support
properties.  All properties that you want to support through setp and
getp should have a set_property and get_property method in the Artist
class.  Yes this is not ideal given python properties or enthought
traits, but it is a historical legacy for now.  The setter methods use
the docstring with the ACCEPTS token to indicate the type of argument
the method accepts.  Eg in matplotlib.lines.Line2D

    def set_linestyle(self, linestyle):
        """
        Set the linestyle of the line

        ACCEPTS: [ '-' | '--' | '-.' | ':' | 'steps' | 'None' | ' ' | '' ]
        """


Since matplotlib uses a lot of pass through kwargs, eg in every
function that creates a line (plot, semilogx, semilogy, etc...), it
can be difficult for the new user to know which kwargs are supported.
I have developed a docstring interpolation scheme to support
documentation of every function that takes a **kwargs.  The
requirements are:

  1) single point of configuration so changes to the properties don't
     require multiple docstring edits

  2) as automated as possible so that as properties change the docs
     are updated automagically.

I have added a matplotlib.artist.kwdocd to faciliate this.  This
combines python string interpolation in the docstring with the
matplotlib artist introspection facility that underlies setp and getp.
The kwdocd is a single dictionary that maps class name to a docstring
of kwargs.  Here is an example at the bottom of matplotlib.lines

artist.kwdocd['Line2D'] = '\n'.join(artist.ArtistInspector(Line2D).pprint_setters(leadingspace=12))

Then in any function accepting Line2D passthrough kwargs, eg
matplotlib.axes.Axes.plot

    def plot(self, *args, **kwargs):
        """
	Some stuff omitted

        The kwargs are Line2D properties:
%(Line2D)s

        kwargs scalex and scaley, if defined, are passed on
        to autoscale_view to determine whether the x and y axes are
        autoscaled; default True.  See Axes.autoscale_view for more
        information
        """
	pass
    plot.__doc__ = plot.__doc__ % artist.kwdocd

Note there is a problem for Artist __init__ methods, eg
Patch.__init__ which supports Patch kwargs, since the artist inspector
cannot work until the class is fully defined and we can't modify the
Patch.__init__.__doc__ docstring outside the class definition.  I have
made some manual hacks in this case which violates the "single entry
point" requirement above; hopefully we'll find a more elegant solution
before too long

Hey Eric, I'm CC-ing the devel list because some of this has
potentially far reaching consequences.

>>>>> "Eric" == Eric Firing <ef...@ha...> writes:

    Eric> In connection with the colorbar sizing problem I have been
    Eric> thinking about adding a function list to axes, with the
    Eric> functions to be executed early in the draw() method, to
    Eric> handle this sort of thing.  Using this mechanism to handle

I think this is a very nice idea -- I saw a talk at pycon one year about
the utility of adding hooks, eg before_somefunc_hook and after_somefunc_hook
that are customizable *for every function in your API*.  The questions are

  1) what is the right data structure -- a single callable, a
     sequence, or a dictionary registry with connect/disconnect
     semantics (eg the backend event callback handler)

  2) what is the signature (see below)

  3) which artists should support it

Most elegant, but most expensive, is something like a ordered
dictionary which is supported for every artist.  By renaming all the
artist draw methods _draw, we can do this quite simply in
backend_bases.Renderer with

def draw(self, renderer):
    for func in self.before_draw_hook.ordered_values():
        func(self, renderer)

    self._draw(renderer)

    for func in self.after_draw_hook.ordered_values():
        func(self, renderer)


I think this could be really useful and would help handle problems
that crop up all the time -- how to handle physical layout problems
where we need the renderer to get size information like you are
experiencing with colorbar.  This problem also crops up in a lot of
text layout problems.  Unfortunately, draw time layout creates other
problems (sometimes users want this info at the API level), and order
effects becomes very important and potentially complicated.  But that
said, given the current architecture, something like this would be a
definite improvement.

I'm inclined to use an ordered dictionary because one wants the
connect/disconnect semantics that keys make easy, and because ordering
matters.  There is a python cookbook recipe for an ordered dictionary,
and we could easily roll our own.  Because the calls are potentially
expensive, we might want an optimization like

class MyOrderedDictionary:
    def __init__(blah):
        self.isempty = True


so we can do

def draw(renderer):
    if not self.before_draw_hook.isempty:
        for func in self.before_draw_hook.ordered_values():
            func(self, renderer)

    self._draw(renderer)

    if not self.after_draw_hook.isempty:
        for func in self.after_draw_hook.ordered_values():
            func(self, renderer)

Since 99% of the time I guess these will be empty and then we are just
adding a few attribute lookups and a boolean check for each draw.

JDH