matplotlib-devel Mailing List for matplotlib

On Tue, Dec 15, 2009 at 9:57 AM, Andrew Straw <str...@as...> wrote:
>
>   notch_max = med + 1.57*iq/np.sqrt(row)
>   notch_min = med - 1.57*iq/np.sqrt(row)
>
> Is this code actually calculating a meaningful value? If so, what?
>

>From the statistics ignoramus in the room, so take this with a grain
of salt...  I'd write that code as

notch_max = med + (iq/2) * (pi/np.sqrt(row))

and it makes more sense.  The notch limits are an estimate of the
interval of the median, which is (one-half, for each up/down) the
q3-q1 range times a normalization factor which is pi/sqrt(n), where
n==row=len(d).  The 1/sqrt(n) makes some sense, as it's the usual
statistical error normalization factor.  The multiplication by pi, I'm
not so sure, and I can't find that exact formula in any quick stats
reference, but I'm sure someone who actually knows stats can point out
where it comes from.

Note that the code below does:

                if notch_max > q3:
                    notch_max = q3
                if notch_min < q1:
                    notch_min = q1

though matlab explicitly states in:

http://www.mathworks.com/access/helpdesk/help/toolbox/stats/boxplot.html

that

"""
Interval endpoints are the extremes of the notches or the centers of
the triangular markers. When the sample size is small, notches may
extend beyond the end of the box.
"""

So it seems to me that the more principled thing to do would be to
leave those notch markers outside the box if they land there, because
that's a warning of the robustness of the estimation. Clipping them to
q1/q3 is effectively hiding a problem...


cheers,

f

Hi,

I've been reading about box plots and examining the source code for 
boxplot() lately. While there doesn't seem to be a convention about what 
the notch specifies, I can't find any justification (or text describing) 
what exactly the MPL notch is. The source code is:

   # get median and quartiles
   q1, med, q3 = mlab.prctile(d,[25,50,75])
   iq = q3 - q1

   notch_max = med + 1.57*iq/np.sqrt(row)
   notch_min = med - 1.57*iq/np.sqrt(row)

Is this code actually calculating a meaningful value? If so, what?

The original commit was r1098, which doesn't offer a useful comment 
either (only "aaplied several sf patches" ... looking through the SF bug 
tracker, I couldn't find anything relevant from before the commit date 
of 2005-03-28).

The following (uncommitted) test currently fails. The reason is that 
mlab.prctile(x,50) doesn't handle even length sequences according to the 
numpy and wikipedia convention for the definition of median. Do we agree 
that it should pass?

Not only would I commit the test, but I also have a fix to make it pass, 
derived from scipy.stats.scoreatpercentile().

This would affect boxplot, if not more.

def test_prctile():
     # test odd lengths
     x=[1,2,3]
     assert mlab.prctile(x,50)==np.median(x)

     # test even lengths
     x=[1,2,3,4]
     assert mlab.prctile(x,50)==np.median(x)

     # derived from email sent by jason-sage to MPL-user on 20090914
     ob1=[1,1,2,2,1,2,4,3,2,2,2,3,4,5,6,7,8,9,7,6,4,5,5]
     p         = [75]
     expected = [5.5]

     # test vectorized
     actual = mlab.prctile(ob1,p)
     assert np.allclose( expected, actual )

     # test scalar
     for pi, expectedi in zip(p,expected):
         actuali = mlab.prctile(ob1,pi)
         assert np.allclose( expectedi, actuali )