16

I am working with a Python object that implements __add__, but does not subclass int. MyObj1 + MyObj2 works fine, but sum([MyObj1, MyObj2]) led to a TypeError, becausesum() first attempts 0 + MyObj. In order to use sum(), my object needs __radd__ to handle MyObj + 0 or I need to provide an empty object as the start parameter. The object in question is not designed to be empty.

Before anyone asks, the object is not list-like or string-like, so use of join() or itertools would not help.

Edit for details: the module has a SimpleLocation and a CompoundLocation. I'll abbreviate Location to Loc. A SimpleLoc contains one right-open interval, i.e. [start, end). Adding SimpleLoc yields a CompoundLoc, which contains a list of the intervals, e.g. [[3, 6), [10, 13)]. End uses include iterating through the union, e.g. [3, 4, 5, 10, 11, 12], checking length, and checking membership.

The numbers can be relatively large (say, smaller than 2^32 but commonly 2^20). The intervals probably won't be extremely long (100-2000, but could be longer). Currently, only the endpoints are stored. I am now tentatively thinking of attempting to subclass set such that the location is constructed as set(xrange(start, end)). However, adding sets will give Python (and mathematicians) fits.

Questions I've looked at:

I'm considering two solutions. One is to avoid sum() and use the loop offered in this comment. I don't understand why sum() begins by adding the 0th item of the iterable to 0 rather than adding the 0th and 1st items (like the loop in the linked comment); I hope there's an arcane integer optimization reason.

My other solution is as follows; while I don't like the hard-coded zero check, it's the only way I've been able to make sum() work.

# ...
def __radd__(self, other):
    # This allows sum() to work (the default start value is zero)
    if other == 0:
        return self
    return self.__add__(other)

In summary, is there another way to use sum() on objects that can neither be added to integers nor be empty?

Improve this question
3
  • 2
    The reason why sum starts with 0 is that it supports empty sequences, too. sum([]) will return the value of the start argument. If this empty list is supposed to hold objects of your custom type, what should sum return in this case? Commented Jul 24, 2012 at 6:04
  • 1
    Is there anything wrong with explicitly passing sum()'s start arg, as in sum(SomeIterable, MyCustomStartObject)? Commented Jul 24, 2012 at 6:09
  • The object is meaningless if it's empty. It contains a read-only location set by the constructor. So sum([]) doesn't have a logical return value, and I can't pass an empty start object. Commented Jul 24, 2012 at 6:18

6 Answers 6

Reset to default
16

Instead of sum, use:

import operator
from functools import reduce
reduce(operator.add, seq)

in Python 2 reduce was built-in so this looks like:

import operator
reduce(operator.add, seq)

Reduce is generally more flexible than sum - you can provide any binary function, not only add, and you can optionally provide an initial element while sum always uses one.

Also note: (Warning: maths rant ahead)

Providing support for add w/r/t objects that have no neutral element is a bit awkward from the algebraic points of view.

Note that all of:

  • naturals
  • reals
  • complex numbers
  • N-d vectors
  • NxM matrices
  • strings

together with addition form a Monoid - i.e. they are associative and have some kind of neutral element.

If your operation isn't associative and doesn't have a neutral element, then it doesn't "resemble" addition. Hence, don't expect it to work well with sum.

In such case, you might be better off with using a function or a method instead of an operator. This may be less confusing since the users of your class, seeing that it supports +, are likely to expect that it will behave in a monoidic way (as addition normally does).

Thanks for expanding, I'll refer to your particular module now:

There are 2 concepts here:

  • Simple locations,
  • Compound locations.

It indeed makes sense that simple locations could be added, but they don't form a monoid because their addition doesn't satisfy the basic property of closure - the sum of two SimpleLocs isn't a SimpleLoc. It's, generally, a CompoundLoc.

OTOH, CompoundLocs with addition looks like a monoid to me (a commutative monoid, while we're at it): A sum of those is a CompoundLoc too, and their addition is associative, commutative and the neutral element is an empty CompoundLoc that contains zero SimpleLocs.

If you agree with me (and the above matches your implementation), then you'll be able to use sum as following:

sum( [SimpleLoc1, SimpleLoc2, SimpleLoc3], start=ComplexLoc() )

Indeed, this appears to work.

I am now tentatively thinking of attempting to subclass set such that the location is constructed as set(xrange(start, end)). However, adding sets will give Python (and mathematicians) fits.

Well, locations are some sets of numbers, so it makes sense to throw a set-like interface on top of them (so __contains__, __iter__, __len__, perhaps __or__ as an alias of +, __and__ as the product, etc).

As for construction from xrange, do you really need it? If you know that you're storing sets of intervals, then you're likely to save space by sticking to your representation of [start, end) pairs. You could throw in an utility method that takes an arbitrary sequence of integers and translates it to an optimal SimpleLoc or CompoundLoc if you feel it's going to help.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

The object contains a location interval, e.g. [start,end). When summed, it yields a compound location, e.g. [start1,end1),[start2,end2),...[start_n_, end_n_). So [start, start) would be a neutral object, but it's not currently treated as empty. And technically since it's joining intervals, it should use union, but I don't want to explicitly store all of the intervening integers. I should see if <strike>a set generator</strike> xrange works with union.
Thanks for the clarification! I've expanded my answer.
We are now considering treating a SimpleLocation as a CompoundLocation with only one interval. I'm not entirely sure what we are going to do, but I'm accepting because your answer has helped me think about the mathematical logic of this structure. Thanks so much!
@mgilson "one" was referring to an initial element. It is 0 by default but you can provide your own as the start parameter.
probably not-obvious note for future readers like me: in Python 2.x reduce is builtin while in 3.x has to be imported from functools
4

I think that the best way to accomplish this is to provide the __radd__ method, or pass the start object to sum explicitly.

In case you really do not want to override __radd__ or provide a start object, how about redefining sum()?

>>> from __builtin__ import sum as builtin_sum
>>> def sum(iterable, startobj=MyCustomStartObject):
...     return builtin_sum(iterable, startobj)
...

Preferably use a function with a name like my_sum(), but I guess that is one of the things you want to avoid (even though globally redefining builtin functions is probably something that a future maintainer will curse you for)

Improve this answer

1 Comment

Yes, I can definitely see somebody kicking me for redefining sum(). But thank you for your answer!
3

Actually, implementing __add__ without the concept of an "empty object" makes little sense. sum needs a start parameter to support the sums of empty and one-element sequences, and you have to decide what result you expect in these cases:

sum([o1, o2]) => o1 + o2  # obviously
sum([o1]) => o1  # But how should __add__ be called here?  Not at all?
sum([]) => ?  # What now?
Improve this answer

1 Comment

I've commented on another answer that there could hypothetically be an empty object (empty location interval). But in an end use application, an empty location would be meaningless, possibly indicative of user error, so I'm hesitant to just invisibly drop the empty object, yet I don't want to throw a warning every time sum() is called.
2

You could use an object that's universally neutral wrt. addition:

class Neutral:
    def __add__(self, other):
        return other

print(sum("A BC D EFG".split(), Neutral())) # ABCDEFG
Improve this answer

1 Comment

I like this a lot. However, the documentation currently urges end users to use +, so I would like an implementation that allows the use of sum() without the start argument.
0

You could so something like:

from operator import add
try:
    total = reduce(add, whatever) # or functools.reduce in Py3.x
except TypeError as e:
    # I'm not 100% happy about branching on the exception text, but
    # figure this msg isn't likely to be changed after so long...
    if e.args[0] == 'reduce() of empty sequence with no initial value':
        pass # do something appropriate here if necessary
    else:
        pass # Most likely that + isn't usable between objects...
Improve this answer

1 Comment

Thank you for this suggestion. I'm biased against reduce() because I'm leery of code that has to change between 2 and 3k.
0

In addition (ba dum tss) to Kos' explanation about the meaning of addition and its neutral element there may be situations where you want an addition without an initial 0+:

  • you don't want to implement the special case in your classes radd
  • your object is large and copying it (0+o) would be costly
  • you want to implicitly check for empty sums
  • ...

Therefore, a sum with more options, available with other accumulators in my operationcounter package:

MISSING = object()
"""Sentinel to mark empty parameters."""

def sum_default(iterable, *, initial=MISSING, default=0):
    """Return the sum of all elements in the iterable.

    Like `sum` but with an optional initial element and an optional default
    return value.

    - If `iterable` is empty and `initial` and `default` are `MISSING`, a
      `TypeError` is raised.
    - If `iterable` is empty and `initial` is `MISSING`, but `default` is not,
      then `default` is returned.
    - If `initial` is `MISSING`, then there is truly no initial `0+=`.
    """
    if initial is not MISSING:
        return sum(iterable, initial)
    else:
        it = iter(iterable)
        try:
            initial = next(it)
        except StopIteration:
            if default is not MISSING:
                return default
            else:
                raise TypeError("accumulation of empty iterable with no" \
                        + " initial or default value")
        return sum(it, start=initial)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.