Efficiently adding elements to a list in Python

Question

I'm iterating through a list of over three million items and am assigning them integer values. For organization, I have made a dictionary whose keys are the integers and values are the list of items with that score. A priori, I do not know how many items will have a certain score, so I'm using the + operator to append onto the list as follows:

for e in xs:
   myDict[val(e)] = myDict.get(val,[]) + [e]

My questions are:

Is there a cleaner way of doing this?
What's the time complexity of the + operation? Is it creating an entirely new list, copying the elements from the original list, and adding them in?
What if I was adding an element to a set?

Re "Is it creating an entirely new list, copying the elements from the original list, and adding them in?": That would be an example of the Shlemiel the painter’s algorithm. — Peter Mortensen
– Peter Mortensen, Commented Feb 18, 2023 at 17:23

Mike Müller · Accepted Answer · 2015-12-20 01:35:55Z

4

Use append:

for e in xs:
   myDict.setdefault(val(e), []).append(e)

This avoids building a new list every time. The operation list1 + list2 needs to build a new list in each iteration and hence allocate memory. append is more efficient because a list as pre-allocated memory at the end. For example, building a list with append starting from an empty list to a list with 10 million entries takes a bit more than 100 memory allocations.

The setdefault method of dictionaries returns the corresponding value if the key exist. If the key is not in th dictionary, it returns the default. In this case the default is a list. Due to the mutability of lists, we can append to the empty list in the first and to the partially filled list in each subsequent iteration.

An alternative to using setdefault() is collections.defaultdict. Do some profiling to find out which one is faster.

edited Dec 20, 2015 at 1:35

answered Dec 19, 2015 at 23:36

Mike Müller

86k21 gold badges174 silver badges165 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user217285 Over a year ago

Aha thank you. So does the setdefault method return the reference to the value of the key then? Otherwise the appending would not do anything.

ShadowRanger Over a year ago

Yes. If the key exists, it returns it, otherwise, it sets the key to be the second argument and then returns that.

Peter Mortensen · Accepted Answer · 2023-02-18 17:32:54Z

2

Yes, append and use a collections.defaultdict:

from collections import defaultdict

d = defaultdict(list)
for e in xs:
   d[val(e)].append(e)

appending is an 0(1) operation, your approach is linear 0(n+k) as you create a new list each time.

If you were in a situation where you were adding multiple items, you would extend your list.

One thing to note is that my_list += some_list is equivalent to mylist.extend(some_list), but it is very different from my_list = my_list + some_list. The former adds to the original list, and the latter is doing what your code is doing, concatenating two lists to create a completely new list.

The complexity for extend or += is 0(k) where k is the length of some_list.

wiki.python has a list of the complexity of common operations in Python.

edited Feb 18, 2023 at 17:32

Peter Mortensen

31.4k22 gold badges110 silver badges134 bronze badges

answered Dec 20, 2015 at 0:26

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

2 Comments

user217285 Over a year ago

Is there absolutely no difference between += and extend? That is, can I do += [e] and .extend([e]) and see no differences even on the nanosecond level?

Padraic Cunningham Over a year ago

There will be nanosecond differences but they both do the same job, I imagine extend may be the very slightly faster operation

Collectives™ on Stack Overflow

Efficiently adding elements to a list in Python

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related