Understanding python tuple to list performance optimization

Question

I have a tuple, say, atup = (1,3,4,5,6,6,7,78,8) and produced dynamically by list of tuples when iterated (generator yield). Each tuple needs to get converted to list so each elements of tuple can be transformed further and used in a method. While doing this, I was surprised to learn that just doing list(atup) is much faster than using list comprehension like this [i for i in atup]. Here is what I did:

Performance Test 1:

timeit.timeit('list((1,3,4,5,6,6,7,78,8))', number=100000)
0.02268475245609025

Performance Test 2:

timeit.timeit('[i for i in (1,3,4,5,6,6,7,78,8)]', number=100000)
0.05304025196801376

Can you please explain this ?

chepner · Accepted Answer · 2019-03-26 16:34:54Z

2

The list comprehension has to iterate over the tuple at the Python level:

>>> dis.dis("[i for i in (1,2,3)]")
  1           0 LOAD_CONST               0 (<code object <listcomp> at 0x1075c0c90, file "<dis>", line 1>)
              2 LOAD_CONST               1 ('<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_CONST               5 ((1, 2, 3))
              8 GET_ITER
             10 CALL_FUNCTION            1
             12 RETURN_VALUE

list iterates over the tuple itself, and uses the C API to do it without going through (as much of) the Python data model.

>>> dis.dis("list((1,2,3))")
  1           0 LOAD_NAME                0 (list)
              2 LOAD_CONST               3 ((1, 2, 3))
              4 CALL_FUNCTION            1
              6 RETURN_VALUE

The Python-level iteration is more clearly seen in Python 2, which implements list comprehensions in a different fashion.

>>> def f():
...   return [i for i in (1,2,3)]
...
>>> dis.dis(f)
  2           0 BUILD_LIST               0
              3 LOAD_CONST               4 ((1, 2, 3))
              6 GET_ITER
        >>    7 FOR_ITER                12 (to 22)
             10 STORE_FAST               0 (i)
             13 LOAD_FAST                0 (i)
             16 LIST_APPEND              2
             19 JUMP_ABSOLUTE            7
        >>   22 RETURN_VALUE

As @blhsing points out, you can get disassemble the code object generated by the list comprehension in Python 3 to see the same thing.

>>> code = compile('[i for i in (1,2,3)]', '', 'eval')
>>> dis(code.co_consts[0])
  1           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                 8 (to 14)
              6 STORE_FAST               1 (i)
              8 LOAD_FAST                1 (i)
             10 LIST_APPEND              2
             12 JUMP_ABSOLUTE            4
        >>   14 RETURN_VALUE

edited Mar 26, 2019 at 16:34

answered Mar 26, 2019 at 16:12

chepner

538k77 gold badges595 silver badges747 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

blhsing Over a year ago

Note that in Python 3 one would simply have to disassemble the code object stored in the co_consts list attribute of the parent code object as indexed to see the content of its byte codes. For example: repl.it/repls/SomberUniformAngles

chepner Over a year ago

@blhsing Oh, thanks. It hadn't occurred to me to try to disassemble the code object.

juanpa.arrivillaga Over a year ago

In python 3.7 I've noticed dis recurses into any nested code objects automatically, and accepts a depth parameter!

chepner Over a year ago

Oh! So it does.

blhsing · Accepted Answer · 2019-03-26 16:12:15Z

1

The list constructor is implemented purely in C and has therefore minimal overhead, while with a list comprehension the Python compiler has to build a temporary function, build an iterator, store the iterator's output as variable i, and load the variable i to append to a list, which are a lot more Python byte codes to execute than simply loading a tuple and calling the list constructor.

answered Mar 26, 2019 at 16:12

blhsing

109k9 gold badges89 silver badges132 bronze badges

Collectives™ on Stack Overflow

Understanding python tuple to list performance optimization

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related