759 questions
1
vote
0
answers
162
views
What is the documented behaviour (if any) when someone imports `pkg.__init__[ as pkg]` instead of `import pkg`?
To be clear, I'm not suggesting anyone actually should import pkg.__init__ directly. This is to understand potential pitfalls if someone decides to convert a module-only distribution into a package, ...
- 4,194
3
votes
1
answer
179
views
What is the length of a python bytecode instruction in CPython?
Python docs on the dis module state that the length of a python bytecode instruction in CPython is 2 bytes (https://docs.python.org/3/library/dis.html)
However, when I disassemble a function and look ...
- 1,203
6
votes
2
answers
290
views
Is L[a:b]=L[c:d] optimized in Python to avoid creating a new list?
I am unable to find anything on the official Python documentation whether
L[a:b] = L[c:d]
creates a new (temporary) list for L[c:d] before the in-place modification of L. The tutorial says that:
All ...
- 142
1
vote
0
answers
77
views
How is import os.path possible? [duplicate]
Since os is a module instead of a package, import os.path should fail. For comparison:
>>> import os.sys
Traceback (most recent call last):
File "<python-input-0>", line 1, ...
- 2,447
2
votes
2
answers
168
views
How is `self` accessed in Python methods?
I am wondering how self works under-the-hood in Python classes.
My current limited understanding is that when a class is defined, e.g.
class Foo:
def __init__(self, x: int):
self.x = x
...
- 293
1
vote
1
answer
198
views
How can I store ids in Python without paying the 28-byte-per-int price?
My Python code stores millions of ids in various data structures, in order to implement a classic algorithm. The run time is good, but the memory usage is awful.
These ids are ints. I assume that ...
- 9,367
0
votes
1
answer
106
views
Why does Python pass an instance to a class attribute that is a function?
I define a class attribute and give it a function. When I call this function, the instance is passed on as the first argument (as if it's an instance function call with a self).
I would not expect an ...
- 821
2
votes
2
answers
95
views
Performance impact of inheriting from many classes
I am investigating the performance impact of a very broad inheritance setup.
Start with 260 distinct attribute names, from a0 through z9.
Create 260 classes with 1 uniquely-named attribute each. ...
- 304k
21
votes
1
answer
2k
views
How/why are {2,3,10} and {x,3,10} with x=2 ordered differently?
Sets are unordered, or rather their order is an implementation detail. I'm interested in that detail. And I saw a case that surprised me:
print({2, 3, 10})
x = 2
print({x, 3, 10})
Output (Attempt ...
- 29k
0
votes
0
answers
95
views
Dynamically create modules inside __init__ if they don't exist
I would like to dynamically create and import modules inside an inner __init__.py file, if one or several of a set of indexed submodules doesn't exist.
I have a set of module layers, say;
top_module/
...
- 1,430
7
votes
1
answer
190
views
Why is bytes(lst) slower than bytearray(lst)?
With lst = [0] * 10**6 I get times like these:
5.4 ± 0.4 ms bytearray(lst)
5.6 ± 0.4 ms bytes(bytearray(lst))
13.1 ± 0.7 ms bytes(lst)
Python:
3.13.0 (main, Nov 9 2024, 10:04:25) [GCC 14.2.1 ...
- 28k
0
votes
1
answer
135
views
Why is `co_code_adaptive` in `PyCodeObject` cast to a `uint16_t` pointer in `_PyCode_CODE` macro?
I was exploring the Python 3.11 source code and came across the _PyCode_CODE macro. I noticed that within this macro, the co_code_adaptive member of PyCodeObject is cast to a uint16_t* pointer. ...
37
votes
1
answer
2k
views
In Python 3.12, why does 'Öl' take less memory than 'Ö'?
I just read PEP 393 and learned that Python's str type uses different internal representations, depending on the content. So, I experimented a little bit and was a bit surprised by the results:
>&...
- 2,447
0
votes
0
answers
66
views
Why is the SWAP instruction missing in Python 3.11's disassembled bytecode for tuple swaps?
In newer versions, ROT_TWO is replaced by SWAP.
However, after disassembling the function using dis.dis(), I noticed that SWAP does not appear in the bytecode.
Python 3.9.0
>>> import dis
>...
- 1
4
votes
0
answers
224
views
AsyncIO CPython hangs with 100% CPU usage
Our Python application is hanging on these 2 particular machines after 10-20 minutes of use. Htop shows 100% CPU usage. I used Pystack to get the stack trace of the running process. The Python side of ...
- 797
3
votes
1
answer
141
views
What is the internal implementation of `copy.deepcopy()` in Python and how to override `__deepcopy__()` correctly?
When reading Antony Hatchkins' answer to "How to override the copy/deepcopy operations for a Python object?", I am confused about why his implementation of __deepcopy()__ does not check memo ...
- 110
0
votes
1
answer
413
views
Is there any way to construct a code object from a code string and assign it to an existing function using Python?
My problem is like this: I need to change how a function behave, but I can't access or change the file with which the function is located. I could import it though, and I would like to change the ...
- 328
0
votes
0
answers
289
views
How to inspect Python file/module information without using `__file__` — perhaps in a module-level `__getattr__(…)` function?
I have a utility package for Python projects I maintain and use. It’s a bundle of lightweight, common, junk-drawer tools (e.g. command parsing, app lifecycle management, document generation, &c &...
- 4,435
1
vote
0
answers
84
views
How does the Python Interpreter check thread duration?
My understanding is that historically, the python Interpreter counted lines of code executed and switched threads after a fixed amount. This was then changed to being time dependent.
What I am trying ...
- 601
8
votes
1
answer
593
views
Where did sys.modules go?
>>> import sys
>>> del sys.modules['sys']
>>> import sys
>>> sys.modules
Traceback (most recent call last):
File "<stdin>", line 1, in <module&...
- 665
5
votes
2
answers
250
views
Why is `if x is None: pass` faster than `x is None` alone?
Timing results in Python 3.12 (and similar with 3.11 and 3.13 on different machines):
When x = None:
13.8 ns x is None
10.1 ns if x is None: pass
When x = True:
13.9 ns x is None
11.1 ns if x is ...
- 10.9k
2
votes
1
answer
139
views
What is the term for a colon before a suite in Python syntax?
I want to know what the technical term for a colon in the context of introducing a suite after a statement is called. I do not mean colons in slices, key-value pairs or type hints, but this kind of ...
- 95
63
votes
1
answer
10k
views
Why is the simpler loop slower?
Called with n = 10**8, the simple loop is consistently significantly slower for me than the complex one, and I don't see why:
def simple(n):
while n:
n -= 1
def complex(n):
while True:...
- 28k
3
votes
1
answer
593
views
What is the `ExceptionTable` in the output of `dis`?
In python3.13, when I try to disassemble [i for i in range(10)], the result is as below:
>>> import dis
>>>
>>> dis.dis('[i for i in range(10)]')
0 RESUME ...
- 2,470
6
votes
1
answer
523
views
What does RESUME opcode actually do?
The documentation is not very informative (at least for me):
opcode:: RESUME (context)
A no-op. Performs internal tracing, debugging and optimization
checks.
The context oparand consists of two parts....
- 2,470
49
votes
3
answers
4k
views
Why list comprehensions create a function internally?
This is disassembly of a list comprehension in python-3.10:
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or &...
- 2,470
0
votes
1
answer
448
views
Which calls in Python may not call `__call__`?
The answer to my question may depend on the interpreter for the code although I'm not sure. If it does, then I would be happy to hear about any widely used Python interpreter, especially CPython ...
8
votes
2
answers
724
views
Can't create Race Condition in Python 3.11 using multiple threads
I believe this is a difference in Python 3.10 and above from older versions. Could someone explain this?
import threading
import time
counter = 0
lock = threading.Lock()
def increment():
global ...
0
votes
1
answer
145
views
Why does Python recursion limit change depending on function?
I was testing stuff when I noticed that python's recursion limit doesn't seem to apply equally to all functions. I'm not sure why or how and couldn't find any documentation explaining this behavior.
...
- 1,022
-1
votes
1
answer
475
views
Python Tuple vs List vs Array memory consumption
I've been reading Fluent code by Luciano Ramalho and in the chapter 'Overview of Built-in Sequences' when describing C struct behind float he states:
".. That's why an array of floats is much ...
- 338
0
votes
1
answer
652
views
Why does my Python thread block the main thread unless I add a print or a sleep?
Does anyone know why running this code causes the script to hang in the thread, unless I uncomment the print, the sleep, or the "if" condition, or remove the try/except? My understanding is ...
- 53
4
votes
2
answers
507
views
Storage of floating point numbers in memory in Python [duplicate]
I know that Python maintains an internal storage of small-ish integers rather than creating them at runtime:
id(5)
4304101544
When repeating this code after some time in the same kernel, the id is ...
- 17.6k
4
votes
1
answer
550
views
Why is set.remove so slow here?
(Extracted from another question.) Removing this set's 200,000 elements one by one like this takes 30 seconds (Attempt This Online!):
s = set(range(200000))
while s:
for x in s:
s.remove(x)...
- 28k
0
votes
1
answer
104
views
How to interpret the error message "Foo() takes no arguments" when specifying a class instance as base class?
The following code:
>>> class Foo: pass
>>> class Spam(Foo()): pass
will of course raise an error message:
Traceback (most recent call last):
File "<stdin>", line ...
- 405
2
votes
1
answer
631
views
Give an example/explanation of the closure parameter of the exec function
Can someone please explain to me when and how I would use the closure parameter of the exec function?
https://docs.python.org/3/library/functions.html#exec
The closure argument specifies a closure–a ...
- 183
2
votes
1
answer
645
views
Why sending a message to a web socket does not yield control to the event loop?
Consider the following code:
main.py
import asyncio
import websockets
async def echo(websocket):
async for message in websocket:
await websocket.send(message)
print(message)
async ...
- 65
1
vote
1
answer
462
views
How to clone a python class object? (not the instance but the class itself)
Imagine you have the following code:
class A:
pass
NewA = ... # copy A
NewA.__init__ = decorator(A.__init__) # but don't change A's init function, just NewA's
I am looking for a way to change ...
- 11
2
votes
1
answer
158
views
Why is the difference between id(2) and id(1) equal to 32?
>>> a = 1
>>> b = 2
>>> id(a), id(b), id(b) - id(a)
(1814458401008, 1814458401040, 32)
Is the memory address returned by id in bits or in bytes? Per the docs:
The current ...
- 927
8
votes
1
answer
1k
views
What's the benefit of asyncio using weakrefs to keep track of tasks?
Python docs for asyncio.create_task state:
Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. ...
- 52.7k
58
votes
2
answers
7k
views
Why is b.pop(0) over 200 times slower than del b[0] for bytearray?
Letting them compete three times (a million pops/dels each time):
from timeit import timeit
for _ in range(3):
t1 = timeit('b.pop(0)', 'b = bytearray(1000000)')
t2 = timeit('del b[0]', 'b = ...
- 28k
1
vote
1
answer
232
views
Prevent Python Interpreter from Exiting if CTRL-D is pressed
I am running a script with python -i main.py. The script starts some C threads and python threads using threading module, then python code ends and it goes to a prompt. How can i prevent python from ...
- 21
17
votes
1
answer
942
views
Are Python 3.11 objects as light as slots?
After Mark Shannon's optimisation of Python objects, is a plain object different from an object with slots?
I understand that after this optimisation in a normal use case, objects have no dictionary.
...
- 950
1
vote
1
answer
99
views
Local imports work in bundled PyInstaller app but in Python source
This issue has plagued me for the last few months, I need a more experienced opinion. We have a CLI Python application that uses a gRPC server to communicate with other backend services. Its ...
- 394
33
votes
2
answers
5k
views
yield from vs yield in for-loop
My understanding of yield from is that it is similar to yielding every item from an iterable. Yet, I observe the different behavior in the following example.
I have Class1
class Class1:
def ...
- 628
2
votes
1
answer
176
views
Python small integer cache: what's different when assigning multiple values?
I'm aware of the CPython implementation that holds a small integer cache in the [-5, 256] range, so I understand that a=2 and b=2 will refer to the same memory address (thus causing a is b to return ...
- 1,397
0
votes
1
answer
115
views
Why PyList_Append is called each time a list is evaluated?
I'm working with CPython3.11.0a3+. I added a break point at PyList_Append and modified the function to stop when the newitem is a dict. The original function:
int
PyList_Append(PyObject *op, PyObject *...
- 2,470
13
votes
1
answer
908
views
How are small sets stored in memory?
If we look at the resize behavior for sets under 50k elements:
>>> import sys
>>> s = set()
>>> seen = {}
>>> for i in range(50_000):
... size = sys.getsizeof(s)...
- 368k
2
votes
0
answers
121
views
Python objects implementation
I am studying the CPython's objects system implementation and I struggling to understand the differences between and purposes of the PyTypeObject and PyType_Type structs.
At first sight, I thought ...
- 21
3
votes
2
answers
1k
views
Please explain to me how does Python interpreter executes modules written in C/C++?
I'm trying to understand how it works. I know that Python interpreter translates python source code to byte code representation for a virtual machine (Python interpreter is a virtual machine) and ...
1
vote
1
answer
657
views
Where is the actual implementation of "__getattribute__"
I can't find the Python source code for __getattribute__. I looked in "object", and "type" classes, but I only see the function declaration. I don't see the actual definition.
- 8,146