16

The way to deal with mutable default arguments in Python is to set them to None.

For example:

def foo(bar=None):
    bar = [] if bar is None else bar
    return sorted(bar)

If I type in the function definition, then the only type for bar says that bar is Optional when, clearly, it is not Optional by the time I expect to run that sorted function on it:

def foo(bar: Optional[List[int]]=None):
    bar = [] if bar is None else bar
    return sorted(bar) # bar cannot be `None` here

So then should I cast?

def foo(bar: Optional[List[int]]=None):
    bar = [] if bar is None else bar
    bar = cast(List[int], bar) # make it explicit that `bar` cannot be `None`
    return sorted(bar)

Should I just hope that whoever reads through the function sees the standard pattern of dealing with default mutable arguments and understands that for the rest of the function, the argument should not be Optional?

What's the best way to handle this?

EDIT: To clarify, the user of this function should be able to call foo as foo() and foo(None) and foo(bar=None). (I don't think it makes sense to have it any other way.)

EDIT #2: Mypy will run with no errors if you never type bar as Optional and instead only type it as List[int], despite the default value being None. However, this is highly not recommended because this behavior may change in the future, and it also implicitly types the parameter as Optional. (See this for details.)

Improve this question

6 Answers 6

Reset to default
5

None is not the only sentinel available. You can choose your own list value to use as a sentinel, replacing it (rather than None) with a new empty list at run time.

_sentinel = []

def foo(bar: List[int]=_sentinel):
    bar = [] if bar is _sentinel else bar
    return sorted(bar)

As long as no one calls foo using _sentinel as an explicit argument, bar will always get a fresh empty list. In a call like foo([]), bar is _sentinel will be false: the two empty lists are not the same object, as the mutability of lists means that you cannot have a single empty list that always gets referenced by [].

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

So for each mutable type I do this for, I would need to add a new sentinel corresponding to that type?
Yes. The use of a single None type to represent the "absence" of a value of any arbitrary type is antithetical to static typing. For example, even in Haskell, which has a Maybe type constructor for representing optional values, the value Nothing :: Maybe a is polymorphic: it's not a single value, but rather a whole family of monomorphic values like Nothing :: Maybe Int, Nothing :: Maybe String, etc.
You should make _sentinel an object() instead of an empty list. Having a mutable default argument is a common pitfall: docs.python-guide.org/writing/gotchas/…
A mutable default is only a problem if you actually mutate the default. We aren't doing that here; the only thing we do with _sentinel is compare it to bar and pass it to sorted. sorted(_sentinel) will always return a new empty list without modifying _sentinel at all.
@chepner But it's still a bad practice. You can end up accidentally mutating it without meaning to do so, and it'll introduce obscure bugs. For simple functions like this it's probably fine, but it'll be harder to make sure you're not mutating the object in more complex code.
|
3

I'm not sure what's the issue here, since using Optional[List[int]] as the type is perfectly fine in mypy: https://mypy-play.net/?mypy=latest&python=3.9&gist=2ee728ee903cbd0adea144ce66efe3ab

In your case, when mypy sees bar = [] if bar is None else bar, it is smart enough to realize that bar cannot be None beyond this point, and thus narrow the type to List[int]. Read more about type narrowing in mypy here: https://mypy.readthedocs.io/en/stable/kinds_of_types.html?highlight=narrow#union-types

Here's some other examples of type narrowing:

from typing import *

a: Optional[int]
assert a is not None
reveal_type(a)  # builtins.int

b: Union[int, float, str]
if isinstance(b, int):
    reveal_type(b)  # builtins.int
else:
    reveal_type(b)  # Union[builtins.float, builtins.str]
Improve this answer

3 Comments

I'm aware that MyPy has this functionality, but I'm worried about the clarity for human programmers. I would like it to be obvious that None is not really an acceptable value for that parameter, but instead is a default and will be replaced with something that is not None as soon as possible. If I were reading the function and jumped straight from the function definition to the return statement, I would be very confused, because the function definition says the parameter can be None, but the function in the return statement cannot take a None parameter.
@ProQ I understand your concern, but I would argue that Optional[List[int]] is still the better choice here. IMO function signatures are for users, users read the signature to understand what arguments they need to provide, and what are their types. Here it's possible to provide None as an argument, so the type should include Optional. It's a different story for those who are reading the function implementation, since they'll need to read through the entire function body before they can understand what the function does, and they'll see the part where you substitute None with default.
If you're still worried about clarity, I would say that a comment or a docstring might work better. Also, I believe mypy by default allows omitting the Optional in the type hints, if the default is None.
2

Why not just cut out the cast when you shadow bar:

def foo(bar: Optional[List[int]]=None):
    bar : List[int] = [] if bar is None else bar
    return sorted(bar)
Improve this answer

9 Comments

mypy reports an error for this kind of variable type redefinition.
@user2357112supportsMonica That's interesting. It works fine in my python editor.
@Kraigolas mypy is a tool that is used to verify type hints and their usage to help enforce typing in python. The editors may be ok with how it looks and python will run just fine but mypy is more strict on type hints and says there is errors.
@Mythalorian Thanks! To me, as types aren't actually checked in python, that restriction that you must cast instead of shadow seems arbitrary and clunky (even in Rust I can shadow a variable as I've done in this answer), so I'll leave the answer up in case OP is fine with a disagreement with mypy, but thanks for pointing that out to me!
I personally am looking for something that doesn't throw a Mypy error, but I think that this answer is still helpful for other people who are okay with Mypy errors (and maybe Mypy will change its ways eventually)
|
0

If you add a default value to a parameter in a function than yes it is optional. Also in your code you're allowing the caller to supply nothing to the function and it still works cause it just creates an empty list.

Also in python even with typing it doesn't enforce the type. That's why its called "type hinting" in the documentation.

So if you want to allow callers to call the function with no arguments and it just sort an empty list then your code here is proper.

def foo(bar=None):
    bar = [] if bar is None else bar
    return sorted(bar)

But if you never want a caller to NOT supply something even None, then you should change you function signature.

def foo(bar: List[int]):
    bar = [] if bar is None else bar
    return sorted(bar)

Now type hinting detects that bar is a List[int]. You have to pass something to foo. So you can do foo(None) which is why you need the None check but now foo() is invalid and throws an error.

If you do want them to not pass in anything then just do this and type hinting still works.

def foo(bar: List[int]=None):
   bar = [] if bar is None else bar
   return sorted(bar)
Improve this answer

3 Comments

This is absolutely a valid point that you make, but bar: List[int] does not make it clear that None will be handled in any way, and it looks like OP wants the type hints to be explicit for someone reading the code to be able to understand.
I see what you're saying but the OP says that the list isnt optional which sounds like they are expecting something to be passed in and not an empty function call like foo(). If that's the case than they should remove the default value and let python throw an exception so the person calling it knows it's a required parameter.
Optional is assumed only because None, rather than an actually list, is used as the default. The Optional[a] type hint does not mean an argument at runtime is optional; it's just short for Union[a,None].
0

How about using Sequence[int]? It will be type-checked that the argument (including its default) is almost readonly.

from collections.abc import Sequence

def foo(bar: Sequence[int] = []) -> list[int]:
    return sorted(bar)

Still bar can be mutated as below, but the risk would not be high. (_sentinel: list[int] = [] can be mutated, too.)

def foo(bar: Sequence[int] = []) -> list[int]:
    if isinstance(bar, list):
        # reveal_type(bar)  # => Revealed type is "builtins.list[Any]"
        bar.append(0)
    return sorted(bar)
Improve this answer

Comments

0

Learned of another workaround for this recently: use an immutable argument type instead if you can. For example: take a function that uses a default column name list for ETL functions if the argument is None, but we can just change them all to tuples and avoid the mutability issue.

def function(
    ...,
    error_group_by_cols: list[str] = None,
) -> None:

    if error_group_by_cols is None:
        error_group_by_cols = copy.deepcopy(METRIC_COLS)

There was no reason we couldn't turn the lists into tuples so we end up with:

def function(
    ...,
    error_group_by_cols: tuple[str] = METRIC_COLS,
) -> None:
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.