Is there a way to use python argparse with nargs='*', choices, AND default?

Question

My use case is multiple optional positional arguments, taken from a constrained set of choices, with a default value that is a list containing two of those choices. I can't change the interface, due to backwards compatibility issues. I also have to maintain compatibility with Python 3.4.

Here is my code. You can see that I want my default to be a list of two values from the set of choices.

parser = argparse.ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd'],
                    default=['a', 'd'])
args = parser.parse_args()
print(args.tests)

All of this is correct:

$ ./test.py a
['a']
$ ./test.py a d
['a', 'd']
$ ./test.py a e
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: 'e' (choose from 'a', 'b', 'c', 'd')

This is incorrect:

$ ./test.py
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: ['a', 'd'] (choose from 'a', 'b', 'c', 'd')

I've found a LOT of similar questions but none that address this particular use case. The most promising suggestion I've found (in a different context) is to write a custom action and use that instead of choices:

https://stackoverflow.com/a/8624107/7660197

That's not ideal. I'm hoping someone can point me to an option I've missed.

Here's the workaround I plan to use if not:

parser.add_argument('tests', nargs='*',
                    choices=['a', 'b', 'c', 'd', 'default'],
                    default='default')

I'm allowed to add arguments as long as I maintain backwards compatibility.

Thanks!

Update: I ended up going with a custom action. I was resistant because this doesn't feel like a use case that should require custom anything. However, it seems like more or less the intended use case of subclassing argparse.Action, and it makes the intent very explicit and gives the cleanest user-facing result I've found.

class TestsArgAction(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        all_tests = ['a', 'b', 'c', 'd']
        default_tests = ['a', 'd']

        if not values:
            setattr(namespace, self.dest, default_tests)
            return

        # If no argument is specified, the default gets passed as a
        # string 'default' instead of as a list ['default']. Probably
        # a bug in argparse. The below gives us a list.
        if not isinstance(values, list):
            values = [values]

        tests = set(values)

        # If 'all', is found, replace it with the tests it represents.
        # For reasons of compatibility, 'all' does not actually include
        # one of the tests (let's call it 'e'). So we can't just do
        # tests = all_tests.
        try:
            tests.remove('all')
            tests.update(set(all_tests))
        except KeyError:
            pass

        # Same for 'default'
        try:
            tests.remove('default')
            tests.update(set(default_tests))
        except KeyError:
            pass

        setattr(namespace, self.dest, sorted(list(tests)))

Another recent one with choices and * nargs (different default), but otherwise the same point - stackoverflow.com/questions/73205632/…. Given the way choices are handled, there isn't a simple way to make this work. — hpaulj
– hpaulj, Commented Aug 18, 2022 at 3:38
That's an interesting approach. (Seems like the enumerate is unnecessary as i is unused). I may end up using that, since I'll likely override the usage message no matter what. The only downside is that it would break if someone specifies an argument multiple times. There's no reason for them to do so in my use case, but I prefer to be forgiving. I can probably use * instead of ? and I think it'll be fine with the usage override. — narwahl
– narwahl, Commented Aug 18, 2022 at 4:26

metatoaster · Accepted Answer · 2022-08-18 04:15:28Z

2

The behavior noted as incorrect is caused by the fact that the raw default value ['a', 'd'] is not inside the specified choices (see: relevant code as found in Python 3.4.10; this check method is effectively unchanged as of Python 3.10.3). I will reproduce the code from the Python argparse.py source code:

    def _check_value(self, action, value):
        # converted value must be one of the choices (if specified)
        if action.choices is not None and value not in action.choices:
            args = {'value': value,
                    'choices': ', '.join(map(repr, action.choices))}
            msg = _('invalid choice: %(value)r (choose from %(choices)s)')
            raise ArgumentError(action, msg % args)

When a default value is specified as a list, that entire value is passed to that _check_value method and thus it will fail (as any given list will not match any strings inside another list). You can actually verify that by setting a breakpoint with pdb in that method and trace through the values by stepping through each line, or alternatively test and verify the stated limitations with the following code:

import argparse
DEFAULT = ['a', 'd']
parser = argparse.ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd', DEFAULT],
                    default=DEFAULT)
args = parser.parse_args()
print(args.tests)

Then run python test.py

$ python test.py
['a', 'd']

This clearly passed because that very same DEFAULT value is present in the list of choices.

However, calling -h or passing any unsupported value will result in:

$ python test.py z
usage: test.py [-h] [{a,b,c,d,['a', 'd']} ...]
test.py: error: argument tests: invalid choice: 'z' (choose from 'a', 'b', 'c', 'd', ['a', 'd'])
$ python test.py -h
usage: test.py [-h] [{a,b,c,d,['a', 'd']} ...]

positional arguments:
  {a,b,c,d,['a', 'd']}
...

Which may or may not be ideal depending on use case as the output looks weird if not confusing. If this output is going to be user-facing it's probably not ideal, but if this is to maintain some internal system call emulation that won't leak out to users, the messages are probably not visible so this may be an acceptable workaround. Hence, I do not recommend this approach if the clarity of the choice message being generated is vital (which is >99% of typical use cases).

However, given that custom action is considered not ideal, I will assume overriding the ArgumentParser class may be a possible choice, and given that _check_value has not changed between 3.4 and 3.10, this might represent the most minimum additional code to nip out the incompatible check (with the specified use case as per the question):

class ArgumentParser(argparse.ArgumentParser):
    def _check_value(self, action, value):
        if value is action.default:
            return
        return super()._check_value(action, value)

This would ensure that the default value be considered a valid choice (return None if the value is the action's default, otherwise return the default check) before using the default implementation that is unsuitable for the requirement as outlined in the question; do please note that this prevents deeper inspection of what that action.default provides being a valid one (if that's necessary, custom Action class is most certainly the way to go).

Might as well show the example usage with the custom class (i.e. copy/pasted the original code, remove the argparse. to use the new custom class):

parser = ArgumentParser()
parser.add_argument('tests', nargs='*', choices=['a', 'b', 'c', 'd'],
                    default=['a', 'd'])
args = parser.parse_args()
print(args.tests)

Usage:

$ python test.py
['a', 'd']
$ python test.py a z
usage: test.py [-h] [{a,b,c,d} ...]
test.py: error: argument tests: invalid choice: 'z' (choose from 'a', 'b', 'c', 'd')
$ python test.py -h
usage: test.py [-h] [{a,b,c,d} ...]

positional arguments:
  {a,b,c,d}

optional arguments:
  -h, --help  show this help message and exit

edited Aug 18, 2022 at 4:15

answered Aug 18, 2022 at 2:34

metatoaster

19.2k5 gold badges65 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

narwahl Over a year ago

Thank you for the thorough answer. This is in-fact user facing (even though most of the users will be people who are contributing to this software -- this is test-suite code). So adding the default list to choices would be too messy. I'll think about overriding ArgumentParser. I think if I'm going to do anything that I can't accomplish in a one-liner, it will be the custom action though.

metatoaster Over a year ago

You're welcome. Custom Action class is safest; the answer I gave was done as such to highlight exactly what/where/why this is not supported out of the box by the argparse module and to produce the bare minimum code to workaround given the restrictions outlined in the question.

narwahl Over a year ago

I eventually went with a custom Action class. See the update to the description if you're interested. Thanks again!

hpaulj Over a year ago

The latest pull merger related to star-choices, github.com/python/cpython/pull/92565

metatoaster Over a year ago

@hpaulj unfortunately that only changes where the validation logic occurs (making the choices fully optional, though this is a breaking change for users that relied on existing, if wrong, behavior (they should have used +)), and has nothing to do with making default value of type list having each item being validated. So, no, the "incorrect" behavior highlighted here remains to be an issue.

Collectives™ on Stack Overflow

Is there a way to use python argparse with nargs='*', choices, AND default?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related