Best way to pass function specified in file x as commandline parameter to file y in python

Question

I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset.

I would like to make it possible for the user to inject a function defined in another python file which is called in my script to transform the data.

Why? The only thing the user has to do is write the function which brings his data into the right format, then the existing code does the rest.

I'm aware of the fact that I could have the user write the function in the same file and call it, or to have an import statement etc.

So as a minimal example, I would like to have file y.py

def main(argv):
    # Parse args etc, let's assume it is there.

    dataset = tf.data.TFRecordDataset(args.filename)
    dataset = dataset.map(args.function)
    # Continue with doing stuff that is independent from actual content

So what I'd like to be able to do is something like this

    python y.py --func x.py my_func

And use the function defined in x.py my_func in dataset.map(...)

Is there a way to do this in python and if yes, which is the best way to do it?

So you have a function my_func defined in x.py? Why not from x import my_func and then call my_func(...) as you normally would with a locally defined function? Or import x then call x.my_func(...)? — user5386938
– user5386938, Commented Oct 25, 2019 at 12:15
@JustinEzequiel As mentioned in the question, I'm aware of this solution, but I don't want to use this, but to be able to pass the function to calling the script. — hechth
– hechth, Commented Oct 25, 2019 at 12:18
Then have you seen importlib.import_module? It's your use-case that I don't get. — user5386938
– user5386938, Commented Oct 25, 2019 at 12:30
"I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset." This is my use case. "Is there a way to do this in python and if yes, which is the best way to do it?" This is the question. So your answer is, there is no way to do it or rather there is no scenario in which this makes sense? — hechth
– hechth, Commented Oct 25, 2019 at 12:36

BenedictWilkins · Accepted Answer · 2019-10-25 13:47:55Z

1

Pass the name of the file as an argument to your script (and function name)
Read the file into a string, possibly extracting the given function
use Python exec() to execute the code

An example:

file = "def fun(*args): \n  return args"
func = "fun(1,2,3)"

def execute(func, file):
    program = file + "\nresult = " + func
    local = {}
    exec(program, local)
    return local['result']

r = execute(func, file)
print(r)

Similar to here however we must use locals() as we are not calling exec in global scope.

Note: the use of exec is somewhat dangerous, you should be sure that the function is safe - if you are using it then its fine!

Hope this helps.

edited Oct 25, 2019 at 13:47

answered Oct 25, 2019 at 12:06

BenedictWilkins

1,29313 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

hechth Over a year ago

Since exec returns None, how can it serve as a function which is expected to return the transformed element? Or how can it return whatever that specified function returns?

BenedictWilkins Over a year ago

it is discussed here. There may be a better way than to use exec for sure!

hechth Over a year ago

Okay so I can make it work by using another wrapper function, this would be a solution which exactly solves the problem specified in the question. So I'm waiting for more feedback but if nothing better comes up this can be an accepted answer if you edit it slightly to contain this info as well ;)

BenedictWilkins Over a year ago

@hechth I have updated with an example. I hope this is what you were after.

hechth · Accepted Answer · 2019-10-25 13:40:39Z

Ok so I have composed the answer myself now using the information from comments and this answer.

    import importlib, inspect, sys, os
    # path is given path to file, funcion_name is name of function and args are the function arguments

    # Create package and module name from path
    package = os.path.dirname(path).replace(os.path.sep,'.')
    module_name = os.path.basename(path).split('.')[0]

    # Import module and get members
    module = importlib.import_module(module_name, package)
    members = inspect.getmembers(module)

    # Find matching function
    function = [t[1] for t in members if t[0] == function_name][0]
    function(args)

This exactly solves the question, since I get a callable function object which I can call, pass around, use it as a normal function.

Collectives™ on Stack Overflow

Best way to pass function specified in file x as commandline parameter to file y in python

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related