Edit parameters in a python file from another python file

Question

I've got a configuration file for an application in the form of a python file containing a dict and I'm running an AWS lambda function to get this conf file from S3, change three variables in it and then push a copy back to S3. Is there any easy way to do this? A coworker said to try Jinja templating, but that seems limited to just HTML files?

Thanks

An example of the python config file is below. I need to change the "alpha" and "cycles" variables

import zutil

alpha = 2.13
cycles = 100


def my_transform(x, y, z):
    v = [x, y, z]
    v = zutil.rotate_vector(v, alpha, 0.0)
    return {'v1': v[0], 'v2': v[1], 'v3': v[2]}

parameters = {
    # units for dimensional quantities
    'units': 'SI',
    # reference state
    'reference': 'IC_1',
    'time marching': {
        'unsteady': {
            'total time': 1.0,
            'time step': 1.0,
            'order': 'second',
        },
        'scheme': {
            'name': 'lu-sgs',
            'stage': 1,
            #'name' : 'runge kutta',
            #'stage': 5,
        },
        'lu-sgs': {
            'Number Of SGS Cycles': 8,
            'Min CFL': 0.1,
            'Max CFL': 5.0,
            'Include Backward Sweep': True,
            'Include Relaxation': True,
            'Jacobian Update Frequency': 1,
            'Jacobian Epsilon': 1.0e-08,
            'CFL growth': 1.05,
            'Use Rusanov Flux For Jacobian': 'true',
            'Finite Difference Jacobian': 'false',
        },

        'multigrid': 10,
        'cfl': 2.5,
        'cfl transport': 2.5 * 0.5,
        'ramp': {'initial': 1.0, 'growth': 1.1},
        'cycles': cycles,
    },

    'equations': 'RANS',

    'RANS': {
        'order': 'euler_second',
        'limiter': 'vanalbada',
        'precondition': 'true',
        'turbulence': {
                   'model': 'sst',
        },
    },

    'material': 'air',
    'air': {
        'gamma': 1.4,
        'gas constant': 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No': 0.72,
        'Turbulent Prandtl No': 0.9,
    },
    'IC_1': {
        'temperature': 310.928,
        'pressure': 101325.0,
        'alpha': alpha,  # User defined variable used for post processing
        'V': {
            'vector': zutil.vector_from_angle(alpha, 0.0),
            'Mach': 0.85,
        },
        'Reynolds No': 5.0e6,
        'Reference Length': 275.8,
        'turbulence intensity': 1.e-4,
        'eddy viscosity ratio': 0.1,
    },
    'BC_1': {
        'ref': 7,
        'type': 'symmetry',
    },
    'BC_2': {
        'ref': 3,
        'type': 'wall',
        'kind': 'noslip',
    },
    'BC_3': {
        'ref': 9,
        'type': 'farfield',
        'condition': 'IC_1',
        'kind': 'riemann',
    },
    'write output': {
        'format': 'vtk',
                  'surface variables': ['V', 'p', 'T', 'rho', 'walldist', 'yplus', 'mach', 'cp', 'eddy', 'pressureforce', 'frictionforce'],
                  'volume variables': ['V', 'p', 'T', 'rho', 'walldist', 'mach', 'cp', 'eddy'],
                  'frequency': 500,
    },
    'report': {
        'frequency': 10,
        'forces': {
            'FR_1': {
                'name': 'wall',
                'zone': [9, 10, 11, 12, 13],
                'transform': my_transform,
                'reference area': 594720.0 * 0.5,  # half model area # half model area # half model area
            },
        },
    },
}

Jinja is a template library and it is certainly not limited to only HTML. But I am not sure why you need to have a template in the first place if all you read is pure python dict from a python file. Although to me it looks like it is a strange way of dealing with confs. Why don't you try to have a conf file in YAML or JSON and that way it will be easy to read, manipulate and write back to S3. Here is a tutorial to access files from / to S3 with AWS lambda. — SRC
– SRC, Commented Jun 20, 2017 at 8:32
@SRC, The application running requires the python file as the format of the config, I'll throw an example into the OP. I suppose I could read the .py, dump the dict into a JSON file, process that and then dump that back into another python file. As for the reading writing to S3, I've got that working already. — Lewiky
– Lewiky, Commented Jun 20, 2017 at 8:37
Although, this is a terrible and bad solution and I strongly suggest you review your code / architecture so that you do not need to use this solution but if you have a string (or a file containing few lines) and want to execute it as a python code then you can use eval or exec — SRC
– SRC, Commented Jun 20, 2017 at 8:54
I have no idea how else you can achieve this. However seems to me that this link may help. The last answer. — SRC
– SRC, Commented Jun 20, 2017 at 9:20

milo · Accepted Answer · 2017-06-20 09:19:26Z

Jinja2 can certainly do it. but if it's worth to do is another question

I modify the your file a little to make it renderable by jinja2

import zutil

alpha = {{ alpha | default(2.13) }}
cycles = {{ cycles | default(100)}}

def my_transform(x,y,z):
    v = [x,y,z]
    v =  zutil.rotate_vector(v,alpha,0.0)
    return {'v1' : v[0], 'v2' : v[1], 'v3' : v[2]}

parameters = { 

 # units for dimensional quantities
'units' : 'SI',

# reference state
'reference' : 'IC_1',

'time marching' : { 
                   'unsteady' : {
                                 'total time' : 1.0,
                                 'time step' : 1.0,
                                 'order' : 'second',
                                },
                   'scheme' : {
                             'name' : 'lu-sgs',
                               'stage': 1,
                               #'name' : 'runge kutta',
                               #'stage': 5,
                               },
                   'lu-sgs' : {
                               'Number Of SGS Cycles' : 8,
                               'Min CFL' : 0.1,
                               'Max CFL' : 5.0,
                               'Include Backward Sweep' : True,
                               'Include Relaxation' : True,
                               'Jacobian Update Frequency' : 1,
                               'Jacobian Epsilon' : 1.0e-08,
                               'CFL growth' : 1.05,
                               'Use Rusanov Flux For Jacobian' : 'true',
                               'Finite Difference Jacobian' : 'false',
                              },

                   'multigrid' : 10,
                   'cfl': 2.5,
                   'cfl transport' : 2.5*0.5,
                   'ramp': { 'initial': 1.0, 'growth': 1.1 },
                   'cycles' : cycles,
                  },

'equations' : 'RANS',

'RANS' : {
               'order' : 'euler_second',
               'limiter' : 'vanalbada',
               'precondition' : 'true',                                          
               'turbulence' : {
                               'model' : 'sst',
                              },
               },

'material' : 'air',
'air' : {
        'gamma' : 1.4,
        'gas constant' : 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No' : 0.72,
        'Turbulent Prandtl No' : 0.9,
        },
'IC_1' : {
          'temperature':310.928,
          'pressure':101325.0,
          'alpha': alpha, # User defined variable used for post processing
          'V': {
                'vector' : zutil.vector_from_angle(alpha,0.0),
                'Mach' : 0.85,
                },
           'Reynolds No' : 5.0e6,
           'Reference Length' : 275.8, 
          'turbulence intensity':1.e-4,
          'eddy viscosity ratio':0.1,
          },
'BC_1' : {
          'ref' : 7,
          'type' : 'symmetry',
         },
'BC_2' : {
          'ref' : 3,
          'type' : 'wall',
          'kind' : 'noslip',
         },
'BC_3' : {
          'ref' : 9,
          'type' : 'farfield',
          'condition' : 'IC_1',
          'kind' : 'riemann',
         },
'write output' : {
                  'format' : 'vtk',
                  'surface variables': ['V','p','T','rho','walldist','yplus','mach','cp','eddy','pressureforce','frictionforce'],
                  'volume variables': ['V','p','T','rho','walldist','mach','cp','eddy'],
                  'frequency' : 500,
                 },      
'report' : {
           'frequency' : 10,
        'forces' : {
            'FR_1' : {
                'name' : 'wall',
                'zone' : [9,10,11,12,13],
                'transform' : my_transform,
                'reference area' : 594720.0*0.5, # half model area # half model area # half model area 
            },
        },
},                      
}

Here is how to render it using jinja2. suppose the path_1 is path of your config file. path_2 is the path of new config file

from jinja2 import Environment, FileSystemLoader

new_config_contex = {'alpha':3, 'cycles': 200}
path, template_filename = os.path.split(path_1)
env = Environment(loader=FileSystemLoader(path))
new_conf_file_content=env.get_template(template_filename).render(new_config_contex)
with open(path_2, "wb") as f:
    f.write(new_conf_file_content)

Thanks for the advice, took it and extended out as in my answer I posted.

Lewiky · Accepted Answer · 2017-06-20 12:27:18Z

I found a solution, it's not pretty and as SRC mentioned, it's a bad solution that shouldn't be used in anything real, but it works.

I took advice from milo and converted my reference control.py file into a Jinja template as shown here:

import jinja2
from sys import argv

pyConf = """
import zutil

alpha = {{alpha}}
cycles = {{cycles}}

def my_transform(x,y,z):
    v = [x,y,z]
    v =  zutil.rotate_vector(v,alpha,0.0)
    return {'v1' : v[0], 'v2' : v[1], 'v3' : v[2]}

parameters = {

 # units for dimensional quantities
'units' : 'SI',

# reference state
'reference' : 'IC_1',

'time marching' : {
                   'unsteady' : {
                                 'total time' : 1.0,
                                 'time step' : 1.0,
                                 'order' : 'second',
                                },
                   'scheme' : {
                             'name' : 'lu-sgs',
                               'stage': 1,
                               #'name' : 'runge kutta',
                               #'stage': 5,
                               },
                   'lu-sgs' : {
                               'Number Of SGS Cycles' : 8,
                               'Min CFL' : 0.1,
                               'Max CFL' : 5.0,
                               'Include Backward Sweep' : True,
                               'Include Relaxation' : True,
                               'Jacobian Update Frequency' : 1,
                               'Jacobian Epsilon' : 1.0e-08,
                               'CFL growth' : 1.05,
                               'Use Rusanov Flux For Jacobian' : 'true',
                               'Finite Difference Jacobian' : 'false',
                              },

                   'multigrid' : 10,
                   'cfl': 2.5,
                   'cfl transport' : 2.5*0.5,
                   'ramp': { 'initial': 1.0, 'growth': 1.1 },
                   'cycles' : cycles,
                  },

'equations' : 'RANS',

'RANS' : {
               'order' : 'euler_second',
               'limiter' : 'vanalbada',
               'precondition' : 'true',
               'turbulence' : {
                               'model' : 'sst',
                              },
               },

'material' : 'air',
'air' : {
        'gamma' : 1.4,
        'gas constant' : 287.0,
        'Sutherlands const': 110.4,
        'Prandtl No' : 0.72,
        'Turbulent Prandtl No' : 0.9,
        },
'IC_1' : {
          'temperature':310.928,
          'pressure':101325.0,
          'alpha': alpha, # User defined variable used for post processing
          'V': {
                'vector' : zutil.vector_from_angle(alpha,0.0),
                'Mach' : 0.85,
                },
           'Reynolds No' : 5.0e6,
           'Reference Length' : 275.8,
          'turbulence intensity':1.e-4,
          'eddy viscosity ratio':0.1,
          },
'BC_1' : {
          'ref' : 7,
          'type' : 'symmetry',
         },
'BC_2' : {
          'ref' : 3,
          'type' : 'wall',
          'kind' : 'noslip',
         },
'BC_3' : {
          'ref' : 9,
          'type' : 'farfield',
          'condition' : 'IC_1',
          'kind' : 'riemann',
         },
'write output' : {
                  'format' : 'vtk',
                  'surface variables': ['V','p','T','rho','walldist','yplus','mach','cp','eddy','pressureforce','frictionforce'],
                  'volume variables': ['V','p','T','rho','walldist','mach','cp','eddy'],
                  'frequency' : 500,
                 },
'report' : {
           'frequency' : 10,
        'forces' : {
            'FR_1' : {
                'name' : 'wall',
                'zone' : [9,10,11,12,13],
                'transform' : my_transform,
                'reference area' : 594720.0*0.5, # half model area # half model area # half model area
            },
        },
},
}
"""
template = jinja2.Template(pyConf)
print template.render(alpha = argv[1], cycles = argv[2])

Then I amended my lambda function, to get this template, execute it, which prints the now rendered control file to stdio, where in the lambda function I have redirected the stdio for my exec function, where I can catch it and stream it into an S3 Object, which then gets pushed back to S3.

import boto3
import jinja2 
import sys
import StringIO
import contextlib

@contextlib.contextmanager
def stdoutIO(stdout=None):
    old = sys.stdout
    if stdout is None:
        stdout = StringIO.StringIO()
    sys.stdout = stdout
    yield stdout
    sys.stdout = old

s3 = boto3.resource('s3')

def main(json_input, context):
    client = boto3.client('s3')
    body = client.get_object(Bucket = "cfdworkflow2", Key='Control/control.py').get('Body').read()
    c_body = convertBody(body)
    s3.Object('cfdworkflow2', 'Control/parsedControl.py').put(Body=c_body)

def convertBody(body):
    sys.argv = ['','3','100']
    with stdoutIO() as s:
        exec(body)
    return s.getvalue()

Found the idea of redirecting the stdio here

To extend I'll pass through the stdin to the lambda function as the parameters to the jinja template.

Collectives™ on Stack Overflow

Edit parameters in a python file from another python file

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related