load variables from in memory setting file

Question

i have settings.py file that looks like:

DEBUG = True
CACHE_IP = '0.0.0.0'
DB_SETTINGS = {
    'SERVERS': ['127.0.0.1:80'],
    'MAX_IDLE_TIME' : 180,
}

this file is stored at external cloud storage.

I have a process the get the url to this file, and load it into memory (not to disk). when loading it into memory it will be string type and looks like:

"DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    'SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"

How for example i can get the values from this string? i mean, how can i get a dictionary where key=verb name from file, and value = value from file? is there any pythonic way for this?

so my dictionary will look like:

{'DEBUG': True, 'CACHE_IP' = '0.0.0.0', 'DB_SETTINGS': {...}}

I would suggest storing your settings on the cloud as json data so you can use python's json parsers to get the data in a dict format. — MCBama
– MCBama, Commented Dec 6, 2017 at 18:13
i cant. this file is stored as settings.py. this is settings file for another python app — Eyal Ch
– Eyal Ch, Commented Dec 6, 2017 at 18:16
I'll second martineau. The string you provide is rather difficult to take apart because of there is no common format you can use to pick everything apart. If you're able to do so you might also consider changing how the source file is formatted. (e.g. with easy-to-parse key = val [/n]) — ikom
– ikom, Commented Dec 6, 2017 at 18:57
If you can't convert it to json then you should be able to load the entire file into memory as a file object instead of a string and then iterate over the rows to build a dictionary using urllib like in this question. the requests library should be able to do it too. — MCBama
– MCBama, Commented Dec 6, 2017 at 19:32

Amit Tripathi · Accepted Answer · 2017-12-07 16:54:38Z

Your requirement is very specific but this kind of requirements do arise. So, I tried to solve this using method below:

In [102]: import ast # Do not use eval(its unsafe)

In [103]: string = "DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    '
     ...: SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"
     ...:

In [104]: def parse_key_val(string):
     ...:     tmp_dict = {}
     ...:     for r in re.split('(?<![{},])\n (?!{)', string):
     ...:         key, val = r.split(" = ")
     ...:         if '{' in val:
     ...:             val = ast.literal_eval(val)
     ...:         tmp_dict[key] = val
     ...:     return tmp_dict
     ...:

In [105]: parse_key_val(string)
Out[105]:
{'CACHE_IP': "'0.0.0.0'",
 'DB_SETTINGS': {'MAX_IDLE_TIME': 180, 'SERVERS': ['127.0.0.1:80']},
 'DEBUG': 'True'}

This solution works great for the string you provided but it might(or not) fail in some complex cases. But this solution gives a good starting point of how to start. You might need to make some modifications to make it work for complex cases.

Now coming to the program, the important parts here to understand are:

re.split('(?<![{},])\n (?!{)', string)

and

val = ast.literal_eval(val)

The pattern (?<![{},])\n (?!{) in re.split means, split the string on character \n but only if it is not after { or } or , and not before {. This regex is created to not split \n line character in dict strings.

Not the second part val = ast.literal_eval(val) builds on top of the above regex step as above step doesn't break the dict string. The resulting dict part of string after split is:

"DB_SETTINGS = {\n    'SERVERS': ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n"

With ast.literal_eval the task of parsing dict, nested dict and list inside it very easy. That it!

One more thing, do not use eval as one of the solution suggests here, it's a security risk.

You can read more about ast here

Ram · Accepted Answer · 2017-12-06 18:21:03Z

1

One way is to use the eval function to execute the text.

eval("DEBUG = True\n CACHE_IP = '0.0.0.0'\n DB_SETTINGS = {\n    'SERVERS':       ['127.0.0.1:80'],\n    'MAX_IDLE_TIME' : 180,\n}\n")
settings_dict = {}
settings_dict['DEBUG'] = DEBUG
settings_dict['CACHE_IP'] = DEBUG
settings_dict['DB_SETTINGS'] = DB_SETTINGS

I don't know of a way to 'automate' the dictionary creation.

answered Dec 6, 2017 at 18:21

Ram

513 bronze badges

4 Comments

Ben Over a year ago

Note that eval just executes code, so if someone puts import shutil; shutil.rmtree('/home/username') in there, you've just wiped your home directory! eval is a security risk- please be very careful when using it!

Amit Tripathi Over a year ago

eval won't work with the above string. You will get DEBUG = True SyntaxError: invalid syntax

Eyal Ch Over a year ago

its not safe to use eval

Ram Over a year ago

oops, eval only evaluates expressions like the dictionary string. As such import statements don't work either. There could still be some creative security compromise. The accepted answer is a better solution.

Collectives™ on Stack Overflow

load variables from in memory setting file

2 Answers 2

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related