ConfigParser and String interpolation with env variable

Question

it's a little bit I'm out of python syntax and I have a problem in reading a .ini file with interpolated values.

this is my ini file:

[DEFAULT]
home=$HOME
test_home=$home

[test]
test_1=$test_home/foo.csv
test_2=$test_home/bar.csv

Those lines

from ConfigParser import SafeConfigParser

parser = SafeConfigParser()
parser.read('config.ini')

print parser.get('test', 'test_1')

does output

$test_home/foo.csv

while I'm expecting

/Users/nkint/foo.csv

EDIT:

I supposed that the $ syntax was implicitly included in the so called string interpolation (referring to the manual):

On top of the core functionality, SafeConfigParser supports interpolation. This means values can contain format strings which refer to other values in the same section, or values in a special DEFAULT section.

But I'm wrong. How to handle this case?

Michele d'Amico · Accepted Answer · 2014-10-27 13:06:04Z

43

First of all according to the documentation you should use %(test_home)s to interpolate test_home. Moreover the key are case insensitive and you can't use both HOME and home keys. Finally you can use SafeConfigParser(os.environ) to take in account of you environment.

from ConfigParser import SafeConfigParser
import os


parser = SafeConfigParser(os.environ)
parser.read('config.ini')

Where config.ini is

[DEFAULT]
test_home=%(HOME)s

[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv

answered Oct 27, 2014 at 13:06

Michele d'Amico

24.1k10 gold badges78 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

nkint Over a year ago

I can't modify the ini syntax because other software already use it and it is with the $ syntax

Michele d'Amico Over a year ago

You should preprocess the file before use SafeConfigParse(). But you can just replace $ syntax by %()s syntax. the real problem will be home=$(HOME) that become recursive. You could assume that the capital words came from environ and replace by something like __ENV__KEY after change the sys.environ dict too. I can't do it now... but you can tray by yourself

Chris Over a year ago

The one thing that got me about this implementation is if there are '%' marks in environment variables themselves, then the parser will throw an error. I had to write a function that would filter out those keys/values. Otherwise, I like this impl.

Taylor D. Edmiston Over a year ago

I like the simplicity of this answer, but was concerned how it casually imports all environment variables instead of just the relevant ones, so I passed on only the desired subset to ConfigParser: {k: v for k, v in os.environ.items() if k in ('MY_ENV_1', 'MY_ENV_2')}.

iron9 Over a year ago

It should be noted that SafeConfigParser only exists in python 2. It is replaced by ConfigParser in python 3.

|

Alex Markov · Accepted Answer · 2020-06-19 07:53:27Z

15

You can write custom interpolation in case of Python 3:

import configparser
import os


class EnvInterpolation(configparser.BasicInterpolation):
    """Interpolation which expands environment variables in values."""

    def before_get(self, parser, section, option, value, defaults):
        value = super().before_get(parser, section, option, value, defaults)
        return os.path.expandvars(value)


cfg = """
[section1]
key = value
my_path = $PATH
"""

config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])

edited Jun 19, 2020 at 7:53

answered Mar 28, 2018 at 8:32

Alex Markov

3313 silver badges7 bronze badges

4 Comments

srand9 Over a year ago

Hey @Alex, I want to fetch Environment variables and values from other sections also. How can I do this? Extended Interpolation doesn't look straight forward.

Andrei Pozolotin Over a year ago

please add missing call to super().before_get(...)

Alex Markov Over a year ago

@srand9, I have fixed the example, now you can use %-notation to reference values from other sections of the same config. See, docs for more details

Vser Over a year ago

I've kind of jeopardized your answer to post my own answer addressing @srand9 concern (which is mine too). I hope you don't mind :)

Vser · Accepted Answer · 2021-06-21 13:35:42Z

5

Based on @alex-markov answer (and code) and @srand9 comment, the following solution works with environment variables and cross-section references.

Note that the interpolation is now based on ExtendedInterpolation to allow cross-sections references and on before_read instead of before_get.

#!/usr/bin/env python3
import configparser
import os


class EnvInterpolation(configparser.ExtendedInterpolation):
    """Interpolation which expands environment variables in values."""

    def before_read(self, parser, section, option, value):
        value = super().before_read(parser, section, option, value)
        return os.path.expandvars(value)


cfg = """
[paths]
foo : ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""

config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
print(config['section1']['my_path'])

answered Jun 21, 2021 at 13:35

Vser

8119 silver badges20 bronze badges

1 Comment

Brent Over a year ago

This works until you need to iterate over multiple items in a section, for k, v in config.items() it iterates and returns every environment variable set instead of the values in the ini

gagnonlg · Accepted Answer · 2016-05-24 13:58:53Z

4

If you want to expand some environment variables, you can do so using os.path.expandvars before parsing a StringIO stream:

import ConfigParser
import os
import StringIO

with open('config.ini', 'r') as cfg_file:
    cfg_txt = os.path.expandvars(cfg_file.read())

config = ConfigParser.ConfigParser()
config.readfp(StringIO.StringIO(cfg_txt))

edited May 24, 2016 at 13:58

answered May 23, 2016 at 19:00

gagnonlg

613 bronze badges

Comments

MrE · Accepted Answer · 2017-11-12 07:19:34Z

3

the trick for proper variable substitution from environment is to use the ${} syntax for the environment variables:

[DEFAULT]
test_home=${HOME}

[test]
test_1=%(test_home)s/foo.csv
test_2=%(test_home)s/bar.csv

answered Nov 12, 2017 at 7:19

MrE

21.2k15 gold badges92 silver badges113 bronze badges

1 Comment

Daniel Over a year ago

In this case you are forced to use the basic ConfigParser interpolation, right?

Mauro Baraldi · Accepted Answer · 2019-07-26 17:10:59Z

2

ConfigParser.get values are strings, even if you set values as integer or True. But ConfigParser has getint, getfloat and getboolean.

settings.ini

[default]
home=/home/user/app
tmp=%(home)s/tmp
log=%(home)s/log
sleep=10
debug=True

config reader

>>> from ConfigParser import SafeConfigParser
>>> parser = SafeConfigParser()
>>> parser.read('/home/user/app/settings.ini')
>>> parser.get('defaut', 'home')
'/home/user/app'
>>> parser.get('defaut', 'tmp')
'/home/user/app/tmp'
>>> parser.getint('defaut', 'sleep')
10
>>> parser.getboolean('defaut', 'debug')
True

Edit

Indeed you could get name values as environ var if you initialize SafeConfigParser with os.environ. Thanks for the Michele's answer.

edited Jul 26, 2019 at 17:10

answered Oct 27, 2014 at 13:16

Mauro Baraldi

6,6042 gold badges35 silver badges48 bronze badges

1 Comment

nkint Over a year ago

I can't modify the ini syntax because other software already use it and it is with the $ syntax

Imtiaz Shakil Siddique · Accepted Answer · 2022-08-06 06:29:24Z

Below is a simple solution that

Can use default value if no environment variable is provided
Overrides variables with environment variables (if found)
needs no custom interpolation implementation

Example: my_config.ini

[DEFAULT]
HOST=http://www.example.com
CONTEXT=${HOST}/auth/
token_url=${CONTEXT}/oauth2/token

ConfigParser:

import os
import configparser

config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
ini_file = os.path.join(os.path.dirname(__file__), 'my_config.ini')
    
# replace variables with environment variables(if exists) before loading ini file
with open(ini_file, 'r') as cfg_file:
    cfg_env_txt = os.path.expandvars(cfg_file.read())

config.read_string(cfg_env_txt)

print(config['DEFAULT']['token_url'])

Output:

If no environtment variable $HOST or $CONTEXT is present this config will take the default value
user can override the default value by creating $HOST, $CONTEXT environment variable
works well with docker container

vik_78 · Accepted Answer · 2019-04-19 15:39:37Z

Quite late, but maybe it can help someone else looking for the same answers that I had recently. Also, one of the comments was how to fetch Environment variables and values from other sections. Here is how I deal with both converting environment variables and multi-section tags when reading in from an INI file.

INI FILE:

[PKG]
# <VARIABLE_NAME>=<VAR/PATH>
PKG_TAG = Q1_RC1

[DELIVERY_DIRS]
# <DIR_VARIABLE>=<PATH>
NEW_DELIVERY_DIR=${DEL_PATH}\ProjectName_${PKG:PKG_TAG}_DELIVERY

Python Class that uses the ExtendedInterpolation so that you can use the ${PKG:PKG_TAG} type formatting. I add the ability to convert the windows environment vars when I read in INI to a string using the builtin os.path.expandvars() function such as ${DEL_PATH} above.

import os
from configparser import ConfigParser, ExtendedInterpolation

class ConfigParser(object):

    def __init__(self):
        """
        initialize the file parser with
        ExtendedInterpolation to use ${Section:option} format
        [Section]
        option=variable
        """
        self.config_parser = ConfigParser(interpolation=ExtendedInterpolation())

    def read_ini_file(self, file='./config.ini'):
        """
        Parses in the passed in INI file and converts any Windows environ vars.

        :param file: INI file to parse
        :return: void
        """
        # Expands Windows environment variable paths
        with open(file, 'r') as cfg_file:
            cfg_txt = os.path.expandvars(cfg_file.read())

        # Parses the expanded config string
        self.config_parser.read_string(cfg_txt)

    def get_config_items_by_section(self, section):
        """
        Retrieves the configurations for a particular section

        :param section: INI file section
        :return: a list of name, value pairs for the options in the section
        """
        return self.config_parser.items(section)

    def get_config_val(self, section, option):
        """
        Get an option value for the named section.

        :param section: INI section
        :param option: option tag for desired value
        :return: Value of option tag
        """
        return self.config_parser.get(section, option)

    @staticmethod
    def get_date():
        """
        Sets up a date formatted string.

        :return: Date string
        """
        return datetime.now().strftime("%Y%b%d")

    def prepend_date_to_var(self, sect, option):
        """
        Function that allows the ability to prepend a
        date to a section variable.

        :param sect: INI section to look for variable
        :param option: INI search variable under INI section
        :return: Void - Date is prepended to variable string in INI
        """
        if self.config_parser.get(sect, option):
            var = self.config_parser.get(sect, option)
            var_with_date = var + '_' + self.get_date()
            self.config_parser.set(sect, option, var_with_date)

Franzi · Accepted Answer · 2019-01-02 12:29:27Z

It seems in the last version 3.5.0, ConfigParser was not reading the env variables, so I end up providing a custom Interpolation based on the BasicInterpolation one.

class EnvInterpolation(BasicInterpolation):
    """Interpolation as implemented in the classic ConfigParser,
    plus it checks if the variable is provided as an environment one in uppercase.
    """

    def _interpolate_some(self, parser, option, accum, rest, section, map,
                          depth):
        rawval = parser.get(section, option, raw=True, fallback=rest)
        if depth > MAX_INTERPOLATION_DEPTH:
            raise InterpolationDepthError(option, section, rawval)
        while rest:
            p = rest.find("%")
            if p < 0:
                accum.append(rest)
                return
            if p > 0:
                accum.append(rest[:p])
                rest = rest[p:]
            # p is no longer used
            c = rest[1:2]
            if c == "%":
                accum.append("%")
                rest = rest[2:]
            elif c == "(":
                m = self._KEYCRE.match(rest)
                if m is None:
                    raise InterpolationSyntaxError(option, section,
                                                   "bad interpolation variable reference %r" % rest)
                var = parser.optionxform(m.group(1))
                rest = rest[m.end():]
                try:
                    v = os.environ.get(var.upper())
                    if v is None:
                        v = map[var]
                except KeyError:
                    raise InterpolationMissingOptionError(option, section, rawval, var) from None
                if "%" in v:
                    self._interpolate_some(parser, option, accum, v,
                                           section, map, depth + 1)
                else:
                    accum.append(v)
            else:
                raise InterpolationSyntaxError(
                    option, section,
                    "'%%' must be followed by '%%' or '(', "
                    "found: %r" % (rest,))

The difference between the BasicInterpolation and the EnvInterpolation is in:

   v = os.environ.get(var.upper())
   if v is None:
       v = map[var]

where I'm trying to find the var in the enviornment before checking in the map.

vschroeter · Accepted Answer · 2024-04-22 22:08:27Z

The extended-configparser package provides an EnvInterpolator class that does the job. It allows interpolation of environment variables as well as options from other sections (like the configparser.ExtendedInterpolation.

import configparser
from extended_configparser.interpolator import EnvInterpolation

parser = configparser.ConfigParser(interpolation=EnvInterpolation())
parser.read("config.cfg")

# Allow combining environment & section interpolation:
cfg = """
[paths]
foo = ${HOME}
[section1]
key = value
my_path = ${paths:foo}/path
"""

config.read_string(cfg)
print(config['section1']['my_path'])

In contrast to the interpolator from Vser's answer, it preserves the ability to get the raw value of an option.

import configparser
from extended_configparser.interpolator import EnvInterpolation

cfg = """
[paths]
foo = ${HOME}
"""

config = configparser.ConfigParser(interpolation=EnvInterpolation())
config.read_string(cfg)
config.get("paths", "foo", raw=True) == r"${HOME}"

Collectives™ on Stack Overflow

ConfigParser and String interpolation with env variable

EDIT:

10 Answers 10

7 Comments

4 Comments

1 Comment

Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

EDIT:

10 Answers 10

7 Comments

4 Comments

1 Comment

Comments

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related