python: getting actual environment variables to modify and pass to subprocess

os.putenv() does not update os.environ as its docs say explicitly. C putenv() (in a CPython extension module) does not update os.environ too (as documented: changes in the environment after os import are not reflected in os.environ).

os.getenv(var) is just os.environ.get(var). There is related Python issue as @ShadowRanger has mentioned.

If you need it; you could access C environ from Python using ctypes e.g. (tested on Ubuntu, it might work on OS X (you might need to call _NSGetEnviron() there), it is unlikely to work on Windows (use _wenviron there)):

import ctypes

libc = ctypes.CDLL(None)
environ = ctypes.POINTER(ctypes.c_char_p).in_dll(libc, 'environ')

environ is a pointer to an array of C (NUL-terminated) strings (char*) where the last item is NULL. To enumerate values in Python 2:

for envvar in iter(iter(environ).next, None):
    print envvar

Output

LC_PAPER=en_GB.UTF-8
LC_ADDRESS=en_GB.UTF-8
CLUTTER_IM_MODULE=xim
LC_MONETARY=en_GB.UTF-8
VIRTUALENVWRAPPER_PROJECT_FILENAME=.project
SESSION=ubuntu
...

To get it as a dictionary that you could modify and pass to a child process:

env = dict(envvar.split(b'=', 1) for envvar in iter(iter(environ).next, None))

To synchronize with os.environ:

os.environ.clear() # NOTE: it clears C environ too!
getattr(os, 'environb', os.environ).update(env) # single source Python 2/3 compat.

Here're several convenience functions:

#!/usr/bin/env python
import ctypes
import functools
import os


_environ = None


def get_libc_environb_items():
    """Get envvars from C environ as bytestrings (unsplit on b'=')."""
    global _environ
    if _environ is None:
        libc = ctypes.CDLL(None)
        _environ = ctypes.POINTER(ctypes.c_char_p).in_dll(libc, 'environ')
    return iter(functools.partial(next, iter(_environ)), None)


def get_libc_environb():
    """Get a copy of C environ as a key,value mapping of bytestrings."""
    return dict(k_v.split(b'=', 1) for k_v in get_libc_environb_items()
                if b'=' in k_v)  # like CPython



def get_libc_environ():
    """Get a copy of C environ as a key,value mapping of strings."""
    environb = get_libc_environb()
    # XXX sys.getfilesystemencoding()+'surrogateescape'
    fsdecode = getattr(os, 'fsdecode', None)
    if fsdecode is None:  # Python 2
        return environb  # keep bytestrings as is (`str` type)
    else:  # Python 3, decode to Unicode
        return {fsdecode(k): fsdecode(v) for k, v in environb.items()}


def synchronize_environ():
    """Synchronize os.environ with C environ."""
    libc_environ = get_libc_environ()
    os.environ.clear()
    os.environ.update(libc_environ)


def test():
    assert 'SPAM' not in os.environ
    assert 'SPAM' not in get_libc_environ()
    os.putenv('SPAM', 'egg')
    assert 'SPAM' not in os.environ
    assert os.getenv('SPAM') is None
    assert get_libc_environ()['SPAM'] == 'egg'
    assert os.popen('echo $SPAM').read().strip() == 'egg'
    synchronize_environ()
    assert os.environ['SPAM'] == 'egg'


if __name__ == "__main__":
    test()
    from pprint import pprint
    pprint(get_libc_environ())

It works on CPython 2, CPython 3, pypy. It doesn't work on Jython.