This is a self-answered question. I've found that there aren't any good examples of this specific situation, and the seemingly related questions don't address this use case.
My C++ application previously embedded Python 2.7 using a virtual environment. This was done using Py_SetPythonHome(venv_path) followed by Py_Initialize(). Currently, I am migrating the app to Python 3. Python 3.8 introduces the new Python Initialization Configuration API, which is now the preferred method of initialization. Additionally, it introduces the concept of an "isolated" initialization, which I would like to use here. However, when I try to use this API in a similar way to the Python 2.7 initialization by setting config.home, I get an initialization error that suggest that the base Python libraries could not be found.
My virtual environment is initialized as follows:
py -3.10 -m venv C:\path\to\venv
When I execute the following code:
PyConfig config;
PyConfig_InitIsolatedConfig(&config);
auto venv_path = L"C:\\path\\to\\venv";
PyConfig_SetString(&config, &config.home, venv_path);
status = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);
if (PyStatus_Exception(status)) {
std::cout << "status.func: " << (status.func ? status.func : "N/A") << '\n';
std::cout << "status.err_msg: " << (status.err_msg ? status.err_msg : "N/A") << '\n';
}
I get the following output and error message:
Python path configuration:
PYTHONHOME = 'C:\path\to\venv'
PYTHONPATH = (not set)
program name = 'python'
isolated = 1
environment = 0
user site = 0
import site = 1
sys._base_executable = 'C:\\demo\\build\\PythonDemo.exe'
sys.base_prefix = 'C:\\path\\to\\venv'
sys.base_exec_prefix = 'C:\\path\\to\\venv'
sys.platlibdir = 'lib'
sys.executable = 'C:\\demo\\build\\PythonDemo.exe'
sys.prefix = 'C:\\path\\to\\venv'
sys.exec_prefix = 'C:\\path\\to\\venv'
sys.path = [
'C:\\Python310\\python310.zip',
'C:\\path\\to\\venv\\DLLs',
'C:\\path\\to\\venv\\lib',
'C:\\demo\\build',
]
status.func: init_fs_encoding
status.err_msg: failed to get the Python codec of the filesystem encoding
The initialization configuration documentation specifically mentions "The following configuration files are used by the path configuration: pyvenv.cfg [...]", suggesting that virtual environments should be properly handled during initialization. However, it doesn't seem like the initialization is finding that file based on only setting config.home, and assumes that C:\path\to\venv is a complete Python installation rather than a virtual environment.
I've found that manually setting base_prefix and base_exec_prefix to C:\Python310 (partially) resolves the issue. However, I do not want to hardcode the path to Python in my application, as the app's users may have installed Python somewhere else. Besides, the home path provided in pyvenv.cfg should be automatically used for these.
How do I get Py_InitializeFromConfig to properly handle my virtual environment?
