Can I replace %USERPROFILE% and still get KNOWNFOLDERIDs from the registry? - windows

Question

We're developing an open source Python library that runs on Linux, MacOS, and Windows, but we don't have much experience or exposure to Windows in the developer team. The way we setup and run our test suite works fine under Linux and Mac, but is suboptimal on Windows.
Our tests set up a new directory in a temporary location, place a fake .gitconfig with relevant configurations inside it, and have the relevant HOME environment variables point to this location as the home directory in order to pick up the configurations during testing.
The code is shortened and can't be run, but hopefully illustrates the gist of what we do:
with make_tempfile(mkdir=True) as new_home:
pass
for v, val in get_home_envvars(new_home).items():
set_envvar(v, val)
if not os.path.exists(new_home):
os.makedirs(new_home)
with open(os.path.join(new_home, '.gitconfig'), 'w') as f:
f.write("""\
[user]
name = Tester
email = test#example.com
[more configs for testing]
exc = 1
""")
where get_home_envvars() makes sure that the $HOME env variable points to the new, temporary test home. On Windows since Python 3.8, os.path no longer queried the $HOME variable to determine a user's home, but USERPROFILE[1 ][2], so we've just overwritten this variable with the temporary test home:
def get_home_envvars(new_home):
environ = os.environ
out = {'HOME': new_home}
if on_windows:
# requires special handling, since it has a number of relevant variables
# and also Python changed its behavior and started to respect USERPROFILE only
# since python 3.8: https://bugs.python.org/issue36264
out['USERPROFILE'] = new_home
out['HOMEDRIVE'], out['HOMEPATH'] = splitdrive(new_home)
return {v: val for v, val in out.items() if v in os.environ}
However, we have now discovered that this breaks our test setup on Windows, with tests "bleeding" their caches, cookie data bases etc. into the places where we perform our unit tests, and with this creating files and directories that break our test assumptions.
I have a very limited understanding on what happens exactly, but my current hypothesis is this: Our library determines the appropriate locations for caches, logs, cookies, etc upon start by using appdirs [3], which does so by querying the "special folder" IDs/ CSIDLs that Windows has [4]. This information is determined in the Windows registry - which is found based on the USERPROFILE. To quote one specific reply in the Python bug tracker to this change:
This is unfortunate. Modifying USERPROFILE is highly unusual. USERPROFILE is the location of the user's "NTUSER.DAT" registry hive and local application data ("AppData\Local"), including "UsrClass.dat" (the "Software\Classes" registry hive). It's also the default location for a user's known shell folders and home directory. Modifying USERPROFILE shouldn't cause problems with any of this, but I'm not completely at ease with it.
After our testsuite setup is done, we start new processes that run our tests. The new processes only get to see the new USERPROFILE, and appdirs returns the paths it finds by sending them through normpath, which unfortunately interprets the empty string returned by _get_win_folder for a CSIDL that now can't be found anymore as a relative path (.):
# snippet from appdirs source code
path = os.path.normpath(_get_win_folder("CSIDL_COMMON_APPDATA"))
And based on this, we end up configuring the current working directory of each test as the place for user data, user caches, etc.
My question is: How could I fix this? Based on my probably incomplete understanding, I currently think it ultimately boils down to the question how to treat or mock the USERPROFILE. I need to have it pointed to a registry in order to derive the "special folder" IDs (be it with appdirs or more modern replacements of it) - but I also need it to point to the fake home with test-specific Git configurations. I believe the latter requires overwriting USERPROFILE in Python3.8 and newer. I'm wondering if there is a way to copy or mock the registry and place it under the new home? Set relevant CSIDLs/KNOWNFOLDERIDs in some other way? Hardcode other temporary locations to use as cache directories etc? Or maybe there is a more clever way to run a test suite under Windows that does not require a fake home?
I would be very grateful to learn from more experienced Windows developers what to do, or also what not to do. Many thanks in advance.
[1] https://docs.python.org/3.11/library/os.path.html#os.path.expanduser
[2] https://bugs.python.org/issue36264
[3] https://github.com/ActiveState/appdirs
[4] https://learn.microsoft.com/en-us/windows/win32/shell/csidl