Use base executable to set virtualenv Python path

[charliermarsh]

Summary

See extensive discussion in #8433 (comment).

This PR brings us into alignment with the standard library by using sys._base_executable rather than canonicalizing the executable path.

The benefits are primarily for Homebrew, where we'll now resolve to paths like /opt/homebrew/opt/[email protected]/bin instead of the undesirable /opt/homebrew/Cellar/[email protected]/3.9.19_1/Frameworks/Python.framework/Versions/3.9/bin.

Most other users should see no change, though in some cases, nested virtual environments now have slightly different behavior -- namely, they sometimes resolve to the virtual environment Python (at least for Homebrew; not for rtx or uv Pythons though). See here for a breakdown.

Closes #1640.
Closes #1795.

[charliermarsh]

Oh, I did find one Python that does not define sys._base_executable: /Library/Frameworks/Python.framework/Versions/3.7/bin/python3

[charliermarsh]

CPython errors if sys._base_executable is undefined, at least on latest. But we have to support creating environments from other Pythons. I found that in 3.7 at least, they used this logic.

[konstin]

i checked and it's also set in pypy3.10 on linux

[charliermarsh]

Okay, it looks like we have the ~same problem with our test setup: we create symlinks to Pythons, but they lose the connection to the base interpreter.

[ofek]

What do you think about flag(s) to modify behavior to account for situations that are broken by the default implementation?

[charliermarsh]

What is an example of a real situation that is broken by this implementation?

[charliermarsh]

This is my attempt to solve the problem demonstrated by our test suite. We symlink from the installed Python directory to (e.g.) /Users/crmarsh/.local/share/uv/tests/.tmpIl4Pjf/python/3.12/python3. In python-build-standalone, sys._base_executable for that symlink just points to the current Python (i.e., it points to /Users/crmarsh/.local/share/uv/tests/.tmpIl4Pjf/python/3.12/python3). The trick I'm employing here is that we resolve symlinks as long as the binary is not in the scripts path.

[konstin]

Does this only occur with python build standalone? In this case, we should patch this in pbs

[charliermarsh]

Patch it to what?

[konstin]

i think we have to fix astral-sh/python-build-standalone#380, or at least i expect it's closely related.

[charliermarsh]

What do you think of the solution itself? Alternatively, we could determine whether this is a uv-managed Python, and resolve the symlink if so. (But we might then fail if users install PBS builds through some other mechanism and symlink those.)

[konstin]

(replied in the main thread)

[konstin]

I can't test the homebrew situation, though unlike #8433, it doesn't solve my pyenv use case:

I have a /home/konsti/.local/bin/python3.11 that is symlinked to a pyenv 3.11 in /home/konsti/.pyenv/versions/3.11.7/bin (pyenv tags folders with the patch version, you can install multiple patch releases at the same time). It would be nice if I could change the symlink when a new patch version of 3.11 is released, without breaking all existing venvs. With this branch, we resolve through the symlink:

$ uv-branch venv -p /home/konsti/.local/bin/python3.11 branch
$ uv-main venv -p /home/konsti/.local/bin/python3.11 main
$ ls -lah main/bin/python
lrwxrwxrwx 1 konsti konsti 50 Okt 24 11:45 main/bin/python -> /home/konsti/.pyenv/versions/3.11.7/bin/python3.11
$ ls -lah branch/bin/python
lrwxrwxrwx 1 konsti konsti 46 Okt 24 11:45 branch/bin/python -> /home/konsti/.pyenv/versions/3.11.7/bin/python

This is not a blocker if it solves the homebrew situation more similar to std than #8433, I just want to note this drawback here.

[zanieb]

Similarly, this does not solve the version stability goals for our managed Python installations as mentioned in #8433 (comment) and as implemented in #8458. As @konstin concludes, I think this is okay since we're aligning with CPython, but it'd be nice to have some plan to address this. It's possible the solution is "create something that looks like Homebrew instead", where you have a whole lib directory structure for minor Python versions. I'm not excited about the additional indirection, but it does seem roughly more correct. We may be able to encourage pyenv to move in the same direction? Perhaps we should see if they have something to say about this?

[zanieb]

I guess an open question for me is if we want to provide a setting for this, i.e., UV_VENV_PYTHON_LINK = resolve | base | no-resolve (don't focus on the names here), so people can recover the existing behavior if we break their setup (and pyenv users can get the behavior they need immediately)

[charliermarsh]

Critically, though, it would be actively wrong to use /home/konsti/.local/bin as the virtual environment home. The same is true for the managed Python design, right? That's not a valid Python installation. It might work, but it also might not, and it could do things we don't expect or can't test (I just don't know how home is used precisely). Homebrew is actually doing something fairly different whereby it's setting sys._base_executable to a stable symlink to a valid Python installation. I think that problem needs to be solved in the Python distributions and layouts, and not in uv...

(I know you both know some of this, just repeating it for clarity.)

[zanieb]

Critically, though, it would be actively wrong to use /home/konsti/.local/bin as the virtual environment home. The same is true for the managed Python design, right? That's not a valid Python installation.

Totally agree. I wonder if we can just omit home entirely in that case? I mean, since it's not valid it basically has no affect and falls back to information present in the Python installation. I tried removing it from a Homebrew virtual environment's pyvenv.cfg and nothing broke.

Switching to a "base executable" strategy by default (and presuming home is always correct) but not writing home for an alternative strategy where we don't follow links might be reasonable? Of course, this doesn't work for our managed distributions (probably because of #8429) but I think that's a separate problem we can address.

[charliermarsh]

Yeah agreed. I think we need to learn more about how home works and the intended design (and see if we can fix the managed distribution behavior upstream). At least this change here brings us into alignment with the standard library by default though, rather than further from it, which seems good.

[zanieb]

Yeah all for changing the default. I'm also leaning further towards the idea that we should support configuring the strategy as an escape hatch for workflows this change breaks though; even if we only support this new mode (base-executable) and our existing behavior (resolve) to start.

[charliermarsh]

I asked in astral-sh/python-build-standalone#382 if there's a way we can reliably detect relocatable Pythons.

[charliermarsh]

Ok, to workaround the python-build-standalone bug, I'm now doing:

1. For python-build-standalone (using the heuristic from the linked issue), resolve all symlinks.
2. Follow CPython for all other interpreters.

The only risk here is that the heuristic could have false positives (interpreters with a legitimate prefix of /install), but I guess we just accept that. Even in that case, the strategy is still ~fine -- you just get the behavior we have today on main.

[konstin]

Here's how sys._base_executable and pyvenv.cfg's home look on my ubuntu 24.04 for a system interpreter, a pyenv interpreter and a python build standalone.

Basic usage, no symlinks

System interpreter

$ /usr/bin/python3 -c "import sys; print(sys._base_executable)"
/usr/bin/python3
$ rm -rf venv && /usr/bin/python3 -m venv venv && cat venv/pyvenv.cfg | grep "home "
home = /usr/bin

Pyenv interpreter

$ /home/konsti/.pyenv/versions/3.11.7/bin/python3.11 -c "import sys; print(sys._base_executable)"
/home/konsti/.pyenv/versions/3.11.7/bin/python3.11
$ rm -rf venv && /home/konsti/.pyenv/versions/3.11.7/bin/python3.11 -m venv venv && cat venv/pyvenv.cfg | grep "home "
home = /home/konsti/.pyenv/versions/3.11.7/bin

Python build standalone

$ /home/konsti/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/bin/python3 -c "import sys; print(sys._base_executable)"
/home/konsti/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/bin/python3
$ rm -rf venv && /home/konsti/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/bin/python3 -m venv venv && cat venv/pyvenv.cfg | grep "home "
home = /home/konsti/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/bin

Symlinked interpreters

ln -s /usr/bin/python3 python-system
ln -s /home/konsti/.pyenv/versions/3.11.7/bin/python3.11 python-pyenv
ln -s /home/konsti/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu/bin/python3 python-pbs

$ ./python-system -c "import sys; print(sys._base_executable)"
/home/konsti/projects/a/python-system
$ ./python-pyenv -c "import sys; print(sys._base_executable)"
/home/konsti/projects/a/python-pyenv
$ ./python-pbs -c "import sys; print(sys._base_executable)"
/home/konsti/projects/a/python-pbs

venvs from symlinked interpreters

$ rm -rf venv && ./python-system -m venv --without-pip venv && cat venv/pyvenv.cfg | grep "home "
home = /home/konsti/projects/a
$ venv/bin/python -c "import sys; print(sys._base_executable)"
/usr/bin/python3.12
$ rm -rf venv && ./python-pyenv -m venv --without-pip venv && cat venv/pyvenv.cfg | grep "home "
home = /home/konsti/projects/a
$ venv/bin/python -c "import sys; print(sys._base_executable)"
/home/konsti/.pyenv/versions/3.11.7/bin/python3.11
$ rm -rf venv && ./python-pbs -m venv --without-pip venv && cat venv/pyvenv.cfg | grep "home "
home = /home/konsti/projects/a
$ venv/bin/python -c "import sys; print(sys._base_executable)"
[ERROR: ModuleNotFoundError: No module named 'encodings']

[charliermarsh]

I'm just concerned that users will use such a setting without understanding all the nuances and break their own setups or do things that are slightly and silently wrong. The tradeoffs are really complicated, and any bugs that emerge as a result of this change are in line with the standard library and bugs in those Python distributions. I understand the push though... I can add it as an environment variable, as an escape hatch.

[charliermarsh]

We refined the behavior a bit in #9723.