Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explaining odd behavior with dependency locking and installation. #9701

Closed
NellyWhads opened this issue Dec 7, 2024 · 4 comments · Fixed by #9370
Closed

Explaining odd behavior with dependency locking and installation. #9701

NellyWhads opened this issue Dec 7, 2024 · 4 comments · Fixed by #9370

Comments

@NellyWhads
Copy link

When using the following pyproject.toml, I get some weird behavior:

pyptoject.toml

[project]
name = "dep-test"
version = "0.1.0"
requires-python = ">=3.8,<3.13"

[tool.uv]
environments = ["platform_system == 'Linux'", "platform_system == 'Darwin'"]
# Conflicting groups specified at the project level
conflicts = [[{ group = "mmcv21_torch21" }, { group = "mmcv22_torch24" }]]
# Repo-wide constraints
constraint-dependencies = [
  "torch>=2.1,<2.5",
  "torchvision>=0.16,<0.20",
  "mmcv>=2.1,<2.3",
]

[dependency-groups]
my_group = ["torch", "torchvision"]
mmcv21_torch21 = [
  "mmcv~=2.1.0 ; platform_system == 'Linux' and python_version >= '3.8' and python_version < '3.12'",
  # Specific dependencies which are not encoded in the wheel
  "torch~=2.1.0",
  "torchvision~=0.16.0",
  "setuptools",
]
mmcv22_torch24 = [
  "mmcv~=2.2.0 ; platform_system == 'Linux' and python_version >= '3.8' and python_version < '3.13'",
  # Specific dependencies which are not encoded in the wheel
  "torch~=2.4.0",
  "torchvision~=0.19.0",
  "setuptools",
]

[tool.uv.sources]
torch = [{ index = "torch-cu118", marker = "platform_system == 'Linux'" }]
torchvision = [{ index = "torch-cu118", marker = "platform_system == 'Linux'" }]
mmcv = [
  # mmcv 2.1.0, torch 2.1.x
  { group = "mmcv21_torch21", marker = "platform_system == 'Linux' and python_version >= '3.8' and python_version < '3.9'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.1.0/mmcv-2.1.0-cp38-cp38-manylinux1_x86_64.whl" },
  { group = "mmcv21_torch21", marker = "platform_system == 'Linux' and python_version >= '3.9' and python_version < '3.10'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.1.0/mmcv-2.1.0-cp39-cp39-manylinux1_x86_64.whl" },
  { group = "mmcv21_torch21", marker = "platform_system == 'Linux' and python_version >= '3.10' and python_version < '3.11'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.1.0/mmcv-2.1.0-cp310-cp310-manylinux1_x86_64.whl" },
  { group = "mmcv21_torch21", marker = "platform_system == 'Linux' and python_version >= '3.11' and python_version < '3.12'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.1.0/mmcv-2.1.0-cp311-cp311-manylinux1_x86_64.whl" },
  # mmcv 2.2.0, torch 2.4.x
  { group = "mmcv22_torch24", marker = "platform_system == 'Linux' and python_version >= '3.8' and python_version < '3.9'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.4.0/mmcv-2.2.0-cp38-cp38-manylinux1_x86_64.whl" },
  { group = "mmcv22_torch24", marker = "platform_system == 'Linux' and python_version >= '3.9' and python_version < '3.10'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.4.0/mmcv-2.2.0-cp39-cp39-manylinux1_x86_64.whl" },
  { group = "mmcv22_torch24", marker = "platform_system == 'Linux' and python_version >= '3.10' and python_version < '3.11'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.4.0/mmcv-2.2.0-cp310-cp310-manylinux1_x86_64.whl" },
  { group = "mmcv22_torch24", marker = "platform_system == 'Linux' and python_version >= '3.11' and python_version < '3.12'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.4.0/mmcv-2.2.0-cp311-cp311-manylinux1_x86_64.whl" },
  { group = "mmcv22_torch24", marker = "platform_system == 'Linux' and python_version >= '3.12' and python_version < '3.13'", url = "https://download.openmmlab.com/mmcv/dist/cu118/torch2.4.0/mmcv-2.2.0-cp312-cp312-manylinux1_x86_64.whl" },
]

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

To verify everything is as expected, I run uv sync --group my_group && uv pip show torch torchvision

I'm befuddled with the outcome.

Installed versions on MacOS

Name: torch
Version: 2.1.2
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: torchvision
---
Name: torch
Version: 2.4.1
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: torchvision
---
Name: torchvision
Version: 0.16.2
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: numpy, pillow, torch
Required-by:
---
Name: torchvision
Version: 0.19.1
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: numpy, pillow, torch
Required-by:

  • Why (and how) did uv install 2 versions of torch and torchvision on MacOS? Or is this just a readout from the lock file?
    • In the .venv site-packages, I only see one installation of each package (the newer versions).
    • My intuition tells me uv pip show is both parsing the lock file for versions and also reading from the environment, and somehow getting confused.
    • Running the show without installing / after cleaning up the venv yields only the observed, installed versions.
  • I can solve this behavior by specifying the Linux platform for torch and torchvision in the mmcv* groups, but I don't yet understand why it's necessary - should it be?

I understand I'm not using the exact prescribed installation setup described in the docs, but I'm trying to understand the tool's behavior instead of just hard-coding a solution.

@charliermarsh
Copy link
Member

I think you're running into a variety of bugs that are fixed by #9370. Check out the linked issues there.

@NellyWhads
Copy link
Author

NellyWhads commented Dec 7, 2024

Is there a quick way to install uv directly from that branch to test it?

Also, are you an agentic LLM? Or are your fingers just crazy fast?

@zanieb
Copy link
Member

zanieb commented Dec 7, 2024

There are a limited set of binary artifacts attached to the CI run for the commit https://github.com/astral-sh/uv/actions/runs/12204703226?pr=9370 (scroll to the bottom) — you can download one of those if you don't want to build from source.

We're all robots

@NellyWhads
Copy link
Author

You've done it again

$ ~/Downloads/uv sync --group my_group && ~/Downloads/uv pip show torch torchvision 
Using CPython 3.10.15
Creating virtual environment at: .venv
⠙ Resolving dependencies...                                                                                                                    warning: Missing version constraint (e.g., a lower bound) for `setuptools`
warning: Missing version constraint (e.g., a lower bound) for `torch`
warning: Missing version constraint (e.g., a lower bound) for `torchvision`
Resolved 73 packages in 1.56s
Installed 12 packages in 256ms
 + filelock==3.16.1
 + fsspec==2024.10.0
 + jinja2==3.1.4
 + markupsafe==2.1.5
 + mpmath==1.3.0
 + networkx==3.1
 + numpy==1.24.4
 + pillow==10.4.0
 + sympy==1.13.3
 + torch==2.4.1
 + torchvision==0.19.1
 + typing-extensions==4.12.2
Name: torch
Version: 2.4.1
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: torchvision
---
Name: torchvision
Version: 0.19.1
Location: /Users/neil.wadhvana/workspaces/main/torc-robotics/pytorc/projects/dep_test/.venv/lib/python3.10/site-packages
Requires: numpy, pillow, torch
Required-by:

Teach me your ways. The world is moving too slowly for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants