Support PyTorch CPU and GPU dependency #9734
Comments
|
Thank you @harteros for the pointer :) |
BurntSushi
added a commit
that referenced
this issue
Dec 10, 2024
This PR adds a notion of "conflict markers" to the lock file as an attempt to address #9289. The idea is to encode a new kind of boolean expression indicating how to choose dependencies based on which extras are activated. As an example of what conflict markers look like, consider one of the cases brought up in #9289, where `anyio` had unconditional dependencies on two different versions of `idna`. Now, those are gated by markers, like this: ```toml [[package]] name = "anyio" version = "4.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "idna", version = "3.5", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-project-foo'" }, { name = "idna", version = "3.6", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-7-project-bar' or extra != 'extra-7-project-foo'" }, { name = "sniffio" }, ] ``` The odd extra values like `extra-7-project-foo` are an encoding of not just the conflicting extra (`foo`) but also the package it's declared for (`project`). We need both bits of information because different packages may have the same extra name, even if they are completely unrelated. The `extra-` part is a prefix to distinguish it from groups (which, in this case, would be encoded as `group-7-project-foo` if `foo` were a dependency group). And the `7` part indicates the length of the package name which makes it possible to parse out the package and extra name from this encoding. (We don't actually utilize that property, but it seems like good sense to do it in case we do need to extra information from these markers.) While this preserves PEP 508 compatibility at a surface level, it does require utilizing this encoding scheme in order to evaluate them when they're present (which only occurs when conflicting extras/groups are declared). My sense is that the most complex part of this change is not just adding conflict markers, but their simplification. I tried to address this in the code comments and commit messages. Reviewers should look at this commit-by-commit. Fixes #9289, Fixes #9546, Fixes #9640, Fixes #9622, Fixes #9498, Fixes #9701, Fixes #9734
|
@harteros Is this solved? Cuz i'm having the same problems with it. Here is what I had to put in the configuration: Every torch dependent package is now being listed under certain extras to make sure we don't install CUDA libs on cpu env. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I followed the guide for supporting PyTorch CPU and GPU dependency. It works well until I added a package
acceleratethat depends on torch, then this package would depend on both version and bothcpuandgpuversion are installed.pyproject.tomluv treeWe found a hack to pretend that there are 2 distinct packages
acceleratefor CPU and GPU by defining a separatte (but identic source):Is it the expected behavior of the resolver? Or are we handling something incorrectly?
Thanks a lot.
The text was updated successfully, but these errors were encountered: