Very large lockfile and clean up

[jbcdnr]

We noticed that our uv.lock file grows in size, up to 2.5MB and 35k lines with a lot of resolution-markers and the time to uv lock explodes to more than 30 min and sometimes OOM. We have a pretty bad hygien for our pyproject.toml with a lot of * versions and some conflicts so the resolver might have a hard time.

We came up with a simple script to extract all the versions from the lockfile and copy paste them into pyproject.toml, then rerun lock and revert pyproject right after. This speeds up the follow up lock calls and make the lock file only 0.4MB big. Is it expected behavior? Is our cleanup script actually something that exists as part of the uv command?

import pathlib
import re

content = pathlib.Path("uv.lock").read_text()

print("dependencies = [")
for package_version_match in re.finditer(
    r'name = "([^"]+)"\nversion = "([\d\.]+)"', content
):
    package_name, package_version = package_version_match.groups()
    print(f'    "{package_name}=={package_version}",')
print("]")

Thank you for your help.

[konstin]

It's hard to say without the input pyproject.toml; A lot of resolution-markers sounds like there's a lot of resolver forking that resolves multiple versions that you don't need. One option would be pinning those packages that appear multiple times in the lockfile (to coerce them into a single version) or to set tool.uv.environments.

[jbcdnr]

Unfortunately I cannot share the whole pyproject.toml. Thanks for the pointer at environments, most of the lockfile resolution-markers entries were for darwin/linux, so resolving only for linux should avoid that this happens again.

I am still curious if there could be a way to clean up the lockfile like we did but in uv.

[charliermarsh]

Does resolution-markers contain duplicates?

[charliermarsh]

My guess is that resolution-markers somehow grows exponentially over time... Are you able to share, like, a subset of the pyproject.toml?

[charliermarsh]

Separately, do you have any conflicts defined in your pyproject.toml?

[jbcdnr]

Does resolution-markers contain duplicates?

Yes, it was containing many lines like

resolution-markers = [
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform == 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform != 'darwin' and sys_platform != 'linux'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
    "sys_platform == 'darwin'",
...

[jbcdnr]

Separately, do you have any conflicts defined in your pyproject.toml?

Yes, we have a conflict defined between two extra, torch-cpu and torch-gpu that follow the PyTorch guide. We also had a hack for to support PyTorch dependency #9734 but this is fixed in 0.5.8.

[jbcdnr]

My guess is that resolution-markers somehow grows exponentially over time... Are you able to share, like, a subset of the pyproject.toml?

Unfortunately I cannot, but I can run what you want on our project. Even though I am not sure I can reproduce the growing lock file again, I would have to add some random package to force resolution a couple of times.

[jbcdnr]

@charliermarsh it looks like bumping to 0.5.8 and deleting our hack for accelerate #9734 also simplifies the resolution markers in our uv.lock file. So maybe this issue is non reproducible at main.