Support Poetry v2 dependency format

[ryancausey]

Tell us more.

Poetry v2 has updated their pyproject.toml specification to support the "project" section in the pyproject.toml specification.

This means that there are now more potential locations where dependencies can be defined within a pyproject.toml file:

1. project.dependencies
2. project.optional-dependencies
3. tool.poetry.dependencies
4. tool.poetry.group.< name >.dependencies

The other rub is that the project.dependencies list doesn't support all of the dependency specifiers that one of the dependency locations under too.poetry does: https://python-poetry.org/docs/dependency-specification/

It would be nice to have renovate support this new dependency location and formats under the project heading.

Minimal reproduction: https://github.com/ryancausey/poetry-v2-support-reproduction

[yann-combarnous]

Yep, PEP 621 support for Poetry 2 would be nice.

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team

[ryancausey]

The minimal reproduction has been added.

[uli-f]

I use the renovate app in GitHub for a repository with a Python application whose dependencies I manage with poetry. This worked just brilliantly - until I updated to poetry v2.0.0. Here is what I observe:

poetry.lock written with poetry version	PR opened by renovate changes pyproject.toml	PR opened by renovate changes poetry.lock
<v2.0.0	✅	✅
>=v2.0.0	✅	❌

This essentially means that renovate stops working for me for any Python repository as soon as I update to poetry >=v2.0.0.

I'd appreciate it if support for poetry.lock files written with poetry >=v2.0.0 could be added to renovate.

[ryancausey]

If you look at the minimal reproduction I linked in the original issue description, that doesn't seem to be the issue here. Renovate will normally leave an artifact generation failed message if it runs into issues while generating artifacts, such as the lock file, while updating a dependency. The reproduction uses the new poetry v2 lock file format, there is no error message about failing to generate artifacts, and the lock file is still not updated.

Normally I've only seen renovate fail to generate the lock file when there is some kind of dependency requirement conflict that has to be resolved manually. In those cases it leaves me a message on the PR that the artifact generation failed.

[ryancausey]

I should clarify: I don't know if there's a new "v2" lock file format. It seems like running poetry lock will fix whatever differences exist, so there shouldn't be any v1 vs v2 lock file issues when using the tool directly assuming the pyproject.toml is correctly formatted. I just thought I recalled renovate trying to not use the tools directly to avoid needing extra dependencies installed alongside the renovate bot.

[uli-f]

@rarkins Thank you for guiding me along. I checked the logs of a (1) successful renovate PR (prior to poetry.lock being written by poetry v2) and (2) an unsuccessful renovate PR (after poetry.lock being written by poetry v2).

(1):

DEBUG: Starting search at index 667 (branch="renovate/rdkit-2024.x")
{
  "packageFile": "pyproject.toml"
  "depName": "rdkit"
}

DEBUG: Found match at index 667 (branch="renovate/rdkit-2024.x")
{
  "packageFile": "pyproject.toml"
  "depName": "rdkit"
}

DEBUG: Contents updated (branch="renovate/rdkit-2024.x")
{
  "packageFile": "pyproject.toml"
  "depName": "rdkit"
}

DEBUG: updateArtifacts for updatedPackageFiles (branch="renovate/rdkit-2024.x")
DEBUG: poetry.updateArtifacts(pyproject.toml) (branch="renovate/rdkit-2024.x")
DEBUG: Updating poetry.lock (branch="renovate/rdkit-2024.x")
DEBUG: Using python version from pyproject.toml (branch="renovate/rdkit-2024.x")
DEBUG: Using poetry version 1.8.5 from poetry.lock header (branch="renovate/rdkit-2024.x")
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/renovate/cache/containerbase (branch="renovate/rdkit-2024.x")
DEBUG: Using containerbase dynamic installs (branch="renovate/rdkit-2024.x")
DEBUG: Resolved stable matching version (branch="renovate/rdkit-2024.x")
{
  "toolName": "python"
  "constraint": "^3.11"
  "resolvedVersion": "3.13.1"
}

DEBUG: Executing command (branch="renovate/rdkit-2024.x")
{
  "command": "install-tool python 3.13.1"
}

DEBUG: exec completed (branch="renovate/rdkit-2024.x")
{
  "durationMs": 11500
  "stdout": "[10:48:55.540] INFO (923): Installing tool [email protected]...\ninstalling v2 tool python v3.13.1\nlinking tool python v3.13.1\nPython 3.13.1\npip 24.3.1 from /opt/containerbase/tools/python/3.13.1/lib/python3.13/site-packages/pip (python 3.13)\n[10:49:05.794] INFO (923): Install tool python succeeded in 10.2s.\n"
  "stderr": ""
}

DEBUG: Executing command (branch="renovate/rdkit-2024.x")
{
  "command": "install-tool poetry 1.8.5"
}

DEBUG: exec completed (branch="renovate/rdkit-2024.x")
{
  "durationMs": 19403
  "stdout": "[10:49:06.938] INFO (1127): Installing pip [email protected]...\nPoetry (version 1.8.5)\n[10:49:25.048] INFO (1127): Install tool poetry succeeded in 18.1s.\n"
  "stderr": ""
}

DEBUG: Executing command (branch="renovate/rdkit-2024.x")
{
  "command": "poetry update --lock --no-interaction rdkit"
}

DEBUG: exec completed (branch="renovate/rdkit-2024.x")
{
  "durationMs": 3118
  "stdout": "Updating dependencies\nResolving dependencies...\n\nWriting lock file\n"
  "stderr": "Creating virtualenv rdkit-microservice-m_oVIF5K-py3.13 in /home/ubuntu/.cache/pypoetry/virtualenvs\n"
}

DEBUG: Returning updated poetry.lock (branch="renovate/rdkit-2024.x")
DEBUG: Updated 1 package files (branch="renovate/rdkit-2024.x")
DEBUG: Updated 1 lock files (branch="renovate/rdkit-2024.x")
{
  "updatedArtifacts": [
    "poetry.lock"
  ]
}

DEBUG: Getting comments for #320 (branch="renovate/rdkit-2024.x")
DEBUG: http cache: saving https://api.github.com/repos/pendingai/rdkit_microservice/issues/320/comments?per_page=100 (etag=W/"e148ff5bd678c2cecb70d81615f9236a47a006ff20c268a85c6b553dbd0ac094", lastModified=undefined) (branch="renovate/rdkit-2024.x")
DEBUG: Found 2 comments (branch="renovate/rdkit-2024.x")
DEBUG: Getting comments for #320 (branch="renovate/rdkit-2024.x")
DEBUG: http cache: saving https://api.github.com/repos/pendingai/rdkit_microservice/issues/320/comments?per_page=100 (etag=W/"e148ff5bd678c2cecb70d81615f9236a47a006ff20c268a85c6b553dbd0ac094", lastModified=undefined) (branch="renovate/rdkit-2024.x")
DEBUG: Found 2 comments (branch="renovate/rdkit-2024.x")
DEBUG: 2 file(s) to commit (branch="renovate/rdkit-2024.x")
DEBUG: Preparing files for committing to branch renovate/rdkit-2024.x (branch="renovate/rdkit-2024.x")
DEBUG: git commit (branch="renovate/rdkit-2024.x")
{
  "deletedFiles": []
  "ignoredFiles": []
  "result": {
    "author": null,
    "branch": "renovate/rdkit-2024.x",
    "commit": "823aac83023e50adfa42ba0d165ca8a8e96dd149",
    "root": false,
    "summary": {
      "changes": 2,
      "insertions": 28,
      "deletions": 23
    }
  }
}

(2):

DEBUG: Starting search at index 933 (branch="renovate/pydantic-2.x")
{
  "packageFile": "pyproject.toml"
  "depName": "pydantic"
}

DEBUG: Found match at index 933 (branch="renovate/pydantic-2.x")
{
  "packageFile": "pyproject.toml"
  "depName": "pydantic"
}

DEBUG: Contents updated (branch="renovate/pydantic-2.x")
{
  "packageFile": "pyproject.toml"
  "depName": "pydantic"
}

DEBUG: updateArtifacts for updatedPackageFiles (branch="renovate/pydantic-2.x")
DEBUG: No pdm.lock found (branch="renovate/pydantic-2.x")
DEBUG: No uv.lock found (branch="renovate/pydantic-2.x")
DEBUG: Updated 1 package files (branch="renovate/pydantic-2.x")
DEBUG: No updated lock files in branch (branch="renovate/pydantic-2.x")
DEBUG: 1 file(s) to commit (branch="renovate/pydantic-2.x")
DEBUG: Preparing files for committing to branch renovate/pydantic-2.x (branch="renovate/pydantic-2.x")
DEBUG: Setting git author name: renovate[bot] (branch="renovate/pydantic-2.x")
DEBUG: Setting git author email: 29139614+renovate[bot]@users.noreply.github.com (branch="renovate/pydantic-2.x")
DEBUG: git commit (branch="renovate/pydantic-2.x")
{
  "deletedFiles": []
  "ignoredFiles": []
  "result": {
    "author": null,
    "branch": "renovate/pydantic-2.x",
    "commit": "4c53e73b16c5f6a5369fbfa40983a879fd858559",
    "root": false,
    "summary": {
      "changes": 1,
      "insertions": 1,
      "deletions": 1
    }
  }
}

Looking at the logs, it looks like poetry.lock does not get modified by renovate in (2).

@ryancausey already listed (some) of the breaking changes in poetry v2 in the initial post. These are also listed in the poetry release notes. Please let me know if there is anything else that might be useful to investigate this issue.

[ryancausey]

It looks like renovate doesn't think it's a poetry project in (2), so doesn't update the poetry lock file.

[uli-f]

That would make sense. The format of the pyproject.toml changed quite a bit.

Here is my pyproject.toml:

[project]

### project metadata ###
name = "rdkit_microservice"
# version string here is always '0'; when producing an artifact using a github workflow the git tag is used to label the release
version = "0"
description = "selected functionality of rdkit wrapped up in a REST API"
authors = [
    {name = "Uli"},
    {name = "Jacob"}
]
readme = "README.md"
license = "Proprietary"
repository = "https://github.com/pendingai/rdkit_microservice"

### project dependencies ###
requires-python = ">=3.11.9 <3.12"
dependencies = [
    # https://github.com/rdkit/rdkit
    # BSD 3-Clause license
    "rdkit ==2024.9.4",

    # https://github.com/tiangolo/fastapi
    # MIT license
    "fastapi ==0.115.6",

    # https://github.com/pgjones/hypercorn
    # MIT license
    "hypercorn ==0.17.3",

    # https://github.com/pydantic/pydantic
    # MIT license
    "pydantic ==2.10.5",

    # https://github.com/orsinium-labs/svg.py
    # MIT license
    "svg.py ==1.5.0",

    # https://github.com/jordaneremieff/mangum
    # MIT license
    "mangum ==0.19.0",

    # Transitive dependency. Added here to explicitly bump version to
    # https://github.com/pallets/jinja
    # BSD license
    "jinja2 ==3.1.5"
]

[tool.poetry]
requires-poetry = ">=2.0"
include = [
    { path = "resources", format = ["sdist", "wheel"] }
]

[tool.poetry.group.dev.dependencies]
# https://github.com/pytest-dev/pytest
# MIT License
pytest = "8.3.4"

# https://github.com/pytest-dev/pytest-html
# MPL 2.0
pytest-html = "4.1.1"

# https://github.com/encode/httpx
# BSD-3 clause license
httpx = "0.28.1"

# https://github.com/pre-commit/pre-commit
# MIT License
pre-commit = "4.0.1"

# https://github.com/pylint-dev/pylint/
# GNU GPL 2
pylint = "3.3.3"

# https://github.com/python/mypy
# MIT License
mypy = "1.14.1"

# https://github.com/pdoc3/pdoc
# AGPL 3.0
pdoc3 = "0.11.5"

# https://github.com/CycloneDX/cyclonedx-python
# Apache 2.0
cyclonedx-bom = "5.1.1"

[tool.poetry.requires-plugins]
poetry-plugin-export = ">=1.8.0"

[build-system]
requires = ["poetry-core>=2.0"]
build-backend = "poetry.core.masonry.api"

[tool.pytest.ini_options]
pythonpath = ["src"]
testpaths = "tests"
junit_suite_name = "rdkit_microservice"

[tool.mypy]
exclude = [ '^tests/', 'Scripts/activate_this.py' ]
strict = true
disallow_untyped_decorators = false

[megum1n]

Also, poetry allows link1 link2 to override source for any dependency, which is not supported by PEP621.

Renovate ignores those overrides and tries to update dependency to the version of the package with the same name found in PyPI (logger in my case).

pyproject.toml

[project]
dependencies = [
    "logger==1.0.53",
]

[[tool.poetry.source]]
name = "gitlab"
url = "https://<gitlab>/api/v4/groups/<id>/-/packages/pypi/simple"
priority = "supplemental"

[tool.poetry.dependencies]
logger = { source = "gitlab" }

[rarkins]

Converted to issue: #33509