Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: export --prune #9389

Merged
merged 2 commits into from
Nov 24, 2024
Merged

feat: export --prune #9389

merged 2 commits into from
Nov 24, 2024

Conversation

dead10ck
Copy link
Contributor

@dead10ck dead10ck commented Nov 23, 2024

Summary

This adds a --prune flag to the export command to correspond with the --prune flag of the tree command.

The purpose is for generating a requirements.txt that omits a package and all of that package's unique dependencies. This is useful for cases where the project has a dependency on a common core package, but where that package does not need to be installed in the target environment.

For example, a pyspark job needs spark for development, but when installing into a cluster that already has pyspark installed, it is desirable to omit pyspark's whole dependency tree so that only the unique dependencies that your job needs get installed, and do not risk breaking the pyspark dependencies with something incompatible.

Dev groups cannot always cover this case because there are other projects where this common dependency occurs as a transitive. One example is Airflow providers, which include Airflow itself as a dependency, but it is unnecessary and undesirable to include Airflow's dependency tree in the requirements.txt for your DAGs.

Partly related to #7214, though I'm not sure it covers the ask in that one of having this functionality extend to the project's actual published metadata.

Test Plan

An integration test was added, and some manual testing. Let me know if more would be better.

This adds a `--prune` flag to the `export` command to correspond with
the `--prune` flag of the `tree` command.

The purpose is for generating a `requirements.txt` that omits a package
and all of that package's unique dependencies. This is useful for cases
where the project has a dependency on a common core package, but where
that package does not need to be installed in the target environment.

For example, a pyspark job needs spark for development, but when
installing into a cluster that already has pyspark installed, it is
desirable to omit pyspark's whole dependency tree so that only the
unique dependencies that your job needs get installed, and do not risk
breaking the pyspark dependencies with something incompatible.

Dev groups cannot always cover this case because there are other
projects where this common dependency occurs as a transitive. One
example is Airflow providers, which include Airflow itself as a
dependency, but it is unnecessary and undesirable to include Airflow's
dependency tree in the `requirements.txt` for your DAGs.
Copy link
Member

@charliermarsh charliermarsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me, thanks.

@charliermarsh charliermarsh added the enhancement New feature or improvement to existing functionality label Nov 24, 2024
@charliermarsh charliermarsh self-assigned this Nov 24, 2024
@charliermarsh charliermarsh enabled auto-merge (squash) November 24, 2024 01:59
@charliermarsh charliermarsh merged commit e5f5bd6 into astral-sh:main Nov 24, 2024
64 checks passed
@charliermarsh
Copy link
Member

Great PR!

tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Nov 28, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.4` -> `0.5.5` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.5.5`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#055)

[Compare Source](astral-sh/uv@0.5.4...0.5.5)

##### Enhancements

-   Add aliases for build backend requests ([#&#8203;9294](astral-sh/uv#9294))
-   Avoid displaying empty paths ([#&#8203;9312](astral-sh/uv#9312))
-   Allow constraints in `uv tool upgrade` ([#&#8203;9375](astral-sh/uv#9375))
-   Remove conflict between `--no-sync` and `--frozen` in `uv run` ([#&#8203;9400](astral-sh/uv#9400))
-   Respect dependency sources in overrides and constraints ([#&#8203;9455](astral-sh/uv#9455))
-   Show an interpreter-focused message for `--target` and `--prefix` ([#&#8203;9373](astral-sh/uv#9373))
-   Add `--no-extra` flag and setting ([#&#8203;9387](astral-sh/uv#9387))
-   Add `uv export --prune` ([#&#8203;9389](astral-sh/uv#9389))
-   Add dedicated error message for musl install attempts ([#&#8203;9430](astral-sh/uv#9430))
-   Add various grammar changes to conflict error messages ([#&#8203;9369](astral-sh/uv#9369))
-   Annotate default groups in conflict error messages ([#&#8203;9368](astral-sh/uv#9368))
-   Report marker diagnostics during parsing, rather than evaluation ([#&#8203;9338](astral-sh/uv#9338))
-   Use consistent formatting for build system errors ([#&#8203;9340](astral-sh/uv#9340))
-   Use rich diagnostics for build failures ([#&#8203;9335](astral-sh/uv#9335))

##### Preview features

-   Improve build backend excludes ([#&#8203;9281](astral-sh/uv#9281))
-   Include PEP 639 `license-files` metadata during `uv publish` ([#&#8203;9442](astral-sh/uv#9442))

##### Performance

-   Initialize rayon lazily ([#&#8203;9435](astral-sh/uv#9435))
-   Migrate to PubGrub's arena for package names ([#&#8203;9448](astral-sh/uv#9448))

##### Bug fixes

-   Allow dependency groups to include the containing package ([#&#8203;9385](astral-sh/uv#9385))
-   Allow syncing to empty virtual environment directories ([#&#8203;9427](astral-sh/uv#9427))
-   Allow system Python discovery with `--target` and `--prefix` ([#&#8203;9371](astral-sh/uv#9371))
-   Don't warn when `--output-file` is empty ([#&#8203;9417](astral-sh/uv#9417))
-   Fix Python interpreter discovery on non-glibc hosts ([#&#8203;9005](astral-sh/uv#9005))
-   Fix `tool.uv.dependency-metadata.[].version` schema ([#&#8203;9468](astral-sh/uv#9468))
-   Only respect preferences across the same indexes ([#&#8203;9302](astral-sh/uv#9302))
-   Re-compile when `--compile` is passed to an install operation ([#&#8203;9378](astral-sh/uv#9378))
-   Remove `--upgrade`, `--no-upgrade`, and `--upgrade-package` from `uv tool upgrade` ([#&#8203;9318](astral-sh/uv#9318))
-   Remove dev dependencies in `--all-groups --no-dev` ([#&#8203;9300](astral-sh/uv#9300))
-   Surface extras and group conflicts in `uv export` ([#&#8203;9365](astral-sh/uv#9365))
-   Treat deprecated aliases as equivalent in marker algebra ([#&#8203;9342](astral-sh/uv#9342))
-   Treat less compatible tags as lower priority in resolver ([#&#8203;9339](astral-sh/uv#9339))

##### Documentation

-   Avoid referencing `scikit-build` (instead of `scikit-build-core`) ([#&#8203;9320](astral-sh/uv#9320))
-   Expand entry points documentation ([#&#8203;9329](astral-sh/uv#9329))
-   Fix example `pyproject.toml` in project concept documentation ([#&#8203;9298](astral-sh/uv#9298))
-   Fix header level of "Conflicting dependencies" page ([#&#8203;9330](astral-sh/uv#9330))
-   Touch-up the extension module guide ([#&#8203;9293](astral-sh/uv#9293))
-   Update the dependencies documentation ([#&#8203;9359](astral-sh/uv#9359))
-   Reference `--no-progress` option in related environment variable ([#&#8203;9357](astral-sh/uv#9357))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improvement to existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants