Skip to content

Commit

Permalink
Exclude function signatures from tar_repository_cas() output strings …
Browse files Browse the repository at this point in the history
…to reduce the size of pipeline metadata (#1390)
  • Loading branch information
wlandau committed Jan 10, 2025
1 parent ef47522 commit 44fd81c
Show file tree
Hide file tree
Showing 4 changed files with 43 additions and 11 deletions.
2 changes: 1 addition & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Description: Pipeline tools coordinate the pieces of computationally
The methodology in this package
borrows from GNU 'Make' (2015, ISBN:978-9881443519)
and 'drake' (2018, <doi:10.21105/joss.00550>).
Version: 1.9.1.9007
Version: 1.9.1.9008
License: MIT + file LICENSE
URL: https://docs.ropensci.org/targets/, https://github.com/ropensci/targets
BugReports: https://github.com/ropensci/targets/issues
Expand Down
12 changes: 9 additions & 3 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
# targets 1.9.1.9007
# targets 1.9.1.9008

## Invalidating changes

These changes invalidate certain targets in a pipeline and cause them to rerun on the next `tar_make()`.

* Exclude function signatures from `tar_repository_cas()` output strings to reduce the size of pipeline metadata (#1390).

## Summary of performance gains

Expand All @@ -18,14 +24,14 @@ RHEL9 | 167.809 | 37.395 | 4.487472

To take advantage of these speed gains for an existing pipeline, you may have to run `tar_make()` to convert the time stamps and file sizes to a new format. This initial `tar_make()` is slow, but subsequent `tar_make()` calls should be much faster than before the upgrade.

## Specific changes
## Other/specific changes

* Speed up `tar_make()` and `tar_outdated()` by avoiding excessive buffering and disk writes for metadata and reporters when the pipeline is just skipping targets.
* Use a more lookup-efficient data structure for `tar_runtime$file_info` (#1398).
* Fall back on vector aggregation without names (#1401, @guglicap).
* Speed up representation of file sizes in metadata (#1408).
* Add a new `"forecast_interactive"` reporter to `tar_outdated()` to choose `"forecast"` for interactive sessions and `"silent"` for non-interactive ones.
* Add a new `seconds_reporter_outdated` argument to `tar_config_set()` with a default of 0.5 to control the time interval of the reporter of `tar_outdated()` and other passive algorithm functions.
* Add a new `seconds_reporter_outdated` argument to `tar_config_set()` with a default of 1 to control the time interval of the reporter of `tar_outdated()` and other passive algorithm functions.
* Remove target descriptions from the default labels of graph visualizations.

# targets 1.9.1
Expand Down
37 changes: 30 additions & 7 deletions R/class_store_repository_cas_methods.R
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,31 @@ store_repository_cas_methods_init <- function(repository) {
splits <- unlist(strsplit(repository, split = "&", fixed = TRUE))
store_repository_cas_methods_new(
repository = repository,
upload = store_repository_cas_field(splits, pattern = "^upload="),
download = store_repository_cas_field(splits, pattern = "^download="),
exists = store_repository_cas_field(splits, pattern = "^exists="),
list = store_repository_cas_field(splits, pattern = "^list="),
upload = store_repository_cas_field(
splits,
pattern = "^upload=",
prefix = "function(key, path) "
),
download = store_repository_cas_field(
splits,
pattern = "^download=",
prefix = "function(key, path) "
),
exists = store_repository_cas_field(
splits,
pattern = "^exists=",
prefix = "function(key) "
),
list = store_repository_cas_field(
splits,
pattern = "^list=",
prefix = "function(keys) "
),
consistent = as.logical(
store_repository_cas_field(splits, pattern = "^consistent=")
store_repository_cas_field(
splits,
pattern = "^consistent="
)
)
)
}
Expand Down Expand Up @@ -45,6 +64,10 @@ store_repository_cas_methods_validate <- function(methods) {
tar_assert_none_na(methods[["consistent"]])
}

store_repository_cas_field <- function(repository, pattern) {
base64url::base64_urldecode(keyvalue_field(repository, pattern))
store_repository_cas_field <- function(repository, pattern, prefix = "") {
text <- base64url::base64_urldecode(keyvalue_field(repository, pattern))
if (text != "NULL") {
text <- paste0(prefix, text)
}
text
}
3 changes: 3 additions & 0 deletions R/tar_repository_cas.R
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,9 @@ tar_repository_cas <- function(
}

tar_repository_cas_field <- function(key, value) {
if (is.function(value)) {
value <- body(value)
}
encoded <- base64url::base64_urlencode(tar_deparse_safe(value))
paste0(key, "=", encoded)
}

0 comments on commit 44fd81c

Please sign in to comment.