Skip to content

Commit

Permalink
feat(swagger): scylla-manager, move towards just restored bytes
Browse files Browse the repository at this point in the history
In overall/keyspace/table progress we were reporting both
downloaded and restored bytes. There are two problems with it:

1. Users does not really care for the distinction between
downloaded and restored bytes - they just want to know the
amount of bytes that have been successfully restored.

2. With new native Scylla API restore, we no longer separate
restoration into download/load&stream stages.

That's why we should move towards reporting just the amount
of successfully restored bytes, and not keep on adding another
fields to our API for each new type of restore.

This commit:
- Adds restored_bytes/duration to host progress, since it was
lacking there. It does not deprecate the downloaded/streamed
fields because they were just added in the previous minor release,
so that would be confusing. Instead, for now we allow for host
progress to contain detailed operation bandwidth information.
- Deprecates downloaded bytes in overall/keyspace/table progress,
as we want to move towards observing just the restored bytes.

A general note is that we should use metrics for monitoring
performance, not the sctool progress cmd, but since we already
started going in this direction, we need to support it.
  • Loading branch information
Michal-Leszczynski committed Jan 17, 2025
1 parent f7f9ac2 commit bd3a85d
Show file tree
Hide file tree
Showing 5 changed files with 28 additions and 11 deletions.
14 changes: 10 additions & 4 deletions v3/swagger/gen/scylla-manager/models/restore_host_progress.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion v3/swagger/gen/scylla-manager/models/restore_progress.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 15 additions & 4 deletions v3/swagger/scylla-manager.json
Original file line number Diff line number Diff line change
Expand Up @@ -888,6 +888,7 @@
"type": "integer"
},
"downloaded": {
"description": "This field is DEPRECATED. Total bytes downloaded from table (included in restored)",
"type": "integer"
},
"failed": {
Expand Down Expand Up @@ -1055,6 +1056,7 @@
"type": "integer"
},
"downloaded": {
"description": "This field is DEPRECATED. Total bytes downloaded from table (included in restored)",
"type": "integer"
},
"failed": {
Expand Down Expand Up @@ -1091,6 +1093,7 @@
"type": "integer"
},
"downloaded": {
"description": "This field is DEPRECATED. Total bytes downloaded from table (included in restored)",
"type": "integer"
},
"failed": {
Expand Down Expand Up @@ -1118,20 +1121,28 @@
"description": "Host shard count",
"type": "integer"
},
"restored_bytes": {
"description": "Total bytes restored by host",
"type": "integer"
},
"restore_duration": {
"description": "Total time spent by host on restore in milliseconds",
"type": "integer"
},
"downloaded_bytes": {
"description": "Total bytes downloaded by host",
"description": "Total bytes downloaded by host (included in restored_bytes)",
"type": "integer"
},
"download_duration": {
"description": "Total time spent by host on download in milliseconds",
"description": "Total time spent by host on download in milliseconds (included in restore_duration)",
"type": "integer"
},
"streamed_bytes": {
"description": "Total bytes load&streamed by host",
"description": "Total bytes load&streamed by host (included in restored_bytes)",
"type": "integer"
},
"stream_duration": {
"description": "Total time spent by host on load&stream in milliseconds",
"description": "Total time spent by host on load&stream in milliseconds (included in restore_duration)",
"type": "integer"
}
}
Expand Down

0 comments on commit bd3a85d

Please sign in to comment.