Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store #13272

skumawat2025 · 2024-04-17T21:07:22Z

Description

This PR is a sub task of feature request #13022. And is being tracked here #13091.

This PR include below changes -

The translog.ckp file's data is now stored as object metadata associated with the translog.tlog file when uploading to an S3 remote storage.
The translog download flow has been updated to handle mixed index scenarios. In cases where translog files were previously uploaded to the remote storage in different formats (separate translog.tlog and translog.ckp files) as well as the new upload format (checkpoint file as header/metadata to translog.tlog file)
Deletion of translog files

Related Issues

Resolves #13094

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…3 remote store Signed-off-by: Sandeep Kumawat <[email protected]>

…ta upload Signed-off-by: Sandeep Kumawat <[email protected]>

…te upload Signed-off-by: Sandeep Kumawat <[email protected]>

…3 remote store Signed-off-by: Sandeep Kumawat <[email protected]>

…ta upload Signed-off-by: Sandeep Kumawat <[email protected]>

…te upload Signed-off-by: Sandeep Kumawat <[email protected]>

Signed-off-by: Sandeep Kumawat <[email protected]>

…earch into storage-13094

github-actions · 2024-04-17T21:41:45Z

❌ Gradle check result for b22a58d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-18T11:50:31Z

❌ Gradle check result for fde9788: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-19T02:48:35Z

❌ Gradle check result for 57326eb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-04-19T03:38:48Z

❌ Gradle check result for 23f4a64: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-19T03:45:35Z

❌ Gradle check result for e69b2b4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-19T04:11:10Z

❌ Gradle check result for 5dc4ab2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

codecov · 2024-04-21T17:39:19Z

Codecov Report

Attention: Patch coverage is 85.23316% with 57 lines in your changes are missing coverage. Please review.

Project coverage is 71.55%. Comparing base (b15cb0c) to head (bd58d92).
Report is 275 commits behind head on main.

❗ Current head bd58d92 differs from pull request most recent head fd5450f. Consider uploading reports for the commit fd5450f to get more accurate results

Files	Patch %	Lines
...sfer/TranslogCkpAsMetadataFileTransferManager.java	86.76%	4 Missing and 5 partials ⚠️
.../translog/transfer/TranslogCheckpointSnapshot.java	76.47%	8 Missing ⚠️
...slog/transfer/TranslogCkpFilesTransferManager.java	87.27%	1 Missing and 6 partials ⚠️
...slog/transfer/TranslogCkpFilesTransferTracker.java	88.88%	3 Missing and 4 partials ⚠️
.../org/opensearch/index/remote/RemoteStoreUtils.java	50.00%	0 Missing and 4 partials ⚠️
...ex/translog/transfer/BlobStoreTransferService.java	20.00%	4 Missing ⚠️
...ex/remote/RemoteMigrationIndexMetadataUpdater.java	62.50%	0 Missing and 3 partials ⚠️
...ensearch/index/translog/transfer/FileSnapshot.java	66.66%	3 Missing ⚠️
...rg/opensearch/index/translog/RemoteFsTranslog.java	83.33%	1 Missing and 1 partial ⚠️
...va/org/opensearch/indices/RemoteStoreSettings.java	66.66%	2 Missing ⚠️
... and 8 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13272      +/-   ##
============================================
+ Coverage     71.42%   71.55%   +0.13%     
- Complexity    59978    61172    +1194     
============================================
  Files          4985     5060      +75     
  Lines        282275   287403    +5128     
  Branches      40946    41628     +682     
============================================
+ Hits         201603   205650    +4047     
- Misses        63999    64830     +831     
- Partials      16673    16923     +250

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-05-07T15:33:11Z

❌ Gradle check result for 7437c73: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-08T06:26:58Z

❌ Gradle check result for 06a730d: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-05-08T12:24:19Z

❌ Gradle check result for 828f8e3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

… into storage-issue-13094

github-actions · 2024-05-08T23:18:22Z

✅ Gradle check result for e058a92: SUCCESS

github-actions · 2024-05-10T13:00:19Z

❌ Gradle check result for a993ee5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-10T13:16:42Z

❌ Gradle check result for b5cdfdd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-05-10T14:22:03Z

❌ Gradle check result for 530277c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-05-10T15:56:40Z

❌ Gradle check result for 2d6a296: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sandeep Kumawat <[email protected]>

github-actions · 2024-05-12T19:41:35Z

✅ Gradle check result for bd58d92: SUCCESS

Bukhtawar · 2024-05-13T05:57:12Z

...in/java/org/opensearch/index/translog/transfer/TranslogCkpAsMetadataFileTransferManager.java

+ *
+ * @opensearch.internal
+ */
+public class TranslogCkpAsMetadataFileTransferManager extends TranslogTransferManager {


Please fix the naming of these files, it looks to specific to the PR and not generic

Bukhtawar · 2024-05-13T06:02:13Z

...er/src/main/java/org/opensearch/index/translog/transfer/TranslogCkpFilesTransferTracker.java

+            if (isUploaded(tlogFileName) == false) {
+                recordFileContentLength(tlogFileName, file::getTranslogFileContentLength);
+            }
+            if (isUploaded(ckpFileName) == false) {
+                recordFileContentLength(ckpFileName, file::getCheckpointFileContentLength);
+            }


why not checksum?

Bukhtawar

Looks like there are way too many abstractions with too specific implementations. We just need a metadata provider abstraction instead and simplify this PR

Signed-off-by: Ashish Singh <[email protected]>

github-actions · 2024-05-13T09:11:00Z

❌ Gradle check result for 46bffbe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-13T09:50:13Z

❌ Gradle check result for fd5450f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale · 2024-05-20T11:31:59Z

This is moved to: #13637

skumawat2025 added 8 commits April 16, 2024 06:06

Upload translog.ckp file data as metadata to translog.tlog file for s…

84c8649

…3 remote store Signed-off-by: Sandeep Kumawat <[email protected]>

add support to identify if enabled remote store support object metada…

c56cf29

…ta upload Signed-off-by: Sandeep Kumawat <[email protected]>

Process translog.tlog and translog.ckp files Snapshot as one for remo…

84f9c92

…te upload Signed-off-by: Sandeep Kumawat <[email protected]>

Upload translog.ckp file data as metadata to translog.tlog file for s…

088eb6b

…3 remote store Signed-off-by: Sandeep Kumawat <[email protected]>

add support to identify if enabled remote store support object metada…

264444b

…ta upload Signed-off-by: Sandeep Kumawat <[email protected]>

Process translog.tlog and translog.ckp files Snapshot as one for remo…

cd413a0

…te upload Signed-off-by: Sandeep Kumawat <[email protected]>

empty commit

e3a1ae4

Signed-off-by: Sandeep Kumawat <[email protected]>

Merge branch 'storage-13094' of https://github.com/skumawat2025/OpenS…

b22a58d

…earch into storage-13094

github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Performance v2.14.0 labels Apr 17, 2024

failure tests

fde9788

Signed-off-by: Sandeep Kumawat <[email protected]>

Revert commits of changes for blobStoreTransferService

39a4a13

prepare toUpload fileSnapshots in translogTransferManager

23f4a64

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the storage-issue-13094 branch from 57326eb to 23f4a64 Compare April 19, 2024 03:31

skumawat2025 force-pushed the storage-issue-13094 branch from e69b2b4 to 5dc4ab2 Compare April 19, 2024 03:51

skumawat2025 added the skip-changelog label Apr 19, 2024

skumawat2025 changed the title ~~Upload translog.ckp file data as metadata to translog.tlog file for s3 remote store~~ Upload translog.ckp file data as object metadata to translog.tlog file for s3 remote store Apr 19, 2024

skumawat2025 changed the title ~~Upload translog.ckp file data as object metadata to translog.tlog file for s3 remote store~~ Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store Apr 19, 2024

skumawat2025 added the Storage:Remote label Apr 21, 2024

skumawat2025 self-assigned this Apr 21, 2024

spotless fix

b64d2d9

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the storage-issue-13094 branch from 5dc4ab2 to b64d2d9 Compare April 21, 2024 16:50

skumawat2025 force-pushed the storage-issue-13094 branch from 7437c73 to 06a730d Compare May 8, 2024 06:04

minor fixes

828f8e3

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the storage-issue-13094 branch from 06a730d to 828f8e3 Compare May 8, 2024 11:29

skumawat2025 added 2 commits May 9, 2024 03:57

abstract FileTransferTracker class

01cfd9d

Signed-off-by: Sandeep Kumawat <[email protected]>

Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…

e058a92

… into storage-issue-13094

skumawat2025 force-pushed the storage-issue-13094 branch from a993ee5 to b5cdfdd Compare May 10, 2024 13:05

add support of new translog upload flow for new indices only

530277c

Signed-off-by: Sandeep Kumawat <[email protected]>

skumawat2025 force-pushed the storage-issue-13094 branch from b5cdfdd to 530277c Compare May 10, 2024 13:26

add changes related to remote store migration

2d6a296

Signed-off-by: Sandeep Kumawat <[email protected]>

fix test failures and clean up

bd58d92

Signed-off-by: Sandeep Kumawat <[email protected]>

Bukhtawar reviewed May 13, 2024

View reviewed changes

Make changes for given comments

fd5450f

Signed-off-by: Ashish Singh <[email protected]>

ashking94 force-pushed the storage-issue-13094 branch from 46bffbe to fd5450f Compare May 13, 2024 09:05

skumawat2025 mentioned this pull request May 13, 2024

[Remote Store] Upload translog checkpoint as object metadata to translog #13637

Merged

8 tasks

skumawat2025 removed the v2.14.0 label May 16, 2024

sachinpkale closed this May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store #13272

Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store #13272

skumawat2025 commented Apr 17, 2024 •

edited

Loading

github-actions bot commented Apr 17, 2024

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

codecov bot commented Apr 21, 2024 •

edited

Loading

github-actions bot commented May 7, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 12, 2024

Bukhtawar May 13, 2024

Bukhtawar May 13, 2024

Bukhtawar left a comment

github-actions bot commented May 13, 2024

github-actions bot commented May 13, 2024

sachinpkale commented May 20, 2024

Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store #13272

Upload translog.ckp file data as object metadata with translog.tlog file for s3 remote store #13272

Conversation

skumawat2025 commented Apr 17, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Apr 17, 2024

github-actions bot commented Apr 18, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

github-actions bot commented Apr 19, 2024

codecov bot commented Apr 21, 2024 • edited Loading

Codecov Report

github-actions bot commented May 7, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 8, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 10, 2024

github-actions bot commented May 12, 2024

Bukhtawar May 13, 2024

Choose a reason for hiding this comment

Bukhtawar May 13, 2024

Choose a reason for hiding this comment

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented May 13, 2024

github-actions bot commented May 13, 2024

sachinpkale commented May 20, 2024

skumawat2025 commented Apr 17, 2024 •

edited

Loading

codecov bot commented Apr 21, 2024 •

edited

Loading