feat(storage): support bucket restore in soft delete phase 2 #13894

mahendra-google · 2024-11-28T12:39:02Z

Feature request includes the following:-

Get Bucket Generation
Get a Soft Deleted Bucket (Also soft-delete time and hard-delete time)
List Soft Deleted Buckets
Restore a Soft Deleted Bucket

… related test cases

jskeet

This is not a full code review by any means, but I'm hoping that this little bit will reduce the number of iterations required after Amanda is back from Thanksgiving.

Directory.Packages.props

mahendra-google · 2024-11-28T13:06:58Z

This is not a full code review by any means, but I'm hoping that this little bit will reduce the number of iterations required after Amanda is back from Thanksgiving.

Ok

…version for Google.Cloud.Storage.V1 in apis.json

jskeet · 2024-11-29T10:46:24Z

Unassigning myself from review - I'm on vacation until Tuesday. (I expect Amanda will review before then.)

amanda-tarafa · 2024-12-03T17:59:49Z

I'll review today. @mahendra-google if you can rebase on main to remove conflicts, that'd be great, thanks.

mahendra-google · 2024-12-04T09:28:40Z

I'll review today. @mahendra-google if you can rebase on main to remove conflicts, that'd be great, thanks.

@amanda-tarafa I have removed conflict for Directory.Packages.props file.

mahendra-google · 2024-12-06T05:24:08Z

I'll review today. @mahendra-google if you can rebase on main to remove conflicts, that'd be great, thanks.

@amanda-tarafa I have removed conflict for Directory.Packages.props file.

@amanda-tarafa I have removed project file conflict.

mahendra-google · 2024-12-06T05:34:07Z

@amanda-tarafa Storage Samples requires latest package of Google.Cloud.Storage.V1 having properties of GetBucketOptions , ListBucketOptions, RestoreBucketOptions to work sample as expected. So for samples can I proceed with lastest dlls?

jskeet · 2024-12-11T08:24:35Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/GetBucketOptions.cs

+        public long? Generation { get; set; }
+
+        /// <summary>
+        /// (Optional) Set to true to retrieve a soft-deleted bucket .It will return the bucket metadata only if the bucket exists and is in a soft-deleted state.


Nit while I spot it: the period after the first sentence is in the wrong place.

amanda-tarafa

You need to add unit tests for all the option classes that you modified. See #12060 for examples.

amanda-tarafa · 2024-12-10T23:06:07Z

apis/apis.json

@@ -4967,7 +4967,7 @@
      "description": "Recommended Google client library to access the Google Cloud Storage API. It wraps the Google.Apis.Storage.v1 client library, making common operations simpler in client code. Google Cloud Storage stores and retrieves potentially large, immutable data objects.",
      "dependencies": {
        "Google.Api.Gax.Rest": "default",
-        "Google.Apis.Storage.v1": "1.68.0.3431"
+        "Google.Apis.Storage.v1": "1.68.0.3604"


This file is now on the generator-input folder

This comment hasn't been addressed. This file is now in a different location to when you started this PR. That's the source of the conflict that github is signaling.

You need to rebase this PR on the repo's HEAD, and while you are at it, please squash all the commits. We are very keen on keeping a clean history on the repo and commits like "fixing this" or "addressing that" don't help.

amanda-tarafa · 2024-12-10T23:07:46Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.csproj

@@ -9,8 +9,8 @@
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="ConfigureAwaitChecker.Analyzer" PrivateAssets="All" />
-    <PackageReference Include="Google.Api.Gax.Rest" />
-    <PackageReference Include="Google.Apis.Storage.v1" VersionOverride="[1.68.0.3431, 2.0.0.0)" />
+    <PackageReference Include="Google.Api.Gax.Rest" VersionOverride="[4.9.0, 5.0.0)" />


I don't think we need this.

This comment hasn't been addressed.

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/StorageClientImpl.RestoreBucket.cs

amanda-tarafa · 2024-12-11T08:11:39Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/ListBucketsTest.cs

+
+            foreach (var bucket in actualBuckets)
+            {
+                // Check if list contains only soft deleted buckets created in storage fixture


This is not what you are checking here. You are only saying to tests assertions on buckets whose names en with soft-delete.

Ain't there a metadata field indicating that the bucket has been softdeleted?

Before further filtering soft deleted buckets with soft-delete, I have added SoftDeletedOnly metatdata for ListBucketsOptions to check if bucket has been softdeleted or not.

There are several things wrong with this test

When this test executes there's absolutely no guaratee that you have run a test that has created a soft delete bucket.

You want to test that setting the SoftDeleteOnly option works, so you cannot rely on it working to test it works (as per your comment: "I have added SoftDeletedOnly metatdata for ListBucketsOptions to check if bucket has been softdeleted or not.")

You are not testing that only soft-deleted buckets were returned, you are testing that if a bucket whose name ends in "soft-delete" is returned, then: its name is not null, its generation is not null, it has a soft delete time, and a hard delete time. If this test is passing anywhere it's because there's no "soft-deleted" bucket that has been created.

amanda-tarafa · 2024-12-11T08:20:40Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+        }
+        catch (Exception)
+        {
+            // If bucket is not empty, we delete on a best effort basis.


Same comment as in the other test. You won't need this one you create the bucket in the test.

(And also there's an option that can be passed to indicate "delete the bucket and all its contents", so you really wouldn't even need this).

amanda-tarafa · 2024-12-11T08:24:32Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+            }
+            await _fixture.Client.DeleteBucketAsync(_fixture.SoftDeleteBucketOne);
+        }
+        // We get softDeleted object using get bucket async method for soft deleted bucket. 


You are not getting an object.

amanda-tarafa · 2024-12-11T08:25:19Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+        Assert.NotNull(bucket.Generation);
+        Assert.NotNull(softDeleted.Generation);


I'm not sure what these are meant to be testing, shouldn't you test they are the same value?

Yes I added that here (I was checking that in the second test) also checking that bucket generations before and after delete are not null.

You can remove the null check then.

amanda-tarafa · 2024-12-11T08:27:01Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+    }
+
+    [Fact]
+    public async Task GetSoftDeletedBucket()


What's the difference between this test and the one above? What are the different cases they are testing?

The above test is for checking generation explicitly and this test is for checking bucket name along with soft delete and hard delete time.

We don't need two tests for that.

I will share design doc let me know so that I will add everything in single test

Yes, everything in a single test please.

Ok I will add in single test

…tsOptions.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

…ientImpl.RestoreBucket.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

amanda-tarafa · 2025-01-08T07:12:01Z

apis/apis.json

@@ -4967,7 +4967,7 @@
      "description": "Recommended Google client library to access the Google Cloud Storage API. It wraps the Google.Apis.Storage.v1 client library, making common operations simpler in client code. Google Cloud Storage stores and retrieves potentially large, immutable data objects.",
      "dependencies": {
        "Google.Api.Gax.Rest": "default",
-        "Google.Apis.Storage.v1": "1.68.0.3431"
+        "Google.Apis.Storage.v1": "1.68.0.3604"


This comment hasn't been addressed. This file is now in a different location to when you started this PR. That's the source of the conflict that github is signaling.

You need to rebase this PR on the repo's HEAD, and while you are at it, please squash all the commits. We are very keen on keeping a clean history on the repo and commits like "fixing this" or "addressing that" don't help.

amanda-tarafa · 2025-01-08T07:13:00Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.csproj

@@ -9,8 +9,8 @@
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="ConfigureAwaitChecker.Analyzer" PrivateAssets="All" />
-    <PackageReference Include="Google.Api.Gax.Rest" />
-    <PackageReference Include="Google.Apis.Storage.v1" VersionOverride="[1.68.0.3431, 2.0.0.0)" />
+    <PackageReference Include="Google.Api.Gax.Rest" VersionOverride="[4.9.0, 5.0.0)" />


This comment hasn't been addressed.

amanda-tarafa · 2025-01-08T07:25:58Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/StorageClientImpl.RestoreBucket.cs

+    {
+        ValidateBucketName(bucket);
+        var request = Service.Buckets.Restore(bucket, generation);
+        ApplyEncryptionKey(options?.EncryptionKey, kmsNameFromOptions: null, request);


@JesseLovelace Can you confirm that RestoreBucket supports encryption? I seem to remember this was just for object related operations.

Buckets can have a default encryption key, but that wouldn't be relevant to the restore bucket request. No need to support it as a RestoreBucketOption

amanda-tarafa · 2025-01-08T07:28:36Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/StorageClientImpl.RestoreBucket.cs

+        var request = Service.Buckets.Restore(bucket, generation);
+        ApplyEncryptionKey(options?.EncryptionKey, kmsNameFromOptions: null, request);
+        options?.ModifyRequest(request);
+        RetryOptions retryOptions = options?.RetryOptions ?? RetryOptions.IdempotentRetryOptions;


@JesseLovelace can you confirm that RestoreBucket is idempotent and retryable?

Restore Bucket doesn't support the common idempotency params (ifGenerationMatch etc) but it is safely retryable, so it can probably use the same retry handler

amanda-tarafa · 2025-01-08T07:32:29Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/RestoreBucketOptions.cs

+    /// except for the bucket's access controls. This only affects
+    /// what information is returned when restoration is successful.
+    /// </summary>
+    public Projection? Projection { get; set; }


Please leave a blank line between one property and the next.

amanda-tarafa · 2025-01-08T08:35:44Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+        Assert.NotNull(bucket.Generation);
+        Assert.NotNull(softDeleted.Generation);


You can remove the null check then.

amanda-tarafa · 2025-01-08T08:36:24Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+        var softDeleted = await _fixture.Client.GetBucketAsync(softDeleteBucket.Name, new GetBucketOptions { SoftDeleted = true, Generation = bucket.Generation });
+        Assert.NotNull(bucket.Generation);
+        Assert.NotNull(softDeleted.Generation);
+        Assert.Equal(bucket.Generation, softDeleted.Generation);


Can you also check that the soft delete timespan is set, that is the real confirmation that this is a soft deleted bucket.

As per design doc I have added soft delete timespan in GetSoftDeletedBucket test. I will share design doc

amanda-tarafa · 2025-01-08T08:37:00Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+    }
+
+    [Fact]
+    public async Task GetSoftDeletedBucket()


We don't need two tests for that.

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.Tests/RestoreBucketOptionsTest.cs

…onTests/GetBucketTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

…onTests/ListBucketsTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

…Options.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

…toreBucketOptionsTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

amanda-tarafa · 2025-01-10T06:56:29Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+    [Fact]
+    public async Task GetBucketGeneration()
+    {
+        var softDeleteBucket  = _fixture.CreateBucket(Guid.NewGuid().ToString() + "-soft-delete", false, true);


Yes, you still need to create the bucket using CreateBucket, but you should generate de bucket name using GenerateBucketName. (here and everywhere)

amanda-tarafa · 2025-01-10T06:57:55Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+    public async Task GetBucketGeneration()
+    {
+        var softDeleteBucket  = _fixture.CreateBucket(Guid.NewGuid().ToString() + "-soft-delete", false, true);
+        var bucket = await _fixture.Client.GetBucketAsync(softDeleteBucket.Name, new GetBucketOptions { SoftDeleted = false });


I'm pretty sure you have the generation already in softDeleteBucket. The metadata returned on bucket creation should be the same as the one you are obtaining here, as the bucket shouldn't have changed because it's owned by this test.

amanda-tarafa · 2025-01-10T06:58:48Z

apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.IntegrationTests/GetBucketTest.cs

+    }
+
+    [Fact]
+    public async Task GetSoftDeletedBucket()


Yes, everything in a single test please.

mahendra-google added 5 commits November 26, 2024 04:50

feat(storage): add getbucket generation , get soft deleted bucket and…

e5dd76f

… related test cases

feat(storage): add listsoftdeleted bucket and related test cases

89e845c

few comment changes for getbucket and listbucket

a552728

feat(storage): add restore soft delete bucket.

49a554f

logic changes for creating and deleting soft delete bucket

93aa3d8

mahendra-google requested review from a team as code owners November 28, 2024 12:39

product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Nov 28, 2024

jskeet reviewed Nov 28, 2024

View reviewed changes

Directory.Packages.props Outdated Show resolved Hide resolved

revert changes from Directory.Packages.props , changes in dependency …

842d4df

…version for Google.Cloud.Storage.V1 in apis.json

mahendra-google requested a review from jskeet November 29, 2024 10:36

amanda-tarafa self-assigned this Dec 3, 2024

Merge branch 'main' into restore_bucket_soft_delete_phase_2

bbcab56

Merge branch 'main' into restore_bucket_soft_delete_phase_2

bb900d0

jskeet reviewed Dec 11, 2024

View reviewed changes

amanda-tarafa requested changes Dec 11, 2024

View reviewed changes

mahendra-google and others added 8 commits December 24, 2024 11:25

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/ListBucke…

d34fb56

…tsOptions.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/ListBucke…

a10f00b

…tsOptions.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/StorageCl…

040e360

…ientImpl.RestoreBucket.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

few code changes

ab6be4b

GetBucket test changes

829887f

ListBucket code changes

ac9b68e

Storage fixture and bucket creation related recommended changes.

9778b89

Test cases modified for ListBucketOptionsTest with new added option

1192ea5

mahendra-google added 3 commits December 30, 2024 22:54

Test cases modifications with newly added options for GetBucketOptions

9cd1541

Remove test checks related to generation

836d9ec

Restore bucket options and tests added.

0eb0efe

mahendra-google requested a review from amanda-tarafa January 2, 2025 12:45

amanda-tarafa requested changes Jan 8, 2025

View reviewed changes

mahendra-google and others added 5 commits January 8, 2025 14:36

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.Integrati…

2ee732d

…onTests/GetBucketTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.Integrati…

56fc3ee

…onTests/ListBucketsTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/GetBucket…

c12edbd

…Options.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1/GetBucket…

87892eb

…Options.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

Update apis/Google.Cloud.Storage.V1/Google.Cloud.Storage.V1.Tests/Res…

f3ca3f1

…toreBucketOptionsTest.cs Co-authored-by: Amanda Tarafa Mas <[email protected]>

amanda-tarafa requested changes Jan 10, 2025

View reviewed changes

file-scoped namespace and generate bucket name method added

b6cee9a

		Assert.NotNull(bucket.Generation);
		Assert.NotNull(softDeleted.Generation);

feat(storage): support bucket restore in soft delete phase 2 #13894

Are you sure you want to change the base?

feat(storage): support bucket restore in soft delete phase 2 #13894

Conversation

mahendra-google commented Nov 28, 2024

jskeet left a comment

Choose a reason for hiding this comment

mahendra-google commented Nov 28, 2024

jskeet commented Nov 29, 2024

amanda-tarafa commented Dec 3, 2024

mahendra-google commented Dec 4, 2024

mahendra-google commented Dec 6, 2024

mahendra-google commented Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

amanda-tarafa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahendra-google Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahendra-google Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahendra-google Dec 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahendra-google Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mahendra-google commented Dec 6, 2024 •

edited

Loading

mahendra-google Dec 26, 2024 •

edited

Loading

mahendra-google Dec 30, 2024 •

edited

Loading

mahendra-google Dec 30, 2024 •

edited

Loading

mahendra-google Jan 8, 2025 •

edited

Loading