chore(storage): Instrument existing metadata ops with storage trace package

[cojenco]

Update: supersedes #11051

• Adds a development flag GO_STORAGE_DEV_OTEL_TRACING that will be removed upon experimental feature launch
• Replaces existing metadata ops instrumentation with storage trace package (tracking sheet)
• Adds emulated trace tests

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[cojenco]

@BrennaEpp could you PTAL and help check the CLA, thanks!

[tritone]

Nice start on this!

[tritone]

one minor nit, otherwise looks good to me at this point.

[BrennaEpp]

Thanks Cathy!

[BrennaEpp]

Should this be a transportClientTest instead?

[cojenco]

We'd want to use multiTransportTest here to test trace instrumentation on the transport-agnostic layer. Also, we need to configure NewGRPCClient and NewClient by passing in ClientOption, which cannot be done via transportClientTest. See previous discussion

[BrennaEpp]

transportClientTest has a check to not run if the emulator env var is not set

google-cloud-go/storage/client_test.go

Line 2175 in 48022ac

func transportClientTest(ctx context.Context, t *testing.T, test func(*testing.T, context.Context, string, string, storageClient)) {

We should modify transportClientTest to allow configuration of client opts similar to multiTransportTest. Or add a check to this test specifically so it only runs on the emulator

[BrennaEpp]

I believe this caused this issue #11480