Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trying to cancel an on going remote attesation crashes #1713

Open
javirln opened this issue Jan 8, 2025 · 11 comments
Open

Trying to cancel an on going remote attesation crashes #1713

javirln opened this issue Jan 8, 2025 · 11 comments
Labels
bug Something isn't working component/CLI component/controlplane

Comments

@javirln
Copy link
Member

javirln commented Jan 8, 2025

The way I could reproduce it was:

  • Start a remote attestation from a CI or somewhere else not local
  • Grab the attestation id
  • While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

Example of the error:

$ chainloop attestation reset --trigger cancellation --attestation-id 40c7b6bc-30cd-4a30-a05d-8b6a7462ad0d
ERR loading existing attestation error="failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error"
ERR failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error
@migmartri
Copy link
Member

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

@javirln
Copy link
Member Author

javirln commented Jan 8, 2025

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

It's not that is important just the way I managed to ran into the issue :/

@migmartri
Copy link
Member

While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

so does the attestation-id need to exist? Is a problem that the attestation has already been cancelled?

@migmartri
Copy link
Member

Start a remote attestation from a CI or somewhere else not local

what does it mean not local? what is it important?

It's not that is important just the way I managed to ran into the issue :/

so you can't reproduce it locally?

@migmartri
Copy link
Member

It would be nice if you could provide a full reproduction path, form init to crash

@javirln
Copy link
Member Author

javirln commented Jan 8, 2025

While the workflow is running, try to cancel it with chainloop attestation reset --trigger cancellation --attestation-id

so does the attestation-id need to exist? Is a problem that the attestation has already been cancelled?

It does need to exist, the attestation was still on going

@migmartri
Copy link
Member

Did you try to cancel from another environment / CLI instance by any chance?

@javirln
Copy link
Member Author

javirln commented Jan 8, 2025 •
edited
Loading

I managed to reproduce it locally:

  1. Create an API token
  2. Initialize an attestation using the remote state
  3. Create another API token
  4. With the second API token, try to cancel the on going attestation
$ chainloop --insecure org api-token create --name first-token
WRN API contacted in insecure mode
WRN Both user credentials and $CHAINLOOP_TOKEN set. Ignoring $CHAINLOOP_TOKEN.
┌─────────────┬─────────────┬─────────────────────┬────────────┬────────────┐
│ NAME        │ DESCRIPTION │ CREATED AT          │ EXPIRES AT │ REVOKED AT │
├─────────────┼─────────────┼─────────────────────┼────────────┼────────────┤
│ first-token │             │ 08 Jan 25 17:34 UTC │            │            │
└─────────────┴─────────────┴─────────────────────┴────────────┴────────────┘

Save the following token since it will not printed again:

 eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiIwMjk3NjhiNS0wYWQ2LTQwMzMtYWY1ZS03NTVjM2M0ODg2NWIifQ.qidxgiKe5Ie0cWVffcqW0206euNVJXikhNVGKY2-k5c

$ export CHAINLOOP_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiIwMjk3NjhiNS0wYWQ2LTQwMzMtYWY1ZS03NTVjM2M0ODg2NWIifQ.qidxgiKe5Ie0cWVffcqW0206euNVJXikhNVGKY2-k5c

$ chainloop --insecure attestation init --name reviews-pm --project one-project --replace --remote-state
Flag --name has been deprecated, please use --workflow instead
WRN API contacted in insecure mode
INF Attestation initialized! now you can check its status or add materials to it
┌────────────────┬──────────────────────────────────────┐
│ Initialized At │ 08 Jan 25 17:34 UTC                  │
├────────────────┼──────────────────────────────────────┤
│ Attestation ID │ 924c3c79-eeb0-4306-869c-297f2a3a3a76 │
│ Organization   │ read-only-demo                       │
│ Name           │ reviews-pm                           │
│ Project        │ one-project                          │
│ Version        │ none                                 │
│ Contract       │ one-project-reviews-pm (revision 1)  │
└────────────────┴──────────────────────────────────────┘

$ chainloop --insecure org api-token create --name second-token
WRN API contacted in insecure mode
WRN Both user credentials and $CHAINLOOP_TOKEN set. Ignoring $CHAINLOOP_TOKEN.
┌──────────────┬─────────────┬─────────────────────┬────────────┬────────────┐
│ NAME         │ DESCRIPTION │ CREATED AT          │ EXPIRES AT │ REVOKED AT │
├──────────────┼─────────────┼─────────────────────┼────────────┼────────────┤
│ second-token │             │ 08 Jan 25 17:34 UTC │            │            │
└──────────────┴─────────────┴─────────────────────┴────────────┴────────────┘

Save the following token since it will not printed again:

 eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiJlNDgwYjI1ZC0zM2Q0LTRkNDEtODNjZC01MjdjNDcxNDM3ZjEifQ.BbsQCzVYt8EDoXEzARW2HFn3HgYR9rlztOwlxgXBTKI

$ export CHAINLOOP_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJvcmdfaWQiOiI5M2QwMjI3NS04NTNjLTRhZDYtOWQ2MC04ZjU2MmIxMjNmZDIiLCJvcmdfbmFtZSI6InJlYWQtb25seS1kZW1vIiwiaXNzIjoiY3AuY2hhaW5sb29wIiwiYXVkIjpbImFwaS10b2tlbi1hdXRoLmNoYWlubG9vcCJdLCJqdGkiOiJlNDgwYjI1ZC0zM2Q0LTRkNDEtODNjZC01MjdjNDcxNDM3ZjEifQ.BbsQCzVYt8EDoXEzARW2HFn3HgYR9rlztOwlxgXBTKI

$ chainloop --insecure attestation reset --trigger cancellation --attestation-id 924c3c79-eeb0-4306-869c-297f2a3a3a76
WRN API contacted in insecure mode
ERR loading existing attestation error="failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error"
ERR failed to load crafting state: failed to read state: rpc error: code = Internal desc = server error

And the controlplane logs:

2025-01-08T18:35:03.005+0100	INFO	{"kind": "server", "component": "grpc", "operation": "/controlplane.v1.AttestationStateService/Initialized", "args": "workflow_run_id:\"924c3c79-eeb0-4306-869c-297f2a3a3a76\"", "code": 0, "reason": "", "stack": "", "latency": 0.017220291}
2025-01-08T18:35:03.007+0100	INFO	{"msg": "[authN] processed credentials", "id": "e480b25d-33d4-4d41-83cd-527c471437f1", "type": "API-token"}
2025-01-08T18:35:03.021+0100	ERROR	{"component": "service", "msg": "failed to decrypt attestation state: incorrect passphrase"}
  github.com/go-kratos/kratos/contrib/log/zap/v2.(*Logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/contrib/log/zap/[email protected]/zap.go:41
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Filter).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/filter.go:99
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Helper).Error
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/helper.go:121
  github.com/chainloop-dev/chainloop/pkg/servicelogger.LogAndMaskErr
  /the-folder/projects/chainloop/backend/pkg/servicelogger/logger.go:57
  github.com/chainloop-dev/chainloop/app/controlplane/internal/service.handleUseCaseErr
  /the-folder/projects/chainloop/backend/app/controlplane/internal/service/service.go:158
  github.com/chainloop-dev/chainloop/app/controlplane/internal/service.(*AttestationStateService).Read
  /the-folder/projects/chainloop/backend/app/controlplane/internal/service/attestationstate.go:126
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler.func1
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:187
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.NewGRPCServer.UnaryServerInterceptor.func2
  /the-folder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/protovalidate/protovalidate.go:38
  google.golang.org/grpc.getChainUnaryHandler.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1211
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:35
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.NewSentryContext.func20.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/sentrycontext/sentry_context.go:41
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.WithAttestationContextFromAPIToken.func18.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/apitoken_middleware.go:130
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.WithAttestationContextFromRobotAccount.func17.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/robotaccount_middleware.go:63
  github.com/chainloop-dev/chainloop/app/controlplane/internal/usercontext/attjwtmiddleware.WithJWTMulti.func1.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/attjwtmiddleware/attmiddleware.go:206
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.(*Builder).Build.selector.func25.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/selector/selector.go:125
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.(*Builder).Build.selector.func24.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/selector/selector.go:123
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Prometheus.func5.1.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/wrappers.go:45
  github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3
  /the-folder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:108
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Prometheus.func5.1
  /the-folder/projects/chainloop/backend/app/controlplane/internal/usercontext/wrappers.go:49
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Server.func4.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/logging/logging.go:34
  github.com/go-kratos/kratos/v2/middleware/recovery.Recovery.func2.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/recovery/recovery.go:59
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:40
  google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1202
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:189
  google.golang.org/grpc.(*Server).processUnaryRPC
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1393
  google.golang.org/grpc.(*Server).handleStream
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1804
  google.golang.org/grpc.(*Server).serveStreams.func2.1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1029
2025-01-08T18:35:03.022+0100	ERROR	{"kind": "server", "component": "grpc", "operation": "/controlplane.v1.AttestationStateService/Read", "args": "workflow_run_id:\"924c3c79-eeb0-4306-869c-297f2a3a3a76\"", "code": 500, "reason": "internal error", "stack": "error: code = 500 reason = internal error message = server error metadata = map[] cause = <nil>", "latency": 0.016550125}
  github.com/go-kratos/kratos/contrib/log/zap/v2.(*Logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/contrib/log/zap/[email protected]/zap.go:41
  github.com/go-kratos/kratos/v2/log.(*logger).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/log.go:30
  github.com/go-kratos/kratos/v2/log.(*Filter).Log
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/log/filter.go:99
  github.com/chainloop-dev/chainloop/app/controlplane/internal/server.craftMiddleware.Server.func4.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/logging/logging.go:40
  github.com/go-kratos/kratos/v2/middleware/recovery.Recovery.func2.1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/middleware/recovery/recovery.go:59
  github.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).unaryServerInterceptor.func1
  /the-folder/go/pkg/mod/github.com/go-kratos/kratos/[email protected]/transport/grpc/interceptor.go:40
  google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1202
  github.com/chainloop-dev/chainloop/app/controlplane/api/controlplane/v1._AttestationStateService_Read_Handler
  /the-folder/projects/chainloop/backend/app/controlplane/api/controlplane/v1/attestation_state_grpc.pb.go:189
  google.golang.org/grpc.(*Server).processUnaryRPC
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1393
  google.golang.org/grpc.(*Server).handleStream
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1804
  google.golang.org/grpc.(*Server).serveStreams.func2.1
  /the-folder/go/pkg/mod/google.golang.org/[email protected]/server.go:1029

@jiparis
Copy link
Member

jiparis commented Jan 8, 2025

It makes sense, since the passphrase is generated from the workload token. We can read this in the code:

// In order to encrypt the state at rest, we will encrypt the state using a passphrase
// which comes in the request and is used to derive an encryption key.
// The passphrase that we'll use is the robot-account token that only the workload should have.

We should probably find a better way. But for sure, that error message should be more friendly.

@migmartri Linear
Copy link
Member

ok, that makes sense. The use of a different token was they key that was missing. We definitely need to handle this better but it's not that pressing.

@javier just so I understand if other users can run into this. Why are you using two API tokens, are you doing a mix of CI/local development?

@javirln Linear
Copy link
Member Author

javirln commented Jan 9, 2025

Yes, that's exactly what I was doing. I was wondering if I could cancel an ongoing attestation locally. My use case was the following:

I had several pipelines that were not handled correctly, meaning the CI job failed but it didn't change the attestation status. So I thought about changing the state myself and mark them as failed. Since I didn't have access to the token used to initialized the attestation I used another one and then ran into the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working component/CLI component/controlplane
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@migmartri @jiparis @javirln