ODIC + CLI Secret mismatch

[Nathan-Nesbitt]

This is a continuation of #20729 as I cannot reopen it 😄

I am trying to log into Harbor via CLI using the username + CLI secret. I am using the username for the user and the CLI secret as found here:

I have followed these instructions: https://goharbor.io/docs/2.10.0/administration/configure-authentication/oidc-auth/

But I continue to get the following error when I attempt to log in:

Error response from daemon: login attempt to <DOMAIN> failed with status: 401 Unauthorized

Along with the logs on the server:

failed to verify secret, username: <USERNAME>, error: failed to verify the secret: secret mismatch, username: <USERNAME>
failed to authenticate user:<USERNAME>, error:not supported 

I know that the username is correct, as changing the username to anything else and looking at the logs the error changes to

failed to verify secret, username: <FAKE USERNAME>, error: failed to get oidc user info, error: <QuerySeter> no row found 

Is there any advice on how to debug this? Where does the secret validation happen? Is there another secret that I should be using?

Thanks in advance!

[stonezdj]

Is your OIDC server just a proxy? and does it proxy the request to a backend OIDC server?

[Nathan-Nesbitt]

Is your OIDC server just a proxy? and does it proxy the request to a backend OIDC server?

It's a full OIDC + OAuth 2 server, I don't believe there is a proxy as it's OAuth!

[Nathan-Nesbitt]

@stonezdj is it possible that this is an issue with the headers from the request not being set on response / being stripped?

Not sure what the 401 error would be coming from since it doesn't show up as a log in the core.

[stonezdj]

This error message indicate that the input secret is different with the secret stored in the database.

failed to verify secret, username: <USERNAME>, error: failed to verify the secret: secret mismatch, username: <USERNAME>

code:

harbor/src/pkg/oidc/secret.go

Lines 103 to 105 in ebac530

	if secret != plainSecret {
	return nil, verifyError(fmt.Errorf("secret mismatch, username: %s", username))
	}

[stonezdj]

Is there any special character in your secret?

[Nathan-Nesbitt]

@stonezdj I tried no special characters in the secrets, no change. I do notice a strange pattern tho. There is a difference between the errors.

This is the output in the CLI when the correct password/token is entered but it fails:

Error response from daemon: login attempt to https://<DOMAIN>/v2/ failed with status: 401 Unauthorized

This is the output when the incorrect password is entered:

Error response from daemon: Get "https://<DOMAIN>/v2/": unauthorized:

It looks different, which makes me think that maybe it is something else that is floating up an exception. These are the logs in the NGINX node when the password is correct:

GET /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 200
GET /v2/ HTTP/1.1" 401 87 "-" 

And when the password is incorrect:

GET /v2/ HTTP/1.1" 401
GET /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry HTTP/1.1" 401

Along with an entry in the harbor-core

[/server/middleware/security/basic_auth.go:72] go/go1.21.12  failed to authenticate user:admin, error:Failed to authenticate user, due to error 'Invalid credentials'

Based on this, looks like it's succeeding to hit the auth endpoint but 401's on /v2/ which results in it failing to authenticate properly? Is there a way to increase the log levels so we can see where it is failing when it hits /V2?

[Nathan-Nesbitt]

@stonezdj So reading through the docs, I have the ingress set up properly, so I don't think that's an issue.

If you route the v2 API to harbor-portal it fails to fetch but doesn't fail to log in. Saying that This means you cannot make any CLI requests to upload/download which I think this means it succeeded to log in with core.

This would explain why the login request resulted in a failure, as the user is logged in and the credentials have been found correct. So this is not a credentials issue but rather an issue on the backend with /v2.

This issue appears to be very similar, but has been closed without resolution. #11935

[stonezdj]

Please check if there is any KEY_PATH env setup in harbor-core pod., default it point to /etc/core/key, Is it possible that the /etc/core/key in the harbor-core container changed? If yes, please delete/generate a new secret for current user.

[Nathan-Nesbitt]

@stonezdj I checked the key path, all are set to the default. All keys are configured from the values chart as provided here: https://github.com/goharbor/harbor-helm/blob/main/values.yaml and are pulled from secrets.

I deleted everything, re-created, tested it without OIDC too just to check. Everything is fresh but the issue persists.

What is the KEY_PATH for the helm chart? I don't see any option in either the helm chart and no config is set for this in the config of the core

[Nathan-Nesbitt]

If /service/token is a token endpoint for an oauth server, it would seem it's having an issue requesting the access/refresh token via CLI?

Looks like this is the error during the OAuth process:

[/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]}           
[/server/middleware/log/log.go:31]: attach request id to the logger for the request GET /service/token                                      
[/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry 
[/core/auth/authenticator.go:145]: Current AUTH_MODE is db_auth                                                                                                                   

[/server/middleware/security/basic_auth.go:79]a basic auth security context generated for request GET /service/token

[/core/service/token/token.go:37]: URL for token request: /service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry

[/core/service/token/creator.go:201]: scopes:[]
[/core/service/token/authutils.go:51]: scopes: [] 

[Nathan-Nesbitt]

@stonezdj is there anything else I should check? :)

I looked into the ingress to be sure that the headers were being passed along, tested this with both forwardedHeaders and proxyProtocol being open allowed but the issue persists.

[floyd242]

@Nathan-Nesbitt Did you get anywhere with this? I'm running into this exact same issue and I'm having no luck.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[Nathan-Nesbitt]

Not stale

[liamdebellada]

I am facing the same issue you describe @Nathan-Nesbitt. I'm seeing the same differences between the error messages when the username / password are wrong in docker login. Did you discover anything new?

Actually I just read goharbor/harbor-helm#1830 . Interestingly after removing the registry.credentials from my helm values, I am now able to login.