feat: allow for configuring the docker hub registry endpoint via API

[tuunit]

Comprehensive Summary of your change

This PR adds the possibility to change the registry endpoint of DockerHub via API. This is useful for Harbor instances that run in airgapped environments and have to go through a DMZ or use a reverse proxy to access DockerHub.

Why not just use a normal Docker Registry type instead of the DockerHub adapter?

Because of the logic for library images which only applies to the Registry Type DockerHub:

harbor/src/server/middleware/repoproxy/proxy.go

Lines 128 to 144 in 8bf710a

	func defaultLibrary(ctx context.Context, registryID int64, a lib.ArtifactInfo) (bool, string, error) {
	if registryID <= 0 {
	return false, "", nil
	}
	reg, err := registry.Ctl.Get(ctx, registryID)
	if err != nil {
	return false, "", err
	}
	if reg.Type != model.RegistryTypeDockerHub {
	return false, "", err
	}
	name := strings.TrimPrefix(a.Repository, a.ProjectName+"/")
	if strings.Contains(name, "/") {
	return false, "", nil
	}
	return true, name, nil
	}

By allowing to change the registry endpoint for DockerHub, those kinds of environments can still benefit from the library substitution logic.

Please indicate you've done the following:

• Well Written Title and Summary of the PR
• Label the PR as needed. "release-note/ignore-for-release, release-note/new-feature, release-note/update, release-note/enhancement, release-note/community, release-note/breaking-change, release-note/docs, release-note/infra, release-note/deprecation"
• Accepted the DCO. Commits without the DCO will delay acceptance.
• Made sure tests are passing and test coverage is added if needed.
• Considered the docs impact and opened a new docs issue or PR with docs changes if needed in website repository.

[tuunit]

/label release-note/enhancement

[tuunit]

@Vad1mo any feedback on the approach? Do you want me to write an apitest?

[Vad1mo]

Makes sense to me..

It might be the stupid questions. but why is the URL hardcoded and not changeable in the first place?
Why not add the convenience to the user to preset it and allow the user to enter what he wants.

[tuunit]

I have no idea why it is hardcoded 😅 If you want me to change the UI as well to allow changing it. I can do it as part of this PR or another. Just let me know what you guys would prefer on how to continue

[wy65701436]

We're doing this due to the behavior in the upstream. You can refer to the following link for more details: https://github.com/moby/moby/blob/master/registry/config.go#L43.

@tuunit, If you need to access Docker Hub through a proxy, you should configure the proxy for harbor-core and jobservice, rather than updating the Docker Hub URL.

[tuunit]

@wy65701436 thanks for actually finding the original place in the moby code base 😅

Nevertheless, I already know that is the official registry URL of docker hub. But as explained in my PR description, it is sometimes necessary to run Harbor inside an air gapped / private environment without direct internet access and therefore replications / caching might need to go through a simple reverse proxy to communicate with the public web. For this purpose it would be great to keep the logic of the library completion if missing despite running behind a proxy. More details in the PR summary. (which is only done for the docker-hub adapter)

[Vad1mo]

We're doing this due to the behavior in the upstream. You can refer to the following link for more details: https://github.com/moby/moby/blob/master/registry/config.go#L43.

The strategy Docker is following is for commercial reasons and backwards compatibility only. We should not limit ourselves to that.

If we ask ourselves why people user Harbor. One of the big arguments for Harbor is replication.

1. On-Prem/air gapped environments
2. No Cloud
3. non Standards setups

Regarding this PR or its further thinking

1. Will a prefilled and changeable URL set to registry-1.docker.io be backwards incompatible? → No
2. Will it restrict users? → No
3. Will it open up new possibilities and use cases? → yes

Personally, I don't see any strong argument against it, or any negative consequences.

[MinerYang]

Hi @tuunit
Would you help us to understand more why you need a reverse-proxy for dockerhub only instead of adding it into something like a while list in your air-gapped env?

[wy65701436]

It looks good to me, but could you provide additional information from your side? For example, a screenshot and detailed logs after modifying the URL and successfully replicating images through the reverse proxy would be really helpful. Since we don't have a test case to validate this, it would give us more confidence in the implementation.

I have one concern: after updating the URL through the API, the UI still shows the default value. I believe we could either update the front-end code accordingly or fork Harbor on your side to address this.

[reasonerjt]

after updating the URL through the API, the UI still shows the default value.

This looks like a show-stopper...

[tuunit]

Hi @tuunit Would you help us to understand more why you need a reverse-proxy for dockerhub only instead of adding it into something like a while list in your air-gapped env?

No offence but have you ever worked in government project? Or any air gap environments that doesn't allow ANY internet access except from one single proxy that is highly monitored and controlled?

I've seen setups like this often enough at my previous employer. Harbor might run inside an internal network without any internet access whatsoever so re-routing to a proxy is the only option which will most likely be behind multiple other hardware loadbalancers and firewalls.

[tuunit]

It looks good to me, but could you provide additional information from your side? For example, a screenshot and detailed logs after modifying the URL and successfully replicating images through the reverse proxy would be really helpful. Since we don't have a test case to validate this, it would give us more confidence in the implementation.

I will provide some screenshots and logs later.

I have one concern: after updating the URL through the API, the UI still shows the default value. I believe we could either update the front-end code accordingly or fork Harbor on your side to address this.

This isn't correct, it's even worse right now. The UI shows a different URL after updating it via the API but the registry URL is not used inside the code. So it currently seems like you can change it via the API but it doesn't do anything.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 46.22%. Comparing base (c8c11b4) to head (763e7bc).
Report is 352 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21385      +/-   ##
==========================================
+ Coverage   45.36%   46.22%   +0.86%     
==========================================
  Files         244      247       +3     
  Lines       13333    13883     +550     
  Branches     2719     2875     +156     
==========================================
+ Hits         6049     6418     +369     
- Misses       6983     7126     +143     
- Partials      301      339      +38     

Flag	Coverage Δ
unittests	46.22% <ø> (+0.86%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 491 files with indirect coverage changes