Support public OCI registries without Auth

[absoludity]

Summary

Following on from #6263 , where we've added support for listing repositories and tags for a registry (DockerHub) without credentials via the OCI-Catalog service, we'd like to ensure that the complete process of adding a public OCI catalog, such as the Bitnami Catalog, can be done without requiring credentials - even if it is rate limited.

The issue is that the existing code, which pulls chart data using the OCI Distribution Spec API, requires a credential to interact with that API. It may well be that we can't pull the chart tarball without any credential, so another solution to the same end would be to ensure Kubeapps itself has a docker credential (or similar for other public registries) - not sure yet.

Background and rationale

We want to demonstrate the OCI catalog enabling adding the Bitnami OCI chart catalog.

Acceptance criteria

This will be complete when a user can add the oci://registry-1.docker.io/bitnamicharts as the URL for an AppRepository, without credentials, and have Kubeapps populate the catalog (up until any rate limit is hit - that can't be worked around without a credential).

[absoludity]

The issue is that the existing code, which pulls chart data using the OCI Distribution Spec API, requires a credential to interact with that API. It may well be that we can't pull the chart tarball without any credential, so another solution to the same end would be to ensure Kubeapps itself has a docker credential (or similar for other public registries) - not sure yet.

In today's happy news: it turns out that this is only partly correct - yes, the OCI Distribution Spec API requires a credential to interact with the API, but an anonymous credential can be obtained for public repositories using the WWW-Authenticate header from the original 401 request (tested both with dockerhub and harbor). So we can use the existing code as originally planned just with a small update to request the anon token if a 401 is received!

With Harbor, verified with:

token=$(curl -s "https://demo.goharbor.io/service/token?service=harbor-registry&scope=repository:testpublic/kubeapps/oci-catalog:pull" \          
        | jq -r '.token')

curl -v -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Authorization: Bearer $token" https://demo.goharbor.io/v2/testpublic/kubeapps/oci-catalog/manifests/dev6

and similarly with dockerhub:

token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \                                       | jq -r '.token')

curl -vH "Authorization: Bearer $token" -H "Accept: application/vnd.oci.image.manifest.v1+json" https://registry-1.docker.io/v2/bitnamicharts/apache/manifests/10.1.0 

[absoludity]

So I have the syncer working in dev, but it quickly hits request limits of dockerhub when syncing the Bitnami OCI. The main reason is that for non-OCI, http repositories, this code was optimised to run many queries in parallel to finish the sync quickly (within the cron default of 10 minutes - which it does easily). But we cannot do this for an OCI repository without hitting request limits, so I'm proposing that we:

• make the number of go-routines run during the sync configurable for both non-OCI and OCI (with the current non-OCI defaults unchanged), then
• ensure that an OCI repository use a default of 1 (but can be updated by users - we may later need to make this a per-repo setting, but not now)
• Add a configurable rate-limit that will be used for each go-routine, again, being configurable for both cases with current settings unchanged and the oci one being a sane default for dockerhub
• Update the cronjob interval so that the default for an oci repo is somehow sane based on at least bring it to the users' attention.
• Don't sync all tags, but sync rest on demand?
• Avoid re-importing chart versions which are already imported.
• Avoid filtering all tags every time - there's no need to do so and it is very expensive.
• Update to stop the sync job if a 429 is received, with a useful error logged.

I think we can also, potentially, remove the checksum when handling an OCI catalog (which queries all tags of all apps), since we need to do this query when syncing an app anyway, so combining that with the above point to only sync the versions which are not already present in the DB, we can avoid the extra overhead.