Asset Syncer job authorization failure

[luffykesh]

Describe the bug
Asset syncer fails due to authorization when syncing assets. I am using Google Artifact Repository (GAR). Below are the logs when the syncer runs.
I verified that the BASIC AuthrorizationHeader that is being logged is valid, it does not seem to be the issue.

Logs

I0103 10:50:04.511609       1 root.go:32] "The component 'asset-syncer' has been configured with" serverOptions={"DatabaseURL":"apps-postgresql:5432","DatabaseName":"assets","DatabaseUser":"postgres","DatabasePassword":"REDACTED","Debug":false,"Namespace":"<redacted>","OciRepositories":[<redacted>],"TlsInsecureSkipVerify":false,"FilterRules":"","PassCredentials":false,"UserAgent":"asset-syncer/2.9.0 (kubeapps/2.9.0)","UserAgentComment":"kubeapps/2.9.0","GlobalPackagingNamespace":"apps","KubeappsNamespace":"","AuthorizationHeader":"Basic <redacted>","DockerConfigJson":"","OCICatalogURL":""}
I0103 10:50:04.531332       1 sync.go:90] Current checksum: "3c9bdedbe0c3f0603335e3880636b6b98f9a7dc49b48ce12e6e4a81b4f7e001c". Previous checksum: "ae21f400068c0a9d7566eb53a58501efaa1403dda749c6b39c698e84df72a1b8"
I0103 10:50:04.531467       1 utils.go:962] Starting 10 file importer workers
I0103 10:50:04.714778       1 utils.go:765] Starting 10 workers for importing OCI charts
E0103 10:50:05.244177       1 utils.go:771] "unable to list tags" err="GET \"https://<redacted>/tags/list\": GET \"https://europe-west4-docker.pkg.dev/v2/token?scope=<redacted>&service=europe-west4-docker.pkg.dev\": response status code 403: denied: Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"<redacted>" (or it may not exist)"
E0103 10:50:05.244213       1 utils.go:772] unable to list tags: GET "https://<redacted>/tags/list": GET "https://europe-west4-docker.pkg.dev/v2/token?scope=<redacted>&service=europe-west4-docker.pkg.dev": response status code 403: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "<redacted>" (or it may not exist)
I0103 10:50:05.244275       1 postgresql_utils.go:96] Removing the following charts that are no longer present in the repo: '<redacted>'
I0103 10:50:05.244375       1 utils.go:975] Finished queueing icon jobs
I0103 10:50:05.244411       1 utils.go:984] Enqueuing chart file imports for first versions
I0103 10:50:05.244421       1 utils.go:993] Enqueuing chart file imports for remaining versions
I0103 10:50:05.244432       1 utils.go:1002] Waiting for file import workers to complete.
I0103 10:50:05.244464       1 utils.go:1005] File importing complete
I0103 10:50:05.289656       1 sync.go:162] Chart data syncing complete. Waiting for file imports to complete.
I0103 10:50:05.289700       1 sync.go:165] Repository synced, shallow=false
I0103 10:50:05.292218       1 sync.go:173] Stored repository update in cache, repo.URL= <redacted>
I0103 10:50:05.292248       1 sync.go:174] Successfully added the package repository <redacted> to database

Expected behavior
The Asset Syncer process the authorization config correctly and successfully syncs the charts.

Desktop (please complete the following information):

• Version - 2.9.0
• Kubernetes version - 1.27.4
• Package version - Kubeapps Helm Chart version: 14.1.1

[antgamdia]

Not sure, I haven't personally tested a GAR repo in Kubeapps in a while. Are you using the service account key? Or the short-lived access token? More at: https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key
Anyway, thanks for reporting; we'll try to reproduce it and see what's failing.

[luffykesh]

@antgamdia I'm using service account key for auth

[KihyeokK]

@antgamdia Hello! Is there any updates on this bug?

[antgamdia]

No, unfortunately we haven't had the chance to look at it yet as we have very little engineering bandwidth these days :(

[KihyeokK]

@antgamdia Just a question regarding this issue. We tried running kubeapps version 2.10.0 and the same issue persists. Then we tried running kubeapps version 2.10.0 while only changing asset-syncer image verison to 2.8.0, and things seem to run fine. Is running all other components in latest versions and only running asset-syncer in old version a discouraged way of using kubeapps?

[antgamdia]

It shouldn't be the preferred way to go, but (from memory) I don't think we had significant changes on the asset-syncer from 2.8.0 to 2.10.0, so it shouldn't be a problem if you do so.
It is worth looking into it... unfortunately, I'm no longer working for VMware/Broadcom and can't be of much help here :(

[KihyeokK]

@antgamdia That is unfortunate😢 Thank you for still replying with the update!!