Unauthorized using cosign to verify AWS ECR image signature

[tianlang8158]

1. I have a cronjob to create a secret to store AWS ECR credential and save to connaisseur namespace.

Command used:
TOKEN=$(aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')
kubectl create secret docker-registry ecr-registry
--docker-server=https://<acccount_id>.dkr.ecr.ap-southeast-1.amazonaws.com
--docker-username=AWS
--docker-password="${TOKEN}"

2. And this secret is referenced from configmap connaisseur-config with:
   validators:

   name: cosign-ecr
   trust_roots:

   • key: |
     -----BEGIN PUBLIC KEY-----
     -----END PUBLIC KEY-----
     name: cosign-ecr
     type: cosign
     auth:
     secret_name: ecr-registry

3. However, the following authorization error still been hit when trying to run image from my ECR private repo:

Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Unexpected Cosign exception for image "<account_id>.dkr.ecr.ap-southeast-1.amazonaws.com/nginx:signed": error: GET https://<account_id>.dkr.ecr.ap-southeast-1.amazonaws.com/v2/nginx/manifests/signed: unexpected status code 401 Unauthorized: Not Authorized.

I have already verified the token stored in the secret have access to the image manifest URL.
May I seek advice on how should I troubleshoot further from here?

[ZacHaque]

              AWS_ACCOUNT=XXXXXXXXXXX
              AWS_REGION=ap-southeast-1
              DOCKER_REGISTRY_SERVER=https://${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
              DOCKER_USER=AWS
              DOCKER_PASSWORD=`aws ecr get-login --region ${AWS_REGION} --registry-ids ${AWS_ACCOUNT} | cut -d' ' -f6`
              kubectl delete secret ecr-registry|| true
              kubectl create secret docker-registry ecr-registry\
              --docker-server=$DOCKER_REGISTRY_SERVER \
              --docker-username=$DOCKER_USER \
              --docker-password=$DOCKER_PASSWORD \
              --docker-email=no@email.local
              kubectl rollout restart deploy connaisseur-deployment -n connaisseur

[ZacHaque]

This should help :) @tianlang8158

[xopham]

Neat solution @ZacHaque ! In fact, Connaisseur mounts the configjson into the container and thus restart is required. Really like the solution 🙏

[tianlang8158]

Thanks for the advice @ZacHaque
It seems the issue is because auth.secret_name was edited manually after helm release installed, when i update it in the values.yaml and re-install it with helm, it works..
So only when auth.secret_name been added to values.yaml, then it will mount it to /app/connaisseur-config/default/.docker

name: cosign-ecr
trust_roots:

key: |
-----BEGIN PUBLIC KEY-----
-----END PUBLIC KEY-----
name: cosign-ecr
type: cosign
auth:
secret_name: ecr-registry

Thanks again both!

[xopham]

@tianlang8158 correct and the line kubectl rollout restart deploy connaisseur-deployment -n connaisseur from @ZacHaque ensures that the actual restart is performed

[ZacHaque]

Yes, make sure the service account have rbac permission to restart deployment.