Unable to verify UBI GPG signatures when the images are pulled from a local registry

Environment

• Red Hat Enterprise Linux (RHEL) 8.2
• podman 1.9.2

Issue

• How to verify that RH container images are genuinely coming from Red Hat when received via an local container registry ? (Artifactory, Nexus, Quay)
• When pulling down a container image, the error message None of the signatures were accepted, reasons: Signature for identity registry.redhat.io/XXX/XXX:XXX is not accepted. is displayed.

Resolution

• mirror the UBI image onto the local registry with skopeo:

    skopeo copy --src-creds xxx:xxxx --dst=creds xxx:xxx docker://registry.access.redhat.com/ubi7/ubi:7.9-193 docker://localhost:5000 ubi7/ubi:7.9-193 --remove-signatures
  

Notice the --remove-signatures flag as the private registry does not support storing the GPG signatures.

• define the public key for verifying the UBI

    podman image trust set -t reject default
    podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release -t signedBy registry.access.redhat.com
  

• verify the trust policy

    podman image trust show 
    default                      reject                                                
    registry.access.redhat.com   signedBy   security@redhat.com, security@redhat.com  
  

• define your private registry as a mirror of Red Hat's registry, for this add at this snippet at the end of /etc/containers/registries.conf

    [[registry]]
    prefix = "registry.access.redhat.com/ubi7"
    location = "localhost:5000/ubi7"
  

• comment in /etc/containers/registries.conf the line

    registries = ['registry.access.redhat.com', 'registry.redhat.io', 'docker.io', 'quay.io']
  

otherwise you'll get the error message mixing sysregistry v1/v2 is not supported

• define a custom signature store for the private registry, so that podman looks for the signature at the right place when pulling the image:

    cat /etc/containers/registries.d/localhost\:5000.yaml
    docker:
      localhost:5000:
      sigstore: https://access.redhat.com/webassets/docker/content/sigstore
  

• finally pull an image using its Red Hat image name and tag

     podman pull  registry.access.redhat.com/ubi7/ubi:7.9-193
  

If you add the --log-level=debug switch to the above command, you will notice that the image is pulled from the private registry, but the signatures are pulled from Red Hat's servers.
Notice this solution won't work in older podman versions as podman < 1.9 is affected by This content is not included.a bug where it is downloading the signatures from the wrong place.

Root Cause

It is not enough to have the same image digest for verifying the signature, because the signature of the image is done on image digest + docker reference (myrepo/myimage name), as explained in This content is not included.this blog post in the Introducing: Simple Signing chapter.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.