How to download Image Signatures within OpenShift 4

Environment

• OpenShift Container Platform
  • 4.1

NOTE: For versions >= 4.2 and/or a more permanent way using MachineConfigOperator, please refer to this article instead.

Issue

• In OpenShift 4 is not as easy as setting the proper sigstore file within "/etc/containers/registries.d/" to automatically download the signatures when importing an image (like within 3.x), the procedure could be done modifying the openshift-controller-manager operator as follows.

Resolution

• Change "openshift-controller-manager" operator to "Unmanaged" state:

$ oc patch openshiftcontrollermanagers.operator.openshift.io/cluster -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge
openshiftcontrollermanager.operator.openshift.io/cluster patched

NOTE: The unmanaged state of the operator will be reverted back in the last solution step, this is just a temporary modification.

• Create a proper sigstore file locally, for example:

$ cat registry.access.redhat.com.yaml
docker:
  registry.access.redhat.com:
    sigstore: https://access.redhat.com/webassets/docker/content/sigstore

OR

$ cat registry.redhat.io.yaml
docker:
  registry.redhat.io:
    sigstore: https://registry.redhat.io/containers/sigstore

• Create a configmap under "openshift-controller-manager" namespace:

$ oc create cm sigstore-config --from-file=$PATH/registry.access.redhat.com.yaml -n openshift-controller-manager
configmap/sigstore-config created

• Configure controller-manager to load this configmap:

$ oc set volume ds/controller-manager --add --type=configmap --configmap-name=sigstore-config -m /etc/containers/registries.d/ --name=sigstore-config -n openshift-controller-manager
daemonset.extensions/controller-manager volume updated

• Wait a pair of minutes until controller-manager pods are running again:

$ oc get pods -n openshift-controller-manager
NAME                       READY   STATUS    RESTARTS   AGE
controller-manager-64rpw   1/1     Running   0          8m43s
controller-manager-hnqj2   1/1     Running   0          8m3s
controller-manager-wsnct   1/1     Running   0          7m16s

• Import an image:

$ oc import-image registry.access.redhat.com/openshift3/ose:latest --confirm
imagestream.image.openshift.io/ose imported

Name:			ose
Namespace:		image-test
Created:		Less than a second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2019-10-04T14:55:34Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/image-test/ose
Image Lookup:		local=false
Unique Images:		1
Tags:			1
[...]

• Confirm the signatures presence:

$ oc describe istag ose:latest | grep -A3 "Signatures"
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@8fda666a579c77c5d5b10050c546f8e9a17a369fb1d1b397020d7aeff7d74925
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@6ee4746291188bc51176f575e9e6b6228d8b6282637fb52e1084fbfccace4b7a
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@72fdd2043313c4b1b8b30313a8957bff040486ceb5b0fcf03f340e1f4ebb4170
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@cb3801c027b4f12ecec3a4c8f4e569f036b4dc79555042538d6ecc31e2badfa0
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@e6ce29d8e74e9d517bb79271766178725d73b3728bbc03ce7ec0d4786e6228a1
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@19516ddb46a304698cfd6e39f913e34e310a114a8d054c68adebee33c5d0787c
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@52cc0e7db1a0a82e73adef6035b12e377c79d25cbcc4470ebc0d908c6660c2ac
			Type:	AtomicImageV1
			Status:	Unverified
Image Signatures:	 
			Name:	sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109@83d7cb14def3a19e2445c3a475210194ee638d5d5d2aa70dd188eb814b11c688
			Type:	AtomicImageV1
			Status:	Unverified

• (Optional) Verify image signatures:

$ oc adm verify-image-signature sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109 --expected-identity registry.access.redhat.com/openshift3/ose@sha256:7ef8d00f4c364732c81623c7fc1b99710e7a6cfe1950fde55ecd1a8f6fb1d109 --public-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release --save

NOTE: Using the --save flag on already verified image together with invalid GPG key or invalid expected identity will cause the saved verification status to be removed and the image will become "unverified", for that reason is strongly recommended to run this command first without --save option.

IMPORTANT NOTE: This signature verification command is broken since the addition of registry.redhat.io extra signatures and will not be fixed, please refer to this This content is not included.BZ#1705984 for more details if needed.

• Return the operator to "Managed" status to revert the changes and recover the upgrade capabilities (downloaded signatures will be retained):

$ oc patch openshiftcontrollermanagers.operator.openshift.io/cluster -p '{"spec":{"managementState": "Managed"}}' --type=merge
openshiftcontrollermanager.operator.openshift.io/cluster patched

Root Cause

• Image Signature automation within OCP4 is still something in progress, please refer to the following links for more information:

This content is not included.BZ#1705984
This content is not included.BZ#1722568

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.