Manual podman pull fails within OCP 4 nodes

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • Trying to pull images with podman pull fails within OpenShift 4 nodes:

    sh-4.4# podman pull registry.redhat.io/openshift4/ose-logging-fluentd
Failed to pull image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:xxxxxxx": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials.

    sh-4.4# podman pull image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest
[...]
Trying to pull image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest...
Error: Error initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator:latest: Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift4/ose-cluster-logging-operator: unauthorized: authentication required

  • After a successful podman login, podman pull fails with unauthorized error.

Resolution

For running podman pull withing an OpenShift 4 node, it is needed to include the credentials for the registry with --authfile parameter. The registry credentials configured in the cluster via the global pull secret can be used with --authfile /var/lib/kubelet/config.json.

An example to pull an image from a node (change the [image_name] with the name of the required image):

$ oc get nodes
[...]
$ oc debug node/[node_name]
[...]

sh-4.4# chroot /host bash

[root@node_name /]# podman pull --authfile /var/lib/kubelet/config.json [image_name]

For additional troubleshooting, it is possible to set the log level to debug the following way:

# podman --log-level debug pull --authfile /var/lib/kubelet/config.json [image_name]

If still failing with credentials error when using the above credentials, configure a valid global pull secret as per how to change the global pull secret in OCP 4.

Root Cause

By default, podman login doesn't store the credentials, as they are in clear text, and podman pull will fail even after a successful podman login. Including the credentials for the registry with --authfile parameter will be needed.

Diagnostic Steps

Connect to an OCP 4 node:

$ oc get nodes
[...]
$ oc debug node/[node_name]
Creating debug namespace/openshift-debug-node-xxxxx ...
Starting pod/[node]-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.0.1
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host bash

Check that after a successful podman login withing a OCP 4 node, the pull fails:

[root@node_name /]# podman login
Login Succeeded!
[root@node_name /]# podman pull registry.redhat.io/openshift4/ose-cluster-logging-operator
[...]
Failed to pull image "registry.redhat.io/openshift4/ose-cluster-logging-operator@sha256:ed07723ee730fee6e4758410c65271dac0b035f89c14595ed4a92db4fb83f037": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials.
[...]

Try the podman pull with --authfile /var/lib/kubelet/config.json

[root@node_name /]# podman pull --authfile /var/lib/kubelet/config.json registry.redhat.io/openshift4/ose-cluster-logging-operator
Trying to pull registry.redhat.io/openshift4/ose-cluster-logging-operator...
Getting image source signatures
Copying blob xxxxxxxxxxxx skipped: already exists
[...]
Copying blob xxxxxxxxxxxx done
[...]
Writing manifest to image destination
Storing signatures
[...]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.