After Upgrading from OpenShift Pipelines to 1.9 the operator is no longer working due to a failing CRD conversion

Environment

• Red Hat OpenShift Pipelines 1.9

Issue

• The tekton-controller is throwing the following messages:

W0223 10:50:27.142577 1 reflector.go:424] knative.dev/pkg/controller/controller.go:732: failed to list *v1beta1.PipelineRun: conversion webhook for tekton.dev/v1alpha1, Kind=PipelineRun failed: conversion not supported for type [kind=PipelineRun group=tekton.dev version=v1alpha1]

• tektoninstallerset shows the following error:

Install failed with message: CustomResourceDefinition.apiextensions.k8s.io "clustertasks.tekton.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Resolution

Red Hat is aware about the problem and investigating a potential solution in This content is not included.SRVKP-2935. To raise questions or obtain further information, contact Red Hat Technical Support.

Workaround 1

Removing v1alpha1 from the respective storedVersions in the CustomResourceDefinition (CRD) using the below patch should prevent the situation from happening and thus allow a successful upgrade to Red Hat OpenShift Pipelines 1.9

  for object in tasks pipelines taskruns pipelineruns clustertasks; do
      INDEX=$(kubectl get crd ${object}.tekton.dev -o json  | jq '.status.storedVersions | map(. == "v1alpha1") | index(true)')
      kubectl patch crd ${object}.tekton.dev --subresource='status' --type=json -p="[{'op': 'remove', 'path': '/status/storedVersions/$INDEX'}]"
  done

Workaround 2

NOT RECOMMENDED workaround to remove all the Tekton related v1alpha1 resources from etcd:

ETCD=$(oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running" -o jsonpath="{.items[0].metadata.name}")

oc exec -n openshift-etcd $ETCD -c etcdctl -- etcdctl get / --prefix --keys-only | grep "kubernetes.io/tekton.dev" | while read KEY; do apiVersion=$(oc exec -n openshift-etcd $ETCD -c etcdctl -- sh -c "etcdctl get $KEY --print-value-only" | jq 'select(.apiVersion | contains ("alpha")).apiVersion' -r ) && if [ "$apiVersion" = "tekton.dev/v1alpha1" ]; then oc exec -n openshift-etcd $ETCD -c etcdctl -- etcdctl del $KEY; fi; done

Diagnostic Steps

• After the Red Hat OpenShift Pipelines 1.9 update seems completed, check the TektonInstallerSet using the below command to see whether all components have been successfully updated or are stuck with the message reported.

    $ oc get tektoninstallerset
    NAME                                      READY   REASON
    addon-custom-clustertask-6dbvm            True
    addon-custom-communityclustertask-m2lcs   True
    addon-custom-consolecli-6z4wr             True
    addon-custom-openshiftconsole-xn52r       True
    addon-custom-pipelinestemplate-jfndf      True
    addon-custom-triggersresources-rwdx2      True
    addon-versioned-clustertasks-1-6-wk92v    True
    addon-versioned-clustertasks-1-7-fdnlg    True
    addon-versioned-clustertasks-1-8-wlvtv    True
    addon-versioned-clustertasks-1.9-vwf2h    True
    pipeline-main-deployment-tzd7h            False   Webhook: requeue after: 10s
    pipeline-main-static-mqxm2                False   Install failed with message: CustomResourceDefinition.apiextensions.k8s.io "clustertasks.tekton.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
    pipeline-pre-blv9p                        True
    rhosp-rbac-pcprw                          True
    trigger-main-deployment-7qf4q             True
    trigger-main-static-x86zw                 True
    validating-mutating-webhoook-lt5h8        True
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.