Operator installation or upgrade fails in RHOCP 4

Issue

• Operator installation/upgrade fails stating:

Bundle unpacking failed. Reason: DeadlineExceeded, and Message: Job was active longer than specified deadline".

Resolution

This issue has been reported to Red Hat engineering. It is being tracked in Bug OCPBUGS-6771.

With the release of OCP 4.14.14 and 4.15.0 we have released the core functionality to fix the “bundle unpacking failed. Reason: DeadlineExceeded”.
There is an additional annotation to the operatorGroup for every to be installed operator needs to be added before applying it as follows: operatorframework.io/bundle-unpack-min-retry-interval

Failing jobs will auto-retry with the operatorGroup annotation "operatorframework.io/bundle-unpack-min-retry-interval" set to a non-zero interval like "5m".

Prerequisite

Download jq

Workarounds

For issues related to operator installation:

1. Find the corresponding job and configmap (usually named the same) in the openshift-marketplace and grep for the operator name or keyword in its contents.

$ oc get job -n openshift-marketplace -o json | jq -r '.items[] | select(.spec.template.spec.containers[].env[].value|contains ("<operator_name_keyword>")) | .metadata.name'

2. Delete the found job and the corresponding configmap, (named the same as the job) found in the openshift-marketplace namespace.

$ oc delete job <job-name> -n openshift-marketplace
$ oc delete configmap <job-name> -n openshift-marketplace

3. Uninstall the failed operator installation using the web UI from the associated namespace.

4. Delete the install plan created.

Note: Validate the InstallPlan before deleting it, to make sure the Operator in scope is the only Operator listed in the InstallPlan (see How can Operators be updated independently from each other? for more details).

$ oc delete ip <operator_installplan_name> -n <user_namespace>

5. Now delete the subscription, and csv associated with the namespace.

$ oc delete sub <operator_subscription_name> -n <user_namespace>
$ oc delete csv <operator_csv_name> -n <user_namespace>

6. Reinstall the operator via the web UI and the installation should complete successfully.

For issues related to operator upgrade:

1. Find the corresponding job and configmap (usually named the same) in the openshift-marketplace and grep for the operator name or keyword in its contents.

$ oc get job -n openshift-marketplace -o json | jq -r '.items[] | select(.spec.template.spec.containers[].env[].value|contains ("<operator_name_keyword>")) | .metadata.name'

At some point, we only need to delete the corresponding jobs, the operator will re-installed. If the action is not working, you can proceed the next step.

2. Delete the found job and the corresponding configmap, (named the same as the job) found in the openshift-marketplace namespace.

$ oc delete job <job-name> -n openshift-marketplace
$ oc delete configmap <job-name> -n openshift-marketplace

Make sure the subscription, cluster service version and install plan is properly installed

oc get sub,csv,ip -n <namespace> | grep <your-operator-name>

Additional Information

I create my custom jq commands to retrieve job names and config map names in openshift-marketplace containing a specific keyword in environment variables or binary data. Because when you install some operator and there is exising job or confimap that stuck on your system, the new operator installation is got stuck too. So, makesure before you install new operators, there is no stuck jobs and configmap.

oc get job -n openshift-marketplace -o json | jq -r '.items[] | select(.spec.template.spec.containers[].env[].value|contains ("<operator_name_keyword>")) | .metadata.name'

Purpose: Retrieves all jobs in the openshift-marketplace namespace and filters them based on environment variable values.

oc get configmap -n openshift-marketplace -o json | jq -r '.items[] | select(.binaryData != null and (.binaryData | to_entries | map(.key) | any(test("<operator_name_keyword>")))) | .metadata.name'

Purpose: Retrieves all config maps in the openshift-marketplace namespace and filters based on the presence of a key in the binary data.

In both cases, replace <operator_name_keyword> with the keyword you’re interested in filtering by.

Resource :

Operator installation or upgrade fails with DeadlineExceeded in RHOCP 4 — Red Hat Customer Portal

Troubleshooting Operator issues — Troubleshooting | Support | OpenShift Container Platform 4.15