• kernel-core
• kernel-devel
• kernel-headers
• kernel-modules
• kernel-modules-extra

• kernel-rt-core
• kernel-rt-devel
• kernel-rt-modules
• kernel-rt-modules-extra

• elfutils-libelf-devel
• kmod
• binutilskabi-dw
• kernel-abi-whitelists
• dependencies for the above

Prerequisites

• You obtained the image This content is not included.pull secret from Red Hat OpenShift Cluster Manager.
• You installed the OpenShift CLI (oc).

Prerequisites

• You have a running OpenShift Container Platform cluster.
• You set the Image Registry Operator state to Managed for your cluster.
• You installed the OpenShift CLI (oc).
• You are logged into the OpenShift CLI as a user with cluster-admin privileges.

Prerequisites

• You have access to an OpenShift Container Platform cluster.
• You installed the OpenShift CLI (oc).
• You are logged in as a user with cluster-admin privileges.

Procedure

• Method 1: Install the NFD Operator by using the CLI:

  1. Create the following Namespace custom resource (CR) that defines the openshift-nfd namespace, and then save the YAML in the nfd-namespace.yaml file. Set cluster-monitoring to "true".

     apiVersion: v1
     kind: Namespace
     metadata:
       name: openshift-nfd
       labels:
         name: openshift-nfd
         openshift.io/cluster-monitoring: "true"

     Copy code to clipboard

  2. Create the namespace by running the following command:

     $ oc create -f nfd-namespace.yaml

     Copy code to clipboard

  3. Create the following OperatorGroup CR and save the YAML in the nfd-operatorgroup.yaml file:

     apiVersion: operators.coreos.com/v1
     kind: OperatorGroup
     metadata:
       generateName: openshift-nfd-
       name: openshift-nfd
       namespace: openshift-nfd
     spec:
       targetNamespaces:
       - openshift-nfd

     Copy code to clipboard

  4. Create the OperatorGroup CR by running the following command:

     $ oc create -f nfd-operatorgroup.yaml

     Copy code to clipboard

  5. Create the following Subscription CR and save the YAML in the nfd-sub.yaml file:

     Example Subscription

     apiVersion: operators.coreos.com/v1alpha1
     kind: Subscription
     metadata:
       name: nfd
       namespace: openshift-nfd
     spec:
       channel: "stable"
       installPlanApproval: Automatic
       name: nfd
       source: redhat-operators
       sourceNamespace: openshift-marketplace

     Copy code to clipboard

  6. Create the subscription object by running the following command:

     $ oc create -f nfd-sub.yaml

     Copy code to clipboard

  7. Change to the openshift-nfd project:

     $ oc project openshift-nfd

     Copy code to clipboard

• Method 2: Install the NFD Operator by using the web console:

  1. In the OpenShift Container Platform web console, click Ecosystem → Software Catalog.
  2. Choose Node Feature Discovery from the list of available Operators, and then click Install.
  3. On the Install Operator page, select A specific namespace on the cluster, and then click Install. You do not need to create a namespace because it is created for you.

Verification

• To verify a CLI installation, run the following command and confirm that the output shows a Running status:

  $ oc get pods

  Copy code to clipboard

  Example output

  NAME                                      READY   STATUS    RESTARTS   AGE
  nfd-controller-manager-7f86ccfb58-vgr4x   2/2     Running   0          10m

  Copy code to clipboard

• To verify a web console installation, navigate to the Ecosystem → Installed Operators page and ensure that Node Feature Discovery is listed in the openshift-nfd project with a Status of InstallSucceeded.

  Note

  During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

Prerequisites

• You have access to an OpenShift Container Platform cluster.
• You have installed the OpenShift CLI (oc).
• You have logged in as a user with cluster-admin privileges.
• You have installed the NFD Operator.

Procedure

• To enable the Topology Updater workers in NFD, set the topologyupdater variable to true in the NodeFeatureDiscovery CR, as described in the section Using the Node Feature Discovery Operator.

Procedure

• To modify any setting, create a ConfigMap with the name kmm-operator-manager-config in the Operator namespace with the relevant data and restart the controller using the following command:

  $ oc rollout restart -n "$namespace" deployment/kmm-operator-controller

  Copy code to clipboard

  The value of $namespace depends on your installation method. For example:

  apiVersion: v1
  data:
    controller_config.yaml: |
      worker:
        firmwareHostPath: /example/different/firmware/path
  kind: ConfigMap
  metadata:
    name: kmm-operator-manager-config
    namespace: openshift-kmm

  Copy code to clipboard
  Note

  If you want to configure KMM Hub, create the ConfigMap using the name kmm-operator-hub-manager-config in the KMM Hub controller’s namespace.

  Table 4.1. Operator configuration parameters

  Parameter	Description
  healthProbeBindAddress	Defines the address on which the Operator monitors for kubelet health probes. The recommended value is :8081.
  job.gcDelay	Defines the duration for which successful build pods should be preserved before they are deleted. For information about the valid values for this setting, see Content from pkg.go.dev is not included.ParseDuration. The default value is 0s.
  leaderElection.enabled	Determines whether leader election is used to ensure that only one replica of the KMM Operator is running at any time. For more information, see Content from kubernetes.io is not included.Leases. The default value is true.
  leaderElection.resourceID	Determines the name of the resource that leader election uses for holding the leader lock. The default value for KMM is kmm.sigs.x-k8s.io. The default value for KMM-hub is kmm-hub.sigs.x-k8s.io.
  metrics.bindAddress	Determines the bind address for the metrics server. Set this to "0" to disable the metrics server. The default value is 0.0.0.0:8443.
  metrics.disableHTTP2	If true, disables HTTP/2 for the metrics server as a mitigation for This content is not included.CVE-2023-44487. The default value is true.
  metrics.enableAuthnAuthz	Determines if metrics are authenticated using TokenReviews and authorized using SubjectAccessReviews with the kube-apiserver. For authentication and authorization, the controller needs a ClusterRole with the following rules: apiGroups: authentication.k8s.io, resources: tokenreviews, verbs: create apiGroups: authorization.k8s.io, resources: subjectaccessreviews, verbs: create To scrape metrics, for example, using Prometheus, the client needs a ClusterRole with the following rule: nonResourceURLs: "/metrics", verbs: get The default value is true.
  metrics.secureServing	Determines whether the metrics are served over HTTPS instead of HTTP. The default value is true.
  webhook.disableHTTP2	If true, disables HTTP/2 for the webhook server, as a mitigation for This content is not included.CVE-2023-44487. The default value is true.
  webhook.port	Defines the port on which the Operator monitors webhook requests. The default value is 9443.
  worker.runAsUser	Determines the value of the runAsUser field of the worker container’s security context. For more information, see Content from kubernetes.io is not included.SecurityContext. The default value is 9443.
  worker.seLinuxType	Determines the value of the seLinuxOptions.type field of the worker container’s security context. For more information, see Content from kubernetes.io is not included.SecurityContext. The default value is spc_t.
  worker.firmwareHostPath	If set, the value of this field is written by the worker container into the /sys/module/firmware_class/parameters/path file on the node. For more information see Content from openshift-kmm.netlify.app is not included.Setting the kernel’s firmware search path. The default value is /var/lib/firmware.

• Labels that match the module’s .spec.selector field.
• A kernel version matching one of the items in the module’s .spec.moduleLoader.container.kernelMappings field.
• If ordered upgrade (ordered_upgrade.md) is configured in the module, a label that matches its .spec.moduleLoader.container.version field.

• Pulls the kmod image configured in the Module resource. Kmod images are standard OCI images that contain .ko files.
• Extracts the image in the pod’s filesystem.
• Runs modprobe with the specified arguments to perform the necessary action.

• In most cases, <prefix> should be equal to /opt. This is the Module CRD’s default value.
• kernel-version must not be empty and must be equal to the kernel version the kernel modules were built for.

Prerequisites

• The keySecret and certSecret secrets have been created in the same namespace as the rest of the resources.

Procedure

• Apply the YAML file:

  ---
  apiVersion: kmm.sigs.x-k8s.io/v1beta1
  kind: Module
  metadata:
    name: example-module
  spec:
    moduleLoader:
      serviceAccountName: default
      container:
        modprobe:
          moduleName: '<module_name>'
        kernelMappings:
          # the kmods will be deployed on all nodes in the cluster with a kernel that matches the regexp
          - regexp: '^.*\.x86_64$'
            # the container to produce containing the signed kmods
            containerImage: <container_image_name>
            sign:
              # the image containing the unsigned kmods (we need this because we are not building the kmods within the cluster)
              unsignedImage: <unsigned_image_name>
              keySecret: # a secret holding the private secureboot key with the key 'key'
                name: <private_key_secret_name>
              certSecret: # a secret holding the public secureboot key with the key 'cert'
                name: <certificate_secret_name>
              filesToSign: # full path within the unsignedImage container to the kmod(s) to sign
                - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko
    imageRepoSecret:
      # the name of a secret containing credentials to pull unsignedImage and push containerImage to the registry
      name: repo-pull-secret
    selector:
      kubernetes.io/arch: amd64

  Copy code to clipboard

  where:

  <module_name>

  Specifies the name of the kmod to load.

  <container_image_name>

  Specifies the name of the container image. For example, quay.io/myuser/my-driver:<kernelversion.

  <unsigned_image_name>

  Specifies the name of the unsigned image. For example, quay.io/myuser/my-driver:<kernelversion.

Prerequisites

• The keySecret and certSecret secrets have been created in the same namespace as the rest of the resources.

Procedure

• Apply the YAML file:

  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: example-module-dockerfile
    namespace: <namespace>
  data:
    dockerfile: |
      ARG DTK_AUTO
      ARG KERNEL_VERSION
      FROM ${DTK_AUTO} as builder
      WORKDIR /build/
      RUN git clone -b main --single-branch https://github.com/rh-ecosystem-edge/kernel-module-management.git
      WORKDIR kernel-module-management/ci/kmm-kmod/
      RUN make
      FROM registry.access.redhat.com/ubi9/ubi:latest
      ARG KERNEL_VERSION
      RUN yum -y install kmod && yum clean all
      RUN mkdir -p /opt/lib/modules/${KERNEL_VERSION}
      COPY --from=builder /build/kernel-module-management/ci/kmm-kmod/*.ko /opt/lib/modules/${KERNEL_VERSION}/
      RUN /usr/sbin/depmod -b /opt
  ---
  apiVersion: kmm.sigs.x-k8s.io/v1beta1
  kind: Module
  metadata:
    name: example-module
    namespace: <namespace>
  spec:
    moduleLoader:
      serviceAccountName: default
      container:
        modprobe:
          moduleName: simple_kmod
        kernelMappings:
          - regexp: '^.*\.x86_64$'
            containerImage: <final_driver_container_name>
            build:
              dockerfileConfigMap:
                name: example-module-dockerfile
            sign:
              keySecret:
                name: <private_key_secret_name>
              certSecret:
                name: <certificate_secret_name>
              filesToSign:
                - /opt/lib/modules/4.18.0-348.2.1.el8_5.x86_64/kmm_ci_a.ko
    imageRepoSecret:
      name: repo-pull-secret
    selector: # top-level selector
      kubernetes.io/arch: amd64

  Copy code to clipboard

  where:

  metadata.namespace

  Specifies the namespace where the module will be deployed.

  spec.moduleLoader.serviceAccountName

  Specifies the service account that will be used to run the module. The default service account does not have the required permissions to run a module that is privileged. For information on creating a service account, see "Creating service accounts".

  spec.imageRepoSecret

  Specifies that it is used as imagePullSecrets in the DaemonSet object and to pull and push for the build and sign features.

• If the operator parameter is set to Equal:

  • the key parameters are the same;
  • the value parameters are the same;
  • the effect parameters are the same.

• If the operator parameter is set to Exists:

  • the key parameters are the same;
  • the effect parameters are the same.

Prerequisites

• Before upgrading, set the kmm.node.kubernetes.io/version-module.<module_namespace>.<module_name>=$moduleVersion label on all the nodes that are used by the kernel module.
• End all user application workloads on the node or move them to another node.
• Unload the currently loaded kernel module.
• Ensure that the user workload (the application running in the cluster that is accessing kernel module) is not running on the node before the kernel module unloads and that the workload is back running on the node after the new kernel module version has been loaded.

• In this release, wildcard and glob pattern support for the filesToSign field has been added for signing kernel modules in a specific folder. Previously, you had to specify the exact path to each .ko file you wanted to sign. Now, you can add the full path to explicit files, as previously required, or any glob patterns supported by the Ash shell.

  The DirName value from moduleLoader.container.modprobe is propagated into the sign image. The validation webhook also verifies that all filesToSign entries fall under the configured DirName.

  For more information, see Specifying files to sign.

• In this release, you can use the AutomountServiceAccountToken parameter to disable the auto-mounting of the projected volume. You can set AutomountServiceAccountToken to false to disable auto-mounting and mount the configmaps and tokens necessary for the DevicePlugin application.

  Kubernetes automatically mounts the service account token and root Certificate Authoritys (CAs) into the /var/run/secrets/kubernetes.io/serviceaccount of the device-plugin pods using projected volumes. In some cases, you may want to use additional custom CAs or tokens for the device-plugin but Kubernetes does not allow mounting them to the same path unless the auto-mount is disabled.

• In this release, new security settings have been added to prevent filesystem tampering and remove privileged kernel operations. The new settings enhance container security by restricting system capabilities and enabling read-only root filesystems across manager, webhook-server, and Operator components to strengthen security posture and reduce attack surface. For more information, see Content from redhat.atlassian.net is not included.KMM RapiDAST "Root file system is not read-only" while checking DAST.

• In this release, when installing KMM using OLM, you can add additional tolerations to the Operator where worker nodes have custom taints or no accessible control-plane nodes.

  By default, the KMM Operator is installed on control plane nodes when possible and includes tolerations that allow for the KMM Operator to be scheduled on the nodes. In environments where the control plane is not accessible, the KMM Operator is installed on worker nodes.

• In this release, a new optional imageRebuildTriggerGeneration counter field has been added in the Module, ManagedClusterModule, and ModuleImagesConfig CRDs that allows you to force Kernel Module Management (KMM) to reverify and rebuild module images when using ephemeral image registries. When this counter’s value changes, the system automatically clears cached image statuses and reverifies the image existence, potentially triggering rebuilds.

• Unwarranted kernel module removal due to memory and disk limitations.

  • Cause: Memory and disk limitations can cause the removal of kernel modules.
  • Consequence: The kernel module is removed.
  • Fix: This release contains a new set of internal tolerations such as DiskPressure, MemoryPressure, and PIDPressure that are propagated through the module loader data flow and node selection process. These internal tolerations are appended to module-specific tolerations when handling module scheduling.
  • Result: The internal tolerations improve module scheduling to account for additional system resource pressure conditions. Modules can now be deployed across a broader range of cluster nodes, enhancing resource utilization and deployment flexibility.

• Kernel Module Management (KMM) cannot delete module names that contain a ..

  • Cause: The Kernel Module Management (KMM) Operator cannot delete modules that contain a . in their name.
  • Consequence: The module intended for deletion hangs indefinitely.
  • Fix: The finalizer regexp used to remove the node label has been modified to complete deletion.

  • Result: When the node label has been correctly deleted, the module deletes successfully.

    For more information, see Content from redhat.atlassian.net is not included.MGMT-19647.

• Build and sign pods do not inherit Module tolerations.

  • Cause: Build and sign pods created through the ModuleImagesConfig (MIC) and ModuleBuildSignConfig (MBSC) flows do not inherit tolerations from the parent Module.
  • Consequence: Build pods fail scheduling on nodes with custom taints. Build pods should have the same tolerations as defined in the Module.spec.tolerations parameter, allowing them to schedule on tainted nodes where kernel headers are available.
  • Fix: Added a Tolerations field to module build and image configurations that specifies the tolerations for build and sign pods.
  • Result: You can now specify tolerations for build and sign pods, enabling control over pod scheduling on nodes with taints. This fix supports standard toleration properties including effect, key, operator, and duration settings.

• Issues have been encountered when loading out-of-tree (OOT) drivers for QDU x100 DU PCIe cards from v4.14. For more information, see This content is not included.Case 04245147.

• Kernel Module Management (KMM) version 2.5 does not run on Red Hat OpenShift Service on AWS (ROSA) clusters or any other cluster that doesn’t install the MachineConfig CRD.

  • Cause: This happens because the BMC controller that monitors MachineConfig objects on clusters cannot find these objects on ROSA clusters because they do not exist.
  • Consequence: Causes the BMC controller to fail and the KMM controller pods to continually restart.
  • Fix: In this version, the Operator verifies that the MachineConfig CRD is present on a cluster and runs the BMC controller on a cluster only when the MachineConfig CRD is present.
  • Result: ROSA controller pods start successfully.

• Starting with this version, you can use the KMM Operator to manage the lifecycle of kmod images that you installed by using the Day 1 utility. When a Day 1 kmod image is transitioned to the KMM Operator by using a Module, a BootMachineConfig (BMC) CRD is also created in the cluster. The BMC CRD fixes sudden reboot issues by ensuring that the MachineConfig gets updated with the correct values without triggering a node reboot. For more information, see Managing Day 1 kmod images.

• The Kernel Module Management Operator (KMMO) 2.5 provides a version.ready label to indicate that the new version of the kernel module is loaded and is ready to use. For more information, see Customizing upgrades for kernel modules.

• KMM Operator support on IBM Power architecture

  RHEL does not provide a real-time Kernel for IBM Power, so do not deploy or validate any real-time features for KMM 2.5 on IBM Power compute nodes.

• KMM Operator support on IBM Z architecture

  Kernel Module Management (KMM) 2.5 is now supported on IBM Z architecture. However, RHEL does not provide a real-time Kernel for IBM Z. Therefore, you should not deploy or validate any real-time features for KMM 2.5 on IBM Z worker nodes.

• PreflightValidationOCP from KMM 2.4 does not synchronize status between v1beta1 and v1beta2.

  • Cause: This happens because the v1beta1 -→ v1beta2 conversion webhook was not well defined in the CRD.
  • Consequence: The status only shows in v1beta2 and not in v1beta1.
  • Fix: A conversion between v1beta1 and v1beta2 has been added into the PreflightValidationOCP CRD.
  • Result: The PreflightValidationOCP status is now shown in v1beta1 and v1beta2.

• PreflightValidationOCP in KMM 2.4 incorrectly pushes images to registry.

  • Cause: PreflightValidationOCP in KMM 2.4 version is pushing images to the registry despite the pushBuiltImage: false setting.
  • Consequence: The new PreflightValidationOCP for KMM 2.4 ignores the pushBuiltImage: false and pushes it to registry.
  • Fix: The internal logic has been updated to ensure the push behavior for built images is correctly propagated across all relevant workflows.
  • Result: Built images are no longer incorrectly pushed to the registry when pushBuiltImage: false.

• The kmm-operator-controller pod encounters OOMKilled errors.

  • Cause: The kmm-operator-controller pod repeatedly encounters OOMKilled errors despite having a 1Gi memory limit. This issue occurs during cycles of installing and uninstalling the x100 Operator and associated kernel modules, suggesting a potential memory leak related to module management.
  • Consequence: This issue occurs even though the container has both resources.limits.memory and resources.requests.memory set to 1Gi. After several cycles, the manager container is repeatedly terminated by OOMKilled, despite having a 1Gi memory limit. The pod status shows Running with a 0/1 ready state, and the restart count grows continuously.
  • Fix: A fix that completely removes all code and configuration related to managing Cluster and Service CA ConfigMaps has been implemented. Update to KMM 2.4 or higher.
  • Result: Installing and uninstalling the x100 operator and associated kernel modules runs as expected.

• OpenShift 4.20 includes Kernel Module Management (KMM) Operator version 2.3.0

  • Cause: The OpenShift 4.20 catalog includes Kernel Module Management (KMM) Operator version 2.3.0 as the latest version instead of the required KMM versions 2.4.0 and 2.4.1.
  • Consequence: The documentation was outdated and required an update.
  • Fix: The 4.20 catalogues have been updated to include KMM Operator versions 2.4.0 and 2.4.1.
  • Result: The catalogues are now up-to-date.

• HashAnnotationDiffer function could produce unexpected results

  • Cause: A potential bug in the HashAnnotationDiffer function within the Kernel Module Management (KMM) could be exposed by a change in implementation.
  • Consequence: While this potential bug is currently mitigated by the NMC logic, a change in implementation could expose this bug in real-time.
  • Fix: The HashAnnotationDiffer method in internal/pod/workerpodmanager.go was updated to correctly handle cases where both input pods are nil or only one is nil. The logic now returns false (no difference) when both are nil, and true (difference) when only one is nil.
  • Result: The HashAnnotationDiffer function runs as expected.

• In this release, you now have the option to configure the Kernel Module Management (KMM) module to not load an out-of-tree kernel driver and use the in-tree driver instead, and run only the device plugin. For more information, see Using in-tree modules with the device plugin.

• In this release, KMM configurations are now persistent after cluster and KMM Operator upgrades and redeployments of KMM.

  In earlier releases, a cluster or KMM upgrade, or any other action, such as upgrading a non-default configuration like the firmware path that redeploys KMM, could create the need to reconfigure KMM. In this release, KMM configurations now remain persistent regardless of any of such actions.

  For more information, see Configuring the Kernel Module Management Operator.

• Improvements have been added to KMM so that GPU Operator vendors do not need to replicate KMM functionality in their code, but instead use KMM as is. This change greatly improves Operators' code size, tests, and reliability.

• In this release, KMM no longer uses HTTP(S) direct requests to check if a kmod image exists. Instead, CRI-O is used internally to check for the images. This mitigates the need to access container image registries directly from HTTP(S) requests and manually handle tasks such as reading /etc/containers/registries.conf for mirroring configuration, accessing the image cluster resource for TLS configuration, mounting the CAs from the node, and maintaining your own cache in Hub & Spoke.

• The KMM and KMM-hub Operators have been assigned the "Meets Best Practices" label in the Red Hat Catalog.

• You can now install KMM on compute nodes, if needed. Previously, it was not possible to deploy workloads on the control-plane nodes. Because the compute nodes do not have the node-role.kubernetes.io/control-plane or node-role.kubernetes.io/master labels, the Kernel Module Management Operator might need further configurations. An internal code change has resolved this issue.

• In this release, the heartbeat filter for the NMC reconciler has been updated to filter the following events on nodes:

  • node.spec
  • metadata.labels
  • status.nodeInfo
  • status.conditions[] (NodeReady only) and still filtering heartbeats

• In this release, the preflight validation resource in the cluster has been modified. You can use the preflight validation to verify kernel modules to be installed on the nodes after cluster upgrades and possible kernel upgrades. Preflight validation also reports on the status and progress of each module in the cluster that it attempts or has attempted to validate. For more information, see Preflight validation for Kernel Module Management (KMM) Modules.

• A requirement when creating a kmod image is that both the .ko kernel module files and the cp binary must be included, which is required for copying files during the image loading process. For more information, see Creating a kmod image.

• The capabilities field that refers to the Operator maturity level has been changed from Basic Install to Seamless upgrades. Basic Install indicates that the Operator does not have an upgrade option. This is not the case for KMM, where seamless upgrades are supported.

• Webhook deployment has been renamed from webhook-server to webhook.

  • Cause: Generating files with controller-gen generated a service called webhook-service that is not configurable. And, when deploying KMM with Operator Lifecycle Manager (OLM), OLM deploys a service for the webhook called -service.
  • Consequence: Two services were generated for the same deployment. One generated by controller-gen and added to the bundle manifests and the other that the OLM created.
  • Fix: Make OLM find an already existing service called webhook-service in the cluster because the deployment is called webhook.
  • Result: A second service is no longer created.

• Using imageRepoSecret object in conjunction with DTK as the image stream results in authorization required error.

  • Cause: On the Kernel Module Management (KMM) Operator, when you set imageRepoSecret object in the KMM module, and the build’s resulting container image is defined to be stored in the cluster’s internal registry, the build fails to push the final image and generate an authorization required error.
  • Consequence: The KMM Operator does not work as expected.
  • Fix: When the imageRepoSecret object is user-defined, it is used as both a pull and push secret by the build process. To support using the cluster’s internal registry, you must add the authorization token for that registry to the imageRepoSecret object. You can obtain the token from the "build" service account of the KMM module’s namespace.
  • Result: The KMM Operator works as expected.

• Creating or deleting the image or creating an MCM module does not load the module on the spoke.

  • Cause: In a hub and spoke environment, when creating or deleting the image in registry, or when creating a ManagedClusterModule (MCM), the module on the spoke cluster is not loaded.
  • Consequence: The module on the spoke is not created.
  • Fix: Remove the cache package and image translation from the hub and spoke environment.
  • Result: The module on the spoke is created for the second time the MCM object is created.

• KMM cannot pull images from the private registry while doing in-cluster builds.

  • Cause: The Kernel Module Management (KMM) Operator cannot pull images from private registry while doing in-cluster builds.
  • Consequence: Images in private registries that are used in the build process can not be pulled.
  • Fix: The imageRepoSecret object configuration is now also used in the build process. The imageRepoSecret object specified must include all registries that are being used.
  • Result: You can now use private registries when doing in-cluster builds.

• KMM worker pod is orphaned when deleting a module with a container image that can not be pulled.

  • Cause: A Kernel Module Management (KMM) Operator worker pod is orphaned when deleting a module with a container image that can not be pulled.
  • Consequence: Failing worker pods are left on the cluster and at no point being collected for garbage.
  • Fix: KMM, now collects orphaned failing pods upon the modules deletion for garbage.
  • Result: The module is successfully deleted, and all associated orphaned failing pods are also deleted.

• The KMM Operator tries to create a MIC even when the node selector does not match.

  • Cause: The Kernel Module Management (KMM) Operator tries to create a 'ModuleImagesConfig' (MIC) resource even when the node selector does not match with any actual nodes and fails.
  • Consequence: The KMM Operator reports an error when reconciling a module that does not target any node.
  • Fix: The Images field in the MIC resource is now optional.
  • Result: The KMM Operator can successfully create the MIC resource even when there are no images in it.

• KMM does not reload the kernel module in case the node reboot sequence is too quick.

  • Cause: The Kernel Module Management (KMM) Operator does not reload the kernel module in case the node reboot sequence is too quick. The reboot is determined based on the timestamp of the status condition being later than the timestamp in the Node Machine Configuration (NMC) status.
  • Consequence: When the reboot happens quickly, in less time than the grace period, the node state does not change. After the node reboots, KMM does not load the kernel module again.
  • Fix: Instead of relying on the condition state, NMC can rely on the Status.NodeInfo.BootID field. This field is set by kubelet based on the /proc/sys/kernel/random/boot_id file of the server node, so it is updated after each reboot.
  • Result: The more accurate timestamps enable the Kernel Module Management (KMM) Operator to reload the kernel module after the node reboot sequence.

• Filtering out node heartbeats events for the Node Machine Configuration (NMC) controller.

  • Cause: The NMC controller gets spammed with events from node heartbeats. The node heartbeats let the Kubernetes API server know that the node is still connected and functional.
  • Consequence: The spamming causes a constant reconciliation even when there is no module, and therefore no NMC, are applied to the cluster.
  • Fix: The NMC controller now filter the node’s heartbeat from its reconciliation loop.
  • Result: The NMC controller only gets real events and filters out node heartbeats.

• NMC status contains toleration values, even though there are no tolerations in the NMC.spec or in the module.

  • Cause: The Node Machine Configuration (NMC) status contains toleration values, even though there are no tolerations in the NMC.spec or in the module.
  • Consequence: Tolerations other than Kernel Module Management-specific tolerations can appear in the status.
  • Fix: The NMC status now gets its toleration from a dedicated annotation rather than from the worker pod.
  • Result: The NMC status only contains the module’s tolerations.

• The KMM Operator version 2.4 fails to start properly and cannot list the \modulebuildsignconfigs\ resource.

  • Cause: On the Kernel Module Management (KMM) Operator, when the Operator is installed using Red Hat Konflux, it does not start properly because the log files contain errors.
  • Consequence: The KMM Operator does not work as expected.
  • Fix: The Cluster Service Version (CSV) file is updated to list the \modulebuildsignconfigs\ and the moduleimagesconfig resources .
  • Result: The KMM Operator works as expected.

• The Red Hat Konflux build does not include version and git commit ID in the Operator logs.

  • Cause: On the Kernel Module Management (KMM) Operator, when the Operator was built using Communications Platform as a Service (CPaas), the build included the Operator version and git commit ID in the log files. However, with Red Hat Konflux these details are not included in the log files.
  • Consequence: Important information is missing from the log files.
  • Fix: Some modifications are introduced in Konflux to resolve this issue.
  • Result: The KMM Operator build now includes the Operator version and git commit ID in the log files.

• The KMM Operator does not load the module after node with taint is rebooted.

  • Cause: The Kernel Module Management (KMM) Operator does not reload the kernel module in case the node reboot sequence is too quick. The reboot is determined based on the timestamp of the status condition being later than the timestamp in the Node Machine Configuration (NMC) status.
  • Consequence: When the reboot happens quickly, in less time than the grace period, the node state does not change. After the node reboots, KMM does not load the kernel module again.
  • Fix: Instead of relying on the condition state, NMC can rely on the Status.NodeInfo.BootID field. This field is set by kubelet based on the /proc/sys/kernel/random/boot_id file of the server node, so it is updated after each reboot.
  • Result: The more accurate timestamps enable the Kernel Module Management (KMM) Operator to reload the kernel module after the node reboot sequence.

• Redeploying a module that uses in-cluster builds fails with the ImagePullBackOff policy.

  • Cause: On the Kernel Module Management (KMM) Operator, the image pull policy for the puller pod and the worker pod is different.
  • Consequence: An image can be considered as existing while, in fact, it is not.
  • Fix: Make the image pull policy of the pull pod the same as the pull policy defined in the KMM module since its the same policy that is used by the worker pod.
  • Result: The MIC represents the state of the image in the same way the worker pod accesses it.

• The MIC controller creates two pull-pods when it should create just one.

  • Cause: On the Kernel Module Management (KMM) Operator, the ModuleImagesConfig (MIC) controller may create multiple pull-pods for the same image.
  • Consequence: Resources are not used appropriately or as intended.
  • Fix: The CreateOrPatch MIC API receives a slice of ImageSpecs, as the input is created by going over the target nodes and adding their images to the slice, so any duplicate ImageSpecs, are now filtered out.
  • Result: The KMM Operator works as expected.

• The job.dcDelay example in the documentation should specify 0s instead of 0.

  • Cause: The Kernel Module Management (KMM) Operator default job.gcDelay duration field is 0s but the documentation mentions the value as 0.
  • Consequence: Entering a custom value of 60 instead of 60s or 1m might result in an error due to the wrong input type.
  • Fix: The job.gcDelay field in the documentation is updated to default value of 0s.
  • Result: Users are less likely to get confused.

• The KMM Operator Hub environment does not work because of missing MIC and MBSC CRDs.

  • Cause: The Kernel Module Management (KMM) Operator hub environment only generates Custom Resource Definitions (CRD) files based on the api-hub/ directory. As a result, this does not contain some CRDs that are required for the KMM Operator Hub environment, such as, ModuleImagesConfig (MIC) resource and Managed Kubernetes Service (MBSC).
  • Consequence: The KMM Operator hub environment cannot work because it tries to start controllers reconciling CRDs that do not exist in the cluster.
  • Fix: The fix generates all CRD files into the config/crd-hub/bases directory, but only applies the resources to the cluster that it actually needs.
  • Result: The KMM Operator hub environment works as expected.

• The KMM OperatorHub environment cannot build when finalisers are not set on a resource.

  • Cause: The Kernel Module Management (KMM) Operator displays an error with the ManagedClusterModule controller failing to build. This is due to the missing ModuleImagesConfig (MIC) resource finalizers and Role-based Action Control (RBAC) permissions for the KMM OperatorHub environment.
  • Consequence: The KMM OperatorHub environment cannot build images.
  • Fix: The RBAC permissions are updated to allow updating finalizers on the MIC resource, and then the appropriate rules created.
  • Result: The KMM OperatorHub environment builds images without errors with the ManagedClusterModule controller.

• The PreflightValidationOCP custom resource, with a kernelVersion: tesdt causes the KMM Operator to panic.

  • Cause: Creating a PreflightValidationOCP custom resource (CR), with a kernelVersion flag that is set to tesdt, causes the Kernel Module Management (KMM) Operator to generate a panic runtime error.
  • Consequence: Entering invalid kernel versions causes the KMM Operator to panic.
  • Fix: A webhook - a method for one application to automatically send real-time data to another application when a specific event occurs - is now added to the PreflightValidationOCP CR.
  • Result: The PreflightValidationOCP CR with invalid kernel versions can no longer be applied to the cluster, therefore, preventing the Operator from generating a panic runtime error.

• The PreFflightValidationOCP custom resource, with a kernelVersion flag that is different that the one of the cluster, does not work.

  • Cause: Creating a PreflightValidationOCP custom resource (CR), with a kernelVersion flag that is different from the one of the cluster, does not work.
  • Consequence: The Kernel Module Management (KMM) Operator is unable to find the Driver Toolkit (DTK) input image for the new kernel version.
  • Fix: You must use the PreflightValidationOCP CR and explicitly set the dtkImage field in the CR.
  • Result: Using the fields kernelVersion and dtkImage the feature can build installed modules for target OpenShift Container Platform versions.

• The KMM Operator version 2.4 documentation is updated with PreflightValidationOCP information.

  • Cause: Previously, when creating an PreflightValidationOCP CR, you were required to supply the release-image. This has now changed and you need to set the kernelVersion the dtkImage fields.
  • Consequence: The documentation was outdated and required an update.
  • Fix: The documentation is updated with the new support details.
  • Result: The KMM preflight feature is documented as expected.

• The ModuleUnloaded event does not appear when a module is Unloaded.

  • Cause: When a module is Loaded (using the create a ModuleLoad event) or Unloaded ` (using the create a `ModuleUnloaded event) the events might not appear. This happens when you load and unload the kernel module in a quick succession.
  • Consequence: The ModuleLoad and the ModuleUnloaded events might not appear in OpenShift Container Platform.
  • Fix: Introduce an alerting mechanism for this potential behavior and for awareness when working with modules.
  • Result: Not yet available.

• In this release, KMM uses version 1.23 of the Golang programming language to ensure test continuity for partners.

• You can now schedule KMM pods by defining taints and tolerations. For more information, see Using tolerations for kernel module scheduling.

• KMM is now using the CRI-O container engine to pull container images in the worker pod instead of using HTTP calls directly from the worker container. For more information, see Example Module CR.

• The Kernel Module Management (KMM) Operator images are now based on rhel-els-minimal container images instead of the rhel-els images. This change results in a greatly reduced image footprint, while still maintaining FIPS compliance.
• In this release, the firmware search path has been updated to copy the contents of the specified path into the path specified in worker.setFirmwareClassPath (default: /var/lib/firmware). For more information, see Example Module CR.

• For each node running a kernel matching the regular expression, KMM now checks if you have included a tag or a digest. If you have not specified a tag or digest in the container image, then the validation webhook returns an error and does not apply the module. For more information, see Example Module CR.