OpenShift Data Foundation Multus support for existing cluster

Updated

Prerequisite

Before moving to the next stage, make sure to read about Multus requirements and architecture here in the Multus architecture for OpenShift Data Foundation before moving the cluster from a non-Multus network to a Multus network.

Configure nodes

As mentioned in the above doc, the recommended way to configure the node is using OpenShift NodeNetworkConfigurationPolicy, and this method requires the NMState Operator to be installed and enabled.

Example of nodeNetworkConfigurationPolicy(NNCP) of node compute-0 and node compute-0 annotation:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  annotations:
    nmstate.io/webhook-mutating-timestamp: "1743035283553699179"
  creationTimestamp: "2025-03-27T00:28:03Z"
  generation: 1
  name: ceph-public-net-shim-compute-0
  resourceVersion: "271933"
  uid: c049b2b8-0a41-4e34-8b07-28ef5321abc8
spec:
  desiredState:
    interfaces:
    - description: Shim interface used to connect host to OpenShift Data Foundation
        public Multus network
      ipv4:
        address:
        - ip: 192.168.252.1
          prefix-length: 24
        dhcp: false
        enabled: true
      mac-vlan:
        base-iface: ens224
        mode: bridge
        promiscuous: true
      name: odf-pub-shim
      state: up
      type: mac-vlan
  nodeSelector:
    kubernetes.io/hostname: compute-0
    node-role.kubernetes.io/worker: ""

Note NNCP are required only if the public network is going to be used.

Setup NAD (network attachment definition)

Note examples below include both public and cluster NAD, which is not required; choose according to the requirements.

Before setting up NAD, check the NAD requirements mentioned here and see here how to configure NAD

Example public-net

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  creationTimestamp: "2025-03-27T14:23:34Z"
  generation: 1
  name: public-net
  namespace: default
  resourceVersion: "261458"
  uid: bb3b38f9-8c56-41fb-acfc-f71015edda16
spec:
  config: '{"cniVersion": "0.3.1", "type": "macvlan", "master": "ens224", "mode":
    "bridge", "ipam": {"type": "whereabouts", "range": "192.168.20.0/24", "routes":
    [{"dst": "192.168.252.0/24"}]}}'

Example cluster-net

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  creationTimestamp: "2025-03-27T14:24:09Z"
  generation: 1
  name: cluster-net
  namespace: default
  resourceVersion: "261599"
  uid: 11e8f2e7-bbc6-4df7-9034-ee449ae5cfcc
spec:
  config: '{"cniVersion": "0.3.1", "type": "macvlan", "master": "ens224", "mode":
    "bridge", "ipam": {"type": "whereabouts", "range": "192.168.30.0/24"}}'

Run the validation tool

To run the validation tool, follow this doc and make sure you get this

RESULT: multus validation test succeeded

The output of the validation tool indicates that the network configurations are correct.

Edit StorageCluster CR

After the successful output of the multus validation tool, edit the storageCluster CR according to the requirements

    network:
      multiClusterService: {}
      provider: multus
      selectors:
        cluster: default/cluster-net
        public: default/public-net

The above example contains both cluster and public net.

Validate with Ceph output

Run the command ceph osd dump | grep osd, the output will look like

osd.0 up   in  weight 1 up_from 66 up_thru 100 down_at 62 last_clean_interval [10,61) [v2:192.168.20.8:6800/2327319507,v1:192.168.20.8:6801/2327319507] [v2:192.168.30.1:6800/2327319507,v1:192.168.30.1:6801/2327319507] exists,up a8bfa5e6-11db-4b6d-99e2-00902fd8c5bd
osd.1 up   in  weight 1 up_from 74 up_thru 102 down_at 70 last_clean_interval [14,69) [v2:192.168.20.9:6800/3990558992,v1:192.168.20.9:6801/3990558992] [v2:192.168.30.2:6800/3990558992,v1:192.168.30.2:6801/3990558992] exists,up 8e4cf4cd-3627-4a25-ba93-a75fffb017c7
osd.2 up   in  weight 1 up_from 78 up_thru 98 down_at 76 last_clean_interval [24,75) [v2:192.168.20.10:6800/2297210135,v1:192.168.20.10:6801/2297210135] [v2:192.168.30.3:6800/2297210135,v1:192.168.30.3:6801/2297210135] exists,up c42d04e4-e8ae-4ac1-8128-510f2a39cd2e

Note the osd is running on both public and cluster NAD range, which are 192.168.20.0/24 for public and 192.168.30.0/20for cluster.

Product(s)
Article Type