1. Ensure that you are logged in to Jira. If you do not have a Jira account, you must create a This content is not included.Red Hat Jira account.
2. Launch the This content is not included.Create Issue form.

3. Complete the Summary, Description, and Reporter fields.

   In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue.

4. Click Create.

1. Install the OpenShift sandboxed containers Operator on the OpenShift Container Platform cluster.
2. Optional: Install the Node Feature Discovery (NFD) Operator to configure node eligibility checks.
3. Optional: Install the Local Storage Operator to configure a local block storage device.
4. Create the KataConfig custom resource.
5. Optional: Modify the number of virtual machines running on each worker node.
6. Optional: Modify the pod overhead.
7. Configure your workload for OpenShift sandboxed containers.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: false
     checkNodeEligibility: false
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata on specific nodes, specify the key and value, for example, kata: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata as a runtime class on the worker nodes.

   Wait for the kata installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata is installed on the cluster.

4. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy code to clipboard

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata    kata-remote        152m

   Copy code to clipboard

   You can also see the default kata runtime class in addition to kata.

Procedure

1. Obtain the RuntimeClass object by running the following command:

   $ oc describe runtimeclass kata

   Copy code to clipboard

2. Update the overhead.podFixed.memory and cpu values and save as the file as runtimeclass.yaml:

   kind: RuntimeClass
   apiVersion: node.k8s.io/v1
   metadata:
     name: kata
   overhead:
     podFixed:
       memory: "500Mi"
       cpu: "500m"

   Copy code to clipboard

3. Apply the changes by running the following command:

   $ oc apply -f runtimeclass.yaml

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata
   # ...

   Copy code to clipboard

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

1. Install the OpenShift sandboxed containers Operator on the OpenShift Container Platform cluster.
2. Enable ports to allow internal communication with peer pods.
3. Create the peer pods config map.
4. Create the KataConfig custom resource.
5. Optional: Modify the number of virtual machines running on each worker node.
6. Disable insecure options by customizing the Kata Agent policy.
7. Optional: If you select a custom peer pod VM image from a private registry such as registry.access.redhat.com, you must configure a pull secret for peer pods.
8. Optional: You can select a custom peer pod VM image.
9. Configure your workload for OpenShift sandboxed containers.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

   Copy code to clipboard

Procedure

1. Log in to your OpenShift Container Platform cluster and retrieve the instance ID:

   $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
     -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

   Copy code to clipboard

2. Retrieve the AWS region:

   $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')

   Copy code to clipboard

3. Retrieve the security group IDs and store them in an array:

   $ AWS_SG_IDS=($(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
     --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
     --output text --region $AWS_REGION))

   Copy code to clipboard

4. For each security group ID, authorize the peer pods shim to access kata-agent communication, and set up the peer pods tunnel:

   $ for AWS_SG_ID in "${AWS_SG_IDS[@]}"; do \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 15150 --source-group $AWS_SG_ID --region $AWS_REGION; \
     aws ec2 authorize-security-group-ingress --group-id $AWS_SG_ID --protocol tcp --port 9000 --source-group $AWS_SG_ID --region $AWS_REGION; \
   done

   Copy code to clipboard

Procedure

1. Obtain the following values from your AWS instance:

   1. Retrieve and record the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' \
        -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      Copy code to clipboard

      This is used to retrieve other values for the secret object.

   2. Retrieve and record the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.aws.region}') \
        && echo "AWS_REGION: \"$AWS_REGION\""

      Copy code to clipboard

   3. Retrieve and record the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} \
          --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""

      Copy code to clipboard

   4. Retrieve and record the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} \
          --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""

      Copy code to clipboard

   5. Retrieve and record the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} \
        --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
        --region  $AWS_REGION --output json | jq -r '.[][][]' | paste -sd ",") \
          && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""

      Copy code to clipboard

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "aws"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     PODVM_INSTANCE_TYPE: "t3.medium"
     PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large"
     PODVM_AMI_ID: "<podvm_ami_id>"
     AWS_REGION: "<aws_region>"
     AWS_SUBNET_ID: "<aws_subnet_id>"
     AWS_VPC_ID: "<aws_vpc_id>"
     AWS_SG_IDS: "<aws_sg_ids>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy code to clipboard

   PODVM_INSTANCE_TYPE

   Defines the default instance type that is used if the instance type is not defined in the workload object.

   PODVM_INSTANCE_TYPES

   Specify the allowed instance types, without spaces, for creating the pod. You can define smaller instance types for workloads that need less memory and fewer CPUs or larger instance types for larger workloads.

   PODVM_AMI_ID

   This value is populated when you run the KataConfig CR, using an AMI ID based on your cluster credentials. If you create your own AMI, specify the correct AMI ID.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy code to clipboard

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy code to clipboard

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata-remote    kata-remote        152m

   Copy code to clipboard

   You can also see the default kata runtime class in addition to kata-remote.

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy code to clipboard

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy code to clipboard

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy code to clipboard

2. Check the status stanza of the YAML file.

   If the PODVM_AMI_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy code to clipboard

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy code to clipboard

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }

   Copy code to clipboard

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy code to clipboard

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy code to clipboard

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy code to clipboard

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy code to clipboard

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy code to clipboard

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy code to clipboard

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy code to clipboard

Procedure

1. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy code to clipboard

2. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy code to clipboard

2. Optional: To use a manually defined instance type, add the following annotation with the instance type that you defined in the config map:

   apiVersion: v1
   kind: <object>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.machine_type: <machine_type>
   # ...

   Copy code to clipboard

3. Optional: To use an automatic instance type, add the following annotations:

   apiVersion: v1
   kind: <Pod>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
       io.katacontainers.config.hypervisor.default_memory: <memory>
   # ...

   Copy code to clipboard

   The workload will run on an automatic instance type based on the amount of memory available.

4. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

1. Prepare your network by configuring outbound connectivity for the peer pods.
2. Install the OpenShift sandboxed containers Operator on the OpenShift Container Platform cluster.
3. Create the peer pods config map.
4. Optional: Create the Azure secret.
5. Create the KataConfig custom resource.
6. Optional: Modify the number of virtual machines running on each worker node.
7. Disable insecure options by customizing the Kata Agent policy.
8. Optional: If you select a custom peer pod VM image from a private registry such as registry.access.redhat.com, you must configure a pull secret for peer pods.
9. Optional: You can select a custom peer pod VM image.
10. Configure your workload for OpenShift sandboxed containers.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

   Copy code to clipboard

Procedure

1. Obtain the following values from your Azure instance:

   1. Retrieve and record the Azure resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
        -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
        && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

      Copy code to clipboard

   2. Retrieve and record the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Name:name}" --output tsv)

      Copy code to clipboard

      This value is used to retrieve the Azure subnet ID.

   3. Retrieve and record the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list \
        --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
        --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
         && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

      Copy code to clipboard

   4. Retrieve and record the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

      Copy code to clipboard

   5. Retrieve and record the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
        --query "{Location:location}" --output tsv) \
        && echo "AZURE_REGION: \"$AZURE_REGION\""

      Copy code to clipboard

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "azure"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
     AZURE_INSTANCE_SIZES: "Standard_B2als_v2,Standard_D2as_v5,Standard_D4as_v5,Standard_D2ads_v5"
     AZURE_SUBNET_ID: "<azure_subnet_id>"
     AZURE_NSG_ID: "<azure_nsg_id>"
     AZURE_IMAGE_ID: ""
     AZURE_REGION: "<azure_region>"
     AZURE_RESOURCE_GROUP: "<azure_resource_group>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy code to clipboard

   AZURE_INSTANCE_SIZE

   Defines the default instance size that is used if the instance size is not defined in the workload object.

   AZURE_IMAGE_ID

   Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod VM image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster.

   AZURE_INSTANCE_SIZES

   Specify the allowed instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy code to clipboard

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

   Copy code to clipboard

2. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

   Copy code to clipboard

3. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy code to clipboard

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy code to clipboard

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata-remote    kata-remote        152m

   Copy code to clipboard

   You can also see the default kata runtime class in addition to kata-remote.

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy code to clipboard

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy code to clipboard

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy code to clipboard

2. Check the status stanza of the YAML file.

   If the AZURE_IMAGE_ID parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy code to clipboard

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy code to clipboard

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }

   Copy code to clipboard

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy code to clipboard

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy code to clipboard

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy code to clipboard

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy code to clipboard

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy code to clipboard

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy code to clipboard

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy code to clipboard

Procedure

1. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<custom_image_id>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy code to clipboard

2. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy code to clipboard

2. Optional: To use a manually defined instance size, add the following annotation with the instance size that you defined in the config map:

   apiVersion: v1
   kind: <object>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.machine_type: <machine_type>
   # ...

   Copy code to clipboard

3. Optional: To use an automatic instance size, add the following annotations:

   apiVersion: v1
   kind: <Pod>
   metadata:
     annotations:
       io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
       io.katacontainers.config.hypervisor.default_memory: <memory>
   # ...

   Copy code to clipboard

   The workload will run on an automatic instance size based on the amount of memory available.

4. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

1. Install the OpenShift sandboxed containers Operator on the OpenShift Container Platform cluster.
2. Enable ports to allow internal communication with peer pods.
3. Create the peer pods config map.
4. Create the pod VM image config map.
5. Create the KataConfig custom resource.
6. Optional: Modify the number of virtual machines running on each worker node.
7. Disable insecure options by customizing the Kata Agent policy.
8. Configure your workload for OpenShift sandboxed containers.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Automatic
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

7. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

8. Watch the process by running the following command:

   $ watch oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   Example output

   NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
   openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

   Copy code to clipboard

Procedure

1. Set the project ID variable by running the following command:

   $ export GCP_PROJECT_ID="<project_id>"

   Copy code to clipboard

2. Log in to Google Cloud by running the following command:

   $ gcloud auth login

   Copy code to clipboard

3. Set the Google Cloud project ID by running the following command:

   $ gcloud config set project ${GCP_PROJECT_ID}

   Copy code to clipboard

4. Open port 15150 by running the following command:

   $ gcloud compute firewall-rules create allow-port-15150-restricted \
      --project=${GCP_PROJECT_ID} \
      --network=default \
      --allow=tcp:15150 \
      --source-ranges=<external_ip_cidr-1>[,<external_ip_cidr-2>,...] 1

   Copy code to clipboard

   1

   Specify one or more IP addresses or ranges in CIDR format, separated by commas. For example, 203.0.113.5/32,198.51.100.0/24.

Procedure

1. Log in to your Compute Engine instance to set the following environmental variables:

   1. Get the project ID by running the following command:

      $ GCP_PROJECT_ID=$(gcloud config get-value project)

      Copy code to clipboard

   2. Get the zone by running the following command:

      $ GCP_ZONE=$(gcloud config get-value compute/zone)

      Copy code to clipboard

   3. Retrieve a list of network names by running the following command:

      $ gcloud compute networks list --format="value(name)"

      Copy code to clipboard

   4. Specify the network by running the following command:

      $ GCP_NETWORK=<network_name>

      Copy code to clipboard

      Only auto-mode networks are supported. Custom networks are not supported at this time.

2. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "gcp"
     VXLAN_PORT: "9000"
     PROXY_TIMEOUT: "5m"
     GCP_MACHINE_TYPE: "e2-medium"
     GCP_PROJECT_ID: "<project_id>"
     GCP_ZONE: "<gcp_zone>"
     GCP_NETWORK: "<gcp_network>"
     TAGS: "key1=value1,key2=value2"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy code to clipboard

   GCP_MACHINE_TYPE

   Defines the default machine type that is used if the machine type is not defined in the workload object.

   TAGS

   You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

3. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy code to clipboard

Procedure

1. Clone the OpenShift sandboxed containers repository by running the following command:

   $ git clone https://github.com/openshift/sandboxed-containers-operator.git

   Copy code to clipboard

2. Navigate to sandboxed-containers-operator/config/peerpods/podvm/bootc by running the following command:

   $ cd sandboxed-containers-operator/config/peerpods/podvm/bootc

   Copy code to clipboard

3. Log in to registry.redhat.io by running the following command:

   $ podman login registry.redhat.io

   Copy code to clipboard

   You must log in to registry.redhat.io, because the podman build process must access the Containerfile.rhel container image hosted on the registry.

4. Set the image path for your container registry by running the following command:

   $ IMG="<container_registry_url>/<username>/podvm-bootc:latest"

   Copy code to clipboard

5. Build the pod VM bootc image by running the following command:

   $ podman build -t ${IMG} -f Containerfile.rhel .

   Copy code to clipboard

6. Log in to your container registry by running the following command:

   $ podman login <container_registry_url>

   Copy code to clipboard

7. Push the image to your container registry by running the following command:

   $ podman push ${IMG}

   Copy code to clipboard

   For testing and development, you can make the image public.

8. Verify the podvm-bootc image by running the following command:

   $ podman images

   Copy code to clipboard

   Example output

   REPOSITORY                               TAG     IMAGE ID      CREATED         SIZE
   example.com/example_user/podvm-bootc     latest  88ddab975a07  2 seconds ago   1.82 GB

   Copy code to clipboard

Procedure

1. Create a podvm-image-cm.yaml manifest with the following content:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     IMAGE_TYPE: pre-built
     PODVM_IMAGE_URI: <container_registry_url>/<username>/podvm-bootc:latest
     IMAGE_BASE_NAME: "podvm-image"
     IMAGE_VERSION: "0-0-0"
   
     INSTALL_PACKAGES: "no"
     DISABLE_CLOUD_CONFIG: "true"
     UPDATE_PEERPODS_CM: "yes"
     BOOT_FIPS: "no"
   
     BOOTC_BUILD_CONFIG: |
       [[customizations.user]]
       name = "peerpod"
       password = "peerpod"
       groups = ["wheel", "root"]
   
       [[customizations.filesystem]]
       mountpoint = "/"
       minsize = "5 GiB"
   
       [[customizations.filesystem]]
       mountpoint = "/var/kata-containers"
       minsize = "15 GiB"

   Copy code to clipboard

2. Create the config map by running the following command:

   $ oc create -f podvm-image-cm.yaml

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy code to clipboard

5. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy code to clipboard

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata-remote    kata-remote        152m

   Copy code to clipboard

   You can also see the default kata runtime class in addition to kata-remote.

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy code to clipboard

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy code to clipboard

Procedure

1. Obtain the config map you created for the peer pods:

   $ oc get configmap peer-pods-cm -n openshift-sandboxed-containers-operator -o yaml

   Copy code to clipboard

2. Check the status stanza of the YAML file.

   If the PODVM_IMAGE_NAME parameter is populated, the pod VM image was created successfully.

Troubleshooting

1. Retrieve the events log by running the following command:

   $ oc get events -n openshift-sandboxed-containers-operator --field-selector involvedObject.name=osc-podvm-image-creation

   Copy code to clipboard

2. Retrieve the job log by running the following command:

   $ oc logs -n openshift-sandboxed-containers-operator jobs/osc-podvm-image-creation

   Copy code to clipboard

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }

   Copy code to clipboard

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy code to clipboard

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy code to clipboard

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy code to clipboard

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

1. Install the OpenShift sandboxed containers Operator on the OpenShift Container Platform cluster.
2. Optional: Configure the libvirt volume.
3. Optional: Create a custom peer pod VM image.
4. Create the peer pods secret.
5. Create the peer pods config map.
6. Create the pod VM image config map.
7. Create the KVM host secret.
8. Create the KataConfig custom resource.
9. Optional: Modify the number of virtual machines running on each worker node.
10. Disable insecure options by customizing the Kata Agent policy.
11. Optional: If you select a custom peer pod VM image from a private registry such as registry.access.redhat.com, you must configure a pull secret for peer pods.
12. Optional: You can select a custom peer pod VM image.
13. Configure your workload for OpenShift sandboxed containers.

Procedure

1. Create an osc-namespace.yaml manifest file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Create the namespace by running the following command:

   $ oc apply -f osc-namespace.yaml

   Copy code to clipboard

3. Create an osc-operatorgroup.yaml manifest file:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: sandboxed-containers-operator-group
     namespace: openshift-sandboxed-containers-operator
   spec:
     targetNamespaces:
     - openshift-sandboxed-containers-operator

   Copy code to clipboard

4. Create the operator group by running the following command:

   $ oc apply -f osc-operatorgroup.yaml

   Copy code to clipboard

5. Create an osc-subscription.yaml manifest file:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: sandboxed-containers-operator
     namespace: openshift-sandboxed-containers-operator
   spec:
     channel: stable
     installPlanApproval: Manual
     name: sandboxed-containers-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: sandboxed-containers-operator.v1.11.1

   Copy code to clipboard

6. Create the subscription by running the following command:

   $ oc create -f osc-subscription.yaml

   Copy code to clipboard

7. Get the InstallPlan CR for the OpenShift sandboxed containers Operator by running the following command:

   $ oc get installplan -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   • Installation example output

     NAME            CSV                                      APPROVAL  APPROVED
     install-bl4fl   sandboxed-containers-operator.v1.11.1    Manual    false

     Copy code to clipboard

   • Upgrade example output

     NAME            CSV                                     APPROVAL   APPROVED
     install-jdzrb   sandboxed-containers-operator.v1.11.1   Manual     false
     install-pfk8l   sandboxed-containers-operator.v1.11.0   Manual     true

     Copy code to clipboard

8. Approve the manual installation by running the following command:

   $ oc patch installplan <installplan_name> -p '{"spec":{"approved":true}}' --type=merge -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   <installplan_name>

   Specify the InstallPlan resource. For example, install-jdzrb.

9. Verify that the Operator is correctly installed by running the following command:

   $ oc get csv -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   This command can take several minutes to complete.

10. Watch the process by running the following command:

    $ watch oc get csv -n openshift-sandboxed-containers-operator

    Copy code to clipboard

    Example output

    NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

    Copy code to clipboard

Procedure

1. Log in to the KVM host.

2. Set the name of the libvirt pool by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   Copy code to clipboard

   You need the LIBVIRT_POOL value to create the secret for the libvirt provider.

3. Set the name of the libvirt volume by running the following command:

   $ export LIBVIRT_VOL_NAME=<libvirt_volume>

   Copy code to clipboard

   You need the LIBVIRT_VOL_NAME value to create the secret for the libvirt provider.

4. Set the path of the default storage pool location, by running the following command:

   $ export LIBVIRT_POOL_DIRECTORY="/var/lib/libvirt/images/"

   Copy code to clipboard

5. Create a libvirt pool by running the following command:

   $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"

   Copy code to clipboard

6. Start the libvirt pool by running the following command:

   $ virsh pool-start $LIBVIRT_POOL

   Copy code to clipboard

7. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

   Copy code to clipboard

Procedure

1. Create a Dockerfile.podvm-oci file:

   FROM scratch
   
   ARG PODVM_IMAGE_SRC
   ENV PODVM_IMAGE_PATH="/image/podvm.qcow2"
   
   COPY $PODVM_IMAGE_SRC $PODVM_IMAGE_PATH

   Copy code to clipboard

2. Build a container with the pod VM QCOW2 image by running the following command:

   $ docker build -t podvm-libvirt \
     --build-arg PODVM_IMAGE_SRC=<podvm_image_source> \ 1
     --build-arg PODVM_IMAGE_PATH=<podvm_image_path> \ 2
     -f Dockerfile.podvm-oci .

   Copy code to clipboard

   1

   Specify the QCOW2 image source on the host.

   2

   Optional: Specify the path of the QCOW2 image if you do not use the default, /image/podvm.qcow2.

Procedure

1. Create a peer-pods-secret.yaml manifest file according to the following example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: peer-pods-secret
     namespace: openshift-sandboxed-containers-operator
   type: Opaque
   stringData:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_URI: "<libvirt_gateway_uri>" 1
     REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 2

   Copy code to clipboard

   1

   Specify the libvirt URI.

   2

   Specify the Red Hat offline token, which is required for the Operator-built image.

2. Create the secret by running the following command:

   $ oc create -f peer-pods-secret.yaml

   Copy code to clipboard

Procedure

1. Create a peer-pods-cm.yaml manifest file according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: peer-pods-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     CLOUD_PROVIDER: "libvirt"
     LIBVIRT_POOL: "<libvirt_pool>"
     LIBVIRT_VOL_NAME: "<libvirt_volume>"
     LIBVIRT_DIR_NAME: "/var/lib/libvirt/images/<directory_name>"
     LIBVIRT_NET: "default"
     PEERPODS_LIMIT_PER_NODE: "10"
     ROOT_VOLUME_SIZE: "6"
     DISABLECVM: "true"

   Copy code to clipboard

   LIBVIRT_POOL

   If you have manually configured the libvirt pool, use the same name as in your KVM host configuration.

   LIBVIRT_VOL_NAME

   If you have manually configured the libvirt volume, use the same name as in your KVM host configuration.

   LIBVIRT_DIR_NAME

   Specify the libvirt directory for storing virtual machine disk images, such as .qcow2, or .raw files. To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.

   LIBVIRT_NET

   Specify a libvirt network if you do not want to use the default network.

   PEERPODS_LIMIT_PER_NODE

   You can increase this value to run more peer pods on a node. The default value is 10.

   ROOT_VOLUME_SIZE

   You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.

2. Create the config map by running the following command:

   $ oc create -f peer-pods-cm.yaml

   Copy code to clipboard

Procedure

1. Create a libvirt-podvm-image-cm.yaml manifest according to the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: libvirt-podvm-image-cm
     namespace: openshift-sandboxed-containers-operator
   data:
     PODVM_DISTRO: "rhel"
     DOWNLOAD_SOURCES: "no" 1
     CAA_SRC: "https://github.com/confidential-containers/cloud-api-adaptor" 2
     CAA_REF: "main" 3
     CONFIDENTIAL_COMPUTE_ENABLED: "yes"
     UPDATE_PEERPODS_CM: "yes"
     ORG_ID: "<rhel_organization_id>"
     ACTIVATION_KEY: "<rhel_activation_key>" 4
     PODVM_IMAGE_URI: "oci::<image_repo_url>:<image_tag>::<image_path>" 5
     SE_BOOT: "true" 6
     BASE_OS_VERSION: "<rhel_image_os_version>" 7
     SE_VERIFY: "false" 8

   Copy code to clipboard

   1

   Specify yes if you want to use the custom Cloud API Adaptor source to build the pod VM image.

   2

   Optional: Specify the URL of the Cloud API Adaptor custom image.

   3

   Optional: Specify the branch or tag of the Cloud API Adaptor custom image.

   4

   Specify your RHEL activation key.

   5

   Optional: If you created a custom peer pod VM image, specify the container registry URL, the image tag, and the image path (default: /image/podvm.qcow2). Otherwise, set the value to "".

   6

   The default value, true, enables IBM Secure Execution for the default Operator-built image. If you use a custom peer pod VM image, set it to false .

   7

   Specify the RHEL image operating system version. IBM Z® Secure Execution supports RHEL 10.1 and later versions.

   8

   Specify false if you do not want to verify Secure Execution with the digicert CA certificate. The default value is true.

2. Create the config map by running the following command:

   $ oc create -f libvirt-podvm-image-cm.yaml

   Copy code to clipboard

   The libvirt pod VM image config map is created for your libvirt provider.

Procedure

1. Generate an SSH key pair by running the following command:

   $ ssh-keygen -f ./id_rsa -N ""

   Copy code to clipboard

2. Copy the public SSH key to your KVM host:

   $ ssh-copy-id -i ./id_rsa.pub <LIBVIRT_IP> 1

   Copy code to clipboard

   1

   Specify the libvirt IP address of your KVM host or the LPAR where the peer pod VM is running. For example, 192.168.122.1.

3. Create the Secret object by running the following command:

   $ oc create secret generic ssh-key-secret \
     -n openshift-sandboxed-containers-operator \
     --from-file=id_rsa.pub=./id_rsa.pub \
     --from-file=id_rsa=./id_rsa

   Copy code to clipboard

4. Delete the SSH keys you created:

   $ shred --remove id_rsa.pub id_rsa

   Copy code to clipboard

Procedure

1. Create an example-kataconfig.yaml manifest file according to the following example:

   apiVersion: kataconfiguration.openshift.io/v1
   kind: KataConfig
   metadata:
     name: example-kataconfig
   spec:
     enablePeerPods: true
     logLevel: info
   #  kataConfigPoolSelector:
   #    matchLabels:
   #      <label_key>: '<label_value>'

   Copy code to clipboard

   <label_key>: '<label_value>'

   Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.

2. Create the KataConfig CR by running the following command:

   $ oc create -f example-kataconfig.yaml

   Copy code to clipboard

   The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.

   Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.

3. Monitor the installation progress by running the following command:

   $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

   Copy code to clipboard

   When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.

4. Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:

   $ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator

   Copy code to clipboard

   Example output

   Name: peer-pods-cm
   Namespace: openshift-sandboxed-containers-operator
   Labels: <none>
   Annotations: <none>
   
   Data
   ====
   CLOUD_PROVIDER: libvirt
   
   BinaryData
   ====
   Events: <none>

   Copy code to clipboard

5. Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:

   $ watch oc get mcp/kata-oc

   Copy code to clipboard

6. Verify the daemon set by running the following command:

   $ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds

   Copy code to clipboard

7. Verify the runtime classes by running the following command:

   $ oc get runtimeclass

   Copy code to clipboard

   Example output

   NAME           HANDLER             AGE
   kata            kata                34m
   kata-remote    kata-remote        152m

   Copy code to clipboard

   You can also see the default kata runtime class in addition to kata-remote.

Procedure

1. Check the current limit by running the following command:

   $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     -o jsonpath='{.spec.limit}{"\n"}'

   Copy code to clipboard

2. Specify a new value for the limit key by running the following command:

   $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
     --type merge --patch '{"spec":{"limit":"<value>"}}'

   Copy code to clipboard

Procedure

1. Create a custom policy.rego file by modifying the default policy:

   package agent_policy
   
   default AddARPNeighborsRequest := true
   default AddSwapRequest := true
   default CloseStdinRequest := true
   default CopyFileRequest := true
   default CreateContainerRequest := true
   default CreateSandboxRequest := true
   default DestroySandboxRequest := true
   default GetMetricsRequest := true
   default GetOOMEventRequest := true
   default GuestDetailsRequest := true
   default ListInterfacesRequest := true
   default ListRoutesRequest := true
   default MemHotplugByProbeRequest := true
   default OnlineCPUMemRequest := true
   default PauseContainerRequest := true
   default PullImageRequest := true
   default ReadStreamRequest := false
   default RemoveContainerRequest := true
   default RemoveStaleVirtiofsShareMountsRequest := true
   default ReseedRandomDevRequest := true
   default ResumeContainerRequest := true
   default SetGuestDateTimeRequest := true
   default SignalProcessRequest := true
   default StartContainerRequest := true
   default StartTracingRequest := true
   default StatsContainerRequest := true
   default StopTracingRequest := true
   default TtyWinResizeRequest := true
   default UpdateContainerRequest := true
   default UpdateEphemeralMountsRequest := true
   default UpdateInterfaceRequest := true
   default UpdateRoutesRequest := true
   default WaitProcessRequest := true
   default ExecProcessRequest := false
   default SetPolicyRequest := false
   default WriteStreamRequest := false
   
   ExecProcessRequest if {
       input_command = concat(" ", input.process.Args)
       some allowed_command in policy_data.allowed_commands
       input_command == allowed_command
   }
   
   policy_data := {
     "allowed_commands": [
           "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
     ]
   }

   Copy code to clipboard

   The default policy allows all API calls. Adjust the true or false values to customize the policy further based on your needs.

2. Convert the policy.rego file to a Base64-encoded string by running the following command:

   $ base64 -w0 policy.rego

   Copy code to clipboard

   Record the output.

3. Add the Base64-encoded policy string to the my-pod.yaml manifest:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod
     annotations:
       io.katacontainers.config.agent.policy: <base64_encoded_policy>
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <container_name>
       image: registry.access.redhat.com/ubi9/ubi:latest
       command:
       - sleep
       - "36000"
       securityContext:
         privileged: false
         seccompProfile:
           type: RuntimeDefault

   Copy code to clipboard

4. Create the pod by running the following command:

   $ oc create -f my-pod.yaml

   Copy code to clipboard

Procedure

1. Set the NS variable to the namespace where you deploy your peer pods:

   $ NS=<namespace>

   Copy code to clipboard

2. Copy the pull secret to the peer pod namespace:

   $ oc get secret pull-secret -n openshift-config -o yaml \
     | sed "s/namespace: openshift-config/namespace: ${NS}/" \
     | oc apply -n "${NS}" -f -

   Copy code to clipboard

   You can use the cluster pull secret, as in this example, or a custom pull secret.

3. Optional: Link the pull secret to the default service account:

   $ oc secrets link default pull-secret --for=pull -n ${NS}

   Copy code to clipboard

4. Alternatively, add the pull secret to the peer pod manifest:

   apiVersion: v1
   kind: <Pod>
   spec:
     containers:
     - name: <container_name>
       image: <image_name>
     imagePullSecrets:
     - name: pull-secret
   # ...

   Copy code to clipboard

Procedure

1. Set the LIBVIRT_POOL variable by running the following command:

   $ export LIBVIRT_POOL=<libvirt_pool>

   Copy code to clipboard

2. Set the LIBVIRT_VOL_NAME variable to a new libvirt volume by running the following command:

   $ export LIBVIRT_VOL_NAME=<new_libvirt_volume>

   Copy code to clipboard

3. Create a libvirt volume for the pool by running the following command:

   $ virsh -c qemu:///system \
     vol-create-as --pool $LIBVIRT_POOL \
     --name $LIBVIRT_VOL_NAME \
     --capacity 20G \
     --allocation 2G \
     --prealloc-metadata \
     --format qcow2

   Copy code to clipboard

4. Upload the custom peer pod VM image to the new libvirt volume:

   $ virsh -c qemu:///system vol-upload \
     --vol $LIBVIRT_VOL_NAME <custom_podvm_image.qcow2> \
     --pool $LIBVIRT_POOL --sparse

   Copy code to clipboard

5. Create a my-pod-manifest.yaml file according to the following example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: my-pod-manifest
     annotations:
       io.katacontainers.config.hypervisor.image: "<new_libvirt_volume>"
   spec:
     runtimeClassName: kata-remote
     containers:
     - name: <example_container>
       image: registry.access.redhat.com/ubi9/ubi:9.3
       command: ["sleep", "36000"]

   Copy code to clipboard

6. Create the pod by running the following command:

   $ oc create -f my-pod-manifest.yaml

   Copy code to clipboard

Procedure

1. Add spec.runtimeClassName: kata-remote to the manifest of each pod-templated workload object as in the following example:

   apiVersion: v1
   kind: <object>
   # ...
   spec:
     runtimeClassName: kata-remote
   # ...

   Copy code to clipboard

2. Apply the changes to the workload object by running the following command:

   $ oc apply -f <object.yaml>

   Copy code to clipboard

   OpenShift Container Platform creates the workload object and begins scheduling it.

Procedure

1. In the OpenShift Container Platform web console, navigate to Observe → Metrics.

2. In the input field, enter the query for the metric you want to observe.

   All kata-related metrics begin with kata. Typing kata displays a list of all available kata metrics.

Procedure

1. Search for the pods by running the following command:

   $ oc get pods -A -o json | jq -r '.items[] | \
     select(.spec.runtimeClassName == "<runtime>").metadata.name'

   Copy code to clipboard

2. Delete each pod by running the following command:

   $ oc delete pod <pod>

   Copy code to clipboard

Procedure

1. Delete the KataConfig CR by running the following command:

   $ oc delete kataconfig example-kataconfig

   Copy code to clipboard

2. Verify the CR removal by running the following command:

   $ oc get kataconfig example-kataconfig

   Copy code to clipboard

   Example output

   No example-kataconfig instances exist

   Copy code to clipboard

Procedure

1. Delete the subscription by running the following command:

   $ oc delete subscription sandboxed-containers-operator -n openshift-sandboxed-containers-operator

   Copy code to clipboard

2. Delete the namespace by running the following command:

   $ oc delete namespace openshift-sandboxed-containers-operator

   Copy code to clipboard

Procedure

1. Delete the KataConfig CRD by running the following command:

   $ oc delete crd kataconfigs.kataconfiguration.openshift.io

   Copy code to clipboard

2. Verify that the CRD was deleted by running the following command:

   $ oc get crd kataconfigs.kataconfiguration.openshift.io

   Copy code to clipboard

   Example output

   Unknown CRD kataconfigs.kataconfiguration.openshift.io

   Copy code to clipboard