Deploying confidential containers

OpenShift sandboxed containers 1.11

Protecting containers and data by leveraging trusted execution environments

Red Hat Customer Content Services

Abstract

Confidential containers provide a confidential computing environment to protect containers and data by leveraging trusted execution environments. You install the OpenShift sandboxed containers Operator on an OpenShift Container Platform cluster for your confidential containers workload after configuring an attestation service such as Red Hat build of Trustee in a trusted environment.

Preface

Chapter 1. Overview

Learn about confidential containers and ensure that your OpenShift Container Platform environment is compatible.

1.1. About confidential containers

Confidential containers provides a confidential computing environment to protect containers and data by leveraging Content from en.wikipedia.org is not included.Trusted Execution Environments.

For more information, see This content is not included.Exploring the OpenShift confidential containers solution.

1.2. Compatibility with OpenShift Container Platform

The required functionality for Red Hat OpenShift Container Platform is supported by two main components:

Kata runtime: The Kata runtime is included with Red Hat Enterprise Linux CoreOS (RHCOS) and receives updates with every OpenShift Container Platform release. When enabling peer pods with the Kata runtime, the OpenShift sandboxed containers Operator requires external network connectivity to pull the necessary image components and helper utilities to create the pod virtual machine (VM) image.
OpenShift sandboxed containers Operator: The OpenShift sandboxed containers Operator is a Rolling Stream Operator, which means the latest version is the only supported version. It works with all currently supported versions of OpenShift Container Platform.

The Operator depends on the features that come with the RHCOS host and the environment it runs in.

Note

You must install RHCOS on the worker nodes. Red Hat Enterprise Linux (RHEL) nodes are not supported.

The following compatibility matrix for OpenShift sandboxed containers and OpenShift Container Platform releases identifies compatible features and environments.

Table 1.1. Supported architectures

Architecture	OpenShift Container Platform version
x86_64	4.17 or later
s390x	4.17 or later

There are two ways to deploy the Kata containers runtime:

Bare metal
Peer pods

You can deploy OpenShift sandboxed containers by using peer pods on Microsoft Azure, AWS Cloud Computing Services, or Google Cloud. With the release of OpenShift sandboxed containers 1.11, the OpenShift sandboxed containers Operator requires OpenShift Container Platform version 4.17 or later.

Table 1.2. Feature availability by OpenShift Container Platform version

Major release version		4.17	4.18	4.19	4.20
Minor release version		4.17.45+	4.18.30+	4.19.20+	4.20.6+
Feature	Platform
Confidential containers	Bare metal	—	—	—	TP
	Azure peer pods	GA	GA	GA	GA
	IBM Z peer pods	TP	TP	TP	TP
	IBM Z bare metal	—	—	—	TP
GPU support	Bare metal	—	—	—	—
	Azure	DP	DP	DP	DP
	AWS	DP	DP	DP	DP
	Google Cloud	DP	DP	DP	DP

Important

GPU support for peer pods is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Table 1.3. Supported cloud platforms

Platform	GPU	Confidential containers
Azure	DP	GA
AWS	DP	—
Google Cloud	DP	—

Table 1.4. Supported on-premise platforms

Platform	GPU	Confidential containers
Bare metal	—	TP
IBM Z	—	TP

Additional resources

1.3. Providing feedback on Red Hat documentation

You can provide feedback or report an error by submitting the Create Issue form in Jira:

Ensure that you are logged in to Jira. If you do not have a Jira account, you must create a This content is not included.Red Hat Jira account.
Launch the This content is not included.Create Issue form.
Complete the Summary, Description, and Reporter fields.
In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue.
Click Create.

Chapter 2. Deploying confidential containers on bare metal

You can deploy confidential containers workloads on a Red Hat OpenShift Container Platform cluster running on bare metal.

Important

Confidential containers on bare metal is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

2.1. Preparation

Review these prerequisites and concepts before you deploy confidential containers on bare metal.

2.1.1. Prerequisites

You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.
You have installed the latest version of Red Hat OpenShift Container Platform on the cluster where you are running your confidential containers workload.

2.1.2. Initializing pods at runtime by using initdata

You can initialize a pod with workload-specific data at runtime by creating and applying initdata.

This approach enhances security by reducing the exposure of confidential information and improves flexibility by eliminating custom image builds. For example, initdata can include three configuration settings:

An X.509 certificate for secure communication.
A cryptographic key for authentication.
An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

The initdata content configures the following components:

Attestation Agent (AA), which verifies the trustworthiness of the pod by sending evidence for attestation.
Confidential Data Hub (CDH), which manages secrets and secure data access within the pod VM.
Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

You create an initdata.toml file and convert it to a gzip-format Base64-encoded string.

You apply initdata to a confidential containers pod by adding an annotation to the pod manifest.

2.1.3. Kata runtime deployment modes

You can choose how the Operator installs and configures the Kata runtime using the deployment modes MachineConfig, DaemonSet, or DaemonSetFallback. You specify the data.deploymentMode key in the osc-feature-gates config map. This flexibility allows the Operator to work consistently in clusters with or without the Machine Config Operator (MCO).

MachineConfig: For clusters that use the Machine Config Operator (MCO). If the deploymentMode key is missing in the config map, the Operator defaults to the MachineConfig for backward compatibility.
DaemonSet: For clusters without the MCO. The Operator uses a DaemonSet to install kata-containers RPMs and manage CRI-O configuration by using host drop-in files. Installation progress is tracked through node labels (for example, installing, installed).
DaemonSetFallback: Enables conditional deployment based on the cluster environment. When set, the operator checks for the presence of the MCO. It uses DaemonSet if the MachineConfig add-on is unavailable and defaults to MachineConfig otherwise.

2.2. Deployment overview

You deploy confidential containers on bare metal by performing the following steps:

Create MachineConfig for Intel® Trust Domain Extensions (TDX).
Install the OpenShift sandboxed containers Operator.
Configure the remote attestation infrastructure for Intel® TDX workloads.
Enable the confidential containers feature gate.
Create the KataConfig CR.
Verify the attestation process.

2.3. Creating a machine config for Intel TDX

If you use Intel® Trust Domain Extensions (TDX), you must create a MachineConfig object before you install the Red Hat build of Trustee Operator.

Procedure

Create a tdx-machine-config.yaml manifest file according to the following example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: <role> 1
  name: 99-enable-intel-tdx
spec:
  kernelArguments:
  - kvm_intel.tdx=1
  - nohibernate
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - path: /etc/modules-load.d/vsock.conf
          mode: 0644
          contents:
            source: data:text/plain;charset=utf-8;base64,dnNvY2stbG9vcGJhY2sK

1: Specify master for single-node OpenShift or kata-oc for a multi-node cluster.

Create the config map by running the following command:
```
$ oc create -f tdx-machine-config.yaml
```
Updating the machine config triggers node reboot.

Verification

Verify that the machine config is correctly configured by running the following command:
```
$ oc get oc get machineconfig 99-enable-intel-tdx
```

2.4. Installing the OpenShift sandboxed containers Operator

You install the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

Create an osc-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

Create the namespace by running the following command:
```
$ oc apply -f osc-namespace.yaml
```

Create an osc-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

Create the operator group by running the following command:
```
$ oc apply -f osc-operatorgroup.yaml
```

Create an osc-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.1

Create the subscription by running the following command:
```
$ oc create -f osc-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n openshift-sandboxed-containers-operator
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n openshift-sandboxed-containers-operator

Example output

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

2.5. Configuring auto-detection of TEEs

You must configure your nodes so that the OpenShift sandboxed containers Operator can detect the Trusted Execution Environments (TEEs).

You label the nodes by installing and configuring the Node Feature Discovery (NFD) Operator.

2.5.1. Creating a NodeFeatureDiscovery custom resource

You create a NodeFeatureDiscovery custom resource (CR) to define the configuration parameters that the Node Feature Discovery (NFD) Operator checks to automatically detect your TEE.

Prerequisites

You have installed the NFD Operator. For more information, see Node Feature Discovery Operator in the OpenShift Container Platform documentation.

Procedure

Create a my-nfd.yaml manifest file according to the following example:

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  operand:
    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
    imagePullPolicy: Always
    servicePort: 12000
  workerConfig:
    configData: |

Create the NodeFeatureDiscovery CR:
```
$ oc create -f my-nfd.yaml
```

2.5.2. Creating the NodeFeatureRule custom resource

Create a NodeFeatureRule custom resource for your Trusted Execution Environment (TEE).

Procedure

Create a custom resource manifest named my-nodefeaturerule.yaml:

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: consolidated-hardware-features
  namespace: openshift-nfd
spec:
  rules:
    - name: "runtime.kata"
      labels:
        feature.node.kubernetes.io/runtime.kata: "true"
      matchAny:
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: { op: Exists }
                VMX: { op: Exists }
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: { op: Exists }
                kvm_intel: { op: Exists }
        - matchFeatures:
            - feature: cpu.cpuid
              matchExpressions:
                SSE42: { op: Exists }
                SVM: { op: Exists }
            - feature: kernel.loadedmodule
              matchExpressions:
                kvm: { op: Exists }
                kvm_amd: { op: Exists }

    - name: "amd.sev-snp"
      labels:
        amd.feature.node.kubernetes.io/snp: "true"
      extendedResources:
        sev-snp.amd.com/esids: "@cpu.security.sev.encrypted_state_ids"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            SVM: { op: Exists }
        - feature: cpu.security
          matchExpressions:
            sev.snp.enabled: { op: Exists }

    - name: "intel.sgx"
      labels:
        intel.feature.node.kubernetes.io/sgx: "true"
      extendedResources:
        sgx.intel.com/epc: "@cpu.security.sgx.epc"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            SGX: { op: Exists }
            SGXLC: { op: Exists }
        - feature: cpu.security
          matchExpressions:
            sgx.enabled: { op: IsTrue }
        - feature: kernel.config
          matchExpressions:
            X86_SGX: { op: Exists }

    - name: "intel.tdx"
      labels:
        intel.feature.node.kubernetes.io/tdx: "true"
      extendedResources:
        tdx.intel.com/keys: "@cpu.security.tdx.total_keys"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            VMX: { op: Exists }
        - feature: cpu.security
          matchExpressions:
            tdx.enabled: { op: Exists }

Create the NodeFeatureRule CR by running the following command:
```
$ oc create -f my-nodefeaturerule.yaml
```

Note

A relabeling delay of up to 1 minute might occur.

2.6. Deploying remote attestation for Intel TDX

You must deploy the Intel® remote attestation infrastructure to enable quote generation for Intel® Trust Domain Extensions (TDX) pod virtual machines.

The infrastructure includes the following components:

In-cluster Provisioning Certificate Caching Service (PCCS)
Automatic per-node Provisioning Certification Key (PCK) Cert ID Retrieval Tool based platform (re-)registration
Per-node Quote Generation Service (QGS)

Note

The system does not back up the PCCS database automatically. Cluster administrators must implement a manual backup strategy for the database file located at /var/cache/pccs/ on the deployment node, typically a control plane node. If you do not have a valid backup, you must trigger an SGX Factory Reset in the BIOS to re-provision the required platform manifests.

Prerequisites

You have installed the Intel® device plugins Operator and created an instance of the Intel® Software Guard Extensions device plugin. For details, see Installing from the software catalog by using the web console in the OpenShift Container Platform documentation.
The node on which you deploy PCCS has Internet access.

Procedure

Configure the remote attestation project:
1. Create the intel-dcap namespace by running the following command:
```
$ oc create namespace intel-dcap
```
2. Switch to the intel-dcap project by running the following command:
```
$ oc project intel-dcap
```
3. Update the Security Context Constraint by running the following command:
```
$ oc adm policy add-scc-to-user privileged -z default
```
Switch to the default project by running the following command:
```
$ oc project default
```
Set the PCCS variables by running the following commands:
```
$ export PCCS_API_KEY="${PCCS_API_KEY:-}"
```
To obtain the API key for the Intel® Software Guard Extensions and Intel® TDX Provisioning Certification Service, navigate to the Content from api.portal.trustedservices.intel.com is not included.Intel Trusted Services API portal, sign in, and subscribe to the Provisioning Certification Service. The API key is displayed on the Manage Subscriptions page.
```
$ export PCCS_USER_TOKEN="${PCCS_USER_TOKEN:-mytoken}"
```
For details about PCCS tokens, see the Content from download.01.org is not included.Design Guide for Intel® SGX Provisioning Certificate Caching Service (Intel® SGX PCCS).
```
$ export PCCS_ADMIN_TOKEN="${PCCS_ADMIN_TOKEN:-mytoken}"
```
```
$ export PCCS_NODE=$(oc get nodes \
  -l 'node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=' \
  -o jsonpath='{.items[0].metadata.name}')
```

Set the cluster proxy variable by running the appropriate command:

$ export CLUSTER_HTTPS_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.httpsProxy})"

$ export CLUSTER_NO_PROXY="$(oc get proxy/cluster \
  -o jsonpath={.spec.noProxy})"

Create the PCCS secrets:

Set the PCCS secrets variables by running the following commands:
```
$ export PCCS_USER_TOKEN_HASH=$(echo -n "$PCCS_USER_TOKEN" | sha512sum | tr -d '[:space:]-')
```
```
$ export PCCS_ADMIN_TOKEN_HASH=$(echo -n "$PCCS_ADMIN_TOKEN" | sha512sum | tr -d '[:space:]-')
```
```
$ PCCS_PEM_CERT_PATH=$(mktemp -d)
```
Note
This directory is automatically deleted at reboot. To re-use the PCCS certificate and key, you must create a persistent directory.

Generate an RSA key pair and output the private key as a PCCS certificate by running the following command:

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
  -keyout $PCCS_PEM_CERT_PATH/private.pem \
  -out $PCCS_PEM_CERT_PATH/certificate.pem \
  -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

Set the PCCS certificate variables by running the following commands:

$ export PCCS_PEM=$(cat "$PCCS_PEM_CERT_PATH"/private.pem | base64 | tr -d '\n')

$ export PCCS_CERT=$(cat "$PCCS_PEM_CERT_PATH"/certificate.pem | base64 | tr -d '\n')

Create the PCCS secrets by running the following command:

$ oc create secret generic pccs-secrets \
    --namespace intel-dcap \
    --from-literal=PCCS_API_KEY="$PCCS_API_KEY" \
    --from-literal=PCCS_USER_TOKEN_HASH="$PCCS_USER_TOKEN_HASH" \
    --from-literal=USER_TOKEN="$PCCS_USER_TOKEN" \
    --from-literal=PCCS_ADMIN_TOKEN_HASH="$PCCS_ADMIN_TOKEN_HASH"

Create the PCCS by running the following command:

$ oc apply -f <(curl -sSf https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/refs/tags/v1.11.1/scripts/install-helpers/baremetal-coco/intel-dcap/pccs.yaml.in|envsubst)

Create the QGS by running the following command:

$ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/refs/tags/v1.11.1/scripts/install-helpers/baremetal-coco/intel-dcap/qgs.yaml

Additional resources

2.7. Creating the osc-feature-gates config map

You enable the confidential containers feature gate and specify the deployment mode by creating the config map.

Procedure

Create a my-feature-gate.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: <deployment_mode>
```
where
<deployment_mode>
On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:
MachineConfig for clusters that always use the MCO
DaemonSet for clusters that never use the MCO
DaemonSetFallback for clusters that sometimes use the MCO
Create the my-feature-gates config map by running the following command:
```
$ oc create -f my-feature-gate.yaml
```

2.8. Creating the KataConfig custom resource

You must create the KataConfig custom resource (CR) to install kata-cc as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-cc as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

A large OpenShift Container Platform deployment with a greater number of worker nodes.
Activation of the BIOS and Diagnostics utility.
Deployment on a hard disk drive rather than an SSD.
Deployment on physical nodes such as bare metal, rather than on virtual nodes.
A slow CPU and network.

Procedure

Create an example-kataconfig.yaml manifest file according to the following example:
```
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: false
  checkNodeEligibility: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
```
<label_key>: '<label_value>'
Optional: If you have applied node labels to install kata-cc on specific nodes, specify the key and value, for example, kata-cc: 'true'.
Create the KataConfig CR by running the following command:
```
$ oc create -f example-kataconfig.yaml
```
The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.
Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.
Monitor the installation progress by running the following command:
```
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.

Verify the runtime classes by running the following command:

$ oc get runtimeclass

Example output

NAME           HANDLER             AGE
kata            kata                34m
kata-cc    kata-tdx        152m

2.9. Creating initdata

You create initdata to securely initialize a pod with sensitive or workload-specific data at runtime, thus avoiding the need to embed this data in a virtual machine image. This approach provides additional security by reducing the risk of exposure of confidential information and eliminates the need for custom image builds.

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

Procedure

Obtain the Red Hat build of Trustee URL by running the following command:

Create the initdata.toml file:

algorithm = <algorithm>
version = "0.1.0"
[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''
"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

algorithm: Specify sha256, sha384, or sha512.
url: Specify the Red Hat build of Trustee
<kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.
image_security_policy_uri: Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:
```
$ cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
Record this string to use in the pod manifest.
Calculate the hash of an initdata.toml file and assign its value to the hash variable by running the following command:
```
$ hash=$(<algorithm> initdata.toml | cut -d' ' -f1)
```
Assign 32 bytes of 0s to the initial_pcr variable by running the following command:
```
$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
```
Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:
```
$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
```
Record the PCR8_HASH value for the RVPS config map.

2.10. Applying initdata to a pod

Prerequisite

You have created an initdata string.

Procedure

Add the initdata string to the pod manifest and save the file as my-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-cc
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

where

<initdata_string>: Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.
<container_name>: Specify a container name.

Create the pod by running the following command:
```
$ oc create -f my-pod.yaml
```

2.11. Verifying attestation

You can verify the attestation process by creating a test pod to retrieve a specific resource from Red Hat build of Trustee.

Important

This procedure is an example to verify that attestation is working. Do not write sensitive data to standard I/O, because the data can be captured by using a memory dump. Only data written to memory is encrypted.

Procedure

Create a test-pod.yaml manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 1
spec:
  runtimeClassName: kata-cc
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
metadata:
  name: coco-test-pod
  labels:
    app: coco-test-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 2
spec:
  runtimeClassName: kata-cc
  containers:
    - name: test-container
      image: registry.access.redhat.com/ubi9/ubi:9.3
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

1 2: Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

Create the pod by running the following command:
```
$ oc create -f test-pod.yaml
```
Log in to the pod by running the following command:
```
$ oc exec -it ocp-cc-pod -- bash
```
Fetch the Red Hat build of Trustee resource by running the following command:
```
$ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status
```
Example output
```
success #/
```

Chapter 3. Deploying confidential containers on Microsoft Azure Red Hat OpenShift

You can deploy confidential containers workloads on Microsoft Azure Red Hat OpenShift.

3.1. Preparation

Review these prerequisites and concepts before you deploy confidential containers on Azure Red Hat OpenShift.

3.1.1. Prerequisites

You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.
You have installed the latest version of Red Hat OpenShift Container Platform on the cluster where you are running your confidential containers workload.
IMPORTANT

Microsoft Azure Red Hat OpenShift clusters use OpenShift Container Platform versions 4.17.26 or 4.18.26 by default. You must update your cluster to 4.17.45 or 4.18.31 before you deploy confidential containers. See Performing a cluster update in the OpenShift Container Platform documentation.

You have enabled ports 15150 and 9000 for communication in the subnet used for worker nodes and the pod virtual machine (VM). The ports enable communication between the Kata shim running on the worker node and the Kata agent running on the pod VM.

3.1.2. Outbound connections

To enable peer pods to communicate with external networks, such as the public internet, you must configure outbound connectivity for the pod virtual machine (VM) subnet. This involves setting up a NAT gateway and, optionally, defining how the subnet integrates with your cluster’s virtual network (VNet) in Azure.

Peer pods and subnets: Peer pods operate in a dedicated Azure subnet that requires explicit configuration for outbound access. This subnet can either be the default worker subnet used by OpenShift Container Platform nodes or a separate, custom subnet created specifically for peer pods.
VNet peering: When using a separate subnet, VNet peering connects the peer pod VNet to the cluster’s VNet, ensuring internal communication while maintaining isolation. This requires non-overlapping CIDR ranges between the VNets.

You can configure outbound connectivity in two ways:

Default worker subnet: Modify the existing worker subnet to include a NAT gateway. This is simpler and reuses cluster resources, but it offers less isolation.
Peer pod VNet: Set up a dedicated VNet and subnet for peer pods, attach a NAT gateway, and peer it with the cluster VNet. This provides greater isolation and flexibility at the cost of additional complexity.

3.1.3. Peer pod resource requirements

You must ensure that your cluster has sufficient resources.

Peer pod virtual machines (VMs) require resources in two locations:

The worker node. The worker node stores metadata, Kata shim resources (containerd-shim-kata-v2), remote-hypervisor resources (cloud-api-adaptor), and the tunnel setup between the worker nodes and the peer pod VM.
The cloud instance. This is the actual peer pod VM running in the cloud.

The CPU and memory resources used in the Kubernetes worker node are handled by the Content from kubernetes.io is not included.pod overhead included in the RuntimeClass (kata-remote) definition used for creating peer pods.

The total number of peer pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the PEERPODS_LIMIT_PER_NODE attribute in the peer-pods-cm config map.

The extended resource is named kata.peerpods.io/vm, and enables the Kubernetes scheduler to handle capacity tracking and accounting.

You can edit the limit per node based on the requirements for your environment after you install the OpenShift sandboxed containers Operator.

A Content from kubernetes.io is not included.mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer pod is only scheduled when resources are available.

The mutating webhook modifies a Kubernetes pod as follows:

The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIMECLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIMECLASS, the webhook exits without modifying the pod.
If the RuntimeClassName values match, the webhook makes the following changes to the pod spec:
1. The webhook removes every resource specification from the resources field of all containers and init containers in the pod.
2. The webhook adds the extended resource (kata.peerpods.io/vm) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes.

Note

The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.

As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces.

3.1.4. Initializing pods at runtime by using initdata

You can initialize a pod with workload-specific data at runtime by creating and applying initdata.

An X.509 certificate for secure communication.
A cryptographic key for authentication.
An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

The initdata content configures the following components:

Attestation Agent (AA), which verifies the trustworthiness of the pod by sending evidence for attestation.
Confidential Data Hub (CDH), which manages secrets and secure data access within the pod VM.
Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

You create an initdata.toml file and convert it to a gzip-format Base64-encoded string.

Then, you create a hash from the initdata file. This hash is required as a reference value for the Reference Value Provider Service (RVPS) config map for Red Hat build of Trustee.

3.2. Deployment overview

You deploy confidential containers on Azure Red Hat OpenShift by performing the following steps:

Prepare your network by configuring outbound connectivity for the peer pods.
Install the OpenShift sandboxed containers Operator.
Enable the confidential containers feature gate.
Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.
Create the peer pods config map. You can add initdata to the config map to create a default global configuration for your peer pods.
Create the KataConfig CR.
Verify the attestation process.
You can add initdata to a pod manifest to override the global initdata configuration you set in the peer pods config map.
Optional: If you select a container image from an authenticated registry, you must configure a pull secret for the pod.
Optional: You can select a custom peer pod VM image.

3.3. Preparing your network

You must prepare your network by configuring outbound connectivity for the peer pods. You can perform this task by using one of the following methods:

Add a NAT gateway to the default worker subnet. This method is simple and reuses cluster resources, but it offers less isolation.
Create a dedicated VNet and subnet for your peer pods, attach a NAT gateway, and peer it with the cluster VNet. This method is more complex but it provides greater isolation and flexibility.

3.3.1. Configuring the default worker subnet

You can configure the default worker subnet for outbound connections by attaching a NAT gateway. This method is simple and reuses cluster resources, but it offers less isolation than a dedicated virtual network.

Prerequisites

The Azure CLI (az) is installed and authenticated.
You have administrator access to the Azure resource group and the VNet.

Procedure

Set the AZURE_RESOURCE_GROUP environment variable by running the following command:

$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
    -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

Set the AZURE_REGION environment variable by running the following command:

$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
    --query "{Location:location}" --output tsv) && \
    echo "AZURE_REGION: \"$AZURE_REGION\""

Set the AZURE_VNET_NAME environment variable by running the following command:

$ AZURE_VNET_NAME=$(az network vnet list \
    -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)

Set the AZURE_SUBNET_ID environment variable by running the following command:

$ AZURE_SUBNET_ID=$(az network vnet subnet list \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --vnet-name "${AZURE_VNET_NAME}" --query "[].{Id:id} \
    | [? contains(Id, 'worker')]" --output tsv)

Set the NAT gateway environment variables for the peer pod subnet by running the following commands:
```
$ export PEERPOD_NAT_GW=peerpod-nat-gw
```
```
$ export PEERPOD_NAT_GW_IP=peerpod-nat-gw-ip
```

Create a public IP address for the NAT gateway by running the following command:

$ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
    -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}" --sku Standard

Create the NAT gateway and associate it with the public IP address by running the following command:

$ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
    -l "${AZURE_REGION}" --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
    -n "${PEERPOD_NAT_GW}"

Update the VNet subnet to use the NAT gateway by running the following command:
```
$ az network vnet subnet update --nat-gateway "${PEERPOD_NAT_GW}" \
    --ids "${AZURE_SUBNET_ID}"
```

Verification

Confirm the NAT gateway is attached to the VNet subnet by running the following command:
```
$ az network vnet subnet show --ids "${AZURE_SUBNET_ID}" \
    --query "natGateway.id" -o tsv
```
The output contains the NAT gateway resource ID. If no NAT gateway is attached, the output is empty.
Example output
```
/subscriptions/12345678-1234-1234-1234-1234567890ab/resourceGroups/myResourceGroup/providers/Microsoft.Network/natGateways/myNatGateway
```

Additional Resources

3.3.2. Creating a dedicated peer pod virtual network

You can configure outbound connections for peer pods by creating a dedicated virtual network (VNet). Then, you create a network address translation (NAT) gateway for the VNet, create a subnet within the VNet, and enable VNet peering with non-overlapping address spaces.

This method is more complex than creating a NAT gateway for the default worker subnet but it provides greater isolation and flexibility.

Prerequisites

The Azure CLI (az) is installed
You have signed in to Azure. See Content from learn.microsoft.com is not included.Authenticate to Azure using Azure CLI.
You have administrator access to the Azure resource group and VNet hosting the cluster.
You have verified the cluster VNet classless inter-domain routing (CIDR) address. The default value is 10.0.0.0/14. If you overrode the default value, you have ensured that you chose a non-overlapping CIDR address for the peer pod VNet. For example, 192.168.0.0/16.

Procedure

Set the environmental variables for the peer pod network:
1. Set the peer pod VNet environment variables by running the following commands:
```
$ export PEERPOD_VNET_NAME="${PEERPOD_VNET_NAME:-peerpod-vnet}"
```
```
$ export PEERPOD_VNET_CIDR="${PEERPOD_VNET_CIDR:-192.168.0.0/16}"
```
2. Set the peer pod subnet environment variables by running the following commands:
```
$ export PEERPOD_SUBNET_NAME="${PEERPOD_SUBNET_NAME:-peerpod-subnet}"
```
```
$ export PEERPOD_SUBNET_CIDR="${PEERPOD_SUBNET_CIDR:-192.168.0.0/16}"
```

Set the environmental variables for Azure:

$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
    -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')

$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP}\
    --query "{Location:location}" --output tsv) && \
    echo "AZURE_REGION: \"$AZURE_REGION\""

$ AZURE_VNET_NAME=$(az network vnet list \
    -g "${AZURE_RESOURCE_GROUP}" --query '[].name' -o tsv)

Set the peer pod NAT gateway environment variables by running the following commands:

$ export PEERPOD_NAT_GW="${PEERPOD_NAT_GW:-peerpod-nat-gw}"

$ export PEERPOD_NAT_GW_IP="${PEERPOD_NAT_PUBLIC_IP:-peerpod-nat-gw-ip}"

Configure the VNET:

Create the peer pod VNet by running the following command:

$ az network vnet create --resource-group "${AZURE_RESOURCE_GROUP}" \
    --name "${PEERPOD_VNET_NAME}" \
    --address-prefixes "${PEERPOD_VNET_CIDR}"

Create a public IP address for the peer pod VNet by running the following command:

$ az network public-ip create -g "${AZURE_RESOURCE_GROUP}" \
    -n "${PEERPOD_NAT_GW_IP}" -l "${AZURE_REGION}"

Create a NAT gateway for the peer pod VNet by running the following command:

$ az network nat gateway create -g "${AZURE_RESOURCE_GROUP}" \
    -l "${AZURE_REGION}" \
    --public-ip-addresses "${PEERPOD_NAT_GW_IP}" \
    -n "${PEERPOD_NAT_GW}"

Create a subnet in the peer pod VNet and attach the NAT gateway by running the following command:

$ az network vnet subnet create \
    --resource-group "${AZURE_RESOURCE_GROUP}" \
    --vnet-name "${PEERPOD_VNET_NAME}" \
    --name "${PEERPOD_SUBNET_NAME}" \
    --address-prefixes "${PEERPOD_SUBNET_CIDR}" \
    --nat-gateway "${PEERPOD_NAT_GW}"

Configure the virtual network peering connection:

Create the peering connection by running the following command:

$ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
    -n peerpod-azure-vnet-to-peerpod-vnet \
    --vnet-name "${AZURE_VNET_NAME}" \
    --remote-vnet "${PEERPOD_VNET_NAME}" --allow-vnet-access \
    --allow-forwarded-traffic

Sync the peering connection by running the following command:

$ az network vnet peering sync -g "${AZURE_RESOURCE_GROUP}" \
    -n peerpod-azure-vnet-to-peerpod-vnet \
    --vnet-name "${AZURE_VNET_NAME}"

Complete the peering connection by running the following command:

$ az network vnet peering create -g "${AZURE_RESOURCE_GROUP}" \
    -n peerpod-peerpod-vnet-to-azure-vnet \
    --vnet-name "${PEERPOD_VNET_NAME}" \
    --remote-vnet "${AZURE_VNET_NAME}" --allow-vnet-access \
    --allow-forwarded-traffic

Verification

Check the peering connection status from the cluster VNet by running the following command:

$ az network vnet peering show -g "${AZURE_RESOURCE_GROUP}" \
    -n peerpod-azure-vnet-to-peerpod-vnet \
    --vnet-name "${AZURE_VNET_NAME}" \
    --query "peeringState" -o tsv

This should return Connected.

Verify that the NAT gateway is attached to the peer pod subnet by running the following command:

$ az network vnet subnet show --resource-group "${AZURE_RESOURCE_GROUP}" \
    --vnet-name "${PEERPOD_VNET_NAME}" --name "${PEERPOD_SUBNET_NAME}" \
    --query "natGateway.id" -o tsv

Additional Resources

Azure documentation: Content from docs.microsoft.com is not included.NAT Gateway Overview
Azure documentation: Content from docs.microsoft.com is not included.VNet Peering Overview

3.4. Installing the OpenShift sandboxed containers Operator

You install the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

Create an osc-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

Create the namespace by running the following command:
```
$ oc apply -f osc-namespace.yaml
```

Create an osc-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

Create the operator group by running the following command:
```
$ oc apply -f osc-operatorgroup.yaml
```

Create an osc-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.1

Create the subscription by running the following command:
```
$ oc create -f osc-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n openshift-sandboxed-containers-operator
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n openshift-sandboxed-containers-operator

Example output

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

3.5. Creating the osc-feature-gates config map

You enable the confidential containers feature gate by creating the config map.

Procedure

Create a my-feature-gate.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: <deployment_mode>
```
where
<deployment_mode>
On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:
MachineConfig for clusters that always use the MCO
DaemonSet for clusters that never use the MCO
DaemonSetFallback for clusters that sometimes use the MCO
Create the my-feature-gates config map by running the following command:
```
$ oc create -f my-feature-gate.yaml
```

3.6. Creating the peer pods config map

You must create the peer pods config map.

Optional: Add initdata to the peer pods config map to create a default configuration for all peer pods.

Procedure

Obtain the following values from your Azure instance:

Retrieve and record the Azure resource group:

$ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster \
  -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') \
  && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""

Retrieve and record the Azure VNet name:

$ AZURE_VNET_NAME=$(az network vnet list \
  --resource-group ${AZURE_RESOURCE_GROUP} \
  --query "[].{Name:name}" --output tsv)

This value is used to retrieve the Azure subnet ID.

Retrieve and record the Azure subnet ID:

$ AZURE_SUBNET_ID=$(az network vnet subnet list \
  --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME \
  --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) \
   && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""

Retrieve and record the Azure network security group (NSG) ID:

$ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} \
  --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""

Retrieve and record the Azure region:

$ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} \
  --query "{Location:location}" --output tsv) \
  && echo "AZURE_REGION: \"$AZURE_REGION\""

Create a peer-pods-cm.yaml manifest file according to the following example:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "azure"
  VXLAN_PORT: "9000"
  PROXY_TIMEOUT: "5m"
  AZURE_INSTANCE_SIZE: "Standard_DC2as_v5"
  AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5"
  AZURE_SUBNET_ID: "<azure_subnet_id>"
  AZURE_NSG_ID: "<azure_nsg_id>"
  AZURE_IMAGE_ID: ""
  AZURE_REGION: "<azure_region>"
  AZURE_RESOURCE_GROUP: "<azure_resource_group>"
  TAGS: "key1=value1,key2=value2"
  PEERPODS_LIMIT_PER_NODE: "10"
  ROOT_VOLUME_SIZE: "6"
  DISABLECVM: "false"
  INITDATA: "<initdata_string>"
```
AZURE_INSTANCE_SIZE
Defines the default instance size that is used if the instance size is not defined in the workload object. "Standard_DC2as_v5" is for AMD SEV-SNP. If your TEE is Intel® Trust Domain Extensions (TDX), specify Standard_EC4eds_v5.
AZURE_IMAGE_ID
Leave this value empty. When you install the Operator, a Job is scheduled to download the default pod VM image from the Red Hat Ecosystem Catalog and upload it to the Azure Image Gallery within the same Azure Resource Group as the OpenShift Container Platform cluster. This image provides root disk integrity protection (dm-verity) and encrypted container storage. See This content is not included.Confidential VMs: The core of confidential containers for details.
AZURE_INSTANCE_SIZES
Specify the allowed instance sizes, without spaces, for creating the pod. You can define smaller instance sizes for workloads that need less memory and fewer CPUs or larger instance sizes for larger workloads.
TAGS
You can configure custom tags as key:value pairs for pod VM instances to track peer pod costs or to identify peer pods in different clusters.
PEERPODS_LIMIT_PER_NODE
You can increase this value to run more peer pods on a node. The default value is 10.
ROOT_VOLUME_SIZE
You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.
INITDATA
Specify the initdata string to create a default configuration for all peer pods. If you add initdata to a peer pod manifest, that setting overrides this global configuration.
Create the config map by running the following command:
```
$ oc create -f peer-pods-cm.yaml
```

3.7. Creating the KataConfig custom resource

You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

A large OpenShift Container Platform deployment with a greater number of worker nodes.
Activation of the BIOS and Diagnostics utility.
Deployment on a hard disk drive rather than an SSD.
Deployment on physical nodes such as bare metal, rather than on virtual nodes.
A slow CPU and network.

Procedure

Create an example-kataconfig.yaml manifest file according to the following example:
```
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
```
<label_key>: '<label_value>'
Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.
Create the KataConfig CR by running the following command:
```
$ oc create -f example-kataconfig.yaml
```
The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.
Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.
Monitor the installation progress by running the following command:
```
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.
Verify the daemon set by running the following command:
```
$ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds
```

Verify the runtime classes by running the following command:

$ oc get runtimeclass

Example output

NAME           HANDLER             AGE
kata            kata                34m
kata-remote    kata-remote        152m

3.8. Creating initdata

You can specify initdata in the peer pods config map, for global configuration, or in a peer pod manifest, for a specific pod. The initdata value in a peer pod manifest overrides the value set in the peer pods config map.

Then, you generate a Platform Configuration Register (PCR) 8 hash from the initdata.toml file for the Reference Value Provider Service (RVPS) config map for Red Hat build of Trustee.

Red Hat build of Trustee uses the RVPS to validate attestation evidence sent by confidential workloads. The RVPS contains trusted reference values, such as file hashes, that are compared to the PCR measurements included in attestation requests. These hashes are not generated by Red Hat build of Trustee.

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

Procedure

Obtain the Red Hat build of Trustee URL by running the following command:

$ TRUSTEE_URL=$(oc get route kbs-service \
  -n trustee-operator-system -o jsonpath='{.spec.host}') \
  && echo $TRUSTEE_URL

Create the initdata.toml file:

algorithm = sha256
version = "0.1.0"
[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''
"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

url: Specify the Red Hat build of Trustee URL. If you configure the Red Hat build of Trustee with insecure_http for testing purposes, use HTTP. Otherwise, use HTTPS. For production systems, avoid using insecure_http unless you configure your environment to handle TLS externally, for example, with a proxy.
<kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.
image_security_policy_uri: Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:
```
$ cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
Record this string to use in the peer pods config map or the peer pod manifest.
Calculate the hash of an initdata.toml file and assign its value to the hash variable by running the following command:
```
$ hash=$(<algorithm> initdata.toml | cut -d' ' -f1)
```
Assign 32 bytes of 0s to the initial_pcr variable by running the following command:
```
$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000
```
Calculate the SHA-256 hash of hash and initial_pcr and assign its value to the PCR8_HASH variable by running the following command:
```
$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1) && echo $PCR8_HASH
```
Record the PCR8_HASH value for the RVPS config map.

3.9. Applying initdata to a pod

You can override the global INITDATA setting you applied in the peer pods config map by applying customized initdata to a specific pod for special use cases, such as development and testing with a relaxed policy, or when using different Red Hat build of Trustee configurations. You can customize initdata by adding an annotation to the workload pod YAML.

Prerequisite

You have created an initdata string.

Procedure

Add the initdata string to the pod manifest and save the file as my-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-remote
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

where

<initdata_string>: Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.
<container_name>: Specify a container name.

Create the pod by running the following command:
```
$ oc create -f my-pod.yaml
```

3.10. Verifying attestation

You can verify the attestation process by creating a test pod to retrieve a specific resource from Red Hat build of Trustee.

Important

Procedure

Create a test-pod.yaml manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 1
spec:
  runtimeClassName: kata-remote
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

1: Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

Create the pod by running the following command:
```
$ oc create -f test-pod.yaml
```
Log in to the pod by running the following command:
```
$ oc exec -it ocp-cc-pod -- bash
```
Fetch the Red Hat build of Trustee resource by running the following command:
```
$ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status
```
Example output
```
success #/
```

3.11. Configuring a pull secret for peer pods

If you select a custom peer pod VM image from a private registry such as registry.access.redhat.com, you must configure a pull secret for peer pods.

Then, you can link the pull secret to the default service account or you can specify the pull secret in the peer pod manifest.

Procedure

Set the NS variable to the namespace where you deploy your peer pods:
```
$ NS=<namespace>
```
Copy the pull secret to the peer pod namespace:
```
$ oc get secret pull-secret -n openshift-config -o yaml \
  | sed "s/namespace: openshift-config/namespace: ${NS}/" \
  | oc apply -n "${NS}" -f -
```
You can use the cluster pull secret, as in this example, or a custom pull secret.
Optional: Link the pull secret to the default service account:
```
$ oc secrets link default pull-secret --for=pull -n ${NS}
```

Alternatively, add the pull secret to the peer pod manifest:

apiVersion: v1
kind: <Pod>
spec:
  containers:
  - name: <container_name>
    image: <image_name>
  imagePullSecrets:
  - name: pull-secret
# ...

3.12. Selecting a custom peer pod VM image

You can select a custom peer pod virtual machine (VM) image, tailored to your workload requirements, by adding an annotation to the pod manifest. The custom image overrides the default image specified in the peer pods config map.

Prerequisites

If the custom peer pod VM image is in a private registry, you have created a pull secret.
You have the ID of a custom pod VM image, which is compatible with your cloud provider or hypervisor.

Procedure

Create a my-pod-manifest.yaml file according to the following example:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod-manifest
  annotations:
    io.katacontainers.config.hypervisor.image: "<custom_image_id>"
spec:
  runtimeClassName: kata-remote
  containers:
  - name: <example_container>
    image: registry.access.redhat.com/ubi9/ubi:9.3
    command: ["sleep", "36000"]

Create the pod by running the following command:
```
$ oc create -f my-pod-manifest.yaml
```

Chapter 4. Deploying confidential containers on IBM Z and IBM LinuxONE with peer pods

You can deploy confidential containers workloads on a Red Hat OpenShift Container Platform cluster running on IBM Z® and IBM® LinuxONE with peer pods.

In the peer pod approach, you leverage libvirt as the provider to launch peer pod virtual machines (VMs) on a logical partition (LPAR). The OpenShift Container Platform cluster is hosted on the same LPAR, either as a single-node or multi-node setup, where all nodes run as VMs (guests).

This approach enables flexible resource sharing and isolation in a virtualized environment, making it suitable for development, testing, or workloads that need isolation without requiring dedicated hardware.

Note

For information about installing confidential containers on IBM Cloud cluster, see Content from cloud.ibm.com is not included.Creating confidential containers in the IBM documentation.

Important

Confidential containers on IBM Z® and IBM® LinuxONE is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

4.1. Preparation

Review these prerequisites and concepts before you deploy confidential containers on IBM Z® and IBM® LinuxONE with peer pods.

Note

IBM® Hyper Protect Confidential Container (HPCC) for Red Hat OpenShift Container Platform is now production-ready. HPCC enables Confidential Computing technology at the enterprise scale by providing a multiparty Hyper Protect Contract, deployment attestation, and validation of container runtime and OCI image integrity.

HPCC is supported by IBM Z17® and IBM® LinuxONE Emperor 5. For more information, see the Content from www.ibm.com is not included.IBM HPCC documentation.

4.1.1. Prerequisites

You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.
You have installed the latest version of Red Hat OpenShift Container Platform on the cluster where you are running your confidential containers workload.
You are using LinuxONE Emperor 4 or later.
You have enabled Secure Unpack Facility on your Logical Partition (LPAR), which is necessary for the IBM Secure Execution. For more information, see Content from www.ibm.com is not included.Enabling the KVM host for IBM Secure Execution.

4.1.2. Peer pod resource requirements

You must ensure that your cluster has sufficient resources.

Peer pod virtual machines (VMs) require resources in two locations:

The worker node. The worker node stores metadata, Kata shim resources (containerd-shim-kata-v2), remote-hypervisor resources (cloud-api-adaptor), and the tunnel setup between the worker nodes and the peer pod VM.
The cloud instance. This is the actual peer pod VM running in the cloud.

The extended resource is named kata.peerpods.io/vm, and enables the Kubernetes scheduler to handle capacity tracking and accounting.

You can edit the limit per node based on the requirements for your environment after you install the OpenShift sandboxed containers Operator.

The mutating webhook modifies a Kubernetes pod as follows:

The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIMECLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIMECLASS, the webhook exits without modifying the pod.
If the RuntimeClassName values match, the webhook makes the following changes to the pod spec:
1. The webhook removes every resource specification from the resources field of all containers and init containers in the pod.
2. The webhook adds the extended resource (kata.peerpods.io/vm) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes.

Note

As a best practice, define a cluster-wide policy to only allow peer pod creation in specific namespaces.

4.1.3. Initializing pods at runtime by using initdata

You can initialize a pod with workload-specific data at runtime by creating and applying initdata.

An X.509 certificate for secure communication.
A cryptographic key for authentication.
An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

The initdata content configures the following components:

Attestation Agent (AA), which verifies the trustworthiness of the pod by sending evidence for attestation.
Confidential Data Hub (CDH), which manages secrets and secure data access within the pod VM.
Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

You create an initdata.toml file and convert it to a gzip-format Base64-encoded string.

4.2. Deployment overview

You deploy confidential containers on IBM Z® and IBM® LinuxONE with peer pods by performing the following steps:

Install the OpenShift sandboxed containers Operator.
Create the peer pods secret.
Enable the confidential containers feature gate.
Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.
Create initdata to initialize a pod with sensitive or workload-specific data at runtime.
Important
Do not use the default permissive Kata Agent policy in a production environment. You must configure a restrictive policy, preferably by creating initdata.
As a minimum requirement, you must disable ExecProcessRequest to prevent a cluster administrator from accessing sensitive data by running the oc exec command on a confidential containers pod.
Create the peer pods config map. You can add initdata to the config map to create a default global configuration for your peer pods.
Create the KataConfig CR.
Verify the attestation process.

4.3. Installing and upgrading the OpenShift sandboxed containers Operator

You can install or upgrade the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Note

You must configure the OpenShift sandboxed containers Operator subscription for manual updates by setting the value of installPlanApproval to Manual. Automatic updates are not supported.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

Create an osc-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

Create the namespace by running the following command:
```
$ oc apply -f osc-namespace.yaml
```

Create an osc-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

Create the operator group by running the following command:
```
$ oc apply -f osc-operatorgroup.yaml
```

Create an osc-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Manual
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.1

Create the subscription by running the following command:
```
$ oc create -f osc-subscription.yaml
```

Get the InstallPlan CR for the OpenShift sandboxed containers Operator by running the following command:

$ oc get installplan -n openshift-sandboxed-containers-operator

Installation example output

NAME            CSV                                      APPROVAL  APPROVED
install-bl4fl   sandboxed-containers-operator.v1.11.1    Manual    false

Upgrade example output

NAME            CSV                                     APPROVAL   APPROVED
install-jdzrb   sandboxed-containers-operator.v1.11.1   Manual     false
install-pfk8l   sandboxed-containers-operator.v1.11.0   Manual     true

Approve the manual installation by running the following command:
```
$ oc patch installplan <installplan_name> -p '{"spec":{"approved":true}}' --type=merge -n openshift-sandboxed-containers-operator
```
<installplan_name>
Specify the InstallPlan resource. For example, install-jdzrb.
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n openshift-sandboxed-containers-operator
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n openshift-sandboxed-containers-operator

Example output

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

4.4. Creating the peer pods secret

You must create a peer pods secret. The secret stores credentials for creating the pod virtual machine (VM) image and peer pod instances.

Prerequisites

LIBVIRT_URI. This value is the default gateway IP address of the libvirt network. Check your libvirt network setup to obtain this value.
Note
If libvirt uses the default bridge virtual network, you can obtain the LIBVIRT_URI by running the following commands:
```
$ virtint=$(bridge_line=$(virsh net-info default | grep Bridge);  echo "${bridge_line//Bridge:/}" | tr -d [:blank:])

$ LIBVIRT_URI=$( ip -4 addr show $virtint | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

$ LIBVIRT_GATEWAY_URI="qemu+ssh://root@${LIBVIRT_URI}/system?no_verify=1"
```
REDHAT_OFFLINE_TOKEN. You have generated this token to download the RHEL image at This content is not included.Red Hat API Tokens.
HOST_KEY_CERTS. The Host Key Document (HKD) certificate enables secure execution on IBM Z®. For more information, see Content from www.ibm.com is not included.Obtaining a host key document from Resource Link in the IBM documentation.

Procedure

Create a peer-pods-secret.yaml manifest file according to the following example:

apiVersion: v1
kind: Secret
metadata:
  name: peer-pods-secret
  namespace: openshift-sandboxed-containers-operator
type: Opaque
stringData:
  CLOUD_PROVIDER: "libvirt"
  LIBVIRT_URI: "<libvirt_gateway_uri>" 1
  REDHAT_OFFLINE_TOKEN: "<rh_offline_token>" 2
  HOST_KEY_CERTS: "<host_key_crt_value>" 3

1: Specify the libvirt URI.
2: Specify the Red Hat offline token, which is required for the Operator-built image.
3: Specify the HKD certificate value to enable IBM Secure Execution for the Operator-built image.

Create the secret by running the following command:
```
$ oc create -f peer-pods-secret.yaml
```

4.5. Creating the osc-feature-gates config map

You enable the confidential containers feature gate by creating the config map.

Procedure

Create a my-feature-gate.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: <deployment_mode>
```
where
<deployment_mode>
On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:
MachineConfig for clusters that always use the MCO
DaemonSet for clusters that never use the MCO
DaemonSetFallback for clusters that sometimes use the MCO
Create the my-feature-gates config map by running the following command:
```
$ oc create -f my-feature-gate.yaml
```

4.6. Creating the peer pods config map

You must create the peer pods config map.

Optional: Add initdata to the peer pods config map to create a default configuration for all peer pods.

Procedure

Create a peer-pods-cm.yaml manifest file according to the following example:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: peer-pods-cm
  namespace: openshift-sandboxed-containers-operator
data:
  CLOUD_PROVIDER: "libvirt"
  LIBVIRT_POOL: "<libvirt_pool>"
  LIBVIRT_VOL_NAME: "<libvirt_volume>"
  LIBVIRT_DIR_NAME: "/var/lib/libvirt/images/<directory_name>"
  LIBVIRT_NET: "default"
  PEERPODS_LIMIT_PER_NODE: "10"
  ROOT_VOLUME_SIZE: "6"
  DISABLECVM: "false"
  INITDATA: "<initdata_string>"
```
LIBVIRT_POOL
If you have manually configured the libvirt pool, use the same name as in your KVM host configuration.
LIBVIRT_VOL_NAME
If you have manually configured the libvirt volume, use the same name as in your KVM host configuration.
LIBVIRT_DIR_NAME
Specify the libvirt directory for storing virtual machine disk images, such as .qcow2, or .raw files. To ensure libvirt has read and write access permissions, use a subdirectory of the libvirt storage directory. The default is /var/lib/libvirt/images/.
LIBVIRT_NET
Specify a libvirt network if you do not want to use the default network.
PEERPODS_LIMIT_PER_NODE
You can increase this value to run more peer pods on a node. The default value is 10.
ROOT_VOLUME_SIZE
You can increase this value for pods with larger container images. Specify the root volume size in gigabytes for the pod VM. The default and minimum size is 6 GB.
INITDATA
Specify the initdata string to create a default configuration for all peer pods. If you add initdata to a peer pod manifest, that setting overrides this global configuration.
Create the config map by running the following command:
```
$ oc create -f peer-pods-cm.yaml
```

4.7. Creating the KataConfig custom resource

You must create the KataConfig custom resource (CR) to install kata-remote as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-remote as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

A large OpenShift Container Platform deployment with a greater number of worker nodes.
Activation of the BIOS and Diagnostics utility.
Deployment on a hard disk drive rather than an SSD.
Deployment on physical nodes such as bare metal, rather than on virtual nodes.
A slow CPU and network.

Procedure

Create an example-kataconfig.yaml manifest file according to the following example:
```
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
```
<label_key>: '<label_value>'
Optional: If you have applied node labels to install kata-remote on specific nodes, specify the key and value, for example, kata-remote: 'true'.
Create the KataConfig CR by running the following command:
```
$ oc create -f example-kataconfig.yaml
```
The new KataConfig CR is created and installs kata-remote as a runtime class on the worker nodes.
Wait for the kata-remote installation to complete and the worker nodes to reboot before verifying the installation.
Monitor the installation progress by running the following command:
```
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-remote is installed on the cluster.
Verify that you have built the peer pod image and uploaded it to the libvirt volume by running the following command:
```
$ oc describe configmap peer-pods-cm -n openshift-sandboxed-containers-operator
```
Example output
```
Name: peer-pods-cm
Namespace: openshift-sandboxed-containers-operator
Labels: <none>
Annotations: <none>

Data
====
CLOUD_PROVIDER: libvirt
DISABLECVM: false 1
LIBVIRT_IMAGE_ID: fa-pp-vol 2

BinaryData
====
Events: <none>
```
1
Enables the Confidential VM during IBM Secure Execution for the Operator-built image.
2
Contains a value if you have built the peer pod image and uploaded it to the libvirt volume.
Monitor the kata-oc machine config pool progress to ensure that it is in the UPDATED state, when UPDATEDMACHINECOUNT equals MACHINECOUNT, by running the following command:
```
$ watch oc get mcp/kata-oc
```
Verify the daemon set by running the following command:
```
$ oc get -n openshift-sandboxed-containers-operator ds/osc-caa-ds
```

Verify the runtime classes by running the following command:

$ oc get runtimeclass

Example output

NAME           HANDLER             AGE
kata            kata                34m
kata-remote    kata-remote        152m

4.8. Creating initdata

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

Procedure

Obtain the Red Hat build of Trustee URL by running the following command:

$ oc get node $(oc get pod -n trustee-operator-system \
  -o jsonpath='{.items[0].spec.nodeName}') \
  -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

Example output

192.168.122.22

Obtain the port by running the following command:

$ oc get svc kbs-service -n trustee-operator-system

Example output

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kbs-service  NodePort    172.30.116.11   <none>        8080:32178/TCP   12d

Create the initdata.toml file:

algorithm = sha256
version = "0.1.0"
[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''
"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

url: Specify the Red Hat build of Trustee IP address and the port, for example, https://192.168.122.22:32178.
<kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.
image_security_policy_uri: Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:
```
$ cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
Record this string to use in the peer pods config map or the peer pod manifest.

4.9. Applying initdata to a pod

Prerequisite

You have created an initdata string.

Procedure

Add the initdata string to the pod manifest and save the file as my-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-remote
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

where

<initdata_string>: Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.
<container_name>: Specify a container name.

Create the pod by running the following command:
```
$ oc create -f my-pod.yaml
```

4.10. Verifying attestation

You can verify the attestation process by creating a BusyBox pod. The pod image deploys the confidential workload where you can retrieve the key.

Important

Procedure

Create a test-pod.yaml manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 1
  labels:
    run: busybox
spec:
  runtimeClassName: kata-remote
  restartPolicy: Never
  containers:
  - name: busybox
    image: quay.io/prometheus/busybox:latest
    imagePullPolicy: Always
    command:
      - "sleep"
      - "3600"

1: Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

Create the pod by running the following command:
```
$ oc create -f test-pod.yaml
```
Log in to the pod by running the following command:
```
$ oc exec -it busybox -n default -- /bin/sh
```

Fetch the Red Hat build of Trustee resource by running the following command:

$ wget http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

Example output

Connecting to 127.0.0.1:8006 (127.0.0.1:8006)
saving to 'key1'
key1                 100% |*******************************************|     8  0:00:00 ETA
'key1' saved

Display the key1 value by running the following command:
```
$ cat key1
```
Example output
```
success #/
```

Chapter 5. Deploying confidential containers on IBM Z and IBM LinuxONE bare-metal servers

You can deploy confidential containers workloads on a Red Hat OpenShift Container Platform cluster running on IBM Z® and IBM® LinuxONE bare-metal servers.

In the bare metal approach, you launch confidential containers virtual machines (VMs) directly on a logical partition (LPAR) that is booted with Red Hat Enterprise Linux CoreOS (RHCOS). The LPAR acts as a compute node in the cluster, providing a dedicated environment for running confidential workloads.

This approach eliminates the need for intermediate peer pod components, resulting in faster boot times, quicker recovery from failures, and simpler storage integration. As a result, it is suitable for production workloads that require high performance, consistent storage behaviour, and resource management that aligns with Kubernetes standards.

Note

For information about installing confidential containers on IBM Cloud cluster, see Content from cloud.ibm.com is not included.Creating confidential containers in the IBM documentation.

Important

Confidential containers on IBM Z® and IBM® LinuxONE bare-metal servers is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

5.1. Preparation

Review these prerequisites and concepts before you deploy confidential containers on IBM Z® and IBM® LinuxONE bare-metal servers.

5.1.1. Prerequisites

You have deployed Red Hat build of Trustee on an OpenShift Container Platform cluster in a trusted environment. For more information, see Deploying Red Hat build of Trustee.
You have installed the latest version of Red Hat OpenShift Container Platform on the cluster where you are running your confidential containers workload.

5.1.2. Initializing pods at runtime by using initdata

You can initialize a pod with workload-specific data at runtime by creating and applying initdata.

An X.509 certificate for secure communication.
A cryptographic key for authentication.
An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

The initdata content configures the following components:

Attestation Agent (AA), which verifies the trustworthiness of the pod by sending evidence for attestation.
Confidential Data Hub (CDH), which manages secrets and secure data access within the pod VM.
Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

You create an initdata.toml file and convert it to a gzip-format Base64-encoded string.

You add the initdata string as an annotation to a pod manifest, allowing customization for individual workloads.

5.2. Deployment overview

You deploy confidential containers on IBM Z® and IBM® LinuxONE bare-metal servers by performing the following steps:

Install the OpenShift sandboxed containers Operator.
Configure auto-detection of TEEs.
Enable the confidential containers feature gate.
Create initdata to initialize a peer pod with sensitive or workload-specific data at runtime.
Upload a Secure Execution image to the container registry.
Create the kata-addon-artifacts config map.
Create initdata to initialize a pod with sensitive or workload-specific data at runtime.
Important
Do not use the default permissive Kata Agent policy in a production environment. You must configure a restrictive policy, preferably by creating initdata.
As a minimum requirement, you must disable ExecProcessRequest to prevent a cluster administrator from accessing sensitive data by running the oc exec command on a confidential containers pod.
Apply initdata to a pod.
Create the KataConfig CR.
Verify the attestation process.

5.3. Installing and upgrading the OpenShift sandboxed containers Operator

You can install or upgrade the OpenShift sandboxed containers Operator by using the command line interface (CLI).

Note

You must configure the OpenShift sandboxed containers Operator subscription for manual updates by setting the value of installPlanApproval to Manual. Automatic updates are not supported.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.

Procedure

Create an osc-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sandboxed-containers-operator

Create the namespace by running the following command:
```
$ oc apply -f osc-namespace.yaml
```

Create an osc-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sandboxed-containers-operator-group
  namespace: openshift-sandboxed-containers-operator
spec:
  targetNamespaces:
  - openshift-sandboxed-containers-operator

Create the operator group by running the following command:
```
$ oc apply -f osc-operatorgroup.yaml
```

Create an osc-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sandboxed-containers-operator
  namespace: openshift-sandboxed-containers-operator
spec:
  channel: stable
  installPlanApproval: Manual
  name: sandboxed-containers-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: sandboxed-containers-operator.v1.11.1

Create the subscription by running the following command:
```
$ oc create -f osc-subscription.yaml
```

Get the InstallPlan CR for the OpenShift sandboxed containers Operator by running the following command:

$ oc get installplan -n openshift-sandboxed-containers-operator

Installation example output

NAME            CSV                                      APPROVAL  APPROVED
install-bl4fl   sandboxed-containers-operator.v1.11.1    Manual    false

Upgrade example output

NAME            CSV                                     APPROVAL   APPROVED
install-jdzrb   sandboxed-containers-operator.v1.11.1   Manual     false
install-pfk8l   sandboxed-containers-operator.v1.11.0   Manual     true

Approve the manual installation by running the following command:
```
$ oc patch installplan <installplan_name> -p '{"spec":{"approved":true}}' --type=merge -n openshift-sandboxed-containers-operator
```
<installplan_name>
Specify the InstallPlan resource. For example, install-jdzrb.
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n openshift-sandboxed-containers-operator
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n openshift-sandboxed-containers-operator

Example output

NAME                             DISPLAY                                  VERSION             REPLACES                   PHASE
openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.11.1    1.11.0        Succeeded

5.4. Configuring auto-detection of TEEs

You must configure your nodes so that the OpenShift sandboxed containers Operator can detect the Trusted Execution Environments (TEEs).

You label the nodes by installing and configuring the Node Feature Discovery (NFD) Operator.

5.4.1. Creating a NodeFeatureDiscovery custom resource

Prerequisites

You have installed the NFD Operator. For more information, see Node Feature Discovery Operator in the OpenShift Container Platform documentation.

Procedure

Create a my-nfd.yaml manifest file according to the following example:

apiVersion: nfd.openshift.io/v1
kind: NodeFeatureDiscovery
metadata:
  name: nfd-instance
  namespace: openshift-nfd
spec:
  operand:
    image: registry.redhat.io/openshift4/ose-node-feature-discovery-rhel9:v4.20
    imagePullPolicy: Always
    servicePort: 12000
  workerConfig:
    configData: |

Create the NodeFeatureDiscovery CR:
```
$ oc create -f my-nfd.yaml
```

5.4.2. Creating the NodeFeatureRule custom resource

Procedure

Create a custom resource manifest named my-nodefeaturerule.yaml:

apiVersion: nfd.openshift.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: consolidated-hardware-features
spec:
  rules:
    - name: "runtime.kata"
      labels:
        feature.node.kubernetes.io/runtime.kata: "true"
      matchFeatures:
        - feature: cpu.cpuid
          matchExpressions:
            SIE: { op: Exists }
            ZARCH: { op: Exists }

    - name: "ibm.se.enabled"
      labels:
        ibm.feature.node.kubernetes.io/se: "true"
      matchAll:
        - matchFeatures:
            - feature: cpu.security
              matchExpressions:
                se.enabled: { op: IsTrue }

Create the NodeFeatureRule CR by running the following command:
```
$ oc create -f my-nodefeaturerule.yaml
```

Note

A relabeling delay of up to 1 minute might occur.

5.5. Creating the osc-feature-gates config map

You enable the confidential containers feature gate by creating the config map.

Bare metal solutions on IBM Z® and IBM® LinuxONE now support only a DaemonSet deployment approach. This method uses the prebuilt image pull process to ensure the virtual machine (VM) runs a Secure Execution-enabled kernel image.

Procedure

Create a my-feature-gate.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: osc-feature-gates
  namespace: openshift-sandboxed-containers-operator
data:
  confidential: "true"
  deploymentMode: daemonset
```
where
<deployment_mode>
On OpenShift Container Platform clusters with the Machine Config Operator (MCO), the deploymentMode field is optional and can be omitted. Specifies the strategy for installing and configuring the Kata runtime. Specify the deployment mode:
MachineConfig for clusters that always use the MCO
DaemonSet for clusters that never use the MCO
DaemonSetFallback for clusters that sometimes use the MCO
Create the my-feature-gates config map by running the following command:
```
$ oc create -f my-feature-gate.yaml
```

5.6. Uploading a Secure Execution image to the container registry

You can either use a custom Secure Execution image or an IBM® Hyper Protect Confidential Container (HPCC) image to deploy confidential containers on IBM Z and IBM LinuxONE bare-metal servers.

You must build a Secure Execution image, create a Dockerfile and push the image to your container registry.

Procedure

Build a Secure Execution image.
Create a Dockerfile file for the Secure Execution image:
```
FROM alpine:3.20
RUN mkdir -p /images
COPY ./<image_name> /images/<image_name>
RUN chmod 644 /images/<image_name>
```
<image_name>
Specify the custom Secure Execution image name. For example, se.img.
Build a container image with a custom tag from the Dockerfile:
```
$ docker build -t <registry_name>/<user_name>/kata-se-artifacts:<image_tag> .
```

Push the container image to your registry:

$ docker push <registry_name>/<user_name>/kata-se-artifacts:<image_tag>

5.7. Creating the kata-addon-artifacts config map

You must create the kata-addon-artifacts config map to enable the use of custom kernel artifacts from container images when deploying in daemon set mode.

Important

If you are using the IBM® Hyper Protect Confidential Container (HPCC) image, see Content from www.ibm.com is not included.IBM HPCC documentation for further procedure information.

Procedure

Create the kata-addon-artifacts.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: kata-addon-artifacts
  namespace: openshift-sandboxed-containers-operator
data:
  addonImage: "<container_image_path>"
  kernelPath: "<kernel_path>"
```
<container_image_path>
Specify the path to your container image in the registry. For example, quay.io/openshift_sandboxed_containers/kata-se-artifacts:v1.0.
<kernel_path>
Specify the kernel path inside the image. For example, /images/se.img.
Create the kata-addon-artifacts config map by running the following command:
```
$ oc create -f kata-addon-artifacts.yaml
```

5.8. Creating the KataConfig custom resource

You must create the KataConfig custom resource (CR) to install kata-cc as a runtime class on your worker nodes.

OpenShift sandboxed containers installs kata-cc as a secondary, optional runtime on the cluster and not as the primary runtime.

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. The following factors can increase the reboot time:

A large OpenShift Container Platform deployment with a greater number of worker nodes.
Activation of the BIOS and Diagnostics utility.
Deployment on a hard disk drive rather than an SSD.
Deployment on physical nodes such as bare metal, rather than on virtual nodes.
A slow CPU and network.

Procedure

Create an example-kataconfig.yaml manifest file according to the following example:
```
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
  name: example-kataconfig
spec:
  enablePeerPods: false
  checkNodeEligibility: true
  logLevel: info
#  kataConfigPoolSelector:
#    matchLabels:
#      <label_key>: '<label_value>'
```
<label_key>: '<label_value>'
Optional: If you have applied node labels to install kata-cc on specific nodes, specify the key and value, for example, kata-cc: 'true'.
Create the KataConfig CR by running the following command:
```
$ oc create -f example-kataconfig.yaml
```
The new KataConfig CR is created and installs kata-cc as a runtime class on the worker nodes.
Wait for the kata-cc installation to complete and the worker nodes to reboot before verifying the installation.
Monitor the installation progress by running the following command:
```
$ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"
```
When the status of all workers under kataNodes is installed and the condition InProgress is False without specifying a reason, the kata-cc is installed on the cluster.

Verify the runtime classes by running the following command:

$ oc get runtimeclass

Example output

NAME           HANDLER             AGE
kata            kata                34m
kata-cc    kata-se        152m

5.9. Creating initdata

You can specify initdata in the pod manifest, for a specific pod.

Important

You must delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.

Procedure

Obtain the Red Hat build of Trustee URL by running the following command:

$ oc get node $(oc get pod -n trustee-operator-system \
  -o jsonpath='{.items[0].spec.nodeName}') \
  -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

Example output

192.168.122.22

Obtain the port by running the following command:

$ oc get svc kbs-service -n trustee-operator-system

Example output

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kbs-service  NodePort    172.30.116.11   <none>        8080:32178/TCP   12d

Create the initdata.toml file:

algorithm = sha256
version = "0.1.0"
[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]

url = '<trustee_url>'

[token_configs.kbs]
url = '<trustee_url>'
'''
"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '<trustee_url>'
kbs_cert = """
-----BEGIN CERTIFICATE-----
<kbs_certificate>
-----END CERTIFICATE-----
"""
[image]
image_security_policy_uri = 'kbs:///default/<secret-policy-name>/<key>
'''

"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default ExecProcessRequest := false
default SetPolicyRequest := false
default WriteStreamRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

policy_data := {
  "allowed_commands": [
        "curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status"
  ]
}
'''

url: Specify the Red Hat build of Trustee IP address and the port, for example, https://192.168.122.22:32178.
<kbs_certificate>: Specify the Base64-encoded TLS certificate for the attestation agent.
kbs_cert: Delete the kbs_cert setting if you configure insecure_http = true in the kbs-config config map for Red Hat build of Trustee.
image_security_policy_uri: Optional, only if you enabled the container image signature verification policy. Replace <secret-policy-name> and <key> with the secret name and key, respectively specified in Creating the KbsConfig custom resource.

Convert the initdata.toml file to a gzipped, Base64-encoded string in a text file by running the following command:
```
$ cat initdata.toml | gzip | base64 -w0 > initdata.txt
```
Record this string to use in the pod manifest.

5.10. Applying initdata to a pod

You can override the global INITDATA setting by applying customized initdata to a specific pod for special use cases, such as development and testing with a relaxed policy, or when using different Red Hat build of Trustee configurations. You can customize initdata by adding an annotation to the workload pod YAML.

Prerequisite

You have created an initdata string.

Procedure

Add the initdata string to the pod manifest and save the file as my-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string>
spec:
  runtimeClassName: kata-cc
  containers:
  - name: <container_name>
    image: registry.access.redhat.com/ubi9/ubi:latest
    command:
    - sleep
    - "36000"
    securityContext:
      privileged: false
      seccompProfile:
        type: RuntimeDefault

where

<initdata_string>: Specify the gzipped, Base64-encoded initdata value in a pod annotation to override the global INITDATA setting in the peer pods config map.
<container_name>: Specify a container name.

Create the pod by running the following command:
```
$ oc create -f my-pod.yaml
```

5.11. Verifying attestation

You can verify the attestation process by creating a test pod to retrieve a specific resource from Red Hat build of Trustee.

Important

Procedure

Create a test-pod.yaml manifest file:

apiVersion: v1
kind: Pod
metadata:
  name: ocp-cc-pod
  labels:
    app: ocp-cc-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 1
spec:
  runtimeClassName: kata-cc
  containers:
    - name: skr-openshift
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
metadata:
  name: coco-test-pod
  labels:
    app: coco-test-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: <initdata_string> 2
spec:
  runtimeClassName: kata-cc
  containers:
    - name: test-container
      image: registry.access.redhat.com/ubi9/ubi:9.3
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

1 2: Optional: Setting initdata in a pod annotation overrides the global INITDATA setting in the peer pods config map.

Create the pod by running the following command:
```
$ oc create -f test-pod.yaml
```
Log in to the pod by running the following command:
```
$ oc exec -it ocp-cc-pod -- bash
```
Fetch the Red Hat build of Trustee resource by running the following command:
```
$ curl http://127.0.0.1:8006/cdh/resource/default/attestation-status/status
```
Example output
```
success #/
```

Chapter 6. Uninstalling confidential containers

You uninstall confidential containers by uninstalling OpenShift sandboxed containers and its components on your workload cluster.

Then, you uninstall the Red Hat build of Trustee Operator and its components. See This content is not included.Uninstalling Red Hat build of Trustee for details.

You uninstall OpenShift sandboxed containers by performing the following tasks:

Delete the workload pods.
Delete the KataConfig custom resource (CR).
Uninstall the OpenShift sandboxed containers Operator.
Delete the KataConfig custom resource definition (CRD).

Important

You must delete the workload pods before deleting the KataConfig CR. The pod names usually have the prefix podvm and custom tags, if provided. If you deployed OpenShift sandboxed containers on a cloud provider and any resources remain after following these procedures, you might receive an unexpected bill for those resources from your cloud provider. Once you complete uninstalling OpenShift sandboxed containers on a cloud provider, check the cloud provider console to ensure that the procedures deleted all of the resources.

6.1. Deleting workload pods

You can delete the OpenShift sandboxed containers workload pods by using the CLI.

Prerequisites

You have the JSON processor (jq) utility installed.

Procedure

Search for the pods by running the following command:

$ oc get pods -A -o json | jq -r '.items[] | \
  select(.spec.runtimeClassName == "<runtime>").metadata.name'

Delete each pod by running the following command:
```
$ oc delete pod <pod>
```

Important

When uninstalling OpenShift sandboxed containers deployed using a cloud provider, you must delete all of the pods. Any remaining pod resources might result in an unexpected bill from your cloud provider.

6.2. Deleting the KataConfig custom resource

You delete the KataConfig custom resource (CR) by using the command line.

Procedure

Delete the KataConfig CR by running the following command:
```
$ oc delete kataconfig example-kataconfig
```
Verify the CR removal by running the following command:
```
$ oc get kataconfig example-kataconfig
```
Example output
```
No example-kataconfig instances exist
```

Important

You must ensure that all pods are deleted. Any remaining pod resources might result in an unexpected bill from your cloud provider.

6.3. Uninstalling the OpenShift sandboxed containers Operator

You uninstall the OpenShift sandboxed containers Operator by using the command line.

Procedure

Delete the subscription by running the following command:

$ oc delete subscription sandboxed-containers-operator -n openshift-sandboxed-containers-operator

Delete the namespace by running the following command:
```
$ oc delete namespace openshift-sandboxed-containers-operator
```

6.4. Deleting the KataConfig CRD

You delete the KataConfig custom resource definition (CRD) by using the command line.

Prerequisites

You have deleted the KataConfig custom resource.
You have uninstalled the OpenShift sandboxed containers Operator.

Procedure

Delete the KataConfig CRD by running the following command:
```
$ oc delete crd kataconfigs.kataconfiguration.openshift.io
```
Verify that the CRD was deleted by running the following command:
```
$ oc get crd kataconfigs.kataconfiguration.openshift.io
```
Example output
```
Unknown CRD kataconfigs.kataconfiguration.openshift.io
```

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.