Deploying Red Hat build of Trustee

OpenShift sandboxed containers 1.11

Secure management and attestation of confidential containers workloads

Red Hat Customer Content Services

Abstract

Red Hat build of Trustee provides secure management and attestation of confidential containers workloads. You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment. Then, you deploy confidential containers on a separate cluster for your workload.

Preface

Chapter 1. Overview

Learn about Red Hat build of Trustee features and terminology. You must ensure that your OpenShift Container Platform environment is compatible.

1.1. About Red Hat build of Trustee

Red Hat build of Trustee is a critical component of the confidential containers solution for OpenShift sandboxed containers. Red Hat build of Trustee enables secure management and attestation of confidential workloads running within Trusted Execution Environments (TEEs) on a Red Hat OpenShift Container Platform cluster.

Red Hat build of Trustee is a service that facilitates the deployment and management of confidential containers. It provides attestation and key management services to ensure the integrity and confidentiality of workloads running in TEEs.

You deploy Red Hat build of Trustee on a separate OpenShift Container Platform cluster in a trusted environment, not on the cluster that hosts the primary workload. This separation enhances security by isolating sensitive operations from the untrusted cloud infrastructure.

Red Hat build of Trustee performs the following key functions:

Verifies the integrity of the TEE and the workloads running within it, ensuring that it only executes trusted code and data.
Securely manages cryptographic keys and secrets required by confidential containers, protecting sensitive data from unauthorized access.
Simplifies the configuration of TEEs through the KbsConfig custom resource, enabling seamless integration with OpenShift Container Platform workflows.

By leveraging Red Hat build of Trustee, OpenShift Container Platform users can deploy confidential workloads using familiar tools while maintaining strong security guarantees, even on shared or third-party infrastructure.

1.2. Providing feedback on Red Hat documentation

You can provide feedback or report an error by submitting the Create Issue form in Jira:

Ensure that you are logged in to Jira. If you do not have a Jira account, you must create a This content is not included.Red Hat Jira account.
Launch the This content is not included.Create Issue form.
Complete the Summary, Description, and Reporter fields.
In the Description field, include the documentation URL, chapter or section number, and a detailed description of the issue.
Click Create.

Chapter 2. Deploying Red Hat build of Trustee for workloads running on bare metal servers

You can deploy Red Hat build of Trustee for confidential containers workloads running on bare metal servers. bare metal servers in a disconnected network environment.

Optionally, you can deploy Red Hat build of Trustee with a cluster-wide proxy. For details, see Configuring the cluster-wide proxy in the OpenShift Container Platform documentation.

2.1. Prerequisites

You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing OpenShift Container Platform on bare metal.

2.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

Install the Red Hat build of Trustee Operator.
Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
Create the Reference Value Provider Service (RVPS) config map.
Create the attestation policy config map.
Optional: Create a config map for Intel® TDX.
Optional: Create a secret for custom keys clients.
Optional: Create a secret for container image signature verification.
Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
Create the resource policy config map.
Create the KBSConfig CR.
Create the cluster route.
Create the authentication secret.
Verify the Red Hat build of Trustee configuration.

2.3. Installing the Red Hat build of Trustee Operator

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI tool (oc).

Procedure

Create a trustee-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system

Create the trustee-operator-system namespace by running the following command:
```
$ oc create -f trustee-namespace.yaml
```

Create a trustee-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system

Create the operator group by running the following command:
```
$ oc create -f trustee-operatorgroup.yaml
```

Create a trustee-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator-system
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the subscription by running the following command:
```
$ oc create -f trustee-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n trustee-operator-system
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n trustee-operator-system

Example output

NAME                      DISPLAY                        PHASE
trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded

2.4. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

Create a kbs-config-cm.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.toml: |
    [http_server]
    sockets = ["0.0.0.0:8080"]
    insecure_http = false
    private_key = "/etc/https-key/tls.key"
    certificate = "/etc/https-cert/tls.crt"
    worker_count = 4

    [admin]
    insecure_api = false

    auth_public_key = "/etc/auth-secret/publicKey"

    [attestation_token]
    attestation_token_type = "CoCo"

    [attestation_service]
    type = "coco_as_builtin"
    work_dir = "/opt/confidential-containers/attestation-service"
    policy_engine = "opa"

    [attestation_service.attestation_token_broker]
    type = "Ear"
    policy_dir = "/opt/confidential-containers/attestation-service/policies"


    [attestation_service.attestation_token_config]
    duration_min = 5

    [attestation_service.rvps_config]
    type = "BuiltIn"

    [attestation_service.rvps_config.storage]
    type = "LocalJson"
    file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"

    [[plugins]]
    name = "resource"
    type = "LocalFs"
    dir_path = "/opt/confidential-containers/kbs/repository"

    [policy_engine]
    policy_path = "/opt/confidential-containers/opa/policy.rego"

Create the config map by running the following command:
```
$ oc create -f kbs-config-cm.yaml
```

2.5. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The client collects measurements from the running software, the TEE hardware and firmware and it submits a quote with the claims to the Attestation Server. These measurements must match the trusted digests registered to Red Hat build of Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with.

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

Create an rvps-configmap.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
    ]

Create the RVPS config map by running the following command:
```
$ oc create -f rvps-configmap.yaml
```

2.6. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.

This policy checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 values against the reference values to ensure that the confidential containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. For details, see Content from uapi-group.org is not included.Linux TPM PCR Registry in the UAPI Group Specifications documentation.

Procedure

Create an attestation-policy.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default_cpu.rego: |
    package policy

    import rego.v1
    default executables := 33
    default hardware := 97
    default configuration := 36

    ##### Azure vTPM SNP
    executables := 3 if {
      input.azsnpvtpm.measurement in data.reference.measurement
      input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
    }

    hardware := 2 if {
      input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
      input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
      input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
      input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
    }

    configuration := 2 if {
      input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
      input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
      input.azsnpvtpm.policy_abi_major in data.reference.abi_major
      input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
      input.azsnpvtpm.policy_single_socket in data.reference.single_socket
      input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
    }

    ##### Azure vTPM TDX
    executables := 3 if {
      input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
      input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
      input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
      input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
      input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
    }

    hardware := 2 if {
      # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
      input.aztdxvtpm.quote.header.tee_type == "81000000"
      input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"

      # Check TDX Module version and its hash. Also check OVMF code hash.
      # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
      # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
      # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
    }

    configuration := 2 if {
      # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
    }

Create the attestation policy config map by running the following command:
```
$ oc create -f attestation-policy.yaml
```

2.7. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

Create a tdx-config.yaml manifest file according to the following example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tdx-config
  namespace: trustee-operator-system
data:
  sgx_default_qcnl.conf: |
    {
      "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
    }

Create the tdx-config config map by running the following command:
```
$ oc create -f tdx-config.yaml
```

2.8. Creating a secret with custom keys for clients

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

You have created one or more custom keys.

Procedure

Create a secret for the custom keys according to the following example:
```
$ oc create secret generic attestation-status \
  --from-literal key1=<custom_key1> \ 1
  --from-literal key2=<custom_key2> \
  -n trustee-operator-system
```
1
Specify a custom key.
You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

2.9. Creating a secret for container image signature verification

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use This content is not included.Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

Create a secret for container image signature verification by running the following command:
```
$ oc create secret generic <type> \ 1
  --from-file=<tag>=./<public_key_file> \ 2
  -n trustee-operator-system
```
1
Specify the KBS secret type, for example, img-sig.
2
Specify the secret tag, for example, pub-key, and the public container image signing key.
Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

2.10. Creating the container image signature verification policy

You configure the container image signature verification policy. Signature verification is disabled by default. To enable signature verification for your container images, follow the procedure. For more information, see Content from github.com is not included.containers-policy.json 5.

Note

Both the signature keys and the corresponding policy must be added to Red Hat build of Trustee. The following procedure describes only how to add the policy itself. For more information about signature keys, see Creating the attestation token secret.

Procedure

Create a security-policy-config.json file according to the following example:
```
{
  "default": [
      {
      "type": "reject" 1
      }
  ],
  "transports": {
      "<transport>": { 2
          "<registry>/<image>": 3
          [
              {
                  "type": "sigstoreSigned",
                  "keyPath": "kbs:///default/<type>/<tag>" 4
              }
          ]
      }
  }
}
```
1
By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
2
Specify the image repository for transport, for example, "docker":. For more information, see Content from github.com is not included.containers-transports 5.
3
Specify the container registry and image, for example, "quay.io/my-image".
4
Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
Create the security policy by running the following command:
```
$ oc create secret generic <security-policy-name> \
  --from-file=<osc-key>=./<security-policy-config.json> \
  -n trustee-operator-system
```
The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

2.11. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

Create a resourcepolicy-configmap.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    package policy
    default allow = true
    allow {
      input["submods"]["cpu0"]["ear.status"] == "affirming"
    }
```
policy.rego
The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
package policy
The resource policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.
Create the resource policy config map by running the following command:
```
$ oc create -f resourcepolicy-configmap.yaml
```

2.12. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

Create an edge route by running the following command:

$ oc create route passthrough --service=kbs-service --port kbs-port \
  -n trustee-operator-system

Set the TRUSTEE_HOST variable by running the following command:

$ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})

Verify the route by running the following command:
```
$ echo $TRUSTEE_HOST
```
Example output
```
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
```

2.13. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

Create a private key by running the following command:
```
$ openssl genpkey -algorithm ed25519 > privateKey
```
Create a public key by running the following command:
```
$ openssl pkey -in privateKey -pubout -out publicKey
```

Create a secret by running the following command:

$ oc create secret generic kbs-auth-public-key \
  --from-file=publicKey -n trustee-operator-system

Verify the secret by running the following command:
```
$ oc get secret -n trustee-operator-system
```

2.14. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

Create a kbsconfig-cr.yaml manifest file:

apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources:
  - attestation-status
  - <security-policy-name>
  kbsResourcePolicyConfigMapName: resource-policy
  kbsHttpsKeySecretName: kbs-https-key
  kbsHttpsCertSecretName: kbs-https-certificate
  kbsAttestationCertSecretName: attestation-cert
  kbsAttestationKeySecretName: attestation-key
# tdxConfigSpec:
#   kbsTdxConfigMapName: tdx-config
# kbsServiceType: <service_type>

kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.

Create the KbsConfig CR by running the following command:
```
$ oc create -f kbsconfig-cr.yaml
```

2.15. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

Set the default project by running the following command:
```
$ oc project trustee-operator-system
```

Check the pods by running the following command:

$ oc get pods -n trustee-operator-system

Example output

NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

Set the POD_NAME environmental variable by running the following command:

$ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

Check the pod logs by running the following command:

$ oc logs -n trustee-operator-system $POD_NAME

Example output

[2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime

2.16. Installing Red Hat build of Trustee in a disconnected environment

You can deploy Red Hat build of Trustee for confidential containers workloads running on

2.16.1. Prerequisites

You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing a user-provisioned bare metal cluster on a disconnected environment

2.16.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

Install the Red Hat build of Trustee Operator.
Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
Optional: Download VCEK certificates for AMD SEV-SNP nodes.
Create the Reference Value Provider Service (RVPS) config map.
Create the attestation policy config map.
Optional: Create a config map for Intel® TDX.
Optional: Create a secret for custom keys clients.
Optional: Create a secret for container image signature verification.
Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
Create the resource policy config map.
Create the KBSConfig CR.
Create the cluster route.
Create the authentication secret.
Verify the Red Hat build of Trustee configuration.

2.16.3. Installing the Red Hat build of Trustee Operator

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI tool (oc).
You have disabled the default catalog sources and mirrored the Operator catalog. For details, see Using Operator Lifecycle Manager in disconnected environments in the OpenShift Container Platform documentation.

Procedure

Create a trustee-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system

Create the trustee-operator-system namespace by running the following command:
```
$ oc create -f trustee-namespace.yaml
```

Create a trustee-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system

Create the operator group by running the following command:
```
$ oc create -f trustee-operatorgroup.yaml
```

Create a trustee-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator-system
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the subscription by running the following command:
```
$ oc create -f trustee-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n trustee-operator-system
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n trustee-operator-system

Example output

NAME                      DISPLAY                        PHASE
trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded

2.16.4. Creating HTTPS secrets

Generate keys to securely launch Red Hat build of Trustee and enables services to use HTTPS.

Procedure

Set the DOMAIN variable for the cluster by running the following command:
```
$ DOMAIN=$(oc get ingress.config/cluster -o jsonpath='{.spec.domain}')
```
Set the NS variable for the Red Hat build of Trustee namespace by running the following command:
```
$ NS=trustee-operator-system
```
Set the ROUTE_NAME variable by running the following command:
```
$ ROUTE_NAME=kbs-service
```
Set the ROUTE variable to the full DNS name by running the following command:
```
$ ROUTE="${ROUTE_NAME}-${NS}.${DOMAIN}"
```
Generate a private SSL/TLS key and certificate for Red Hat build of Trustee by running the following command:
```
$ openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout tls.key \
  -out tls.crt \
  -subj "/CN=<custom_cn>/O=<custom_org>" \
  -addext "subjectAltName=DNS:${ROUTE}"
```
- <custom_cn> is a custom CN. For example: kbs-trustee-operator-system.
- <custom_org> is a name of your organization.
Create the kbs-https-certificate secret in the trustee-operator-system namespace by running the following command:
```
$ oc create secret generic kbs-https-certificate --from-file=tls.crt -n trustee-operator-system
```
Create the kbs-https-key secret in the trustee-operator-system namespace by running the following command:
```
$ oc create secret generic kbs-https-key --from-file=tls.key -n trustee-operator-system
```

2.16.5. Creating the attestation token secrets

Generate an attestation token key and certificate for Red Hat build of Trustee.

Procedure

Generate a private elliptic curve SSL key called token.key by running the following command:
```
$ openssl ecparam -name prime256v1 -genkey -noout -out token.key
```
Create the attestation-key secret from the SSL/TLS key in the trustee-operator-system namespace:
```
$ oc create secret generic attestation-key \
  --from-file=token.key \
  -n trustee-operator-system
```
Generate a self-signed SSL/TLS certificate from the private SSL key by running the following command:
```
$ openssl req -new -x509 -key token.key -out token.crt -days 365 \
  -subj "/CN=<custom_cn>/O=<custom_org>"
```
- <custom_cn>: Specify the Common Name. For example: kbs-trustee-operator-system.
- <custom_org>: Specify your organization name.
Create the attestation-cert secret from the SSL/TLS key and certificate in the trustee-operator-system namespace:
```
$ oc create secret generic attestation-cert \
  --from-file=token.crt \
  -n trustee-operator-system
```

Create the attestation-status secret used for verifying the attestation process:

$ oc create secret generic attestation-status \
  --from-literal=status=success \
  -n trustee-operator-system

2.16.6. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

Create a kbs-config-cm.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.toml: |
    [http_server]
    sockets = ["0.0.0.0:8080"]
    insecure_http = false
    private_key = "/etc/https-key/tls.key"
    certificate = "/etc/https-cert/tls.crt"
    worker_count = 4

    [admin]
    insecure_api = false

    auth_public_key = "/etc/auth-secret/publicKey"

    [attestation_token]
    insecure_key = false
    trusted_certs_paths = ["/etc/attestation-cert/token.crt"]
    attestation_token_type = "CoCo"

    [attestation_service]
    type = "coco_as_builtin"
    work_dir = "/opt/confidential-containers/attestation-service"
    policy_engine = "opa"

    [attestation_service.attestation_token_broker]
    type = "Ear"
    policy_dir = "/opt/confidential-containers/attestation-service/policies"

    [attestation_service.attestation_token_broker.signer]
    key_path = "/etc/attestation-key/token.key"
    cert_path = "/etc/attestation-cert/token.crt"

    [attestation_service.attestation_token_config]
    duration_min = 5

    [attestation_service.rvps_config]
    type = "BuiltIn"

    [attestation_service.rvps_config.storage]
    type = "LocalJson"
    file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"

    [[plugins]]
    name = "resource"
    type = "LocalFs"
    dir_path = "/opt/confidential-containers/kbs/repository"

    [policy_engine]
    policy_path = "/opt/confidential-containers/opa/policy.rego"

Create the config map by running the following command:
```
$ oc create -f kbs-config-cm.yaml
```

2.16.7. Downloading VCEK certificates for AMD SEV-SNP

For each AMD SEV-SNP node, you must download a Versioned Chip Endorsement Key (VCEK) certificate so that Red Hat build of Trustee can sign the attestation report generated by the confidential containers virtual machine (VM).

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI tool (oc).
You have installed the podman utility.
You have installed the jq utility.

Procedure

Perform the following steps on the worker node:

Set the nodename variable by running the following command:
```
$ export nodename=<node_name>
```

Retrieve the VCEK URL by running the following command:

$ sudo podman run --privileged \
  quay.io/openshift_sandboxed_containers/coco-tools:latest \
  /tools/snphost show vcek-url \
  | cut -d '?' -f 1  > vcek_base_url_$nodename.txt

Obtain the VCEK URL by running the following command:

$ VCEK_BASE_URL=$(cat vcek_base_url_$nodename.txt) \
&& echo $VCEK_BASE_URL

The command prints the full VCEK URL, including query parameters.

Example output

https://kdsintf.amd.com/vcek/v1/Genoa/126A06A956658EF040D5DF52D438BB78DBB8E8C1C6244B420513B00ECCF1E6E2456C925FD913E8F5A80C209F83052077871919E4F1EBB62B296B36E084B57095

Create a my-pod.yaml pod manifest according to the following example:

apiVersion: v1
kind: Pod
metadata:
  name: coco-guest
  labels:
    app: coco-guest
  annotations:
    io.katacontainers.config.hypervisor.default_memory: "4096"
    io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
    io.katacontainers.config.hypervisor.cc_init_data: <data>
spec:
  runtimeClassName: kata-cc
  nodeName: <node_name>
  containers:
    - name: snp-guest
      image: registry.access.redhat.com/ubi9/ubi:latest
      command:
        - sleep
        - "36000"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault

<data>: Specify the gzip-format, Base64-encoded initdata string. For more information, see Creating initdata.

Create the pod by running the following command:
```
$ oc create pod -f my-pod.yaml
```

Obtain the pod attestation report by running the following command:

$ oc exec -it coco-guest -- curl http://127.0.0.1:8006/aa/evidence?runtime_data=test > <attestation_report>-$nodename.txt

<attestation_report>

Specify the attestation report file name.

Example output

{
  "attestation_report": {
    "version": 2,
    "guest_svn": 0,
    "policy": 196608,
    ...
    "reported_tcb": {
      "bootloader": 3,
      "tee": 0,
      "_reserved": [0, 0, 0, 0],
      "snp": 14,
      "microcode": 213
    },
   ...
  },
  "cert_chain": null
}

Set the bootloader, tee, snp, and microcode values by running the following command:

$ eval $(cat <attestation-report-name> | jq -r '
  .attestation_report.reported_tcb |
  "export bootloader=\(.bootloader);
   export tee=\(.tee);
   export snp=\(.snp);
   export microcode=\(.microcode);"
')

echo "Bootloader: $bootloader"
echo "TEE: $tee"
echo "SNP: $snp"
echo "Microcode: $microcode"

Set the VCEK_URL_NODE1 variable by running the following command:

$ export VCEK_URL_NODE1="$VCEK_BASE_URL?blSPL=$bootloader&teeSPL=$tee&snpSPL=$snp&ucodeSPL=$microcode"

Create a separate directory for each processor type by running the following commands:
```
$ mkdir -p ek/<processor_type>
```
<processor_type>
Specify genoa, milan, or turin.
Download certificates in separate directories for each processor type:
```
$ curl -L -o ek/<processor_type>/vcek_node1.crt $VCEK_URL_NODE1
```
<processor_type>
Specify genoa, milan, or turin.

Create a secret for each processor from the VCEK certificates by running the following command:
```
$ oc create secret generic <processor_type>-secret \
  --from-file ./ek/<processor_type> \
  -n trustee-operator-system
```
<processor_type>
Specify genoa, milan, or turin.

2.16.8. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

Create an rvps-configmap.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
    ]

Create the RVPS config map by running the following command:
```
$ oc create -f rvps-configmap.yaml
```

2.16.9. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.

Procedure

Create an attestation-policy.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default_cpu.rego: |
    package policy

    import rego.v1
    default executables := 33
    default hardware := 97
    default configuration := 36

    ##### Azure vTPM SNP
    executables := 3 if {
      input.azsnpvtpm.measurement in data.reference.measurement
      input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
    }

    hardware := 2 if {
      input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
      input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
      input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
      input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
    }

    configuration := 2 if {
      input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
      input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
      input.azsnpvtpm.policy_abi_major in data.reference.abi_major
      input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
      input.azsnpvtpm.policy_single_socket in data.reference.single_socket
      input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
    }

    ##### Azure vTPM TDX
    executables := 3 if {
      input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
      input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
      input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
      input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
      input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
    }

    hardware := 2 if {
      # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
      input.aztdxvtpm.quote.header.tee_type == "81000000"
      input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"

      # Check TDX Module version and its hash. Also check OVMF code hash.
      # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
      # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
      # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
    }

    configuration := 2 if {
      # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
    }

Create the attestation policy config map by running the following command:
```
$ oc create -f attestation-policy.yaml
```

2.16.10. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

Create a tdx-config.yaml manifest file according to the following example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tdx-config
  namespace: trustee-operator-system
data:
  sgx_default_qcnl.conf: |
    {
      "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
    }

Create the tdx-config config map by running the following command:
```
$ oc create -f tdx-config.yaml
```

2.16.11. Creating a secret with custom keys for clients

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

You have created one or more custom keys.

Procedure

Create a secret for the custom keys according to the following example:
```
$ oc create secret generic attestation-status \
  --from-literal key1=<custom_key1> \ 1
  --from-literal key2=<custom_key2> \
  -n trustee-operator-system
```
1
Specify a custom key.
You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

2.16.12. Creating a secret for container image signature verification

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use This content is not included.Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

Create a secret for container image signature verification by running the following command:
```
$ oc create secret generic <type> \ 1
  --from-file=<tag>=./<public_key_file> \ 2
  -n trustee-operator-system
```
1
Specify the KBS secret type, for example, img-sig.
2
Specify the secret tag, for example, pub-key, and the public container image signing key.
Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

2.16.13. Creating the container image signature verification policy

Note

Procedure

Create a security-policy-config.json file according to the following example:
```
{
  "default": [
      {
      "type": "reject" 1
      }
  ],
  "transports": {
      "<transport>": { 2
          "<registry>/<image>": 3
          [
              {
                  "type": "sigstoreSigned",
                  "keyPath": "kbs:///default/<type>/<tag>" 4
              }
          ]
      }
  }
}
```
1
By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
2
Specify the image repository for transport, for example, "docker":. For more information, see Content from github.com is not included.containers-transports 5.
3
Specify the container registry and image, for example, "quay.io/my-image".
4
Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
Create the security policy by running the following command:
```
$ oc create secret generic <security-policy-name> \
  --from-file=<osc-key>=./<security-policy-config.json> \
  -n trustee-operator-system
```
The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

2.16.14. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

Create a resourcepolicy-configmap.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    package policy
    default allow = true
    allow {
      input["submods"]["cpu0"]["ear.status"] == "affirming"
    }
```
policy.rego
The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
package policy
The resource policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.
Create the resource policy config map by running the following command:
```
$ oc create -f resourcepolicy-configmap.yaml
```

2.16.15. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

Create an edge route by running the following command:

$ oc create route passthrough --service=kbs-service --port kbs-port \
  -n trustee-operator-system

Set the TRUSTEE_HOST variable by running the following command:

$ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})

Verify the route by running the following command:
```
$ echo $TRUSTEE_HOST
```
Example output
```
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
```

2.16.16. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

Create a private key by running the following command:
```
$ openssl genpkey -algorithm ed25519 > privateKey
```
Create a public key by running the following command:
```
$ openssl pkey -in privateKey -pubout -out publicKey
```

Create a secret by running the following command:

$ oc create secret generic kbs-auth-public-key \
  --from-file=publicKey -n trustee-operator-system

Verify the secret by running the following command:
```
$ oc get secret -n trustee-operator-system
```

2.16.17. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

Create a kbsconfig-cr.yaml manifest file:

apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources:
  - attestation-status
  - <security-policy-name>
  kbsResourcePolicyConfigMapName: resource-policy
  kbsHttpsKeySecretName: kbs-https-key
  kbsHttpsCertSecretName: kbs-https-certificate
  kbsAttestationCertSecretName: attestation-cert
  kbsAttestationKeySecretName: attestation-key
# tdxConfigSpec:
#   kbsTdxConfigMapName: tdx-config
# kbsServiceType: <service_type>
  kbsLocalCertCacheSpec:
    secrets:
    - secretName: milan-secret
      mountPath: "/etc/kbs/snp/ek/milan"
    - secretName: genoa-secret
      mountPath: "/etc/kbs/snp/ek/genoa"
    - secretName: turin-secret
      mountPath: "/etc/kbs/snp/ek/turin"

kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.
kbsLocalCertCacheSpec: For workloads running on AMD SEV-SNP nodes, specify the VCEK certificates in the kbsLocalCertCacheSpec stanza.

Create the KbsConfig CR by running the following command:
```
$ oc create -f kbsconfig-cr.yaml
```

2.16.18. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

Set the default project by running the following command:
```
$ oc project trustee-operator-system
```

Check the pods by running the following command:

$ oc get pods -n trustee-operator-system

Example output

NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

Set the POD_NAME environmental variable by running the following command:

$ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

Check the pod logs by running the following command:

$ oc logs -n trustee-operator-system $POD_NAME

Example output

[2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime

Chapter 3. Deploying Red Hat build of Trustee for workloads running on Microsoft Azure Red Hat OpenShift

You can deploy Red Hat build of Trustee for confidential containers workloads running on Azure Red Hat OpenShift.

3.1. Prerequisites

You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing OpenShift Container Platform on bare metal.

3.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

Install the Red Hat build of Trustee Operator.
Create HTTPS secrets.
Create the attestation token secret.
Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
Create the Reference Value Provider Service (RVPS) config map. Initially, you create an empty config map for the reference values. You update the values after you create KBSConfig custom resource (CR).
Create the attestation policy config map.
Optional: Create a config map for Intel® TDX.
Optional: Create a secret for custom keys clients.
Optional: Create a secret for container image signature verification.
Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
Create the resource policy config map.
Create the KBSConfig CR.
Create the cluster route.
Create the authentication secret.
Update the RVPS config map with the reference values.
Verify the Red Hat build of Trustee configuration.

3.3. Installing the Red Hat build of Trustee Operator

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI tool (oc).

Procedure

Create a trustee-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system

Create the trustee-operator-system namespace by running the following command:
```
$ oc create -f trustee-namespace.yaml
```

Create a trustee-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system

Create the operator group by running the following command:
```
$ oc create -f trustee-operatorgroup.yaml
```

Create a trustee-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator-system
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the subscription by running the following command:
```
$ oc create -f trustee-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n trustee-operator-system
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n trustee-operator-system

Example output

NAME                      DISPLAY                        PHASE
trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded

3.4. Creating HTTPS secrets

Generate keys to securely launch Red Hat build of Trustee and enables services to use HTTPS.

Procedure

Set the DOMAIN variable for the cluster by running the following command:
```
$ DOMAIN=$(oc get ingress.config/cluster -o jsonpath='{.spec.domain}')
```
Set the NS variable for the Red Hat build of Trustee namespace by running the following command:
```
$ NS=trustee-operator-system
```
Set the ROUTE_NAME variable by running the following command:
```
$ ROUTE_NAME=kbs-service
```
Set the ROUTE variable to the full DNS name by running the following command:
```
$ ROUTE="${ROUTE_NAME}-${NS}.${DOMAIN}"
```
Generate a private SSL/TLS key and certificate for Red Hat build of Trustee by running the following command:
```
$ openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout tls.key \
  -out tls.crt \
  -subj "/CN=<custom_cn>/O=<custom_org>" \
  -addext "subjectAltName=DNS:${ROUTE}"
```
- <custom_cn> is a custom CN. For example: kbs-trustee-operator-system.
- <custom_org> is a name of your organization.
Create the kbs-https-certificate secret in the trustee-operator-system namespace by running the following command:
```
$ oc create secret generic kbs-https-certificate --from-file=tls.crt -n trustee-operator-system
```
Create the kbs-https-key secret in the trustee-operator-system namespace by running the following command:
```
$ oc create secret generic kbs-https-key --from-file=tls.key -n trustee-operator-system
```

3.5. Creating the attestation token secrets

Generate an attestation token key and certificate for Red Hat build of Trustee.

Procedure

Generate a private elliptic curve SSL key called token.key by running the following command:
```
$ openssl ecparam -name prime256v1 -genkey -noout -out token.key
```
Create the attestation-key secret from the SSL/TLS key in the trustee-operator-system namespace:
```
$ oc create secret generic attestation-key \
  --from-file=token.key \
  -n trustee-operator-system
```
Generate a self-signed SSL/TLS certificate from the private SSL key by running the following command:
```
$ openssl req -new -x509 -key token.key -out token.crt -days 365 \
  -subj "/CN=<custom_cn>/O=<custom_org>"
```
- <custom_cn>: Specify the Common Name. For example: kbs-trustee-operator-system.
- <custom_org>: Specify your organization name.
Create the attestation-cert secret from the SSL/TLS key and certificate in the trustee-operator-system namespace:
```
$ oc create secret generic attestation-cert \
  --from-file=token.crt \
  -n trustee-operator-system
```

Create the attestation-status secret used for verifying the attestation process:

$ oc create secret generic attestation-status \
  --from-literal=status=success \
  -n trustee-operator-system

3.6. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Procedure

Create a kbs-config-cm.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.toml: |
    [http_server]
    sockets = ["0.0.0.0:8080"]
    insecure_http = false
    private_key = "/etc/https-key/tls.key"
    certificate = "/etc/https-cert/tls.crt"
    worker_count = 4

    [admin]
    insecure_api = false

    auth_public_key = "/etc/auth-secret/publicKey"

    [attestation_token]
    insecure_key = false
    trusted_certs_paths = ["/etc/attestation-cert/token.crt"]
    attestation_token_type = "CoCo"

    [attestation_service]
    type = "coco_as_builtin"
    work_dir = "/opt/confidential-containers/attestation-service"
    policy_engine = "opa"

    [attestation_service.attestation_token_broker]
    type = "Ear"
    policy_dir = "/opt/confidential-containers/attestation-service/policies"

    [attestation_service.attestation_token_broker.signer]
    key_path = "/etc/attestation-key/token.key"
    cert_path = "/etc/attestation-cert/token.crt"

    [attestation_service.attestation_token_config]
    duration_min = 5

    [attestation_service.rvps_config]
    type = "BuiltIn"

    [attestation_service.rvps_config.storage]
    type = "LocalJson"
    file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"

    [[plugins]]
    name = "resource"
    type = "LocalFs"
    dir_path = "/opt/confidential-containers/kbs/repository"

    [policy_engine]
    policy_path = "/opt/confidential-containers/opa/policy.rego"

Create the config map by running the following command:
```
$ oc create -f kbs-config-cm.yaml
```

3.7. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

Create an rvps-configmap.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
    ]

Create the RVPS config map by running the following command:
```
$ oc create -f rvps-configmap.yaml
```

3.8. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.

Procedure

Create an attestation-policy.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default_cpu.rego: |
    package policy

    import rego.v1
    default executables := 33
    default hardware := 97
    default configuration := 36

    ##### Azure vTPM SNP
    executables := 3 if {
      input.azsnpvtpm.measurement in data.reference.measurement
      input.azsnpvtpm.tpm.pcr11 in data.reference.snp_pcr11
    }

    hardware := 2 if {
      input.azsnpvtpm.reported_tcb_bootloader in data.reference.tcb_bootloader
      input.azsnpvtpm.reported_tcb_microcode in data.reference.tcb_microcode
      input.azsnpvtpm.reported_tcb_snp in data.reference.tcb_snp
      input.azsnpvtpm.reported_tcb_tee in data.reference.tcb_tee
    }

    configuration := 2 if {
      input.azsnpvtpm.platform_smt_enabled in data.reference.smt_enabled
      input.azsnpvtpm.platform_tsme_enabled in data.reference.tsme_enabled
      input.azsnpvtpm.policy_abi_major in data.reference.abi_major
      input.azsnpvtpm.policy_abi_minor in data.reference.abi_minor
      input.azsnpvtpm.policy_single_socket in data.reference.single_socket
      input.azsnpvtpm.policy_smt_allowed in data.reference.smt_allowed
    }

    ##### Azure vTPM TDX
    executables := 3 if {
      input.aztdxvtpm.tpm.pcr03 in data.reference.tdx_pcr03
      input.aztdxvtpm.tpm.pcr08 in data.reference.tdx_pcr08
      input.aztdxvtpm.tpm.pcr09 in data.reference.tdx_pcr09
      input.aztdxvtpm.tpm.pcr11 in data.reference.tdx_pcr11
      input.aztdxvtpm.tpm.pcr12 in data.reference.tdx_pcr12
    }

    hardware := 2 if {
      # Check the quote is a TDX quote signed by Intel SGX Quoting Enclave
      input.aztdxvtpm.quote.header.tee_type == "81000000"
      input.aztdxvtpm.quote.header.vendor_id == "939a7233f79c4ca9940a0db3957f0607"

      # Check TDX Module version and its hash. Also check OVMF code hash.
      # input.aztdxvtpm.quote.body.mr_seam in data.reference.mr_seam
      # input.aztdxvtpm.quote.body.tcb_svn in data.reference.tcb_svn
      # input.aztdxvtpm.quote.body.mr_td in data.reference.mr_td
    }

    configuration := 2 if {
      # input.aztdxvtpm.quote.body.xfam in data.reference.xfam
    }

Create the attestation policy config map by running the following command:
```
$ oc create -f attestation-policy.yaml
```

3.9. Creating a tdx-config config map

Create a config map for Intel® Trust Domain Extensions (TDX).

Procedure

Create a tdx-config.yaml manifest file according to the following example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tdx-config
  namespace: trustee-operator-system
data:
  sgx_default_qcnl.conf: |
    {
      "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
    }

Create the tdx-config config map by running the following command:
```
$ oc create -f tdx-config.yaml
```

3.10. Creating a secret with custom keys for clients

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

You have created one or more custom keys.

Procedure

Create a secret for the custom keys according to the following example:
```
$ oc create secret generic attestation-status \
  --from-literal key1=<custom_key1> \ 1
  --from-literal key2=<custom_key2> \
  -n trustee-operator-system
```
1
Specify a custom key.
You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

3.11. Creating a secret for container image signature verification

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use This content is not included.Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

Create a secret for container image signature verification by running the following command:
```
$ oc create secret generic <type> \ 1
  --from-file=<tag>=./<public_key_file> \ 2
  -n trustee-operator-system
```
1
Specify the KBS secret type, for example, img-sig.
2
Specify the secret tag, for example, pub-key, and the public container image signing key.
Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

3.12. Creating the container image signature verification policy

Note

Procedure

Create a security-policy-config.json file according to the following example:
```
{
  "default": [
      {
      "type": "reject" 1
      }
  ],
  "transports": {
      "<transport>": { 2
          "<registry>/<image>": 3
          [
              {
                  "type": "sigstoreSigned",
                  "keyPath": "kbs:///default/<type>/<tag>" 4
              }
          ]
      }
  }
}
```
1
By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.
2
Specify the image repository for transport, for example, "docker":. For more information, see Content from github.com is not included.containers-transports 5.
3
Specify the container registry and image, for example, "quay.io/my-image".
4
Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.
Create the security policy by running the following command:
```
$ oc create secret generic <security-policy-name> \
  --from-file=<osc-key>=./<security-policy-config.json> \
  -n trustee-operator-system
```
The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

3.13. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

Create a resourcepolicy-configmap.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    package policy
    default allow = true
    allow {
      input["submods"]["cpu0"]["ear.status"] == "affirming"
    }
```
policy.rego
The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
package policy
The resource policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.
Create the resource policy config map by running the following command:
```
$ oc create -f resourcepolicy-configmap.yaml
```

3.14. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

Create an edge route by running the following command:

$ oc create route passthrough --service=kbs-service --port kbs-port \
  -n trustee-operator-system

Set the TRUSTEE_HOST variable by running the following command:

$ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})

Verify the route by running the following command:
```
$ echo $TRUSTEE_HOST
```
Example output
```
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
```

3.15. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

Create a private key by running the following command:
```
$ openssl genpkey -algorithm ed25519 > privateKey
```
Create a public key by running the following command:
```
$ openssl pkey -in privateKey -pubout -out publicKey
```

Create a secret by running the following command:

$ oc create secret generic kbs-auth-public-key \
  --from-file=publicKey -n trustee-operator-system

Verify the secret by running the following command:
```
$ oc get secret -n trustee-operator-system
```

3.16. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

Create a kbsconfig-cr.yaml manifest file:

apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources:
  - attestation-status
  - <security-policy-name>
  kbsResourcePolicyConfigMapName: resource-policy
  kbsHttpsKeySecretName: kbs-https-key
  kbsHttpsCertSecretName: kbs-https-certificate
  kbsAttestationCertSecretName: attestation-cert
  kbsAttestationKeySecretName: attestation-key
# tdxConfigSpec:
#   kbsTdxConfigMapName: tdx-config
# kbsServiceType: <service_type>

kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
Uncomment tdxConfigSpec.kbsTdxConfigMapName: tdx-config for Intel Trust Domain Extensions.
Uncomment kbsServiceType: <service_type> if you create a service type, other than the default ClusterIP service, to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName.

Create the KbsConfig CR by running the following command:
```
$ oc create -f kbsconfig-cr.yaml
```

3.17. Updating the RVPS config map

You update the Reference Value Provider Service (RVPS) config map with expected measurements, including the Platform Configuration Register (PCR) 8 value, for the trusted execution environment. Red Hat build of Trustee uses these measurements to verify the attestation evidence.

The workload cluster administrator calculates the PCR8 value by performing a SHA256 hash on a configuration or policy file such as initdata:

$ hash=$(sha256sum <config_file> | cut -d' ' -f1)

$ initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

$ PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1)

Prerequisites

PCR8 value, expiration, and algorithm, created by the workload cluster administrator

Procedure

Create an rvps-configmap-update.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
     {
        "name": "svn",
        "expiration": "2027-01-01T00:00:00Z",
        "value" : 1
      },
      {
        "name": "major_version",
        "expiration": "2027-01-01T00:00:00Z",
        "value" : 1
      },
      {
        "name": "minimum_minor_version",
        "expiration": "2027-01-01T00:00:00Z",
        "value" : 4
      }
    ]

Note

Do not use this configuration example in a production environment.

The "value" can be any JSON type (string, number, boolean, array, object). The JSON type must be combined with the operand used in the attestation-policy. See the following examples for valid "value" types:

If the attestation rule is:

input.sample.platform_version.major == data.reference.major_version

The == operand expects to match an integer type in the reference values:

{
"name": "major_version",
"expiration": "2027-01-01T00:00:00Z",
"value" : 1
}

If the attestation rule is:
```
input.sample.svn in data.reference.svn
```
The in operand expects to match an array type in the reference values:
```
{
"name": "svn",
"expiration": "2027-01-01T00:00:00Z",
"value" : [
1
]
}
```

Update the RVPS config map by running the following command:
```
$ oc apply -f rvps-configmap-update.yaml
```

3.18. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

Set the default project by running the following command:
```
$ oc project trustee-operator-system
```

Check the pods by running the following command:

$ oc get pods -n trustee-operator-system

Example output

NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

Set the POD_NAME environmental variable by running the following command:

$ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

Check the pod logs by running the following command:

$ oc logs -n trustee-operator-system $POD_NAME

Example output

[2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime

Chapter 4. Deploying Red Hat build of Trustee for workloads running on IBM Z and IBM LinuxONE

You can deploy Red Hat build of Trustee for confidential containers workloads running on IBM Z® and IBM® LinuxONE.

4.1. Prerequisites

You have installed the latest version of Red Hat OpenShift Container Platform in a trusted environment. For more information, see Installing OpenShift Container Platform on bare metal.
You are using LinuxONE Emperor 4.
You have enabled Secure Unpack Facility on your Logical Partition (LPAR), which is necessary for the IBM Secure Execution. For more information, see Content from www.ibm.com is not included.Enabling the KVM host for IBM Secure Execution.

4.2. Deployment overview

You deploy Red Hat build of Trustee by performing the following steps:

Install the Red Hat build of Trustee Operator.
Optional: Create the kbs-config config map if you are using Intel® Trust Domain Extensions (TDX) remote attestation.
Create the Reference Value Provider Service (RVPS) config map.
Configure the IBM Secure Execution certificates and keys.
Create the IBM persistent storage components.
Create the attestation policy config map.
Optional: Create a secret for custom keys clients.
Optional: Create a secret for container image signature verification.
Create the container image signature verification policy. The container image signature verification policy is disabled by default. For production workloads, you must use signature verification to ensure container images are not tampered with.
Create the resource policy config map.
Create the KBSConfig CR.
Create the cluster route.
Create the authentication secret.
Verify the Red Hat build of Trustee configuration.

4.3. Installing the Red Hat build of Trustee Operator

You install the Red Hat build of Trustee Operator on an OpenShift Container Platform cluster in a trusted environment.

Prerequisites

You have access to the cluster as a user with the cluster-admin role.
You have installed the OpenShift CLI tool (oc).

Procedure

Create a trustee-namespace.yaml manifest file:

apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system

Create the trustee-operator-system namespace by running the following command:
```
$ oc create -f trustee-namespace.yaml
```

Create a trustee-operatorgroup.yaml manifest file:

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system

Create the operator group by running the following command:
```
$ oc create -f trustee-operatorgroup.yaml
```

Create a trustee-subscription.yaml manifest file:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator-system
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Create the subscription by running the following command:
```
$ oc create -f trustee-subscription.yaml
```
Verify that the Operator is correctly installed by running the following command:
```
$ oc get csv -n trustee-operator-system
```
This command can take several minutes to complete.

Watch the process by running the following command:

$ watch oc get csv -n trustee-operator-system

Example output

NAME                      DISPLAY                        PHASE
trustee-operator.v1.0.0   Trustee Operator  1.0.0        Succeeded

4.4. Creating the kbs-config config map

You create the kbs-config config map to configure Red Hat build of Trustee.

Note

The following configuration example turns off security features. Do not use this example in a production environment.

Procedure

Create a kbs-config-cm.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.toml: |
    [http_server]
    sockets = ["0.0.0.0:8080"]
    insecure_http = false
    private_key = "/etc/https-key/tls.key"
    certificate = "/etc/https-cert/tls.crt"
    worker_count = 4

    [admin]
    insecure_api = false

    auth_public_key = "/etc/auth-secret/publicKey"

    [attestation_token]
    insecure_key = true
    attestation_token_type = "CoCo"

    [attestation_service]
    type = "coco_as_builtin"
    work_dir = "/opt/confidential-containers/attestation-service"
    policy_engine = "opa"

    [attestation_service.attestation_token_broker]
    type = "Simple"
    policy_dir = "/opt/confidential-containers/attestation-service/policies"


    [attestation_service.attestation_token_config]
    duration_min = 5

    [attestation_service.rvps_config]
    type = "BuiltIn"

    [attestation_service.rvps_config.storage]
    type = "LocalJson"
    file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"

    [[plugins]]
    name = "resource"
    type = "LocalFs"
    dir_path = "/opt/confidential-containers/kbs/repository"

    [policy_engine]
    policy_path = "/opt/confidential-containers/opa/policy.rego"

Create the config map by running the following command:
```
$ oc create -f kbs-config-cm.yaml
```

4.5. Creating the RVPS config map

You create the Reference Value Provider Service (RVPS) config map, which specifies the reference values for your Trusted Execution Environment (TEE).

The data.reference-values.json stanza must be present, but it can be empty.

Note

Do not use this configuration example in a production environment. Initially, you create an empty RVPS config map. Then, you update the RVPS config map with reference values for your TEE.

Procedure

Create an rvps-configmap.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |
    [
    ]

Create the RVPS config map by running the following command:
```
$ oc create -f rvps-configmap.yaml
```

4.6. Configuring the IBM Secure Execution certificates and keys

You must configure the IBM Secure Execution (SE) certificates and keys for your worker nodes.

Prerequisites

You have the IP address of the bastion node.
You have the internal IP addresses of the worker nodes.

Procedure

Generate the Key Broker Service (KBS) certificate and key.
Obtain the attestation policy fields.
Download the certificates and certificate revocation lists (CRLs).
Generate the RSA keys.
Verify and copy files to the OpenShift Container Platform worker nodes.
Create the secrets in the cluster with the KBS key and certificate.

4.6.1. Generating the Key Broker Service certificate and key

You must generate the Key Broker Service (KBS) certificate and key.

Procedure

Create the kbs.conf configuration file according to the following example:

[req]
default_bits = 2048
default_keyfile = localhost.key
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_ca

[req_distinguished_name]
countryName = Country Name (2-letter code)
countryName_default = <country_name>
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = <state_name>
localityName = Locality Name (eg, city)
localityName_default = <locality_name>
organizationName = Organization Name (eg, company)
organizationName_default = Red Hat
organizationalUnitName = organizationalunit
organizationalUnitName_default = Development
commonName = Common Name (e.g. server FQDN or
YOUR name)
commonName_default = kbs-service
commonName_max = 64

[req_ext]
subjectAltName = @alt_names

[v3_ca]
subjectAltName = @alt_names

[alt_names]
IP.1  = <worker_node_ip>
DNS.1  = localhost
DNS.2  = 127.0.0.1

<worker_node_ip>

Obtain the IP address of a worker node by running the following command:

$ oc get node $(oc get pod -n trustee-operator-system -o jsonpath='{.items[0].spec.nodeName}') -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

Generate the KBS key and self-signed certificate by running the following command:

$ openssl req -x509 -nodes -days 365 \
   -newkey rsa:2048 \
   -keyout kbs.key \
   -out kbs.crt \
   -config kbs.conf \
   -passin pass:

Copy the KBS key to the ibmse directory by running the following command:
```
$ cp kbs.key /tmp/ibmse/kbs.key
```
Copy the KBS certificate to the ibmse directory by running the following command:
```
$ cp kbs.crt /tmp/ibmse/kbs.crt
```

4.6.2. Obtaining the attestation policy fields

You must obtain the attestation policy fields by using Reference Value Provider Service (RVPS).

Procedure

Create a directory to download the GetRvps.sh script by running the following command:
```
$ mkdir -p Rvps-Extraction/
```

Download the script by running the following command:

$ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/GetRvps.sh -O $PWD/GetRvps.sh

Create a subdirectory by running the following command:
```
$ mkdir -p Rvps-Extraction/static-files
```
Go to the static-files directory by running the following command:
```
$ cd Rvps-Extraction/static-files
```

Download the pvextract-hdr tool by running the following command:

$ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/pvextract-hdr -O $PWD/pvextract-hdr

Make the tool executable by running the following command:
```
$ chmod +x pvextract-hdr
```

Download the se_parse_hdr.py script by running the following command:

$ wget https://github.com/openshift/sandboxed-containers-operator/raw/devel/scripts/rvps-extraction/static-files/se_parse_hdr.py -O $PWD/se_parse_hdr.py

Copy your Host Key Document (HKD) certificate to the static-files directory by running the following command:
```
$ cp ~/path/to/<hkd_cert.crt> .
```
The static-files directory contains the following files:
- HKD.crt
- pvextract-hdr
- se_parse_hdr.py
Go to the Rvps-Extraction directory by running the following command:
```
$ cd ..
```
Make the GetRvps.sh script executable by running the following command:
```
$ chmod +x GetRvps.sh
```

Run the script:

$ ./GetRvps.sh

Example output

***Installing necessary packages for RVPS values extraction ***
Updating Subscription Management repositories.
Last metadata expiration check: 0:37:12 ago on Mon Nov 18 09:20:29 2024.
Package python3-3.9.19-8.el9_5.1.s390x is already installed.
Package python3-cryptography-36.0.1-4.el9.s390x is already installed.
Package kmod-28-10.el9.s390x is already installed.
Dependencies resolved.
Nothing to do.
Complete!
***Installation Finished ***
1) Generate the RVPS From Local Image from User pc
2) Generate RVPS from Volume
3) Quit
Please enter your choice:

Enter 2 to generate the Reference Value Provider Service from the volume:
```
Please enter your choice: 2
```
Enter fa-pp for the libvirt pool name:
```
Enter the Libvirt Pool Name: fa-pp
```
Enter the libvirt gateway URI:
```
Enter the Libvirt URI Name: <libvirt-uri> 1
```
1
Specify the LIBVIRT_URI value that you used to create the peer pods secret.

Enter fa-pp-vol for the libvirt volume name:

Enter the Libvirt Volume Name: fa-pp-vol

Example output

Downloading from PODVM Volume...

mount: /mnt/myvm: special device /dev/nbd3p1 does not exist.
Error: Failed to mount the image. Retrying...
Mounting on second attempt passed
/dev/nbd3 disconnected
SE header found at offset 0x014000
SE header written to '/root/Rvps-Extraction/output-files/hdr.bin' (640 bytes)
se.tag:  42f3fe61e8a7e859cab3bb033fd11c61
se.image_phkh:  92d0aff6eb86719b6b1ea0cb98d2c99ff2ec693df3efff2158f54112f6961508
provenance = ewogICAgInNlLmF0dGVzdGF0aW9uX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnRhZyI6IFsKICAgICAgICAiNDJmM2ZlNjFlOGE3ZTg1OWNhYjNiYjAzM2ZkMTFjNjEiCiAgICBdLAogICAgInNlLmltYWdlX3Boa2giOiBbCiAgICAgICAgIjkyZDBhZmY2ZWI4NjcxOWI2YjFlYTBjYjk4ZDJjOTlmZjJlYzY5M2RmM2VmZmYyMTU4ZjU0MTEyZjY5NjE1MDgiCiAgICBdLAogICAgInNlLnVzZXJfZGF0YSI6IFsKICAgICAgICAiMDAiCiAgICBdLAogICAgInNlLnZlcnNpb24iOiBbCiAgICAgICAgIjI1NiIKICAgIF0KfQo=
-rw-r--r--. 1 root root 640 Dec 16 10:57 /root/Rvps-Extraction/output-files/hdr.bin
-rw-r--r--. 1 root root 446 Dec 16 10:57 /root/Rvps-Extraction/output-files/ibmse-policy.rego
-rw-r--r--. 1 root root 561 Dec 16 10:57 /root/Rvps-Extraction/output-files/se-message

Obtain the attestation policy field values by running the following command:

$ cat /root/Rvps-Extraction/output-files/se-sample

Example output

{
    "se.attestation_phkh": [
        "50a59219c5034f23f69a81893b77f80190dab0ab4781d10b6631d6ed23ef38e4"
    ],
    "se.tag": [
        "ded333bce2d721547ee2b59b1b96e7e5"
    ],
    "se.image_phkh": [
        "50a59219c5034f23f69a81893b77f80190dab0ab4781d10b6631d6ed23ef38e4"
    ],
    "se.user_data": [
        "00"
    ],
    "se.version": [
        "256"
    ]
}

4.6.3. Downloading the certificates and certificate revocation lists

You must download the IBM certificates and Certificate Revocation Lists (CRLs).

Procedure

Create a temporary directory for certificates by running the following command:
```
$ mkdir /tmp/ibmse/certs
```

Download the ibm-z-host-key-signing-gen2.crt certificate by running the following command:

$ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-signing-gen2.crt -O /tmp/ibmse/certs/ibm-z-host-key-signing-gen2.crt

Download the DigiCertCA.crt certificate by running the following command:

$ wget https://www.ibm.com/support/resourcelink/api/content/public/DigiCertCA.crt -O /tmp/ibmse/certs/DigiCertCA.crt

Create a temporary directory for the CRLs by running the following command:
```
$ mkdir /tmp/ibmse/crls
```

Download the ibm-z-host-key-gen2.crl file by running the following command:

$ wget https://www.ibm.com/support/resourcelink/api/content/public/ibm-z-host-key-gen2.crl -O /tmp/ibmse/crls/ibm-z-host-key-gen2.crl

Download the DigiCertTrustedRootG4.crl file by running the following command:

$ wget http://crl3.digicert.com/DigiCertTrustedRootG4.crl -O /tmp/ibmse/crls/DigiCertTrustedRootG4.crl

Download the DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl file by running the following command:

$ wget http://crl3.digicert.com/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl -O /tmp/ibmse/crls/DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl

Create a temporary directory for the hdr.bin file by running the following command:
```
$ mkdir -p /tmp/ibmse/hdr/
```
Copy the hdr.bin file to the hdr directory by running the following command:
```
$ cp /root/Rvps-Extraction/output-files/hdr.bin /tmp/ibmse/hdr/
```
Create a temporary directory for Host Key Document (HKD) certificate by running the following command:
```
$ mkdir -p /tmp/ibmse/hkds
```
Copy your HKD certificate to the hkds directory by running the following command:
```
$ cp ~/path/to/<hkd_cert.crt> /tmp/ibmse/hkds/
```

4.6.4. Generating the RSA keys

You must generate the RSA key pair for encryption.

Procedure

Generate an RSA key pair by running the following command:

$ openssl genrsa -aes256 -passout pass:<password> -out /tmp/encrypt_key-psw.pem 4096 1

Create a temporary directory for the RSA keys by running the following command:
```
$ mkdir -p /tmp/ibmse/rsa
```

Create an encrypt_key.pub key by running the following command:

$ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -pubout -out /tmp/ibmse/rsa/encrypt_key.pub

Create an encrypt_key.pem key by running the following command:

$ openssl rsa -in /tmp/encrypt_key-psw.pem -passin pass:<password> -out /tmp/ibmse/rsa/encrypt_key.pem

4.6.5. Verifying and copying files to the worker nodes

You must verify the /tmp/ibmse folder structure and copy the files to the OpenShift Container Platform worker nodes.

Procedure

Verify the structure of the /tmp/ibmse directory by running the following command:

$ tree /tmp/ibmse

Example output

/tmp/ibmse
├──kbs.key
├──kbs.crt
├── certs
  ├── ibm-z-host-key-signing-gen2.crt
  └── DigiCertCA.crt
├── crls
  └── ibm-z-host-key-gen2.crl
  └── DigiCertTrustedRootG4.crl
  └── DigiCertTrustedG4CodeSigningRSA4096SHA3842021CA1.crl
├── hdr
  └── hdr.bin
├── hkds
  └── <hkd_cert.crt>
├── rsa
  └── encrypt_key.pem
  └── encrypt_key.pub

Create a compressed file from the /tmp/ibmse directory by running the following command:
```
$ tar -czf ibmse.tar.gz -C /tmp/ibmse
```
Copy the .tar.gz file to the bastion node in your cluster by running the following command:
```
$ scp /tmp/ibmse.tar.gz root@<ocp_bastion_ip>:/tmp 1
```
Connect to the bastion node over SSH by running the following command:
```
$ ssh root@<ocp_bastion_ip>
```
Copy the .tar.gz file to each worker node by running the following command:
```
$ scp /tmp/ibmse.tar.gz core@<worker_node_ip>:/tmp 1
```

Extract the .tar.gz on each worker node by running the following command:

$ ssh core@<worker_node_ip> 'sudo mkdir -p /opt/confidential-containers/ && sudo tar -xzf /tmp/ibmse.tar.gz -C /opt/confidential-containers/'

Update the ibmse folder permissions by running the following command:
```
$ ssh core@<worker_node_ip> 'sudo chmod -R 755 /opt/confidential-containers/ibmse/'
```

4.6.6. Creating the secrets for Key Broker Service

You must create the secrets in the cluster with the Key Broker Service (KBS) key and certificate.

Procedure

Convert the kbs.crt file to a Base64-encoded string by running the following command:
```
$ cat /tmp/ibmse/kbs.crt | base64 -w 0
```
Record the string for the kbs-https-certificate.yaml manifest.
Create a kbs-https-certificate.yaml manifest file according to the following example:
```
apiVersion: v1
kind: Secret
metadata:
 name: kbs-https-certificate
 namespace: trustee-operator-system
data:
 https.crt: <kbs_crt_value>
```
<kbs_crt_value>
Specify the Base64-encoded string you created from the kbs.crt file.
Create the secret with the KBS certificate by running the following command:
```
$ oc create -f kbs-https-certificate.yaml
```
Convert the kbs.key file to a Base64-encoded string by running the following command:
```
$ cat /tmp/ibmse/kbs.key | base64 -w 0
```
Record the string for the kbs-https-key.yaml manifest.
Create a kbs-https-key.yaml manifest file according to the following example:
```
apiVersion: v1
kind: Secret
metadata:
  name: kbs-https-key
  namespace: trustee-operator-system
data:
 https.key: <kbs_key_value>
```
<kbs_key_value>
Specify the Base64-encoded string you created from the kbs.key file.
Create the secret with the KBS key by running the following command:
```
$ oc create -f kbs-https-key.yaml
```

4.7. Creating the IBM persistent storage components

You must create the persistent volume (PV) and persistent volume claim (PVC) to mount the ibmse folder on the trustee-deployment pod.

Procedure

Create a persistent-volume.yaml manifest file:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: ibmse-pv
  namespace: trustee-operator-system
spec:
  capacity:
    storage: 100Mi
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  local:
    path: /opt/confidential-containers/ibmse
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node-role.kubernetes.io/worker
              operator: Exists

Create the persistent volume by running the following command:
```
$ oc create -f persistent-volume.yaml
```

Create a persistent-volume-claim.yaml manifest file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ibmse-pvc
  namespace: trustee-operator-system
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: ""
  resources:
    requests:
      storage: 100Mi

Create the persistent volume claim by running the following command:
```
$ oc create -f persistent-volume-claim.yaml
```

4.8. Creating the attestation policy config map

You create an attestation policy config map to define attestation policies for Red Hat build of Trustee.

The attestation policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.

Procedure

Create an attestation-policy.yaml manifest file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default_cpu.rego: |
    package policy
    import rego.v1
    default allow = false
    converted_version := sprintf("%v", [input["se.version"]])
    allow if {
        input["se.attestation_phkh"] == "<se.attestation_phkh>"
        input["se.image_phkh"] == "<se.image_phkh>"
        input["se.tag"] == "<se.tag>"
        converted_version == "256"
    }

default_cpu.rego: Do not modify the policy name.

Create the attestation policy config map by running the following command:
```
$ oc create -f attestation-policy.yaml
```

4.9. Creating a secret with custom keys for clients

You can create a secret that contains one or more custom keys for Red Hat build of Trustee clients.

In this example, the attestation-status secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Prerequisites

You have created one or more custom keys.

Procedure

Create a secret for the custom keys according to the following example:
```
$ oc create secret generic attestation-status \
  --from-literal key1=<custom_key1> \ 1
  --from-literal key2=<custom_key2> \
  -n trustee-operator-system
```
1 1 1 1
Specify a custom key.
You specify the attestation-status secret in the spec.kbsSecretResources key of the KbsConfig custom resource manifest.

4.10. Creating a secret for container image signature verification

If you use container image signature verification, you must create a secret that contains the public container image signing key.

The Red Hat build of Trustee Operator uses the secret to verify the signature, ensuring that only trusted and authenticated container images are deployed in your environment.

You can use This content is not included.Red Hat Trusted Artifact Signer or other tools to sign container images.

Procedure

Create a secret for container image signature verification by running the following command:
```
$ oc create secret generic <type> \ 1
  --from-file=<tag>=./<public_key_file> \ 2
  -n trustee-operator-system
```
1
Specify the KBS secret type, for example, img-sig.
2
Specify the secret tag, for example, pub-key, and the public container image signing key.
Record the <type> value. You must add this value to the spec.kbsSecretResources key when you create the KbsConfig custom resource.

4.11. Creating the container image signature verification policy

Note

Procedure

Create a security-policy-config.json file according to the following example:

{
    "default": [
    ],
    "transports": {
        "docker": {
            "<container_registry_url>/<username>/busybox:latest": 1
            [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "kbs:///default/img-sig/pub-key" 2
                }
            ]
        }
    }
}

1: Specify the container registry URL, for example, "quay.io".
2: Specify the type and tag of the container image signature verification secret that you created, for example, img-sig/pub-key.

Create the security policy by running the following command:
```
$ oc create secret generic <security-policy-name> \
  --from-file=<osc-key>=./<security-policy-config.json> \
  -n trustee-operator-system
```
The <security-policy-name> secret is specified in the spec.kbsSecretResources key of the KbsConfig custom resource.

4.12. Creating the resource policy config map

You configure the resource policy config map for the policy engine. This policy determines which resources are accessible to Red Hat build of Trustee.

Note

This policy engine is different from the Attestation Service policy engine, which determines the validity of TEE evidence.

Procedure

Create a resourcepolicy-configmap.yaml manifest file:
```
apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    package policy
    default allow = true
    allow {
      input["tee"] == "se"
    }
```
policy.rego
The name of the resource policy, policy.rego, must match the resource policy defined in the kbs-config config map.
package policy
The resource policy follows the Content from www.openpolicyagent.org is not included.Open Policy Agent specification.
Create the resource policy config map by running the following command:
```
$ oc create -f resourcepolicy-configmap.yaml
```

4.13. Creating the cluster route

You create a secure route with edge TLS termination for the cluster where you installed Red Hat build of Trustee.

External ingress traffic reaches the router pods as HTTPS and passes on to the pods running in the trustee-operator-system namespace as HTTP.

Procedure

Create an edge route by running the following command:

$ oc create route passthrough --service=kbs-service --port kbs-port \
  -n trustee-operator-system

Set the TRUSTEE_HOST variable by running the following command:

$ TRUSTEE_HOST=$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})

Verify the route by running the following command:
```
$ echo $TRUSTEE_HOST
```
Example output
```
kbs-service-trustee-operator-system.apps.memvjias.eastus.aroapp.io
```

4.14. Creating the authentication secret

You create the authentication secret for Red Hat build of Trustee.

Procedure

Create a private key by running the following command:
```
$ openssl genpkey -algorithm ed25519 > privateKey
```
Create a public key by running the following command:
```
$ openssl pkey -in privateKey -pubout -out publicKey
```

Create a secret by running the following command:

$ oc create secret generic kbs-auth-public-key \
  --from-file=publicKey -n trustee-operator-system

Verify the secret by running the following command:
```
$ oc get secret -n trustee-operator-system
```

4.15. Creating the KbsConfig custom resource

Create the KbsConfig custom resource (CR) to launch Red Hat build of Trustee.

Procedure

Create a kbsconfig-cr.yaml manifest file:

apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources:
  - attestation-status
  - <security-policy-name>
  kbsResourcePolicyConfigMapName: resource-policy
  kbsHttpsKeySecretName: kbs-https-key
  kbsHttpsCertSecretName: kbs-https-certificate
  kbsAttestationCertSecretName: attestation-cert
  kbsAttestationKeySecretName: attestation-key
  ibmSEConfigSpec:
    certStorePvc: ibmse-pvc
  KbsEnvVars:
    SE_SKIP_CERTS_VERIFICATION: "false"

kbsSecretResources: Specify the type value of the container image signature verification secret if you created the secret, for example, img-sig.
SE_SKIP_CERTS_VERIFICATION: Set to true only for testing purposes.

Create the KbsConfig CR by running the following command:
```
$ oc create -f kbsconfig-cr.yaml
```

4.16. Verifying the configuration

You verify the Red Hat build of Trustee configuration by checking its pods and logs.

Procedure

Set the default project by running the following command:
```
$ oc project trustee-operator-system
```

Check the pods by running the following command:

$ oc get pods -n trustee-operator-system

Example output

NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

Set the POD_NAME environmental variable by running the following command:

$ POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

Check the pod logs by running the following command:

$ oc logs -n trustee-operator-system $POD_NAME

Example output

[2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 12 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime

Chapter 5. Uninstalling

You uninstall by performing the following tasks:

Delete the KbsConfig custom resource.
Uninstall the Red Hat build of Trustee Operator.
Delete the KbsConfig custom resource definition.

5.1. Deleting the KbsConfig custom resource

You delete the KbsConfig custom resource (CR) by using the command line.

Procedure

Delete the KbsConfig CR by running the following command:
```
$ oc delete kbsconfig kbsconfig
```
Verify the CR removal by running the following command:
```
$ oc get kbsconfig kbsconfig
```
Example output
```
No kbsconfig instances exist
```

Important

You must ensure that all pods are deleted. Any remaining pod resources might result in an unexpected bill from your cloud provider.

5.2. Uninstalling the Red Hat build of Trustee Operator

You uninstall the Red Hat build of Trustee Operator by using the command line.

Procedure

Delete the subscription by running the following command:
```
$ oc delete subscription trustee-operator -n trustee-operator-system
```
Delete the namespace by running the following command:
```
$ oc delete namespace trustee-operator-system
```

5.3. Deleting the KbsConfig CRD

You delete the KbsConfig custom resource definition (CRD) by using the command line.

Prerequisites

You have deleted the KbsConfig custom resource.
You have uninstalled the Red Hat build of Trustee Operator.

Procedure

Delete the KbsConfig CRD by running the following command:
```
$ oc delete crd kataconfigs.kataconfiguration.openshift.io
```
Verify that the CRD was deleted by running the following command:
```
$ oc get crd kataconfigs.kataconfiguration.openshift.io
```
Example output
```
Unknown CRD kataconfigs.kataconfiguration.openshift.io
```

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.