Injecting CA certificate inside OpenJDK container at runtime and build time

Introduction

This article discusses how to import a CA root certificate to trust store inside a OpenJDK container at runtime and build time, building on top of the solution How do I import a CA root certificate to trust store in JBoss EAP for OpenShift?, which tackles this problem at runtime (Option 1 - Mount cacerts as Secret) and build time (Option 2 - Add a CA certification when building a container), but with other resources for runtime injection:

• At build time time, the user has flexibility on the injection of files and permissions for deployment.
• At runtime, although the user can set a difference SCC, it does not give flexibility on the permissions for /etc/pki/ca-trust/extracted

There are several ways of generating JKS, specifically for /etc/pki/ca-trust/extracted (which is RHEL's standard way of doing certs) Service-CA Operator can be used to inject CA root certificates at runtime inside a container. However, similarly to RHEL standard injection, update-ca-trust execution is required with extract argument (for JKS creation), which requires root user, see notes for clarification.

Important notes

• update-ca-trust command requires root user to run and cannot be run on OpenJDK's container inside a OpenShift Pod.
• The default java trust store /etc/pki/ca-trust/extracted/java/cacerts is read-only.
• The Service-CA Operator injection still requires the update-ca Linux command to be executed

Consequently, one cannot update cacerts after OpenJDK container has been started already.
Therefore, the following approaches can be used to inject a CA certificate inside a OpenJDK container:

Build example 1

Example below at Build Time sets an OpenJDK deployment with the ROOT user.

spec:
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: 'openjdk-root:latest'
  resources: {}
  successfulBuildsHistoryLimit: 5
  failedBuildsHistoryLimit: 5
  strategy:
    type: Docker
    dockerStrategy:
      from:
        kind: ImageStreamTag
        namespace: openshift
        name: 'ubi8-openjdk-17:1.18'
      imageOptimizationPolicy: SkipLayers
  postCommit: {}
  source:
    type: Dockerfile
    dockerfile: |-
      FROM ubi8-openjdk-17:1.18
      USER root
      RUN microdnf -y install wget
      RUN wget https://github.com/path/jkube.jar -O jkube.jar 
      RUN mv jkube.jar /deployments/jkube.jar
  runPolicy: Serial
status:
  lastVersion: 8  

Notes about the build above:

• The solution above is sets ROOT as user for running a pod
• Above requires SCC with any-uid

Runtime example 1

spec:
  replicas: 1
  selector:
    matchLabels:
      deployment: openjdk11-init
  template:
    metadata:
      creationTimestamp: null
      labels:
        deployment: openjdk11-init
      annotations:
        openshift.io/generated-by: OpenShiftNewApp
    spec:
      restartPolicy: Always
      initContainers:
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: init-myservice
          command:
            - /bin/bash
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: trusted-ca
              mountPath: /etc/pki/ca-trust/source/anchors/
            - name: new-cacerts
              mountPath: /opt/new-cacerts/
          terminationMessagePolicy: File
          image: registry.access.redhat.com/ubi9/openjdk-17
          args:
            - '-c'
            - update-ca-trust extract && echo "Injection" && ls -las /etc/pki/ca-trust/extracted/ && cp -a /etc/pki/ca-trust/extracted/. /opt/new-cacerts/ && ls -las /opt/new-cacerts/
      serviceAccountName: sa-with-anyuid
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      securityContext: {}
      containers:
        - name: openjdk11-init
          image: 'image-registry.openshift-image-registry.svc:5000/openjdk/openjdk11-injected@sha256:fe07aae3b4fdeec0d1e91f7c00ebb318204468b849475ea1b550a39154d7bddb'
          ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 8443
              protocol: TCP
            - containerPort: 8778
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: new-cacerts
              mountPath: /etc/pki/ca-trust/extracted/
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      serviceAccount: sa-with-anyuid
      volumes:
        - name: new-cacerts
          emptyDir: {}
        - name: trusted-ca
          configMap:
            name: ca-inject
            items:
              - key: service-ca.crt
                path: example.pem
            defaultMode: 420
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

And Configmap with the annotation service.beta.openshift.io/inject-cabundle: 'true', which makes the Service-CA generate the certificates - after the user uses the command:

kind: ConfigMap
apiVersion: v1
metadata:
  name: ca-inject
  namespace: openjdk
  annotations:
    service.beta.openshift.io/inject-cabundle: 'true'
immutable: false

Workflow for the Service-CA:

Streamlined process for the Service-CA Operator injection:

1. Deploy app
2. Create configmap annotate it
3. Run update-ca extract -> but this requires write over the /etc/pki/ca-trust/extracted/pem/ , which requires either a secret to overwrite that or a build to set user as root.

Notes about the injection above:

• The solution above is not trivial and requires a creation of a configmap to be injected via service-ca and then calls the update-ca-trust extract, which requires root for its execution.
• Requires privileged SCC - for the whole pod not just the init container. That can be a concern.
• The priviledged SCC will be available on the namespace for usage of other pods

Java inner works and runtime behavior

Java reads the truststore once when starting the JVM and cache the stores at runtime, it doesn't do that more than once at runtime.
Reason: Java's Secure Socket Extension (JSSE) reads the javax.net.ssl.trustStore property and initializes the default SSLContext and TrustManager the very first time an SSL/HTTPS connection is made. Once it is initialized, it is cached in memory.

Therefore, updating the file on disk or changing the system property afterward has zero effect on the running JVM.

Related solutions

The solution How do I import a CA root certificate to trust store in JBoss EAP for OpenShift?, describes a runtime and build time alternative for injecting CA Root certificates:

• Runtime alternative: generate the certificate generated local and add inside a configmap mounted to a writable location, then set JAVA_OPTS_APPEND for that location for the JVM to pick it up, given /etc/pki/ca-trust/extracted/java/cacerts is read-only.
• Build time alternative: copy the anchors inside the image and then run update-ca-trust as root on the build process, given the update-ca-trust command requires privileges and cannot be run on JBoss EAP for OpenShift Pod.
  The default java trust store /etc/pki/ca-trust/extracted/java/cacerts is read-only.