Procedure

1. If you do not have an existing SSH key pair on your local machine to use for authentication onto your cluster nodes, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t ed25519 -N '' -f <path>/<file_name>

   Copy code to clipboard

   Specifies the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have an existing key pair, ensure your public key is in the your ~/.ssh directory.

   Note

   If you plan to install an OpenShift Container Platform cluster that uses the RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le, and s390x architectures, do not create a key that uses the ed25519 algorithm. Instead, create a key that uses the rsa or ecdsa algorithm.

2. View the public SSH key:

   $ cat <path>/<file_name>.pub

   Copy code to clipboard

   For example, run the following to view the ~/.ssh/id_ed25519.pub public key:

   $ cat ~/.ssh/id_ed25519.pub

   Copy code to clipboard

3. Add the SSH private key identity to the SSH agent for your local user, if it has not already been added. SSH agent management of the key is required for password-less SSH authentication onto your cluster nodes, or if you want to use the ./openshift-install gather command.

   Note

   On some distributions, default SSH private key identities such as ~/.ssh/id_rsa and ~/.ssh/id_dsa are managed automatically.

   1. If the ssh-agent process is not already running for your local user, start it as a background task:

      $ eval "$(ssh-agent -s)"

      Copy code to clipboard

      Example output

      Agent pid 31874

      Copy code to clipboard

      Note

      If your cluster is in FIPS mode, only use FIPS-compliant algorithms to generate the SSH key. The key must be either RSA or ECDSA.

4. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name>

   Copy code to clipboard

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_ed25519.

   Example output

   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   Copy code to clipboard

Procedure

1. Go to the This content is not included.Cluster Type page on the Red Hat Hybrid Cloud Console. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

   Tip

   You can also Content from mirror.openshift.com is not included.download the binaries for a specific OpenShift Container Platform release.

2. Select your infrastructure provider from the Run it yourself section of the page.
3. Select your host operating system and architecture from the dropdown menus under OpenShift Installer and click Download Installer.

4. Place the downloaded file in the directory where you want to store the installation configuration files.

   Important

   • The installation program creates several files on the computer that you use to install your cluster. You must keep the installation program and the files that the installation program creates after you finish installing the cluster. Both of the files are required to delete the cluster.
   • Deleting the files created by the installation program does not remove your cluster, even if the cluster failed during installation. To remove your cluster, complete the OpenShift Container Platform uninstallation procedures for your specific cloud provider.

5. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -xvf openshift-install-linux.tar.gz

   Copy code to clipboard

6. Download your installation This content is not included.pull secret from Red Hat OpenShift Cluster Manager. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

   Tip

   Alternatively, you can retrieve the installation program from the This content is not included.Red Hat Customer Portal, where you can specify a version of the installation program to download. However, you must have an active subscription to access this page.

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.
4. Click Download Now next to the OpenShift v4.21 Linux Clients entry and save the file.

5. Unpack the archive:

   $ tar xvf <file>

   Copy code to clipboard

6. Place the oc binary in a directory that is on your PATH.

   To check your PATH, run the following command:

   $ echo $PATH

   Copy code to clipboard

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the appropriate version from the Version list.
3. Click Download Now next to the OpenShift v4.21 Windows Client entry and save the file.
4. Extract the archive with a ZIP program.

5. Move the oc binary to a directory that is on your PATH variable.

   To check your PATH variable, open the Command Prompt and run the following command:

   C:\> path

   Copy code to clipboard

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.

4. Click Download Now next to the OpenShift v4.21 macOS Clients entry and save the file.

   Note

   For macOS arm64, choose the OpenShift v4.21 macOS arm64 Client entry.

5. Extract the archive.

6. Move the oc binary to a directory on your PATH variable.

   To check your PATH variable, open a terminal and run the following command:

   $ echo $PATH

   Copy code to clipboard

Procedure

1. Obtain the RHCOS VHD cluster image:

   1. Export the URL of the RHCOS VHD to an environment variable.

      $ export COMPRESSED_VHD_URL=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.azurestack.formats."vhd.gz".disk.location')

      Copy code to clipboard

   2. Download the compressed RHCOS VHD file locally.

      $ curl -O -L ${COMPRESSED_VHD_URL}

      Copy code to clipboard

2. Decompress the VHD file.

   Note

   The decompressed VHD file is approximately 16 GB, so be sure that your host system has 16 GB of free space available. The VHD file can be deleted once you have uploaded it.

3. Upload the local VHD to the Azure Stack Hub environment, making sure that the blob is publicly available. For example, you can upload the VHD to a blob using the az cli or the web portal.

Procedure

1. Create an installation directory to store your required installation assets in:

   $ mkdir <installation_directory>

   Copy code to clipboard
   Important

   You must create a directory. Some installation assets, such as bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

2. Customize the provided sample install-config.yaml file template and save the file in the <installation_directory>.

   Note

   You must name this configuration file install-config.yaml.

   Make the following modifications:

   1. Specify the required installation parameters.
   2. Update the platform.azure section to specify the parameters that are specific to Azure Stack Hub.

   3. Optional: Update one or more of the default configuration parameters to customize the installation.

      For more information about the parameters, see "Installation configuration parameters".

3. Back up the install-config.yaml file so that you can use it to install many clusters.

   Important

   Back up the install-config.yaml file now, because the installation process consumes the file in the next step.

Procedure

1. If you have not previously created installation manifest files, do so by running the following command:

   $ openshift-install create manifests --dir <installation_directory>

   Copy code to clipboard

   where <installation_directory> is the directory in which the installation program creates files.

2. Set a $RELEASE_IMAGE variable with the release image from your installation file by running the following command:

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

   Copy code to clipboard

3. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image by running the following command:

   $ oc adm release extract \
     --from=$RELEASE_IMAGE \
     --credentials-requests \
     --included \
     --install-config=<path_to_directory_with_installation_configuration>/install-config.yaml \
     --to=<path_to_directory_for_credentials_requests>

   Copy code to clipboard

   where:

   --included

   Specifies only the manifests that your specific cluster configuration requires.

   <path_to_directory_with_installation_configuration>

   Specifies the location of the install-config.yaml file.

   <path_to_directory_for_credentials_requests>

   Specifies the path to the directory where you want to store the CredentialsRequest objects. If the specified directory does not exist, this command creates it.

   This command creates a YAML file for each CredentialsRequest object.

   Sample CredentialsRequest object

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
     ...

   Copy code to clipboard

4. Create YAML files for secrets in the openshift-install manifests directory that you generated previously. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each CredentialsRequest object.

   Sample CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Copy code to clipboard

   Sample Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     azure_subscription_id: <base64_encoded_azure_subscription_id>
     azure_client_id: <base64_encoded_azure_client_id>
     azure_client_secret: <base64_encoded_azure_client_secret>
     azure_tenant_id: <base64_encoded_azure_tenant_id>
     azure_resource_prefix: <base64_encoded_azure_resource_prefix>
     azure_resourcegroup: <base64_encoded_azure_resourcegroup>
     azure_region: <base64_encoded_azure_region>

   Copy code to clipboard

   Important

   Before upgrading a cluster that uses manually maintained credentials, you must ensure that the CCO is in an upgradeable state.

Procedure

1. From the directory in which the installation program creates files, go to the manifests directory.

2. Add user-ca-bundle to the spec.trustedCA.name field.

   Example cluster-proxy-01-config.yaml file

   apiVersion: config.openshift.io/v1
   kind: Proxy
   metadata:
     creationTimestamp: null
     name: cluster
   spec:
     trustedCA:
       name: user-ca-bundle
   status: {}

   Copy code to clipboard

3. Optional: Back up the manifests/ cluster-proxy-01-config.yaml file. The installation program consumes the manifests/ directory when you deploy the cluster.

Procedure

1. Export the kubeadmin credentials by running the following command:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig

   Copy code to clipboard

   where:

   <installation_directory>

   Specifies the path to the directory that stores the installation files.

2. Verify you can run oc commands successfully using the exported configuration by running the following command:

   $ oc whoami

   Copy code to clipboard

   Example output

   system:admin

   Copy code to clipboard

Procedure

1. Obtain the password for the kubeadmin user from the kubeadmin-password file on the installation host:

   $ cat <installation_directory>/auth/kubeadmin-password

   Copy code to clipboard
   Note

   Or, you can obtain the kubeadmin password from the <installation_directory>/.openshift_install.log log file on the installation host.

2. List the OpenShift Container Platform web console route:

   $ oc get routes -n openshift-console | grep 'console-openshift'

   Copy code to clipboard
   Note

   Or, you can obtain the OpenShift Container Platform route from the <installation_directory>/.openshift_install.log log file on the installation host.

   Example output

   console     console-openshift-console.apps.<cluster_name>.<base_domain>            console     https   reencrypt/Redirect   None

   Copy code to clipboard

3. Navigate to the route detailed in the output of the preceding command in a web browser and log in as the kubeadmin user.

Procedure

1. Obtain the RHCOS VHD cluster image:

   1. Export the URL of the RHCOS VHD to an environment variable.

      $ export COMPRESSED_VHD_URL=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.azurestack.formats."vhd.gz".disk.location')

      Copy code to clipboard

   2. Download the compressed RHCOS VHD file locally.

      $ curl -O -L ${COMPRESSED_VHD_URL}

      Copy code to clipboard

2. Decompress the VHD file.

   Note

   The decompressed VHD file is approximately 16 GB, so be sure that your host system has 16 GB of free space available. The VHD file can be deleted once you have uploaded it.

3. Upload the local VHD to the Azure Stack Hub environment, making sure that the blob is publicly available. For example, you can upload the VHD to a blob using the az cli or the web portal.

Procedure

1. Create an installation directory to store your required installation assets in:

   $ mkdir <installation_directory>

   Copy code to clipboard
   Important

   You must create a directory. Some installation assets, such as bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

2. Customize the provided sample install-config.yaml file template and save the file in the <installation_directory>.

   Note

   You must name this configuration file install-config.yaml.

   Make the following modifications:

   1. Specify the required installation parameters.
   2. Update the platform.azure section to specify the parameters that are specific to Azure Stack Hub.

   3. Optional: Update one or more of the default configuration parameters to customize the installation.

      For more information about the parameters, see "Installation configuration parameters".

3. Back up the install-config.yaml file so that you can use it to install many clusters.

   Important

   Back up the install-config.yaml file now, because the installation process consumes the file in the next step.

Procedure

1. If you have not previously created installation manifest files, do so by running the following command:

   $ openshift-install create manifests --dir <installation_directory>

   Copy code to clipboard

   where <installation_directory> is the directory in which the installation program creates files.

2. Set a $RELEASE_IMAGE variable with the release image from your installation file by running the following command:

   $ RELEASE_IMAGE=$(./openshift-install version | awk '/release image/ {print $3}')

   Copy code to clipboard

3. Extract the list of CredentialsRequest custom resources (CRs) from the OpenShift Container Platform release image by running the following command:

   $ oc adm release extract \
     --from=$RELEASE_IMAGE \
     --credentials-requests \
     --included \
     --install-config=<path_to_directory_with_installation_configuration>/install-config.yaml \
     --to=<path_to_directory_for_credentials_requests>

   Copy code to clipboard

   where:

   --included

   Specifies only the manifests that your specific cluster configuration requires.

   <path_to_directory_with_installation_configuration>

   Specifies the location of the install-config.yaml file.

   <path_to_directory_for_credentials_requests>

   Specifies the path to the directory where you want to store the CredentialsRequest objects. If the specified directory does not exist, this command creates it.

   This command creates a YAML file for each CredentialsRequest object.

   Sample CredentialsRequest object

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
     ...

   Copy code to clipboard

4. Create YAML files for secrets in the openshift-install manifests directory that you generated previously. The secrets must be stored using the namespace and secret name defined in the spec.secretRef for each CredentialsRequest object.

   Sample CredentialsRequest object with secrets

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: <component_credentials_request>
     namespace: openshift-cloud-credential-operator
     ...
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AzureProviderSpec
       roleBindings:
       - role: Contributor
         ...
     secretRef:
       name: <component_secret>
       namespace: <component_namespace>
     ...

   Copy code to clipboard

   Sample Secret object

   apiVersion: v1
   kind: Secret
   metadata:
     name: <component_secret>
     namespace: <component_namespace>
   data:
     azure_subscription_id: <base64_encoded_azure_subscription_id>
     azure_client_id: <base64_encoded_azure_client_id>
     azure_client_secret: <base64_encoded_azure_client_secret>
     azure_tenant_id: <base64_encoded_azure_tenant_id>
     azure_resource_prefix: <base64_encoded_azure_resource_prefix>
     azure_resourcegroup: <base64_encoded_azure_resourcegroup>
     azure_region: <base64_encoded_azure_region>

   Copy code to clipboard

   Important

   Before upgrading a cluster that uses manually maintained credentials, you must ensure that the CCO is in an upgradeable state.

Procedure

1. From the directory in which the installation program creates files, go to the manifests directory.

2. Add user-ca-bundle to the spec.trustedCA.name field.

   Example cluster-proxy-01-config.yaml file

   apiVersion: config.openshift.io/v1
   kind: Proxy
   metadata:
     creationTimestamp: null
     name: cluster
   spec:
     trustedCA:
       name: user-ca-bundle
   status: {}

   Copy code to clipboard

3. Optional: Back up the manifests/ cluster-proxy-01-config.yaml file. The installation program consumes the manifests/ directory when you deploy the cluster.

Procedure

1. Change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory>

   Copy code to clipboard

   where <installation_directory> specifies the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a stub manifest file for the advanced network configuration that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:

   Copy code to clipboard

3. Specify the advanced network configuration for your cluster in the cluster-network-03-config.yml file, such as in the following example:

   The following example enables IPsec for the OVN-Kubernetes network provider:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         ipsecConfig:
           mode: Full

   Copy code to clipboard
4. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program consumes the manifests/ directory when you create the Ignition config files.

5. Remove the Kubernetes manifest files that define the control plane machines and compute MachineSets:

   $ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml

   Copy code to clipboard

   Because you create and manage these resources yourself, you do not have to initialize them.

   • You can preserve the MachineSet files to create compute machines by using the machine API, but you must update references to them to match your environment.

Procedure

1. Change to the directory that contains the installation program and create the manifests:

   $ ./openshift-install create manifests --dir <installation_directory>

   Copy code to clipboard

   For the <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a stub manifest file for the advanced network configuration that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   $ cat <<EOF > <installation_directory>/manifests/cluster-network-03-config.yml
   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
   EOF

   Copy code to clipboard

   For <installation_directory>, specify the directory name that contains the manifests/ directory for your cluster.

3. Open the cluster-network-03-config.yml file in an editor and specify a hybrid networking configuration similar to the following example:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     defaultNetwork:
       ovnKubernetesConfig:
         hybridOverlayConfig:
           hybridClusterNetwork:
           - cidr: 10.132.0.0/14
             hostPrefix: 23
           hybridOverlayVXLANPort: 9898

   Copy code to clipboard

   where:

   spec.defaultNetwork.ovnKubernetesConfig.hybridOverlayConfig.hybridClusterNetwork

   Specifies the CIDR configuration used for nodes on the additional overlay network. The hybridClusterNetwork CIDR must not overlap with the clusterNetwork CIDR.

   spec.defaultNetwork.ovnKubernetesConfig.hybridOverlayConfig.hybridOverlayVXLANPort

   Specifies a custom VXLAN port for the additional overlay network. This is required for running Windows nodes in a cluster installed on vSphere, and must not be configured for any other cloud provider. The custom port can be any open port excluding the default 6081 port. For more information on this requirement, see Content from docs.microsoft.com is not included.Pod-to-pod connectivity between hosts is broken in the Microsoft documentation.

   Note

   Windows Server Long-Term Servicing Channel (LTSC): Windows Server 2019 is not supported on clusters with a custom hybridOverlayVXLANPort value because this Windows server version does not support selecting a custom VXLAN port.

4. Save the cluster-network-03-config.yml file and quit the text editor.
5. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory when creating the cluster.

Procedure

1. Export the kubeadmin credentials by running the following command:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig

   Copy code to clipboard

   where:

   <installation_directory>

   Specifies the path to the directory that stores the installation files.

2. Verify you can run oc commands successfully using the exported configuration by running the following command:

   $ oc whoami

   Copy code to clipboard

   Example output

   system:admin

   Copy code to clipboard

Procedure

1. Obtain the password for the kubeadmin user from the kubeadmin-password file on the installation host:

   $ cat <installation_directory>/auth/kubeadmin-password

   Copy code to clipboard
   Note

   Or, you can obtain the kubeadmin password from the <installation_directory>/.openshift_install.log log file on the installation host.

2. List the OpenShift Container Platform web console route:

   $ oc get routes -n openshift-console | grep 'console-openshift'

   Copy code to clipboard
   Note

   Or, you can obtain the OpenShift Container Platform route from the <installation_directory>/.openshift_install.log log file on the installation host.

   Example output

   console     console-openshift-console.apps.<cluster_name>.<base_domain>            console     https   reencrypt/Redirect   None

   Copy code to clipboard

3. Navigate to the route detailed in the output of the preceding command in a web browser and log in as the kubeadmin user.

Procedure

1. If you do not have an existing SSH key pair on your local machine to use for authentication onto your cluster nodes, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t ed25519 -N '' -f <path>/<file_name>

   Copy code to clipboard

   Specifies the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have an existing key pair, ensure your public key is in the your ~/.ssh directory.

   Note

   If you plan to install an OpenShift Container Platform cluster that uses the RHEL cryptographic libraries that have been submitted to NIST for FIPS 140-2/140-3 Validation on only the x86_64, ppc64le, and s390x architectures, do not create a key that uses the ed25519 algorithm. Instead, create a key that uses the rsa or ecdsa algorithm.

2. View the public SSH key:

   $ cat <path>/<file_name>.pub

   Copy code to clipboard

   For example, run the following to view the ~/.ssh/id_ed25519.pub public key:

   $ cat ~/.ssh/id_ed25519.pub

   Copy code to clipboard

3. Add the SSH private key identity to the SSH agent for your local user, if it has not already been added. SSH agent management of the key is required for password-less SSH authentication onto your cluster nodes, or if you want to use the ./openshift-install gather command.

   Note

   On some distributions, default SSH private key identities such as ~/.ssh/id_rsa and ~/.ssh/id_dsa are managed automatically.

   1. If the ssh-agent process is not already running for your local user, start it as a background task:

      $ eval "$(ssh-agent -s)"

      Copy code to clipboard

      Example output

      Agent pid 31874

      Copy code to clipboard

      Note

      If your cluster is in FIPS mode, only use FIPS-compliant algorithms to generate the SSH key. The key must be either RSA or ECDSA.

4. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name>

   Copy code to clipboard

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_ed25519.

   Example output

   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   Copy code to clipboard

Procedure

1. Go to the This content is not included.Cluster Type page on the Red Hat Hybrid Cloud Console. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

   Tip

   You can also Content from mirror.openshift.com is not included.download the binaries for a specific OpenShift Container Platform release.

2. Select your infrastructure provider from the Run it yourself section of the page.
3. Select your host operating system and architecture from the dropdown menus under OpenShift Installer and click Download Installer.

4. Place the downloaded file in the directory where you want to store the installation configuration files.

   Important

   • The installation program creates several files on the computer that you use to install your cluster. You must keep the installation program and the files that the installation program creates after you finish installing the cluster. Both of the files are required to delete the cluster.
   • Deleting the files created by the installation program does not remove your cluster, even if the cluster failed during installation. To remove your cluster, complete the OpenShift Container Platform uninstallation procedures for your specific cloud provider.

5. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar -xvf openshift-install-linux.tar.gz

   Copy code to clipboard

6. Download your installation This content is not included.pull secret from Red Hat OpenShift Cluster Manager. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

   Tip

   Alternatively, you can retrieve the installation program from the This content is not included.Red Hat Customer Portal, where you can specify a version of the installation program to download. However, you must have an active subscription to access this page.

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.
4. Click Download Now next to the OpenShift v4.21 Linux Clients entry and save the file.

5. Unpack the archive:

   $ tar xvf <file>

   Copy code to clipboard

6. Place the oc binary in a directory that is on your PATH.

   To check your PATH, run the following command:

   $ echo $PATH

   Copy code to clipboard

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the appropriate version from the Version list.
3. Click Download Now next to the OpenShift v4.21 Windows Client entry and save the file.
4. Extract the archive with a ZIP program.

5. Move the oc binary to a directory that is on your PATH variable.

   To check your PATH variable, open the Command Prompt and run the following command:

   C:\> path

   Copy code to clipboard

Procedure

1. Navigate to the This content is not included.Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.

4. Click Download Now next to the OpenShift v4.21 macOS Clients entry and save the file.

   Note

   For macOS arm64, choose the OpenShift v4.21 macOS arm64 Client entry.

5. Extract the archive.

6. Move the oc binary to a directory on your PATH variable.

   To check your PATH variable, open a terminal and run the following command:

   $ echo $PATH

   Copy code to clipboard

Procedure

1. Create an Azure storage account to store the VHD cluster image:

   $ az storage account create -g ${RESOURCE_GROUP} --location ${AZURE_REGION} --name ${CLUSTER_NAME}sa --kind Storage --sku Standard_LRS

   Copy code to clipboard
   Warning

   The Azure storage account name must be between 3 and 24 characters in length and use numbers and lower-case letters only. If your CLUSTER_NAME variable does not follow these restrictions, you must manually define the Azure storage account name. For more information on Azure storage account name restrictions, see Content from docs.microsoft.com is not included.Resolve errors for storage account names in the Azure documentation.

2. Export the storage account key as an environment variable:

   $ export ACCOUNT_KEY=`az storage account keys list -g ${RESOURCE_GROUP} --account-name ${CLUSTER_NAME}sa --query "[0].value" -o tsv`

   Copy code to clipboard

3. Export the URL of the RHCOS VHD to an environment variable:

   $ export COMPRESSED_VHD_URL=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.azurestack.formats."vhd.gz".disk.location')

   Copy code to clipboard
   Important

   The RHCOS images might not change with every release of OpenShift Container Platform. You must specify an image with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image version that matches your OpenShift Container Platform version if it is available.

4. Create the storage container for the VHD:

   $ az storage container create --name vhd --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY}

   Copy code to clipboard

5. Download the compressed RHCOS VHD file locally:

   $ curl -O -L ${COMPRESSED_VHD_URL}

   Copy code to clipboard

6. Decompress the VHD file.

   Note

   The decompressed VHD file is approximately 16 GB, so be sure that your host system has 16 GB of free space available. You can delete the VHD file after you upload it.

7. Copy the local VHD to a blob:

   $ az storage blob upload --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY} -c vhd -n "rhcos.vhd" -f rhcos-<rhcos_version>-azurestack.x86_64.vhd

   Copy code to clipboard

8. Create a blob storage container and upload the generated bootstrap.ign file:

   $ az storage container create --name files --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY}

   Copy code to clipboard

   $ az storage blob upload --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY} -c "files" -f "<installation_directory>/bootstrap.ign" -n "bootstrap.ign"

   Copy code to clipboard

Procedure

1. Copy the template from the ARM template for the VNet section of this topic and save it as 01_vnet.json in your cluster’s installation directory. This template describes the VNet that your cluster requires.

2. Create the deployment by using the az CLI:

   $ az deployment group create -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/01_vnet.json" \
     --parameters baseName="${INFRA_ID}"

   Copy code to clipboard

   baseName specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

Procedure

1. Copy the template from the ARM template for image storage section of this topic and save it as 02_storage.json in your cluster’s installation directory. This template describes the image storage that your cluster requires.

2. Export the RHCOS VHD blob URL as a variable:

   $ export VHD_BLOB_URL=`az storage blob url --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY} -c vhd -n "rhcos.vhd" -o tsv`

   Copy code to clipboard

3. Deploy the cluster image:

   $ az deployment group create -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/02_storage.json" \
     --parameters vhdBlobURL="${VHD_BLOB_URL}" \
     --parameters baseName="${INFRA_ID}" \
     --parameters storageAccount="${CLUSTER_NAME}sa" \
     --parameters architecture="<architecture>"

   Copy code to clipboard

   where:

   vhdBlobURL

   Specifies the blob URL of the RHCOS VHD to be used to create master and worker machines.

   baseName

   Specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

   storageAccount

   Specifies the name of your Azure storage account.

   architecture

   Specifies the system architecture. Valid values are x64 (default) or Arm64.

Procedure

1. Copy the template from the ARM template for the network and load balancers section of this topic and save it as 03_infra.json in your cluster’s installation directory. This template describes the networking and load balancing objects that your cluster requires.

2. Create the deployment by using the az CLI:

   $ az deployment group create -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/03_infra.json" \
     --parameters baseName="${INFRA_ID}"

   Copy code to clipboard

   The baseName specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

3. Create an api DNS record and an api-int DNS record. When creating the API DNS records, the ${BASE_DOMAIN_RESOURCE_GROUP} variable must point to the resource group where the DNS zone exists.

   1. Export the following variable:

      $ export PUBLIC_IP=`az network public-ip list -g ${RESOURCE_GROUP} --query "[?name=='${INFRA_ID}-master-pip'] | [0].ipAddress" -o tsv`

      Copy code to clipboard

   2. Export the following variable:

      $ export PRIVATE_IP=`az network lb frontend-ip show -g "$RESOURCE_GROUP" --lb-name "${INFRA_ID}-internal" -n internal-lb-ip --query "privateIpAddress" -o tsv`

      Copy code to clipboard

   3. Create the api DNS record in a new DNS zone:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z ${CLUSTER_NAME}.${BASE_DOMAIN} -n api -a ${PUBLIC_IP} --ttl 60

      Copy code to clipboard

      If you are adding the cluster to an existing DNS zone, you can create the api DNS record in it instead:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z ${BASE_DOMAIN} -n api.${CLUSTER_NAME} -a ${PUBLIC_IP} --ttl 60

      Copy code to clipboard

   4. Create the api-int DNS record in a new DNS zone:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z "${CLUSTER_NAME}.${BASE_DOMAIN}" -n api-int -a ${PRIVATE_IP} --ttl 60

      Copy code to clipboard

      If you are adding the cluster to an existing DNS zone, you can create the api-int DNS record in it instead:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z ${BASE_DOMAIN} -n api-int.${CLUSTER_NAME} -a ${PRIVATE_IP} --ttl 60

      Copy code to clipboard

Procedure

1. Copy the template from the ARM template for the bootstrap machine section of this topic and save it as 04_bootstrap.json in your cluster’s installation directory. This template describes the bootstrap machine that your cluster requires.

2. Export the bootstrap URL variable:

   $ bootstrap_url_expiry=`date -u -d "10 hours" '+%Y-%m-%dT%H:%MZ'`

   Copy code to clipboard

   $ export BOOTSTRAP_URL=`az storage blob generate-sas -c 'files' -n 'bootstrap.ign' --https-only --full-uri --permissions r --expiry $bootstrap_url_expiry --account-name ${CLUSTER_NAME}sa --account-key ${ACCOUNT_KEY} -o tsv`

   Copy code to clipboard

3. Export the bootstrap ignition variable:

   1. If your environment uses a public certificate authority (CA), run this command:

      $ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url ${BOOTSTRAP_URL} '{ignition:{version:$v,config:{replace:{source:$url}}}}' | base64 | tr -d '\n'`

      Copy code to clipboard

   2. If your environment uses an internal CA, you must add your PEM encoded bundle to the bootstrap ignition stub so that your bootstrap virtual machine can pull the bootstrap ignition from the storage account. Run the following commands, which assume your CA is in a file called CA.pem:

      $ export CA="data:text/plain;charset=utf-8;base64,$(cat CA.pem |base64 |tr -d '\n')"

      Copy code to clipboard

      $ export BOOTSTRAP_IGNITION=`jq -rcnM --arg v "3.2.0" --arg url "$BOOTSTRAP_URL" --arg cert "$CA" '{ignition:{version:$v,security:{tls:{certificateAuthorities:[{source:$cert}]}},config:{replace:{source:$url}}}}' | base64 | tr -d '\n'`

      Copy code to clipboard

4. Create the deployment by using the az CLI:

   $ az deployment group create --verbose -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/04_bootstrap.json" \
     --parameters bootstrapIgnition="${BOOTSTRAP_IGNITION}" \
     --parameters baseName="${INFRA_ID}" \
     --parameters diagnosticsStorageAccountName="${CLUSTER_NAME}sa"

   Copy code to clipboard

   where:

   bootstrapIgnition

   Specifies the bootstrap Ignition content for the bootstrap cluster.

   baseName

   Specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

   diagnosticsStorageAccountName

   Specifies the name of the storage account for your cluster.

Procedure

1. Copy the template from the ARM template for control plane machines section of this topic and save it as 05_masters.json in your cluster’s installation directory. This template describes the control plane machines that your cluster requires.

2. Export the following variable needed by the control plane machine deployment:

   $ export MASTER_IGNITION=`cat <installation_directory>/master.ign | base64 | tr -d '\n'`

   Copy code to clipboard

3. Create the deployment by using the az CLI:

   $ az deployment group create -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/05_masters.json" \
     --parameters masterIgnition="${MASTER_IGNITION}" \
     --parameters baseName="${INFRA_ID}" \
     --parameters diagnosticsStorageAccountName="${CLUSTER_NAME}sa"

   Copy code to clipboard

   where:

   masterIgnition

   Specifies the Ignition content for the control plane nodes (also known as the master nodes).

   baseName

   Specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

   diagnosticsStorageAccountName

   Specifies the name of the storage account for your cluster.

Procedure

1. Change to the directory that contains the installation program and run the following command:

   $ ./openshift-install wait-for bootstrap-complete --dir <installation_directory> \
       --log-level info

   Copy code to clipboard

   where:

   <installation_directory>

   Specifies the path to the directory that you stored the installation files in.

   --log-level info

   Specifies the installation details. Specify warn, debug, or error instead of info to view different installation details.

   If the command exits without a FATAL warning, your production control plane has initialized.

2. Delete the bootstrap resources:

   $ az network nsg rule delete -g ${RESOURCE_GROUP} --nsg-name ${INFRA_ID}-nsg --name bootstrap_ssh_in

   Copy code to clipboard

   $ az vm stop -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap

   Copy code to clipboard

   $ az vm deallocate -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap

   Copy code to clipboard

   $ az vm delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap --yes

   Copy code to clipboard

   $ az disk delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap_OSDisk --no-wait --yes

   Copy code to clipboard

   $ az network nic delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap-nic --no-wait

   Copy code to clipboard

   $ az storage blob delete --account-key ${ACCOUNT_KEY} --account-name ${CLUSTER_NAME}sa --container-name files --name bootstrap.ign

   Copy code to clipboard

   $ az network public-ip delete -g ${RESOURCE_GROUP} --name ${INFRA_ID}-bootstrap-ssh-pip

   Copy code to clipboard
   Note

   If you do not delete the bootstrap server, installation may not succeed due to API traffic being routed to the bootstrap server.

Procedure

1. Copy the template from the ARM template for worker machines section of this topic and save it as 06_workers.json in your cluster’s installation directory. This template describes the worker machines that your cluster requires.

2. Export the following variable needed by the worker machine deployment:

   $ export WORKER_IGNITION=`cat <installation_directory>/worker.ign | base64 | tr -d '\n'`

   Copy code to clipboard

3. Create the deployment by using the az CLI:

   $ az deployment group create -g ${RESOURCE_GROUP} \
     --template-file "<installation_directory>/06_workers.json" \
     --parameters workerIgnition="${WORKER_IGNITION}" \
     --parameters baseName="${INFRA_ID}" \
     --parameters diagnosticsStorageAccountName="${CLUSTER_NAME}sa"

   Copy code to clipboard

   where:

   workerIgnition

   Specifies the Ignition content for the worker nodes.

   baseName

   Specifies the base name to be used in resource names; this is usually the cluster’s infrastructure ID.

   diagnosticsStorageAccountName

   Specifies the name of the storage account for your cluster.

Procedure

1. Export the kubeadmin credentials by running the following command:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig

   Copy code to clipboard

   where:

   <installation_directory>

   Specifies the path to the directory that stores the installation files.

2. Verify you can run oc commands successfully using the exported configuration by running the following command:

   $ oc whoami

   Copy code to clipboard

   Example output

   system:admin

   Copy code to clipboard

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Copy code to clipboard

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.34.2
   master-1  Ready     master  63m  v1.34.2
   master-2  Ready     master  64m  v1.34.2

   Copy code to clipboard

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes until you approve some CSRs.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Copy code to clipboard

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   Copy code to clipboard

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   You must approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates rotate, and more than two certificates are present for each node. You must approve all of these certificates. After you approve the client CSR, the kubelet creates a secondary CSR for the serving certificate, which requires manual approval. The machine-approver then automatically approves later serving certificate renewal requests if the kubelet requests a new certificate with the same parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If you do not approve a request, the oc exec, oc rsh, and oc logs commands cannot succeed, because the API server requires a serving certificate when it connects to the kubelet. Any operation that contacts the kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the node-bootstrapper service account in the system:node or system:admin groups submitted the CSR, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name>

     Copy code to clipboard

     where:

     <csr_name>

     Specifies the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Copy code to clipboard
     Note

     Some Operators might not become available until you approve some CSRs.

4. After you approve your client requests, review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Copy code to clipboard

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   Copy code to clipboard

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name>

     Copy code to clipboard

     where:

     <csr_name>

     Specifies the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

     Copy code to clipboard

6. After you approve all client and server CSRs, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Copy code to clipboard

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.34.2
   master-1  Ready     master  73m  v1.34.2
   master-2  Ready     master  74m  v1.34.2
   worker-0  Ready     worker  11m  v1.34.2
   worker-1  Ready     worker  11m  v1.34.2

   Copy code to clipboard

   Note

   You might need to wait a few minutes after approval of the server CSRs for the machines to reach the Ready status.

Procedure

1. Confirm the Ingress router has created a load balancer and populated the EXTERNAL-IP field:

   $ oc -n openshift-ingress get service router-default

   Copy code to clipboard

   Example output

   NAME             TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
   router-default   LoadBalancer   172.30.20.10   35.130.120.110   80:32288/TCP,443:31215/TCP   20

   Copy code to clipboard

2. Export the Ingress router IP as a variable:

   $ export PUBLIC_IP_ROUTER=`oc -n openshift-ingress get service router-default --no-headers | awk '{print $4}'`

   Copy code to clipboard

3. Add a *.apps record to the DNS zone.

   1. If you are adding this cluster to a new DNS zone, run:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z ${CLUSTER_NAME}.${BASE_DOMAIN} -n *.apps -a ${PUBLIC_IP_ROUTER} --ttl 300

      Copy code to clipboard

   2. If you are adding this cluster to an already existing DNS zone, run:

      $ az network dns record-set a add-record -g ${BASE_DOMAIN_RESOURCE_GROUP} -z ${BASE_DOMAIN} -n *.apps.${CLUSTER_NAME} -a ${PUBLIC_IP_ROUTER} --ttl 300

      Copy code to clipboard

   If you prefer to add explicit domains instead of using a wildcard, you can create entries for each of the cluster’s current routes:

   +

   $ oc get --all-namespaces -o jsonpath='{range .items[*]}{range .status.ingress[*]}{.host}{"\n"}{end}{end}' routes

   Copy code to clipboard

   + .Example output

   oauth-openshift.apps.cluster.basedomain.com
   console-openshift-console.apps.cluster.basedomain.com
   downloads-openshift-console.apps.cluster.basedomain.com
   alertmanager-main-openshift-monitoring.apps.cluster.basedomain.com
   prometheus-k8s-openshift-monitoring.apps.cluster.basedomain.com

   Copy code to clipboard