Procedure

1. Create a Splunk account by navigating to Content from www.splunk.com is not included.Splunk and entering the required credentials.
2. Navigate to the Content from www.splunk.com is not included.Splunk Enterprise Free Trial page, select your platform and installation package, and then click Download Now.
3. Install the Splunk software on your machine. When prompted, create a username, for example, splunk_admin and password.
4. After creating a username and password, a localhost URL will be provided for your Splunk deployment, for example, Content from <sample_url>.remote.csb is not included.http://<sample_url>.remote.csb:8000/. Open the URL in your preferred browser.
5. Log in with the username and password you created during installation. You are directed to the Splunk UI.

Procedure

1. Configure Red Hat Quay to use Splunk or the Splunk HTTP Event Collector (HEC).

   1. If opting to use Splunk, open your Red Hat Quay config.yaml file and add the following configuration fields:

      # ...
      LOGS_MODEL: splunk
      LOGS_MODEL_CONFIG:
          producer: splunk
          splunk_config:
              host: http://<user_name>.remote.csb
              port: 8089
              bearer_token: <bearer_token>
              url_scheme: <http/https>
              verify_ssl: False
              index_prefix: <splunk_log_index_name>
              ssl_ca_path: <location_to_ssl-ca-cert.pem>
              search_timeout: 60
              max_results: 10000
              export_batch_size: 5000
      # ...

      Copy code to clipboard

      where:

      LOGS_MODEL_CONFIG.splunk_config.host

      Specifies the Splunk cluster endpoint.

      LOGS_MODEL_CONFIG.splunk_config.port

      Specifies the Splunk management cluster endpoint port. Differs from the Splunk GUI hosted port. Can be found on the Splunk UI under Settings → Server Settings → General Settings.

      LOGS_MODEL_CONFIG.splunk_config.bearer_token

      Specifies the generated bearer token for Splunk.

      LOGS_MODEL_CONFIG.splunk_config.url_scheme

      Specifies the URL scheme for access the Splunk service. If Splunk is configured to use TLS/SSL, this must be https.

      LOGS_MODEL_CONFIG.splunk_config.verify_ssl

      Specifies whether to enable TLS/SSL. Defaults to True.

      LOGS_MODEL_CONFIG.splunk_config.index_prefix

      Specifies the Splunk index prefix. Can be a new, or used, index. Can be created from the Splunk UI.

      LOGS_MODEL_CONFIG.splunk_config.ssl_ca_path

      Specifies the relative container path to a single .pem file containing a certificate authority (CA) for TLS/SSL validation.

      LOGS_MODEL_CONFIG.splunk_config.search_timeout

      Specifies the timeout for Splunk search queries in seconds. Increase for slow Splunk clusters or complex queries.

      LOGS_MODEL_CONFIG.splunk_config.max_results

      Specifies the maximum number of results to return per search query. Larger values require more memory.

      LOGS_MODEL_CONFIG.splunk_config.export_batch_size

      Specifies the batch size for log export operations.

   2. If opting to use Splunk HEC, open your Red Hat Quay config.yaml file and add the following configuration fields:

      # ...
      LOGS_MODEL: splunk
      LOGS_MODEL_CONFIG:
        producer: splunk_hec
        splunk_hec_config:
          host: prd-p-aaaaaq.splunkcloud.com
          port: 8088
          hec_token: 12345678-1234-1234-1234-1234567890ab
          url_scheme: https
          verify_ssl: False
          index: quay
          splunk_host: quay-dev
          splunk_sourcetype: quay_logs
          timeout: 10
          search_token: <bearer_token>
          search_host: <splunk.example.com>
          search_port: 8089
          search_timeout: 60
          max_results: 10000
          export_batch_size: 5000
      # ...

      Copy code to clipboard

      where:

      LOGS_MODEL_CONFIG.producer

      Specifies splunk_hec when configuring Splunk HEC.

      LOGS_MODEL_CONFIG.splunk_hec_config

      Specifies the logs model configuration for Splunk HTTP Event Collector action logs configuration.

      LOGS_MODEL_CONFIG.splunk_hec_config.host

      Specifies the Splunk cluster endpoint.

      LOGS_MODEL_CONFIG.splunk_hec_config.port

      Specifies the Splunk management cluster endpoint port.

      LOGS_MODEL_CONFIG.splunk_hec_config.hec_token

      Specifies the HEC token for Splunk.

      LOGS_MODEL_CONFIG.splunk_hec_config.url_scheme

      Specifies the URL scheme for access the Splunk service. If Splunk is behind SSL/TLS, must be https.

      LOGS_MODEL_CONFIG.splunk_hec_config.verify_ssl

      Specifies whether to enable (true) or disable (false) SSL/TLS verification for HTTPS connections.

      LOGS_MODEL_CONFIG.splunk_hec_config.index

      Specifies the Splunk index to use.

      LOGS_MODEL_CONFIG.splunk_hec_config.splunk_host

      Specifies the host name to log this event.

      LOGS_MODEL_CONFIG.splunk_hec_config.splunk_sourcetype

      Specifies the name of the Splunk sourcetype to use.

      LOGS_MODEL_CONFIG.splunk_hec_config.timeout

      Specifies the timeout in seconds for HTTP requests to the Splunk HEC endpoint. Prevents requests from hanging indefinitely when Splunk is unresponsive.

      LOGS_MODEL_CONFIG.splunk_hec_config.search_token

      Specifies an optional bearer token for the Splunk search API. Required because HEC tokens are ingest-only and cannot search.

      LOGS_MODEL_CONFIG.splunk_hec_config.search_host

      Specifies the Splunk management host for the search API. Defaults to the HEC host if not specified.

      LOGS_MODEL_CONFIG.splunk_hec_config.search_port

      Specifies the Splunk management port for the search API. Defaults to 8089 if not specified.

      LOGS_MODEL_CONFIG.splunk_hec_config.search_timeout

      Specifies the timeout for Splunk search queries in seconds. Increase for slow Splunk clusters or complex queries.

      LOGS_MODEL_CONFIG.splunk_hec_config.max_results

      Specifies the maximum number of results to return per search query. Larger values require more memory.

      LOGS_MODEL_CONFIG.splunk_hec_config.export_batch_size

      Specifies the batch size for log export operations.

2. If you are configuring ssl_ca_path, you must configure the SSL/TLS certificate so that Red Hat Quay trusts it.

   1. If you are using a standalone deployment of Red Hat Quay, SSL/TLS certificates can be provided by placing the certificate file inside of the extra_ca_certs directory, or inside of the relative container path and specified by ssl_ca_path.

   2. If you are using the Red Hat Quay Operator, create a config bundle secret, including the certificate authority (CA) of the Splunk server. For example:

      $ oc create secret generic --from-file config.yaml=./config_390.yaml --from-file extra_ca_cert_splunkserver.crt=./splunkserver.crt config-bundle-secret

      Copy code to clipboard

      Specify the conf/stack/extra_ca_certs/splunkserver.crt file in your config.yaml. For example:

      # ...
      LOGS_MODEL: splunk
      LOGS_MODEL_CONFIG:
          producer: splunk
          splunk_config:
              host: ec2-12-345-67-891.us-east-2.compute.amazonaws.com
              port: 8089
              bearer_token: eyJra
              url_scheme: https
              verify_ssl: true
              index_prefix: quay123456
              ssl_ca_path: conf/stack/splunkserver.crt
      # ...

      Copy code to clipboard

Procedure

1. Log in to your Red Hat Quay deployment.
2. Click on the name of the organization that you use to create an action log for Splunk.
3. In the navigation pane, click Robot Accounts → Create Robot Account.
4. When prompted, enter a name for the robot account, for example splunkrobotaccount, then click Create robot account.
5. On your browser, open the Splunk UI.
6. Click Search and Reporting.

7. In the search bar, enter the name of your index, for example, <splunk_log_index_name> and press Enter.

   The search results populate on the Splunk UI. Logs are forwarded in JSON format. A response might look similar to the following:

   {
     "log_data": {
       "kind": "authentication",
       "account": "quayuser123",
       "performer": "John Doe",
       "repository": "projectQuay",
       "ip": "192.168.1.100",
       "metadata_json": {...},
       "datetime": "2024-02-06T12:30:45Z"
     }
   }

   Copy code to clipboard

   where:

   kind

   Specifies the type of log event. In this example, authentication indicates that the log entry relates to an authentication event.

   account

   Specifies the user account involved in the event.

   performer

   Specifies the individual who performed the action.

   repository

   Specifies the repository associated with the event.

   ip

   Specifies the IP address from which the action was performed.

   metadata_json

   Specifies additional metadata related to the event, when present.

   datetime

   Specifies the timestamp of when the event occurred.

Procedure

1. Update your config.yaml file:

   1. To display Splunk SDK audit logs on the Red Hat Quay UI, use the following reference:

      LOGS_MODEL: splunk
      LOGS_MODEL_CONFIG:
        producer: splunk
        splunk_config:
          host: <splunk.example.com>
          port: 8089
          bearer_token: <your_bearer_token>
          url_scheme: https
          verify_ssl: false
          index_prefix: quay_logs
          search_timeout: 60
          max_results: 10000
          export_batch_size: 5000

      Copy code to clipboard

      where:

      LOGS_MODEL_CONFIG.splunk_config.host

      Specifies the host name of your Splunk instance.

      LOGS_MODEL_CONFIG.splunk_config.bearer_token

      Specifies the bearer token you generated for Splunk.

      LOGS_MODEL_CONFIG.splunk_config.index_prefix

      Specifies the Splunk index prefix.

   2. To display Splunk HEC logs on the Red Hat Quay UI, include the generated search_token and hec_token. For example:

      LOGS_MODEL: splunk
      LOGS_MODEL_CONFIG:
        producer: splunk_hec
        splunk_hec_config:
          host: <splunk.example.com>
          port: 8088
          hec_token: <your_hec_token>
          search_token: <your_bearer_token>
          url_scheme: https
          verify_ssl: true
          ssl_ca_path: conf/stack/ca.pem
          index: quay_logs
          splunk_host: <quay-server.example.com>
          splunk_sourcetype: access_combined
          timeout: 10
          search_host: <splunk.example.com>
          search_port: 8089
          search_timeout: 60
          max_results: 10000
          export_batch_size: 5000

      Copy code to clipboard

      where:

      LOGS_MODEL_CONFIG.splunk_hec_config.host

      Specifies the host name of your Splunk instance (used for both the HEC endpoint and the search API).

      LOGS_MODEL_CONFIG.splunk_hec_config.port

      Specifies the port number for the Splunk HEC endpoint.

      LOGS_MODEL_CONFIG.splunk_hec_config.hec_token

      Specifies the HEC token you generated for Splunk.

      LOGS_MODEL_CONFIG.splunk_hec_config.search_token

      Specifies the bearer token you generated for Splunk search. This field is optional.

      LOGS_MODEL_CONFIG.splunk_hec_config.splunk_host

      Specifies the host name of your Red Hat Quay instance.

2. Restart your Red Hat Quay instance to apply the changes.

3. Push an example image to your Red Hat Quay instance to generate an audit log by entering the following command. Note that you can push to an organization or a repository.

   $ podman push <quay-server.example.com>/<organization_name>/busybox:test

   Copy code to clipboard

4. On the Red Hat Quay UI, open the Logs view in one of these places:

   • Organizations → <organization_name> → Logs
   • Repositories → <organization_name> / <repository_name> → Logs
   • Superuser → Usage Logs

Procedure

1. Enter the following command to log in to your Red Hat Quay PostgreSQL database:

   $ psql -h <quay-server.example.com> -p 5432 -U <user_name> -d <database_name>

   Copy code to clipboard

   Example output

   psql (16.1, server 13.7)
   Type "help" for help.

   Copy code to clipboard

2. Optional. Enter the following command to display the tables list of your PostgreSQL database:

   quay=> \dt

   Copy code to clipboard

   Example output

                      List of relations
    Schema |            Name            | Type  |  Owner
   --------+----------------------------+-------+----------
    public | logentry                   | table | quayuser
    public | logentry2                  | table | quayuser
    public | logentry3                  | table | quayuser
    public | logentrykind               | table | quayuser
   ...

   Copy code to clipboard

3. Enter the following command to return a list of repository_ids that are required to return log information:

   quay=> SELECT id, name FROM repository;

   Copy code to clipboard

   Example output

    id |        name
   ----+---------------------
     3 | new_repository_name
     6 | api-repo
     7 | busybox
   ...

   Copy code to clipboard

4. Enter the following command to use the logentry3 relation to show log information about one of your repositories:

   SELECT * FROM logentry3 WHERE repository_id = <repository_id>;

   Copy code to clipboard

   Example output

    id | kind_id | account_id | performer_id | repository_id | datetime | ip |    metadata_json
   
    59 | 14 | 2 | 1 | 6 | 2024-05-13 15:51:01.897189 | 192.168.1.130 | {"repo": "api-repo", "namespace": "test-org"}

   Copy code to clipboard

   In this example, the following information is returned:

   {
     "log_data": {
       "id": 59
       "kind_id": "14",
       "account_id": "2",
       "performer_id": "1",
       "repository_id": "6",
       "ip": "192.168.1.100",
       "metadata_json": {"repo": "api-repo", "namespace": "test-org"}
       "datetime": "2024-05-13 15:51:01.897189"
     }
   }

   Copy code to clipboard

   where:

   id

   Specifies the unique identifier for the log entry.

   kind_id

   Specifies the action that was performed. In this example, 14 maps to creating a repository (create_repo).

   account_id

   Specifies the account that performed the action.

   performer_id

   Specifies the performer of the action.

   repository_id

   Specifies the repository that the action was performed on. In this example, 6 correlates to the api-repo repository from the previous step.

   ip

   Specifies the IP address where the action was performed.

   metadata_json

   Specifies metadata information, including the name of the repository and its namespace.

   datetime

   Specifies the time when the action was performed.

Procedure

1. Navigate to the Red Hat Single Sign-On Admin Console.

   1. On the OpenShift Container Platform Web Console, navigate to Network → Route.
   2. Select the Red Hat Single Sign-On project from the drop-down list.
   3. Find the Red Hat Single Sign-On Admin Console in the Routes table.
2. Select the Realm that you use to configure Red Hat Quay.
3. Click Clients under the Configure section of the navigation panel, and then click Create to add a new OIDC client for Red Hat Quay.

4. Enter the following information:

   • Client ID: quay-enterprise
   • Client Protocol: openid-connect
   • Root URL: https://<quay_endpoint>/
5. Click Save. This results in a redirect to the Clients setting panel.
6. Navigate to Access Type and select Confidential.

7. Navigate to Valid Redirect URIs. You must provide three redirect URIs. The value should be the fully qualified domain name of the Red Hat Quay registry appended with /oauth2/redhatsso/callback. For example:

   • https://<quay_endpoint>/oauth2/redhatsso/callback
   • https://<quay_endpoint>/oauth2/redhatsso/callback/attach
   • https://<quay_endpoint>/oauth2/redhatsso/callback/cli
8. Click Save and navigate to the new Credentials setting.
9. Copy the value of the Secret.

Procedure

1. Edit your Red Hat Quay config.yaml file by navigating to Operators → Installed Operators → Red Hat Quay → Quay Registry → Config Bundle Secret. Then, click Actions → Edit Secret. Alternatively, you can update the config.yaml file locally.

2. Add the following information to your Red Hat Quay on OpenShift Container Platform config.yaml file:

   # ...
   RHSSO_LOGIN_CONFIG:
     CLIENT_ID: <client_id>
     CLIENT_SECRET: <client_secret>
     OIDC_SERVER: <oidc_server_url>
     SERVICE_NAME: <service_name>
     SERVICE_ICON: <service_icon>
     VERIFIED_EMAIL_CLAIM_NAME: <example_email_address>
     PREFERRED_USERNAME_CLAIM_NAME: <preferred_username>
     LOGIN_SCOPES: [ 'openid', 'roles' ]
     USE_PKCE: true
     PKCE_METHOD: "S256"
   # ...

   Copy code to clipboard

   where:

   RHSSO_LOGIN_CONFIG

   Specifies the parent key that holds the OIDC configuration settings. In this example, the parent key used is RHSSO_LOGIN_CONFIG. The string can be replaced with any arbitrary string based on your specific needs, for example ABC123. However, the strings GOOGLE and GITHUB are not accepted. These strings are reserved for their respective identity platforms and require a specific config.yaml entry contingent upon which platform you are using.

   CLIENT_ID

   Specifies the client ID of the application that is being registered with the identity provider. For example, quay.

   CLIENT_SECRET

   Specifies the client secret.

   OIDC_SERVER

   Specifies the fully qualified domain name (FQDN) of the Red Hat Single Sign-On instance, appended with /auth/realms/ and the Realm name. You must include the forward slash at the end, for example, https://sso-redhat.example.com/auth/realms/<your_realm_name>/.

   SERVICE_NAME

   Specifies the name that is displayed on the Red Hat Quay login page, for example, Red Hat Single Sign-On.

   SERVICE_ICON

   Specifies the icon on the login screen. For example, /static/img/RedHat.svg.

   VERIFIED_EMAIL_CLAIM_NAME

   Specifies the name of the claim that is used to verify the email address of the user.

   PREFERRED_USERNAME_CLAIM_NAME

   Specifies the name of the claim that is used for the preferred username of the user.

   LOGIN_SCOPES

   Specifies the scopes to send to the OIDC provider when performing the login flow, for example, openid.

   USE_PKCE

   Specifies whether to enable Proof Key for Code Exchange (PKCE). Defaults to false.

   PKCE_METHOD

   Specifies the code challenge method used to generate the code_challenge sent in the initial authorization request. Defaults to S256.

3. Restart your Red Hat Quay on OpenShift Container Platform deployment with Red Hat Single Sign-On enabled.

Procedure

1. Update your config.yaml file with the following information:

   AUTHENTICATION_TYPE: OIDC
   # ...
   OIDC_LOGIN_CONFIG:
     CLIENT_ID:
     CLIENT_SECRET:
     OIDC_SERVER:
     SERVICE_NAME:
     PREFERRED_GROUP_CLAIM_NAME:
     LOGIN_SCOPES: [ 'openid', '<example_scope>' ]
     OIDC_DISABLE_USER_ENDPOINT: false
   # ...
   FEATURE_TEAM_SYNCING: true
   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP: true
   FEATURE_UI_V2: true
   # ...

   Copy code to clipboard

   where:

   CLIENT_ID

   Specifies the registered OIDC client ID for this Red Hat Quay instance. This field is required.

   CLIENT_SECRET

   Specifies the registered OIDC client secret for this Red Hat Quay instance. This field is required.

   OIDC_SERVER

   Specifies the address of the OIDC server that is being used for authentication. This URL should be such that a GET request to <OIDC_SERVER>/.well-known/openid-configuration returns the provider’s configuration information. This field is required.

   SERVICE_NAME

   Specifies the name of the service that is being authenticated. This field is required.

   PREFERRED_GROUP_CLAIM_NAME

   Specifies the key name within the OIDC token payload that holds information about the user’s group memberships. This field allows the authentication system to extract group membership information from the OIDC token so that it can be used with Red Hat Quay. This field is required.

   LOGIN_SCOPES

   Specifies the scopes Red Hat Quay requests during login. Must include 'openid'. Each scope must also be listed in the identity provider’s scopes_supported from /.well-known/openid-configuration. This field is required.

   OIDC_DISABLE_USER_ENDPOINT

   Specifies whether to allow or disable the /userinfo endpoint. If using Azure Entra ID, set this field to true. Defaults to false.

   FEATURE_TEAM_SYNCING

   Specifies whether to allow team membership to be synced from a backing group in the authentication engine. This field is required.

   FEATURE_NONSUPERUSER_TEAM_SYNCING_SETUP

   Optional. If enabled, non-superusers can configure team synchronization.

2. Restart your Red Hat Quay registry.

Procedure

1. Log in to your Red Hat Quay registry by using your OIDC provider.
2. On the Red Hat Quay v2 UI dashboard, click Create Organization.
3. Enter an organization name, for example, test-org.
4. Click the name of the organization.
5. In the navigation pane, click Teams and membership.
6. Click Create new team and enter a name, for example, testteam.

7. On the Create team pop-up:

   1. Optional. Add this team to a repository.
   2. Add a team member, for example, user1, by typing in the user’s account name.
   3. Add a robot account to this team. This page provides the option to create a robot account.
8. Click Next.
9. On the Review and Finish page, review the information that you have provided and click Review and Finish.
10. To enable team synchronization for your Red Hat Quay OIDC deployment, click Enable Directory Sync on the Teams and membership page.

11. You are prompted to enter the group Object ID if your OIDC authenticator is Azure Entra ID, or the group name if using a different provider.

    Warning

    After you enable team syncing, membership of users who are already part of the team is revoked. The OIDC group is the single source of truth. This action is not reversible. Team user membership from within Red Hat Quay is read-only.

12. Click Enable Sync.

13. You are returned to the Teams and membership page. Note that users of this team are removed and are re-added upon logging back in. At this stage, only the robot account is still part of the team.

    A banner at the top of the page confirms that the team is synced:

    This team is synchronized with a group in OIDC and its user membership is therefore read-only.

    Copy code to clipboard

    By clicking the Directory Synchronization Config accordion, the OIDC group that your deployment syncs with appears.

14. Log out of your Red Hat Quay registry.

Verification

1. Log back in to your Red Hat Quay registry.
2. Click Organizations → test-org → test-team → Teams and memberships. user1 now appears as a team member for this team.
3. Navigate to your OIDC provider’s administration console.
4. Navigate to the Users page of your OIDC provider. The name of this page varies depending on your provider.
5. Click the name of the user associated with Red Hat Quay, for example, user1.
6. Remove the user from the group in the configured identity provider.
7. Remove, or unassign, the access permissions from the user.
8. Log in to your Red Hat Quay registry.
9. Click Organizations → test-org → test-team → Teams and memberships. user1 has been removed from this team.

Procedure

1. To set a quota for an organization, you can use the POST /api/v1/organization/{orgname}/quota endpoint:

   $ curl -X POST "https://<quay-server.example.com>/api/v1/organization/<orgname>/quota" \
        -H "Authorization: Bearer <access_token>" \
        -H "Content-Type: application/json" \
        -d '{
            "limit_bytes": 10737418240,
            "limits": "10 Gi"
        }'

   Copy code to clipboard

   Example output:

   "Created"

   Copy code to clipboard

2. Use the GET /api/v1/organization/{orgname}/quota command to see if your organization already has an established quota:

   $ curl -k -X GET -H "Authorization: Bearer <token>" -H 'Content-Type: application/json'  https://<quay-server.example.com>/api/v1/organization/<organization_name>/quota  | jq

   Copy code to clipboard

   Example output:

   [{"id": 1, "limit_bytes": 10737418240, "limit": "10.0 GiB", "default_config": false, "limits": [], "default_config_exists": false}]

   Copy code to clipboard

3. You can use the PUT /api/v1/organization/{orgname}/quota/{quota_id} command to modify the existing quota limitation. For example:

   $ curl -X PUT "https://<quay-server.example.com>/api/v1/organization/<orgname>/quota/<quota_id>" \
        -H "Authorization: Bearer <access_token>" \
        -H "Content-Type: application/json" \
        -d '{
            "limit_bytes": <limit_in_bytes>
        }'

   Copy code to clipboard

   Example output:

   {"id": 1, "limit_bytes": 21474836480, "limit": "20.0 GiB", "default_config": false, "limits": [], "default_config_exists": false}

   Copy code to clipboard

Procedure

1. To set a reject limit, send a POST request to the /api/v1/organization/<organization_name>/quota/<quota_id>/limit endpoint. For example:

   $ curl -k -X POST -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' -d '{"type":"Reject","threshold_percent":80}'  https://<quay-server.example.com>/api/v1/organization/<organization_name>/quota/1/limit

   Copy code to clipboard

   • To set a warning limit, send a POST request to the same endpoint. For example:

     $ curl -k -X POST -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' -d '{"type":"Warning","threshold_percent":50}'  https://<quay-server.example.com>/api/v1/organization/<organization_name>/quota/1/limit

     Copy code to clipboard

Procedure

1. Ensure that the PERMANENTLY_DELETE_TAGS and RESET_CHILD_MANIFEST_EXPIRATION parameters are set to true in your config.yaml file. For example:

   PERMANENTLY_DELETE_TAGS: true
   RESET_CHILD_MANIFEST_EXPIRATION: true

   Copy code to clipboard
2. In the navigation pane, click Repositories.
3. Click the name of the repository, for example, quayadmin/busybox.
4. Select the check box of the image tag that you want to delete, for example, test.

5. Click Actions → Permanently Delete.

   Important

   This action is permanent and cannot be undone.

Procedure

1. In your Red Hat Quay config.yaml file, set FEATURE_IMAGE_PULL_STATS: true. For example:

   # ...
   FEATURE_IMAGE_PULL_STATS: true
   REDIS_FLUSH_INTERVAL_SECONDS: 30
   PULL_METRICS_REDIS:
       host: <redis_host>
       password: <redis_password>
       port: 6379
       db: 1
   # ...

   Copy code to clipboard

   where:

   FEATURE_IMAGE_PULL_STATS

   Specifies whether image pull tracking activity is enabled.

   REDIS_FLUSH_INTERVAL_SECONDS

   Specifies the time, in seconds, at which the Redis flush worker clears old data. Shorter intervals keep data fresher and help prevent Redis from bloating, while longer intervals reduce flush frequency.

   PULL_METRICS_REDIS

   Specifies the connection settings for the Redis database used to store image pull metrics.

2. Restart your Red Hat Quay deployment.

Verification

1. Push an image to your registry by entering the following command. Following this step, you can use your browser to see the tagged image in your repository.

   $ podman push <quay-server.example.com>/<organization>/<image>:<tag>

   Copy code to clipboard

2. Pull the image from your Red Hat Quay registry by entering the following command:

   $ podman pull <quay-server.example.com>/<organization>/<image>:<tag>

   Copy code to clipboard
3. On the Red Hat Quay UI, navigate to Repositories, and then click the name of your repository.

4. Click Tags. The Last Pulled and Pull Count categories show you information about when the image was last pulled, and how many times it has been pulled, respectively. For example:

   

Procedure

1. In your Red Hat Quay config.yaml file, add and set the FEATURE_AUTO_PRUNE environment variable to true. For example:

   # ...
   FEATURE_AUTO_PRUNE: true
   # ...

   Copy code to clipboard

Procedure

1. Update your config.yaml file to add the DEFAULT_NAMESPACE_AUTOPRUNE_POLICY configuration field:

   1. To set the policy method to remove the oldest tags by their creation date until the number of tags provided is left, use the number_of_tags method:

      # ...
      DEFAULT_NAMESPACE_AUTOPRUNE_POLICY:
        method: number_of_tags
        value: 2
      # ...

      Copy code to clipboard

      where: value:: Specifies the number of tags to keep. In this example, two tags remain.

   2. To set the policy method to remove tags with a creation date older than the provided time span, for example, 5d, use the creation_date method:

      DEFAULT_NAMESPACE_AUTOPRUNE_POLICY:
        method: creation_date
        value: 5d

      Copy code to clipboard
2. Restart your Red Hat Quay deployment.

3. Optional. If you need to tag and push images to test this feature:

   1. Tag four sample images that you push to a Red Hat Quay registry. For example:

      $ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test

      Copy code to clipboard

      $ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test2

      Copy code to clipboard

      $ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test3

      Copy code to clipboard

      $ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test4

      Copy code to clipboard

   2. Push the four sample images to the registry with auto-pruning enabled by entering the following commands:

      $ podman push <quay-server.example.com>/quayadmin/busybox:test

      Copy code to clipboard

      $ podman push <quay-server.example.com>/<quayadmin>/busybox:test2

      Copy code to clipboard

      $ podman push <quay-server.example.com>/<quayadmin>/busybox:test3

      Copy code to clipboard

      $ podman push <quay-server.example.com>/<quayadmin>/busybox:test4

      Copy code to clipboard
4. Check that the registry that you pushed the images to shows four tags.
5. By default, the auto-pruner worker at the registry level runs every 24 hours. After 24 hours, the two oldest image tags are removed, leaving the test3 and test4 tags if you followed these instructions. Check your Red Hat Quay organization to ensure that the two oldest tags were removed.

Procedure

1. On the Red Hat Quay v2 UI, click Organizations in the navigation pane.
2. Select the name of an organization to which you apply the auto-pruning feature, for example, test_organization.
3. Click Settings.

4. Click Auto-Prune Policies. For example:

   

5. Click the drop-down menu and select the desired policy, for example, By number of tags.
6. Select the desired number of tags to keep. By default, this is set at 20 tags. For this example, the number of tags to keep is set at 3.

7. Optional. With the introduction of regular expressions, you are provided the following options to fine-tune your auto-pruning policy:

   • Match: When selecting this option, the auto-pruner prunes all tags that match the given regex pattern.

   • Does not match: When selecting this option, the auto-pruner prunes all tags that do not match the regex pattern.

     If you do not select an option, the auto-pruner defaults to pruning all image tags.

     For this example, click the Tag pattern box and select match. In the regex box, enter a pattern to match tags against. For example, to automatically prune all test tags, enter ^test.*.

8. Optional. You can create a second auto-prune policy by clicking Add Policy and entering the required information.

9. Click Save. A notification that your auto-prune policy has been updated appears.

   With this example, the organization is configured to keep the three latest tags that are named ^test.*.

Procedure

1. Enter the following POST /api/v1/organization/<organization_name>/autoprunepolicy/ command to create a new policy that limits the number of tags allowed in an organization:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{"method": "number_of_tags", "value": 10}' http://<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/

   Copy code to clipboard

   Alternatively, you can set tags to expire for a specified time after their creation date:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{
   "method": "creation_date", "value": "7d"}' http://<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/

   Copy code to clipboard

   Example output:

   {"uuid": "73d64f05-d587-42d9-af6d-e726a4a80d6e"}

   Copy code to clipboard

2. Optional. You can add an additional policy to an organization and pass in the tagPattern and tagPatternMatches fields to prune only tags that match the given regex pattern. For example:

   $ curl -X POST \
     -H "Authorization: Bearer <bearer_token>" \
     -H "Content-Type: application/json" \
     -d '{
       "method": "creation_date",
       "value": "7d",
       "tagPattern": "^v*",
       "tagPatternMatches": <true>
     }' \
     "https://<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/"

   Copy code to clipboard

   where:

   tagPatternMatches

   Specifies whether tags that match the regex pattern are pruned. Set to true to prune matching tags. In this example, tags that match ^v* are pruned.

   Example output:

   {"uuid": "ebf7448b-93c3-4f14-bf2f-25aa6857c7b0"}

   Copy code to clipboard

3. You can update your organization’s auto-prune policy by using the PUT /api/v1/organization/<organization_name>/autoprunepolicy/<policy_uuid> command. For example:

   $ curl -X PUT   -H "Authorization: Bearer <bearer_token>"   -H "Content-Type: application/json"   -d '{
       "method": "creation_date",
       "value": "4d",
       "tagPattern": "^v*",
       "tagPatternMatches": true
     }'   "<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/<uuid>"

   Copy code to clipboard

   This command does not return output. Continue to the next step.

4. Check your auto-prune policy by entering the following command:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/

   Copy code to clipboard

   Example output:

   {"policies": [{"uuid": "ebf7448b-93c3-4f14-bf2f-25aa6857c7b0", "method": "creation_date", "value": "4d", "tagPattern": "^v*", "tagPatternMatches": true}, {"uuid": "da4d0ad7-3c2d-4be8-af63-9c51f9a501bc", "method": "number_of_tags", "value": 10, "tagPattern": null, "tagPatternMatches": true}, {"uuid": "17b9fd96-1537-4462-a830-7f53b43f94c2", "method": "creation_date", "value": "7d", "tagPattern": "^v*", "tagPatternMatches": true}]}

   Copy code to clipboard

5. You can delete the auto-prune policy for your organization by entering the following command. Note that deleting the policy requires the UUID.

   $ curl -X DELETE -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/organization/<organization_name>/autoprunepolicy/73d64f05-d587-42d9-af6d-e726a4a80d6e

   Copy code to clipboard

Procedure

1. Enter the following POST command to create a new policy that limits the number of tags for the current user:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{"method": "number_of_tags", "value": 10}' http://<quay-server.example.com>/api/v1/user/autoprunepolicy/

   Copy code to clipboard

   Example output

   {"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859"}

   Copy code to clipboard

2. Check your auto-prune policy by entering the following command:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/

   Copy code to clipboard

   Alternatively, you can include the UUID:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/8c03f995-ca6f-4928-b98d-d75ed8c14859

   Copy code to clipboard

   Example output

   {"policies": [{"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859", "method": "number_of_tags", "value": 10}]}

   Copy code to clipboard

3. You can delete the auto-prune policy by entering the following command. Note that deleting the policy requires the UUID.

   $ curl -X DELETE -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/8c03f995-ca6f-4928-b98d-d75ed8c14859

   Copy code to clipboard

   Example output

   {"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859"}

   Copy code to clipboard

Procedure

1. On the Red Hat Quay v2 UI, click Repository in the navigation pane.
2. Select the name of a repository to which you apply the auto-pruning feature, for example, <organization_name>/<repository_name>.
3. Click Settings.
4. Click Repository Auto-Prune Policies.
5. Click the drop-down menu and select the desired policy, for example, By age of tags.
6. Set a time, for example, 5 and an interval, for example minutes to delete tags older than the specified time frame. For this example, tags older than 5 minutes are marked for deletion.

7. Optional. With the introduction of regular expressions, you are provided the following options to fine-tune your auto-pruning policy:

   • Match: When selecting this option, the auto-pruner prunes all tags that match the given regex pattern.

   • Does not match: When selecting this option, the auto-pruner prunes all tags that do not match the regex pattern.

     If you do not select an option, the auto-pruner defaults to pruning all image tags.

     For this example, click the Tag pattern box and select Does not match. In the regex box, enter a pattern to match tags against. For example, to automatically prune all tags that do not match the test tag, enter ^test.*.

8. Optional. You can create a second auto-prune policy by clicking Add Policy and entering the required information.
9. Click Save. A notification that your auto-prune policy has been updated appears.

Procedure

1. Enter the following POST /api/v1/repository/<repository>/autoprunepolicy/ command to create a new policy that limits the number of tags allowed in a repository:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{"method": "number_of_tags","value": 2}' http://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/

   Copy code to clipboard

   Alternatively, you can set tags to expire for a specified time after their creation date:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{"method": "creation_date", "value": "7d"}' http://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/

   Copy code to clipboard

   Example output

   {"uuid": "ce2bdcc0-ced2-4a1a-ac36-78a9c1bed8c7"}

   Copy code to clipboard

2. Optional. You can add an additional policy and pass in the tagPattern and tagPatternMatches fields to prune only tags that match the given regex pattern. For example:

   $ curl -X POST \
     -H "Authorization: Bearer <access_token>" \
     -H "Content-Type: application/json" \
     -d '{
       "method": "creation_date",
       "value": "7d",
       "tagPattern": "^test.",
       "tagPatternMatches": false
     }' \
     "https://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/"

   Copy code to clipboard

   Where:

   tagPatternMatches

   Specifies that the false value prunes tags that do not match the given regex pattern. In this example, all tags except those that match ^test. are pruned.

3. You can update your policy for the repository by using the PUT /api/v1/repository/<repository>/autoprunepolicy/<policy_uuid> command and passing in the UUID. For example:

   $ curl -X PUT \
     -H "Authorization: Bearer <bearer_token>" \
     -H "Content-Type: application/json" \
     -d '{
       "method": "number_of_tags",
       "value": "5",
       "tagPattern": "^test.*",
       "tagPatternMatches": true
     }' \
     "https://quay-server.example.com/api/v1/repository/<namespace>/<repo_name>/autoprunepolicy/<uuid>"

   Copy code to clipboard

   This command does not return output. Continue to the next step to check your auto-prune policy.

4. Check your auto-prune policy by entering the following command:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/

   Copy code to clipboard

   Alternatively, you can include the UUID:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/ce2bdcc0-ced2-4a1a-ac36-78a9c1bed8c7

   Copy code to clipboard

   Example output

   {"policies": [{"uuid": "ce2bdcc0-ced2-4a1a-ac36-78a9c1bed8c7", "method": "number_of_tags", "value": 10}]}

   Copy code to clipboard

5. You can delete the auto-prune policy by entering the following command. Note that deleting the policy requires the UUID.

   $ curl -X DELETE -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<organization_name>/<repository_name>/autoprunepolicy/ce2bdcc0-ced2-4a1a-ac36-78a9c1bed8c7

   Copy code to clipboard

   Example output

   {"uuid": "ce2bdcc0-ced2-4a1a-ac36-78a9c1bed8c7"}

   Copy code to clipboard

Procedure

1. Enter the following POST /api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/ command to create a new policy that limits the number of tags for the user:

   $ curl -X POST -H "Authorization: Bearer <access_token>" -H "Content-Type: application/json" -d '{"method": "number_of_tags","value": 2}' https://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/

   Copy code to clipboard

   Example output

   {"uuid": "7726f79c-cbc7-490e-98dd-becdc6fefce7"}

   Copy code to clipboard

2. Optional. You can add an additional policy for the current user and pass in the tagPattern and tagPatternMatches fields to prune only tags that match the given regex pattern. For example:

   $ curl -X POST \
     -H "Authorization: Bearer <bearer_token>" \
     -H "Content-Type: application/json" \
     -d '{
       "method": "creation_date",
       "value": "7d",
       "tagPattern": "^v*",
       "tagPatternMatches": true
     }' \
     "http://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/"

   Copy code to clipboard

   Example output

   {"uuid": "b3797bcd-de72-4b71-9b1e-726dabc971be"}

   Copy code to clipboard

3. You can update your policy for the current user by using the PUT /api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/<policy_uuid> command. For example:

   $ curl -X PUT   -H "Authorization: Bearer <bearer_token>"   -H "Content-Type: application/json"   -d '{
       "method": "creation_date",
       "value": "4d",
       "tagPattern": "^test.",
       "tagPatternMatches": true
     }'   "https://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/<policy_uuid>"

   Copy code to clipboard

   Updating a policy does not return output in the CLI.

4. Check your auto-prune policy by entering the following command:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/

   Copy code to clipboard

   Alternatively, you can include the UUID:

   $ curl -X GET -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/7726f79c-cbc7-490e-98dd-becdc6fefce7

   Copy code to clipboard

   Example output

   {"uuid": "81ee77ec-496a-4a0a-9241-eca49437d15b", "method": "creation_date", "value": "7d", "tagPattern": "^v*", "tagPatternMatches": true}

   Copy code to clipboard

5. You can delete the auto-prune policy by entering the following command. Note that deleting the policy requires the UUID.

   $ curl -X DELETE -H "Authorization: Bearer <access_token>" http://<quay-server.example.com>/api/v1/repository/<user_account>/<user_repository>/autoprunepolicy/<policy_uuid>

   Copy code to clipboard

   Example output

   {"uuid": "7726f79c-cbc7-490e-98dd-becdc6fefce7"}

   Copy code to clipboard

Procedure

1. Update your config.yaml file to include the storage engines to which data is replicated. You must list all storage engines to be used:

   # ...
   FEATURE_STORAGE_REPLICATION: true
   # ...
   DISTRIBUTED_STORAGE_CONFIG:
       usstorage:
           - RHOCSStorage
           - access_key: <access_key>
             bucket_name: <example_bucket>
             hostname: my.noobaa.hostname
             is_secure: false
             port: "443"
             secret_key: <secret_key>
             storage_path: /datastorage/registry
       eustorage:
           - S3Storage
           - host: s3.amazon.com
             port: "443"
             s3_access_key: <access_key>
             s3_bucket: <example bucket>
             s3_secret_key: <secret_key>
             storage_path: /datastorage/registry
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
   DISTRIBUTED_STORAGE_PREFERENCE:
       - usstorage
       - eustorage
   # ...

   Copy code to clipboard

2. Optional. If complete replication of all images to all storage engines is required, you can replicate images to the storage engine by manually setting the DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS field. This ensures that all images are replicated to that storage engine. For example:

   # ...
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
       - usstorage
       - eustorage
   # ...

   Copy code to clipboard
   Note

   To enable per-namespace replication, contact Red Hat Quay support.

3. After adding storage and enabling Replicate to storage engine by default for geo-replication, you must sync existing image data across all storage. To do this, you must execute into the container by running the following command:

   $ podman exec -it <container_id>

   Copy code to clipboard

4. To sync the content after adding new storage, enter the following commands:

   # scl enable python27 bash

   Copy code to clipboard

   # python -m util.backfillreplication

   Copy code to clipboard
   Note

   This is a one time operation to sync content after adding new storage.

Procedure

1. Copy the config.yaml file to all machines running Red Hat Quay.

2. For each machine in each region, add a QUAY_DISTRIBUTED_STORAGE_PREFERENCE environment variable with the preferred storage engine for the region in which the machine is running.

   For example, for a machine running in Europe with the config directory on the host available from $QUAY/config:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443  \
      --name=quay \
      -v $QUAY/config:/conf/stack:Z \
      -e QUAY_DISTRIBUTED_STORAGE_PREFERENCE=europestorage \
      registry.redhat.io/quay/quay-rhel9:v3.18.0

   Copy code to clipboard
   Note

   The value of the environment variable specified must match the name of a Location ID as defined in the config panel.

3. Restart all Red Hat Quay containers.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Copy code to clipboard
   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites. Complete this step before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Enter the following command to obtain a list of running containers:

   $ podman ps

   Copy code to clipboard

   Example output:

   CONTAINER ID  IMAGE                                                                     COMMAND         CREATED         STATUS             PORTS                                        NAMES
   92c5321cde38  registry.redhat.io/rhel8/redis-5:1                                        run-redis       11 days ago     Up 11 days ago     0.0.0.0:6379->6379/tcp                       redis
   4e6d1ecd3811  registry.redhat.io/rhel8/postgresql-13:1-109                              run-postgresql  33 seconds ago  Up 34 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay
   d2eadac74fda  registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8:v3.9.0-131  registry        4 seconds ago   Up 4 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay

   Copy code to clipboard

4. Enter the following command to execute a shell inside of the PostgreSQL container:

   $ podman exec -it postgresql-quay -- /bin/bash

   Copy code to clipboard

5. Enter psql by running the following command:

   bash-4.4$ psql

   Copy code to clipboard

6. Enter the following command to reveal a list of sites in your geo-replicated deployment:

   quay=# select * from imagestoragelocation;

   Copy code to clipboard

   Example output:

    id |       name
   ----+-------------------
     1 | usstorage
     2 | eustorage

   Copy code to clipboard

7. Enter the following command to exit the postgres CLI to re-enter bash-4.4:

   \q

   Copy code to clipboard

8. Enter the following command to permanently remove the eustorage site:

   Important

   The following action cannot be undone. Use with caution.

   bash-4.4$ python -m util.removelocation eustorage

   Copy code to clipboard

   Example output:

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

   Copy code to clipboard

Procedure

1. Deploy a PostgreSQL instance for Red Hat Quay.

2. Log in to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

   Copy code to clipboard

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

   Copy code to clipboard

4. Enable the pg_trgm extension inside the database:

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Copy code to clipboard

5. Deploy a Redis instance:

   Note

   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis.
   2. Verify that the Redis instance is accessible from the clusters where Red Hat Quay is running.
   3. Port 6379/TCP must be open.

   4. Run Redis inside the instance:

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

      Copy code to clipboard
6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket is close to the first, or primary, cluster, and the other runs closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Copy code to clipboard
   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If blobs need to be replicated, the script returns UUIDs of blobs that are replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that no more blobs remain to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Identify your Red Hat Quay application pods by entering the following command:

   $ oc get pod -n <quay_namespace>

   Copy code to clipboard

   Example output:

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

   Copy code to clipboard

4. Open an interactive shell session in the usstorage pod by entering the following command:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

   Copy code to clipboard

5. Permanently remove the eustorage site by entering the following command:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Copy code to clipboard

   Example output:

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

   Copy code to clipboard

Procedure

1. If your Red Hat Quay registry is readily available, you can generate service keys inside of the Quay registry container.

   1. Enter the following command to generate a key pair inside of the Quay container:

      $ podman exec quay python3 tools/generatekeypair.py quay-readonly

      Copy code to clipboard

2. If your Red Hat Quay is not readily available, you must generate your service keys inside of a virtual environment.

   1. Change into the directory of your Red Hat Quay deployment and create a virtual environment inside of that directory:

      $ cd <$QUAY>/quay && virtualenv -v venv

      Copy code to clipboard

   2. Activate the virtual environment by entering the following command:

      $ source venv/bin/activate

      Copy code to clipboard

   3. Optional. Install the pip CLI tool if you do not have it installed:

      $ venv/bin/pip install --upgrade pip

      Copy code to clipboard

   4. In your Red Hat Quay directory, create a requirements-generatekeys.txt file with the following content:

      $ cat << EOF > requirements-generatekeys.txt
      cryptography==3.4.7
      pycparser==2.19
      pycryptodome==3.9.4
      pycryptodomex==3.9.4
      pyjwkest==1.4.2
      PyJWT==1.7.1
      Authlib==1.0.0a2
      EOF

      Copy code to clipboard

   5. Enter the following command to install the Python dependencies defined in the requirements-generatekeys.txt file:

      $ venv/bin/pip install -r requirements-generatekeys.txt

      Copy code to clipboard

   6. Enter the following command to create the necessary service keys:

      $ PYTHONPATH=. venv/bin/python /<path_to_cloned_repo>/tools/generatekeypair.py quay-readonly

      Copy code to clipboard

      Example output:

      Writing public key to quay-readonly.jwk
      Writing key ID to quay-readonly.kid
      Writing private key to quay-readonly.pem

      Copy code to clipboard

   7. Enter the following command to deactivate the virtual environment:

      $ deactivate

      Copy code to clipboard

Procedure

1. Enter the following command to enter your Red Hat Quay database environment:

   $ podman exec -it postgresql-quay psql -U postgres -d quay

   Copy code to clipboard

2. Display the approval types and associated notes of the servicekeyapproval by entering the following command:

   quay=# select * from servicekeyapproval;

   Copy code to clipboard

   Example output:

    id | approver_id |          approval_type           |       approved_date        | notes
   ----+-------------+----------------------------------+----------------------------+-------
     1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
     2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
     3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
     4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
     5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
   ...

   Copy code to clipboard

3. Add the service key to your Red Hat Quay database by entering the following query:

   quay=# INSERT INTO servicekey
     (name, service, metadata, kid, jwk, created_date, expiration_date)
     VALUES ('quay-readonly',
              'quay',
              '{}',
              '<contents_of_.kid_file>',
              '<contents_of_.jwk_file>',
              '<created_date_of_read-only>',
              '<expiration_date_of_read-only>');

   Copy code to clipboard

   Example output:

   INSERT 0 1

   Copy code to clipboard

4. Next, add the key approval with the following query:

   quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
     VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
              <include_notes_here_on_why_this_is_being_added>);

   Copy code to clipboard

   Example output:

   INSERT 0 1

   Copy code to clipboard

5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs:

   UPDATE servicekey
   SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
   WHERE name = 'quay-readonly';

   Copy code to clipboard

   UPDATE 1

   Copy code to clipboard

Procedure

1. Shut down all Red Hat Quay instances on all virtual machines. For example:

   $ podman stop <quay_container_name_on_virtual_machine_a>

   Copy code to clipboard

   $ podman stop <quay_container_name_on_virtual_machine_b>

   Copy code to clipboard

2. Enter the following command to copy the contents of the quay-readonly.kid file and the quay-readonly.pem file to the directory that holds your Red Hat Quay configuration bundle:

   $ cp quay-readonly.kid quay-readonly.pem $Quay/config

   Copy code to clipboard

3. Enter the following command to set file permissions on all files in your configuration bundle folder:

   $ setfacl -m user:1001:rw $Quay/config/*

   Copy code to clipboard

4. Modify your Red Hat Quay config.yaml file and add the following information:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy code to clipboard
5. Distribute the new configuration bundle to all Red Hat Quay instances.

6. Start Red Hat Quay by entering the following command:

   $ podman run -d --rm -p 80:8080 -p 443:8443  \
      --name=quay-main-app \
      -v $QUAY/config:/conf/stack:Z \
      -v $QUAY/storage:/datastorage:Z \
      {productrepo}/{quayimage}:{productminv}

   Copy code to clipboard

7. After starting Red Hat Quay, a banner inside your instance informs users that Red Hat Quay is running in read-only mode. Pushes should be rejected and a 405 error should be logged. You can test this by running the following command:

   $ podman push <quay-server.example.com>/quayadmin/busybox:test

   Copy code to clipboard

   Example output:

   613be09ab3c0: Preparing
   denied: System is currently read-only. Pulls will succeed but all write operations are currently suspended.

   Copy code to clipboard

   With your Red Hat Quay deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

8. Optional. After you finish with read-only mode, you can return to normal operations by removing the following information from your config.yaml file. Then, restart your Red Hat Quay deployment:

   # ...
   REGISTRY_STATE: readonly
   INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
   INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
   # ...

   Copy code to clipboard

   $ podman restart <container_id>

   Copy code to clipboard

Procedure

1. Connect to your Red Hat Quay production database by using the methods described earlier.

2. Optional. List service key IDs by running the following query:

   SELECT id, name, expiration_date FROM servicekey;

   Copy code to clipboard

3. Update the key expiration by issuing the following query:

   quay=# UPDATE servicekey SET expiration_date = 'new-date' WHERE id = servicekey_id;

   Copy code to clipboard