Resetting superuser password for Quay through the database

Solution Verified - Updated

Environment

  • Red Hat Quay (all supported versions)
    • internal auth set to Database
  • OpenShift Mirror Registry (all supported versions)

Issue

  • Pulls from the registry start to fail with the Invalid username or password error message and then subsequently Quay starts returning 429 Too many requests on pulls and login attempts.

Resolution

OpenShift/VM deployments of Quay

  1. Generate a new user password for the affected user:

    • OpenShift deployment:
         $ oc get pods -n quay-namespace
     $ oc exec -it quay-pod-name -- python3 -c 'import bcrypt; print(bcrypt.hashpw(b"REAL-PASSWORD-HERE", 
bcrypt.gensalt(12)).decode("utf-8"))'
     $2b$12$d8W84Z50fNWzf4mL6WWmn.kPFj4dfOnQSnsraghETC8V0RIEBk9Lu

    • VM deployment:

      $ podman exec -it quay-container-name python3 -c 'import bcrypt; print(bcrypt.hashpw(b"REAL-PASSWORD-HERE", bcrypt.gensalt(12)).decode("utf-8"))'
$2b$12$d8W84Z50fNWzf4mL6WWmn.kPFj4dfOnQSnsraghETC8V0RIEBk9Lu

  2. Execute a PostgreSQL shell inside Quay's database:

    • For operator managed Quay database:
    $ oc exec -it quay-database-pod-name -- psql
postgres=# \l           # <--- to list databases
postgres=# \c "quay-database-name"
quay-database-name=# UPDATE "user" SET password_hash = 'HASH_FROM_STEP_1' WHERE username = 'your_username';
UPDATE 1
quay-database-name=# UPDATE "user" SET invalid_login_attempts = 0 WHERE username = 'your_username';
UPDATE 1
quay-database-name=# \q
    • VM deployments and non-managed databases:

    a. Install the psql tool which is available in the postgresql package (This content is not included.package description):

     ```
 # yum install postgresql
 ```

    b. Connect to the database by executing:

     ```
 # psql -d quay-database-name -h quay-db-host -p port -U quay-db-name -W 
 Enter password:
 ```

    c. Execute the following queries:

     ```
 quay-database-name=# UPDATE "user" SET password_hash = 'HASH_FROM_STEP_1' WHERE username = 'your_username';
 UPDATE 1
 quay-database-name=# UPDATE "user" SET invalid_login_attempts = 0 WHERE username = 'your_username';
 UPDATE 1
 quay-database-name=# \q
 ```

OpenShift Mirror Registry

  1. Generate a new user password for the affected user:

       $ podman exec -it quay-app python3 -c 'import bcrypt; print(bcrypt.hashpw(b"REAL-PASSWORD-HERE", 
bcrypt.gensalt(12)).decode("utf-8"))'
   $2b$12$d8W84Z50fNWzf4mL6WWmn.kPFj4dfOnQSnsraghETC8V0RIEBk9Lu

  2. OMR stores all Quay information in a SQLite3 database which is usually stored as a Podman volume. You can check the volume name by inspecting the unit file. Assuming that OMR is installed under root user:

    # cat /etc/systemd/system/quay-app.service | grep -i sqlite
      -v sqlite-storage-volume:/sqlite \

  3. Shut down Quay by running:

    # systemctl stop quay-app.service

    The app must be completely shut down and podman ps should not show it as a running container.

  4. Run the ubi9 container on the node with the following command:

    # podman run --rm -it -v sqlite-storage:/sqlite-storage:Z registry.redhat.io/ubi9/ubi:latest

    Install the sqlite utility inside the container (This content is not included.package info), we need it to connect to the SQLite database:

    # yum install -y sqlite

    Connect to the database and execute the following queries:

    # sqlite3 /sqlite-storage/quay_sqlite.db
sqlite> UPDATE user SET password_hash = 'HASH_FROM_STEP_1' WHERE username = 'your_username';
sqlite> UPDATE user SET invalid_login_attempts = 0 WHERE username = 'your_username';
sqlite>.quit
# exit

  5. Restart Quay container by executing:

    # systemctl start quay-app.service

Root Cause

  • Quay stores user information (such as username and e-mail address) in its own database. If internal auth is set to Database, it also stores a bcrypt hash of the user's password in the password_hash field.
  • On each login attempt, whether it's through the UI or through the Docker v2 API by podman/crio, the authentication endpoint checks the password validity against the hash stored in the database.
  • If a wrong password is given too many times, Quay will start throwing a 429 Too many requests to registry HTTP code and will start exponentially increasing the cooldown time for the affected user. This number can grow very large very quickly if the wrong password is used in a pipeline for instance.
  • Super users can change passwords for normal users via the "super user control panel", but this cannot be done for other super users. So a direct change in the database is necessary.
  • Once passwords are reset using this procedure, the password is immediately active.
  • Note that this procedure only works if AUTHENTICATION_TYPE: Database is set in Quay's config.yaml file. If you're using LDAP or OIDC as the internal auth type, passwords cannot be changed using this procedure since user auth management is offloaded to the federated login service. However, the number of failed login attempts can still be reset as that is always stored in Quay's database.
  • We would highly recommend not using real user credentials in any pipelines that talk to the registry. Robot accounts are much better suited for automations, since their tokens never change (unless they are regenerated manually).
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.