When using the Quay Operator, Secrets don't get garbage collected
Environment
- Red Hat Quay Operator
- 3.8+
Issue
- It is observed that whenever a
config changeis made, some secrets are recreated but the older ones remain in the namespace. - Is there an automatic procedure to
garbage collectthem?
Resolution
- Currently, there is no automatic mechanism to prune secrets in quay operator deployment. Workaround: Locate and
manually deletesecrets that are not in use by other resources using below command.
$ oc delete secret <secret-name>
Root Cause
- On every
reconciliation, the Quay operatorrecreatesall secrets regarding PostgreSQL password, Quay config editor password and Quay config itself. After a couple of iterations, the number of secrets in the namespace grows to the point where it's impossible to tell which secret is being used where (apart from reading the time stamps or reading the deployment file for each individual component directly). This is not a great user experience, it is prone to failure and it complicates debugging process. - This is a known issue captured in This content is not included.PROJQUAY-5172 and is getting worked on by the Engineering team.
Diagnostic Steps
- Check secrets in quay namespace as shown below:
$ oc get secrets -n quay
NAME TYPE DATA AGE
builder-dockercfg-dpcg2 kubernetes.io/dockercfg 1 642d
builder-token-fhz5m kubernetes.io/service-account-token 4 642d
builder-token-rpzmz kubernetes.io/service-account-token 4 642d
clairv4-config Opaque 1 629d
default-dockercfg-drrjj kubernetes.io/dockercfg 1 642d
default-token-jvv86 kubernetes.io/service-account-token 4 642d
default-token-qrkxh kubernetes.io/service-account-token 4 642d
deployer-dockercfg-pkjn7 kubernetes.io/dockercfg 1 642d
deployer-token-7lscp kubernetes.io/service-account-token 4 642d
deployer-token-pph9k kubernetes.io/service-account-token 4 642d
quay-operator-dockercfg-wcsf7 kubernetes.io/dockercfg 1 642d
quay-operator-token-j2bbz kubernetes.io/service-account-token 4 642d
quay-operator-token-q5995 kubernetes.io/service-account-token 4 642d
quay-registry-clair-config-secret-57t89mdgbc Opaque 1 642d
quay-registry-clair-config-secret-5h55967k44 Opaque 1 631d
quay-registry-clair-config-secret-b77dk6hm42 Opaque 1 642d
quay-registry-clair-config-secret-c27md669cb Opaque 1 642d
quay-registry-clair-config-secret-h45h5m6mc7 Opaque 1 642d
quay-registry-clair-postgres-dockercfg-b25pj kubernetes.io/dockercfg 1 243d
quay-registry-clair-postgres-token-c7w29 kubernetes.io/service-account-token 4 243d
quay-registry-clair-postgres-token-mcxg4 kubernetes.io/service-account-token 4 243d
quay-registry-config-bundle-whvlq Opaque 1 642d
quay-registry-extra-ca-certs-74t2f6m724 Opaque 1 243d
quay-registry-extra-ca-certs-9f9hg95cmg Opaque 2 194d
quay-registry-quay-app-dockercfg-rhgt5 kubernetes.io/dockercfg 1 460d
quay-registry-quay-app-token-k5hnm kubernetes.io/service-account-token 4 460d
quay-registry-quay-app-token-mmlwt kubernetes.io/service-account-token 4 460d
quay-registry-quay-config-bundle-2th2t Opaque 4 193d
quay-registry-quay-config-bundle-4qbxv Opaque 6 77d
quay-registry-quay-config-bundle-7w6zj Opaque 5 42d
quay-registry-quay-config-bundle-9vf5c Opaque 3 265d
quay-registry-quay-config-bundle-crf4z Opaque 5 84d
quay-registry-quay-config-bundle-d2fsc Opaque 5 285d
quay-registry-quay-config-bundle-dp5qt Opaque 4 109d
quay-registry-quay-config-bundle-ghvmr Opaque 5 298d
quay-registry-quay-config-bundle-grz5b Opaque 4 257d
quay-registry-quay-config-bundle-hx45r Opaque 5 300d
quay-registry-quay-config-bundle-jf8j5 Opaque 4 112d
quay-registry-quay-config-bundle-lvt85 Opaque 6 109d
quay-registry-quay-config-bundle-ph9bz Opaque 5 267d
quay-registry-quay-config-bundle-qlfz4 Opaque 4 138d
quay-registry-quay-config-bundle-qmzgb Opaque 5 84d
quay-registry-quay-config-bundle-sqtcw Opaque 4 194d
quay-registry-quay-config-bundle-z2sx7 Opaque 5 306d
quay-registry-quay-config-editor-credentials-2457b55gf6 Opaque 2 287d
quay-registry-quay-config-editor-credentials-24b22k7fb5 Opaque 2 285d
quay-registry-quay-config-editor-credentials-25hg46924c Opaque 2 249d
quay-registry-quay-config-editor-credentials-2g8t797k4t Opaque 2 257d
quay-registry-quay-config-editor-credentials-2ghcddckg5 Opaque 2 245d
quay-registry-quay-config-editor-credentials-2hhgg8t27b Opaque 2 300d
quay-registry-quay-config-editor-credentials-2khf44b27m Opaque 2 306d
quay-registry-quay-config-editor-credentials-447cb54bmd Opaque 2 253d
quay-registry-quay-config-editor-credentials-4954k7k4db Opaque 2 244d
quay-registry-quay-config-editor-credentials-4mcfmbf26k Opaque 2 280d
quay-registry-quay-config-editor-credentials-4t6bkfb66k Opaque 2 246d
quay-registry-quay-config-editor-credentials-52tfk6c554 Opaque 2 284d
quay-registry-quay-config-editor-credentials-596t8m85fk Opaque 2 294d
quay-registry-quay-config-editor-credentials-5t6cmhcdkb Opaque 2 265d
quay-registry-quay-config-editor-credentials-65tmbhfgkf Opaque 2 264d
quay-registry-quay-config-editor-credentials-6mc4kgd6k6 Opaque 2 281d
quay-registry-quay-config-editor-credentials-76mtdg5kfd Opaque 2 243d
quay-registry-quay-config-editor-credentials-77498b29g4 Opaque 2 257d
quay-registry-quay-config-editor-credentials-77885k5cm8 Opaque 2 301d
quay-registry-quay-config-editor-credentials-7gd55b895g Opaque 2 292d
quay-registry-quay-config-editor-credentials-849h9mcft5 Opaque 2 279d
quay-registry-quay-config-editor-credentials-84cfdk4c42 Opaque 2 274d
quay-registry-quay-config-editor-credentials-86f6khgt7d Opaque 2 300d
quay-registry-quay-config-editor-credentials-872995t9b5 Opaque 2 243d
quay-registry-quay-config-editor-credentials-8c8fhhk57h Opaque 2 291d
quay-registry-quay-config-editor-credentials-8g2hbbhffd Opaque 2 291d
quay-registry-quay-config-editor-credentials-8gkgcb7c9b Opaque 2 273d
quay-registry-quay-config-editor-credentials-8k584kcf28 Opaque 2 292d
quay-registry-quay-config-editor-credentials-8km95gg9mg Opaque 2 299d
quay-registry-quay-config-editor-credentials-9656h54td8 Opaque 2 272d
quay-registry-quay-config-editor-credentials-99m7f77hf8 Opaque 2 251d
quay-registry-quay-config-editor-credentials-9g6tdfh4mc Opaque 2 277d
quay-registry-quay-config-editor-credentials-9m4m55t865 Opaque 2 265d
quay-registry-quay-config-editor-credentials-9m7cc22ggb Opaque 2 258d
quay-registry-quay-config-editor-credentials-b6d7858965 Opaque 2 259d
quay-registry-quay-config-editor-credentials-bt8k2ff5gd Opaque 2 300d
quay-registry-quay-config-editor-credentials-btf6fm8tt5 Opaque 2 285d
quay-registry-quay-config-editor-credentials-btmf9gbhmg Opaque 2 252d
quay-registry-quay-config-editor-credentials-c5fhkbbmcd Opaque 2 266d
quay-registry-quay-config-editor-credentials-c879gt6ht7 Opaque 2 243d
quay-registry-quay-config-editor-credentials-cc85c9kc24 Opaque 2 270d
quay-registry-quay-config-editor-credentials-dbh8ftgf8c Opaque 2 295d
quay-registry-quay-config-editor-credentials-dc62h59ttk Opaque 2 257d
quay-registry-quay-config-editor-credentials-f8286d2gdg Opaque 2 259d
quay-registry-quay-config-editor-credentials-fg6bm7bh48 Opaque 2 267d
quay-registry-quay-config-editor-credentials-g88497ttmh Opaque 2 250d
quay-registry-quay-config-editor-credentials-gmk774khdc Opaque 2 278d
quay-registry-quay-config-editor-credentials-gt2t7c5998 Opaque 2 273d
quay-registry-quay-config-editor-credentials-gtk94bm8td Opaque 2 300d
quay-registry-quay-config-editor-credentials-h6ffm2m6g5 Opaque 2 286d
quay-registry-quay-config-editor-credentials-hf689hg2mk Opaque 2 267d
quay-registry-quay-config-editor-credentials-m8d74kt9b6 Opaque 2 277d
quay-registry-quay-config-editor-credentials-m8htmtt66f Opaque 2 293d
quay-registry-quay-config-editor-credentials-mdthc9kcdh Opaque 2 298d
quay-registry-quay-config-editor-credentials-mt75c4cm27 Opaque 2 271d
quay-registry-quay-config-editor-credentials-t4gf4d27tc Opaque 2 243d
quay-registry-quay-config-editor-credentials-t54t4m2fcb Opaque 2 298d
quay-registry-quay-config-editor-credentials-t7ggb78bc8 Opaque 2 285d
quay-registry-quay-config-editor-credentials-tkg56k6b75 Opaque 2 288d
quay-registry-quay-config-secret-246b7fb57g Opaque 2 158d
quay-registry-quay-config-secret-2484bthb7d Opaque 3 77d
quay-registry-quay-config-secret-28c49t44t4 Opaque 4 88d
quay-registry-quay-config-secret-2ff7kt886m Opaque 3 300d
quay-registry-quay-config-secret-2g7dkb82t6 Opaque 3 77d
quay-registry-quay-config-secret-2gchftd4b2 Opaque 4 88d
quay-registry-quay-config-secret-2gddf5ccb2 Opaque 4 89d
quay-registry-quay-config-secret-2hmkbkc6hc Opaque 4 42d
quay-registry-quay-config-secret-45dgm5c5tt Opaque 3 77d
quay-registry-quay-config-secret-48b76498dd Opaque 3 77d
quay-registry-quay-config-secret-496cf24962 Opaque 4 42d
quay-registry-quay-config-secret-49fk46kcfg Opaque 2 109d
quay-registry-quay-config-secret-4dmhffccc2 Opaque 4 42d
quay-registry-quay-config-secret-54h9c2f85f Opaque 3 42d
quay-registry-quay-config-secret-58d2ck7h8g Opaque 3 81d
quay-registry-quay-config-secret-5b886gb964 Opaque 4 42d
quay-registry-quay-config-secret-5cc7d4k4gd Opaque 2 243d
quay-registry-quay-config-secret-697gkhb66t Opaque 3 300d
quay-registry-quay-config-secret-72f9kmk2bk Opaque 3 77d
quay-registry-quay-config-secret-74h8k84t9k Opaque 2 243d
quay-registry-quay-config-secret-7fc4hb6d95 Opaque 3 81d
quay-registry-quay-config-secret-7g265726h2 Opaque 3 76d
quay-registry-quay-config-secret-7hmd9dmdc4 Opaque 3 81d
quay-registry-quay-config-secret-8275t8bbh8 Opaque 3 77d
quay-registry-quay-config-secret-86dhm59ggm Opaque 3 77d
quay-registry-quay-config-secret-8bbg67dg92 Opaque 3 76d
quay-registry-quay-config-secret-8dmbd2hckc Opaque 3 194d
quay-registry-quay-config-secret-8m7457d5hm Opaque 3 222d
quay-registry-quay-config-secret-94bfhcfcfh Opaque 3 76d
quay-registry-quay-config-secret-955f6hddgd Opaque 2 138d
quay-registry-quay-config-secret-98f25k9t5b Opaque 2 112d
quay-registry-quay-config-secret-992g99b7t8 Opaque 4 74d
quay-registry-quay-config-secret-99b4h6hg86 Opaque 4 70d
quay-registry-quay-config-secret-9bd2dbdcd4 Opaque 3 77d
quay-registry-quay-config-secret-9dg7c4mmcc Opaque 3 84d
quay-registry-quay-config-secret-9hctf59g9c Opaque 3 259d
quay-registry-quay-config-secret-9hhb5chktm Opaque 3 257d
quay-registry-quay-config-secret-b27685c4m6 Opaque 3 77d
quay-registry-quay-config-secret-b4g84k9kmm Opaque 3 81d
quay-registry-quay-config-secret-b7d7bbm9t4 Opaque 3 298d
quay-registry-quay-config-secret-b8795h852k Opaque 4 70d
quay-registry-quay-config-secret-bgbht7bdb7 Opaque 2 111d
quay-registry-quay-config-secret-c27chhbg59 Opaque 4 74d
quay-registry-quay-config-secret-c5929ck7dm Opaque 3 77d
quay-registry-quay-config-secret-ccgtmc6d5k Opaque 4 64d
quay-registry-quay-config-secret-chmhkg4d4d Opaque 4 42d
quay-registry-quay-config-secret-ctddgtmd95 Opaque 3 77d
quay-registry-quay-config-secret-d6md27td89 Opaque 2 398d
quay-registry-quay-config-secret-d8797g9khf Opaque 4 62d
quay-registry-quay-config-secret-dchmkt4b64 Opaque 4 88d
quay-registry-quay-config-secret-dh8btghtbb Opaque 4 70d
quay-registry-quay-config-secret-dkmf2ct6d7 Opaque 3 285d
quay-registry-quay-config-secret-dmfdmfb6gc Opaque 3 81d
quay-registry-quay-config-secret-f2dfcfh7f7 Opaque 3 77d
quay-registry-quay-config-secret-fb64gt6td5 Opaque 3 84d
quay-registry-quay-config-secret-fdhk9gtkb2 Opaque 3 84d
quay-registry-quay-config-secret-fg928bbgm8 Opaque 2 112d
quay-registry-quay-config-secret-fkfb69252h Opaque 4 67d
quay-registry-quay-config-secret-fmfk2mdc27 Opaque 3 77d
quay-registry-quay-config-secret-g8tft486fc Opaque 3 81d
quay-registry-quay-config-secret-gc64m499dd Opaque 3 84d
quay-registry-quay-config-secret-gd8mcfb564 Opaque 3 42d
quay-registry-quay-config-secret-gdctmb64m5 Opaque 4 89d
quay-registry-quay-config-secret-gfb2969gd7 Opaque 3 77d
quay-registry-quay-config-secret-gtb277hd9c Opaque 3 81d
quay-registry-quay-config-secret-h4d4mm87gc Opaque 3 77d
quay-registry-quay-config-secret-h57tb92t5f Opaque 3 196d
quay-registry-quay-config-secret-h7kgfktb86 Opaque 3 77d
quay-registry-quay-config-secret-hdh7k57k8d Opaque 3 398d
quay-registry-quay-config-secret-hgmdh68787 Opaque 3 389d
quay-registry-quay-config-secret-k59287k82t Opaque 3 267d
quay-registry-quay-config-secret-k9kb9tc727 Opaque 2 194d
quay-registry-quay-config-secret-kfcc8th96m Opaque 3 77d
quay-registry-quay-config-secret-m2c4fh4hmh Opaque 3 77d
quay-registry-quay-config-secret-m6f9795c78 Opaque 4 70d
quay-registry-quay-config-secret-m852264896 Opaque 3 231d
quay-registry-quay-config-secret-m974254546 Opaque 3 224d
quay-registry-quay-config-secret-mc6647d4kh Opaque 3 84d
quay-registry-quay-config-secret-mkk7tf9fgd Opaque 3 81d
quay-registry-quay-config-secret-t4h84g9498 Opaque 3 82d
quay-registry-quay-config-secret-t77gh25h2f Opaque 3 82d
quay-registry-quay-config-secret-t79b2mhg2h Opaque 2 193d
quay-registry-quay-config-secret-tb6g5m462f Opaque 4 70d
quay-registry-quay-config-secret-tg59fkd2ct Opaque 3 77d
quay-registry-quay-config-secret-tghhcmkt5b Opaque 3 306d
quay-registry-quay-config-secret-tt9tfd566k Opaque 4 42d
quay-registry-quay-config-tls-2bd7hdf4fh Opaque 1 243d
quay-registry-quay-config-tls-75b4dmm4d4 Opaque 1 259d
quay-registry-quay-config-tls-d6bbbfg5kk Opaque 3 460d
quay-registry-quay-proxy-config-6kth6khg6m Opaque 3 243d
quay-registry-quay-registry-managed-secret-keys Opaque 6 243d
quay-registry-quay-registry-managed-secret-keys-b6cf5hctd4 Opaque 4 315d
$ oc get secrets | wc -l
121
Product(s)
Components
Category
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.