In Quay Geo-Replication, Storage Buckets not synced

Environment

• Red Hat Quay
  • 3.7

Issue

• Why do s3 buckets differ in size and number of objects?
• How to figure out what is the delta between them?

Root Cause

Over a period of time images in Quay registry are deleted. But within quay there is no mechanism as of now to make sure that an image deleted at some point in time is 100% removed from the backing storage. Due to this, it is likely that many layers of such images are lying in the backing storage and causing inconsistencies in all backing stores data counts.

Replicationworkers from the backfill script might take some time to catch up with latest tasks especially when images are constantly being pushed/new layers are being added to the registry. There is always a delay.

A difference in the size of backing s3 storage is not worrysome. Unless it leads pulls failing due layer/s not being present in the table imagestoragelocation that the database expects it to be.

Diagnostic Steps

• Check the size of the quay s3 bucket on each site for their content and size:

    Bucket site A:
    Total objects: 24.081k (24081)
    Total size: 830.088 GiB (891300185024 Byte)
  
    Bucket site B:
    Total objects: 21.081k (21081)
    Total size: 753.233 GiB (808777708664 Byte)
  
    Bucket site C:
    Total objects: 21.667k (21667)
    Total size: 760.407 GiB (816480880859 Byte)
  

• Check DB table if one sees differences as shown in above output.

    quaygeo=> select * from imagestoragelocation;
     id |       name        
    ----+-------------------
    1 | s3_us_east_1
    2 | s3_eu_west_1
    3 | s3_ap_southeast_1
    4 | s3_ap_southeast_2
    5 | s3_ap_northeast_1
    6 | s3_sa_east_1
    7 | local
    8 | s3_us_west_1
    9 | azureStorage
    10 | cremaStorageOCS
    11 | gdcStorageOCS
    (11 rows)
  
    quaygeo=> select count(*) from imagestorageplacement where location_id = 9;
     count 
    ........
     21884
    (1 row)
    quaygeo=> select count(*) from imagestorageplacement where location_id = 10;
     count 
    ........
     21302
    (1 row)
    quaygeo=> select count(*) from imagestorageplacement where location_id = 11;
     count 
    ........
     23791
    (1 row)
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.