Clair scans are not working behind proxy

Environment

• Red Hat Quay Operator Deployment
  • 3.6
• Clair behind proxy

Issue

• Quay debug logs show 401 error while reaching out to Clair through a Proxy, suggesting that quay is not able to communicate to to Clair.

securityworker stdout | 2022-11-08 14:32:52,443 [106] [DEBUG] [urllib3.connectionpool] http://192.168.xx.xx:3128 "GET http://clairv4/indexer/api/v1/index_state HTTP/1.1" 401 843
securityworker stdout | 2022-11-08 14:32:52,474 [106] [ERROR] [util.secscan.v4.api] Security scanner endpoint responded with non-200 HTTP status code: 401

Resolution

• Remove, if any, proxy variables from QuayRegistry CRD in order to keep Quay unproxied.

kind: QuayRegistry
    components:
         - kind: quay
            managed: true
       overrides:
          env:
            - name: DEBUGLOG
              value: "true"
            - name: NO_PROXY                                     --
              value: svc.cluster.local,localhost,quay.example.com  |
            - name: HTTP_PROXY                                     |----- remove the values from this section(if  added)
              value: quayproxy.example.com:<port>                  |
            - name: HTTPS_PROXY                                    |
              value: quayproxy.example.com:<port>                 --    

• Set proxy variables for Clair. They can be copied from RHOCP's cluster proxy file. Add full Clair service name to NO_PROXY in QuayRegistry CRD

Root Cause

• Quay inherited the cluster proxy configuration from RHOCP and hence attempted to connect with Clair through the proxy, resulting in HTTP error code 401.
• Even after adding .cluster.local,.svc noProxy to Clair component in QuayRegistry CRD, Quay is not able to communicate with Clair. Because it uses short hostname of Clair instead of clairv4-{quay-namespace}.svc.cluster.local when it calls Clair service. Thus, the requests were still being proxied and failed.

Diagnostic Steps

• Make sure Clair is accessible through no proxy. And that it has FQDN in NO_PROXY variables set instead of short name like clairv4.

• Check proxy variables being used by quay pods using cURL commands shown below:

$ oc rsh quay-registry-quay-app-pod

sh-4.4$ curl -Iv -x http://192.168.xx.xx:3128 http://clairv4:80
* Rebuilt URL to: http://clairv4:80/
* Uses proxy env variable NO_PROXY == 'quay.apps.example.com,.cluster.local,.svc,10.130.xx.xx/16,10.131.xx.xx/16,127.0.0.1,169.254.xx.xx,172.28.xx.xx/27,googleapis.com,localhost,metadata,metadata.google.internal,metadata.google.internal.'
*   Trying 192.168.xx.xx...
* TCP_NODELAY set
* Connected to 192.168.xx.xx (192.168.xx.xx) port 3128 (#0)
> HEAD http://clairv4:80/ HTTP/1.1
> Host: clairv4
> User-Agent: curl/7.61.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 401 Unauthorized
HTTP/1.1 401 Unauthorized
< Content-Length: 791
Content-Length: 791
< Content-Type: text/html
Content-Type: text/html
< Date: Tue, 15 Nov 2022 12:20:03 GMT
Date: Tue, 15 Nov 2022 12:20:03 GMT
< Expires: Tue, 15 Nov 2022 12:20:03 GMT
Expires: Tue, 15 Nov 2022 12:20:03 GMT
< Server: Mikrotik HttpProxy
Server: Mikrotik HttpProxy
<
* Excess found in a non pipelined read: excess = 791 url = / (zero-length body)
* Connection #0 to host 192.168.xx.xx left intact


sh-4.4$ curl -Iv --noproxy '*' http://clairv4:80
* Rebuilt URL to: http://clairv4:80/
*   Trying 10.131.yy.yy...
* TCP_NODELAY set
* Connected to clairv4 (10.131.yy.yy) port 80 (#0)
> HEAD / HTTP/1.1
> Host: clairv4
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
HTTP/1.1 401 Unauthorized
< Date: Tue, 15 Nov 2022 12:20:22 GMT
Date: Tue, 15 Nov 2022 12:20:22 GMT

<
* Connection #0 to host clairv4 left intact

• Check Quay debug logs why Clair scanning fails:

securityworker stdout | 2022-11-08 14:32:52,443 [106] [DEBUG] [urllib3.connectionpool] http://192.168.100.155:3128 "GET http://clairv4/indexer/api/v1/index_state HTTP/1.1" 401 843
securityworker stdout | 2022-11-08 14:32:52,474 [106] [ERROR] [util.secscan.v4.api] Security scanner endpoint responded with non-200 HTTP status code: 401
securityworker stdout | 2022-11-08 14:32:53,144 [106] [DEBUG] [urllib3.connectionpool] http://192.168.100.155:3128 "GET http://clairv4/indexer/api/v1/index_state HTTP/1.1" 401 843
securityworker stdout | 2022-11-08 14:32:53,175 [106] [ERROR] [util.secscan.v4.api] Security scanner endpoint responded with non-200 HTTP status code: 401

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.