How to avoid that the default ingresscontroller serves routes of all projects when using router sharding in OpenShift 4.x

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform 4.x
  • Red Hat OpenShift Service on AWS

Issue

When configuring router sharing according to the documentation [0], it may be desired to serve specific namespaces only by specific ingress controllers. However, with the default configuration, the default ingress operator will still serve all services due to an absence of namespaceSelector or routeSelector configuration for this operator. Additionally, in environments which use hostNetwork for the ingress operators, such as vSphere and Bare Metal, it is not possible to use network policies to block traffic from those operators due to [1].

For example, in an environment where fh.apps.cluster43.example.com is hosted by the default ingress controller and fh.test1.cluster43.example.com by the test1 ingress controller:

 $  curl fh.apps.cluster43.example.com
Apache default
 $ curl fh.test1.cluster43.example.com
Apache test1
 $ getent hosts fh.apps.cluster43.example.com | awk '{print $1}'
172.16.0.207
 $ getent hosts fh.test1.cluster43.example.com | awk '{print $1}'
172.16.0.224

In the above environment, an attacker could take advantage of this by changing the resolver and point it to the IP of the default ingress controller, using IP 172.16.0.207 (the default namespace's ingress controller):

 $ curl -I fh.test1.cluster43.example.com --resolve fh.test1.cluster43.example.com:80:172.16.0.207 2>/dev/null  | grep HTTP
HTTP/1.1 200 OK
 $ curl fh.test1.cluster43.example.com --resolve fh.test1.cluster43.example.com:80:172.16.0.207 2>/dev/null 
Apache test1

How can one avoid that the default ingress controller serves routes of all projects when using router sharding in OpenShift 4.x?

Network policy does not apply to the host network namespace. Pods with host networking enabled are unaffected by NetworkPolicy object rules.

Resolution

NOTE:
The steps in this article are supported on ROSA (with HCP).
For ROSA (classic architecture), this can be achieved with the ROSA CLI, using the rosa edit ingress --excluded-namespaces command. Please see documentation.

Assuming the aforementioned example case where test1-ingress-controller serves namespace test1:

 $ cat operator-test1.yaml

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: test1-ingress-controller
  namespace: openshift-ingress-operator
spec:
  replicas: 1
  domain: test1.cluster43.example.com
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/worker: ""
        ingressoperator: "test1"
  namespaceSelector:
    matchLabels:
      type: test1
  endpointPublishingStrategy:
    type: HostNetwork
    hostNetwork:
      httpPort: <value>  ---> specify dedicate host ports to open in case router shards are schedule on the same nodes. Only necessary if HostNetwork is used as endpointPublishingStrategy for all routers [0].
      httpsPort: <value>
      statsPort: <value>

Additionally, assume there is another namespace, test2 with the same configuration:

 $ cat operator-test2.yaml

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: test2-ingress-controller
  namespace: openshift-ingress-operator
spec:
  replicas: 1
  domain: test2.cluster43.example.com
  nodePlacement:
    nodeSelector:
      matchLabels:
        node-role.kubernetes.io/worker: ""
        ingressoperator: test2
  namespaceSelector:
    matchLabels:
      type: test2
  endpointPublishingStrategy:
    type: HostNetwork
    hostNetwork:
      httpPort: <value>  ---> specify dedicate host ports to open in case router shards are schedule on the same nodes
      httpsPort: <value>
      statsPort: <value>

No other ingress controller shall serve these namespaces. In that case, it is necessary to modify the default ingress controller and instruct it not to serve these namespaces. Modify the spec of the default ingress controller:

oc edit -n openshift-ingress-operator ingresscontroller default

It should look like this to exclude namespaces test1 and test2:

 $ oc get -n openshift-ingress-operator ingresscontroller default -o yaml | grep spec: -A12
spec:
  namespaceSelector:
    matchExpressions:
    - key: type
      operator: NotIn
      values:
      - test1
      - test2
  nodePlacement:
    nodeSelector:
      matchLabels:
        ingressoperator: default
  replicas: 1

[0] This page is not included, but the link has been rewritten to point to the nearest parent document.This page is not included, but the link has been rewritten to point to the nearest parent document.https://docs.openshift.com/container-platform/4.14/networking/ingress-operator.html#:~:text=is%202.-,endpointPublishingStrategy,-endpointPublishingStrategy%20is%20used

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.