Custom nftables firewall rules in OpenShift

Updated 24 Sept 2025

This document outlines a supported method for implementing custom nftables firewall rules in OpenShift clusters. It is intended for cluster administrators who are responsible for managing network security policies within their OpenShift environments.

This method leverages MachineConfig Custom Resources (CRs) to configure firewall rules on each cluster node. The MachineConfig CR configures the cluster as follows:

Enables the "nftables" systemd service unit.
Writes the custom firewall rules to the /etc/sysconfig/nftables.conf file on selected nodes in the cluster.

Important Note: Because you apply these custom rules when booting the host before the network is operational, it is crucial to carefully design and test the rules to avoid unintended network disruptions during the cluster boot process.

Implementing custom firewall rules

Two options are available when implementing custom firewall rules in an OpenShift cluster using MachineConfig CRs.

Option A: MachineConfig (OpenShift 4.16 or older)
Option B: MachineConfig with Node Disruption Policies (OpenShift 4.17 or newer)

	Option A	Option B
Minimum OCP version	4.12	4.17
Node reboot	Yes	No
Production grade	Yes	Yes

Option A: Implementing custom firewall rules with a MachineConfig CR

Create the 98-nftables-cnf-worker.bu Butane config YAML file with the following content.

variant: openshift
version: 4.16.0
metadata:
  name: 98-nftables-cnf-worker
  labels:
    machineconfiguration.openshift.io/role: worker
systemd:
  units:
    - name: "nftables.service"
      enabled: true
      contents: |
        [Unit]
        Description=Netfilter Tables
        Documentation=man:nft(8)
        Wants=network-pre.target
        Before=network-pre.target
        [Service]
        Type=oneshot
        ProtectSystem=full
        ProtectHome=true
        ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecReload=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecStop=/sbin/nft 'add table inet custom_table; delete table inet custom_table'
        RemainAfterExit=yes
        [Install]
        WantedBy=multi-user.target
storage:
  files:
  - path: /etc/sysconfig/nftables.conf
    mode: 0600
    overwrite: true
    contents:
      inline: |
        table inet custom_table
        delete table inet custom_table
        table inet custom_table {
            <YOUR_CUSTOM_NFTABLES_CHAINS_WITH_RULES>
        }

Important notes:

Replace <YOUR_CUSTOM_NFTABLES_CHAINS_WITH_RULES> with the required nftables chains and rules.
This configuration targets worker nodes by setting the machineconfiguration.openshift.io/role label to worker. Adjust this label if you need the configuration for a different node type.

Generate the MachineConfig CR. Use Butane to generate a MachineConfig object file that contains the 98-nftables-cnf-worker.bu configuration to be delivered to the worker nodes:
```
$ butane 98-nftables-cnf-worker.bu -o 98-nftables-cnf-worker.yaml
```
Apply the MachineConfig CR to the cluster:
```
$ oc apply -f 98-nftables-cnf-worker.yaml
```
This command creates the MachineConfig CR and instructs the cluster to configure worker nodes with the custom firewall rules upon the next boot.

Option B: Implementing custom firewall rules with a `MachineConfig` CR containing node disruption policies

If you implement custom firewall rules by using a MachineConfig CR that contains node disruption policies, you avoid the node having to reboot every time the firewall rules are updated.

Note: Option B is similar to option A with a few prerequisite steps described below. It relies on a new General Availability (GA) node disruption policies feature for MachineConfig CRs introduced in OpenShift 4.17. Please refer to This page is not included, but the link has been rewritten to point to the nearest parent document.Using node disruption policies to minimize disruption from machine config changes.

Configure node restart behaviors after machine config changes are applied:

Edit the MachineConfiguration CR named cluster with the following node disruption policies:

$ oc edit MachineConfiguration cluster -n openshift-machine-config-operator

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  nodeDisruptionPolicy:
    files:
    - actions:
      - restart:
          serviceName: nftables.service
        type: Restart
      path: /etc/sysconfig/nftables.conf
    units:
    - actions:
      - type: DaemonReload
      - type: Reload
        reload:
          serviceName: nftables.service
      name: nftables.service

The first policy is triggered upon changes to the custom firewall rules file and will reload the systemd nftables service unit to re-apply the required firewall configuration.

Implement the custom firewall rules by using a MachineConfig CR. Apply the custom firewall rules as described in "Option A: Implementing custom firewall rules with a MachineConfig CR".

Operational considerations

Implementing custom firewall rules using MachineConfig CRs offers granular control over network traffic on your OpenShift cluster nodes. It is crucial to carefully consider the operational implications before deploying this method:

Early application: The rules are applied at boot time, before the network is fully operational. Ensure the rules don't inadvertently block essential services required during the boot process.
Risk of misconfiguration: Errors in your custom rules can lead to unintended consequences, potentially leading to performance impact or blocking legitimate traffic or isolating nodes. Thoroughly test your rules in a non-production environment before deploying them to your main cluster.
External endpoints: OpenShift requires access to external endpoints to function. Ensure that cluster nodes are permitted access to those endpoints. Please refer to the allowlist table on the official documentation page.
Network flow matrix: Use the information in the network flow matrix to help you manage ingress traffic. You can restrict ingress traffic to essential flows to improve network security. The matrix provides insights into base cluster services but excludes traffic generated by Day-2 Operators.
Cluster version updates and upgrades: Users should exercise caution when updating or upgrading their OpenShift clusters. Recent changes to the platform's firewall requirements might require adjustments to network port permissions. While the current documentation provides guidelines, it is important to note that these requirements can evolve over time. To minimize disruptions, it is strongly recommended to test any updates or upgrades in a staging environment before applying them in production. This will help identify and address potential compatibility issues related to firewall configuration changes.

Additional considerations for nftables rule priority and chain policy

OpenShift rule management: Be aware that components of OpenShift such as Kubernetes and OVN-Kubernetes will create nftables rules at runtime (i.e., later than your custom rules). These runtime rules can potentially override your custom rules depending on factors like chain priority and default chain policy (accept or drop).
Chain priority and default policy: To manage precedence, consider adjusting the priority of your custom chains within the nftables configuration. You might also need to modify the default chain policy (accept or drop) in your custom chains to achieve the required outcome.
Separate nftables table: To avoid conflicts, it is mandatory to define your custom firewall rules in a separate nftables table that is isolated from the tables managed by OpenShift components. This approach ensures your custom rules are not inadvertently overridden by runtime rules from OpenShift.

By following these operational considerations and carefully managing chain priority and policy, you can leverage MachineConfig CRs for custom firewall rules while maintaining a secure OpenShift cluster environment.

Verifying that the nftables firewall rules are successfully applied

Create a Butane config file with custom nftables rules and apply them in your cluster.

The following rules open ports required by the running cluster services and stops other traffic.

variant: openshift
version: 4.16.0
metadata:
  name: 98-nftables-cnf-worker
  labels:
    machineconfiguration.openshift.io/role: worker
systemd:
  units:
    - name: "nftables.service"
      enabled: true
      contents: |
        [Unit]
        Description=Netfilter Tables
        Documentation=man:nft(8)
        Wants=network-pre.target
        Before=network-pre.target
        [Service]
        Type=oneshot
        ProtectSystem=full
        ProtectHome=true
        ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecReload=/sbin/nft -f /etc/sysconfig/nftables.conf
        ExecStop=/sbin/nft 'add table inet custom_table; delete table inet custom_table'
        RemainAfterExit=yes
        [Install]
        WantedBy=multi-user.target
storage:
  files:
  - path: /etc/sysconfig/nftables.conf
    mode: 0600
    overwrite: true
    contents:
      inline: |
        table inet custom_table
        delete table inet custom_table
        table inet custom_table {
            chain custom_chain_INPUT {
                type filter hook input priority 1; policy accept;
                # Drop TCP port 8000 and log
                tcp dport 8000 log prefix "[USERFIREWALL] PACKET DROP: " drop
            }
        }

Review the nftables configuration:

Verify that the nftables.service is loaded and active with the systemctl status nftables command.

        # systemctl status nftables
        ● nftables.service - Netfilter Tables
             Loaded: loaded (/usr/lib/systemd/system/nftables.service; enabled; preset: disabled)
             Active: active (exited) since Thu 2024-07-18 12:24:43 UTC; 9min ago
               Docs: man:nft(8)
            Process: 742861 ExecStart=/sbin/nft -f /etc/sysconfig/nftables.conf (code=exited, status=0/SUCCESS)
           Main PID: 742861 (code=exited, status=0/SUCCESS)
                CPU: 100ms

        Jul 18 12:24:43 cnfdc6.t5g-dev.eng.rdu2.dc.redhat.com systemd[1]: Starting Netfilter Tables...
        Jul 18 12:24:43 cnfdc6.t5g-dev.eng.rdu2.dc.redhat.com systemd[1]: Finished Netfilter Tables.

Check the /etc/sysconfig/nftables.conf file to confirm your custom rules are present.

Test blocked traffic:

Spin up a test server on a port not explicitly allowed in your rules (e.g., port 8000 listening on IPv4 and IPv6):
```
# python3 -m http.server -b :: 8000
Serving HTTP on :: port 8000 (http://[::]:8000/) ...
```

Attempt to connect to this server from a separate machine using curl:

        $ curl -m 5 -v -s 10.6.105.51:8000 1> /dev/null
        *   Trying 10.6.105.51:8000...
        * Connection timed out after 5002 milliseconds
        * Closing connection

        $ curl -m 5 -v -s [2620:52:9:1669:e42:a1ff:fe56:1fe]:8000 1> /dev/null
        *   Trying 2620:52:9:1669:e42:a1ff:fe56:1fe:8000...
        * Connection timed out after 5000 milliseconds
        * Closing connection 0

If the connection times out, it suggests the firewall is blocking traffic on that port as intended.

        # journalctl -f -g USERFIREWALL

        # IPv4
        kernel: [USERFIREWALL] PACKET DROP: IN=br-ex OUT= MAC=0c:42:a1:56:01:fe:52:54:00:b5:d8:06:08:00 SRC=10.6.105.208 DST=10.6.105.51 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52940 DF PROTO=TCP SPT=60236 DPT=8000 WINDOW=64240 RES=0x00 SYN URGP=0

        # IPv6
        kernel: [USERFIREWALL] PACKET DROP: IN=br-ex OUT= MAC=0c:42:a1:56:01:fe:52:54:00:b5:d8:06:86:dd SRC=2620:0052:0009:1669:5054:00ff:feb5:d806 DST=2620:0052:0009:1669:0e42:a1ff:fe56:01fe LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=74763 PROTO=TCP SPT=55458 DPT=8000 WINDOW=64800 RES=0x00 SYN URGP=0

Test allowed traffic:

Replace the drop rule with an accept rule:

# nft replace rule inet custom_table custom_chain_INPUT handle 2 tcp dport 8000 log prefix \"[USERFIREWALL] PACKET ACCEPT: \" accept

From a separate machine, try connecting to the test server.

Successful connections indicate that the allowed traffic is flowing as expected.

        # IPv4
        $ curl -m 5 -v -s 10.6.105.51:8000 1> /dev/null
        *   Trying 10.6.105.51:8000...
        * Connected to 10.6.105.51 (10.6.105.51) port 8000 (#0)
        > GET / HTTP/1.1
        > Host: 10.6.105.51:8000
        > User-Agent: curl/7.76.1
        > Accept: */*
        > 
        * Mark bundle as not supporting multiuse
        * HTTP 1.0, assume close after body
        < HTTP/1.0 200 OK
        < Server: SimpleHTTP/0.6 Python/3.9.18
        < Date: Thu, 18 Jul 2024 12:39:26 GMT
        < Content-type: text/html; charset=utf-8
        < Content-Length: 1022
        < 
        { [1022 bytes data]
        * Closing connection 0

        # IPv6
        $ curl -m 5 -v -s [2620:52:9:1669:e42:a1ff:fe56:1fe]:8000 1> /dev/null
        *   Trying 2620:52:9:1669:e42:a1ff:fe56:1fe:8000...
        * Connected to 2620:52:9:1669:e42:a1ff:fe56:1fe (2620:52:9:1669:e42:a1ff:fe56:1fe) port 8000 (#0)
        > GET / HTTP/1.1
        > Host: [2620:52:9:1669:e42:a1ff:fe56:1fe]:8000
        > User-Agent: curl/7.76.1
        > Accept: */*
        > 
        * Mark bundle as not supporting multiuse
        * HTTP 1.0, assume close after body
        < HTTP/1.0 200 OK
        < Server: SimpleHTTP/0.6 Python/3.9.18
        < Date: Thu, 18 Jul 2024 12:39:28 GMT
        < Content-type: text/html; charset=utf-8
        < Content-Length: 1022
        < 
        { [1022 bytes data]
        * Closing connection 0

SBR

Product(s)

Red Hat OpenShift Container Platform

Category

Secure

Components

Tags

Article Type

General