Capturing Original Client IP from the X-Forwarded-For Header in Ingress and Application Logs

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4

Issue

  • How to view the original client IP through the X-Forwarded-For header in OpenShift Ingress access logs?
  • Viewing the original client IP using the X-Forwarded-For header in applications logs deployed on OpenShift.

Resolution

OpenShift provides two robust methods to to capture the original IP address of the client or end user IP address in the Openshift ingress access logs. This article delivers a comprehensive, step-by-step guide to capturing the authentic client IP in OpenShift Ingress router access logs for a custom IngressController using X-Forwarded-For header on the top-level L7 loadbalancer.

In HAProxy, we can capture HTTP headers such as requests and responses using http-request and http-response headers. This feature is valuable for debugging, monitoring, or logging specific HTTP header information that helps track client behavior or backend responses.

The httpCaptureHeaders option in Openshift IngressController specifies the HTTP headers that you want to capture in the access logs. If the httpCaptureHeaders field is empty, the access logs do not capture the headers.

The httpCaptureHeaders contains two lists of headers to capture in the access logs. The two lists of header fields are request and response. In both lists, the name field must specify the header name and the maxlength field must specify the maximum length of the header. For example:

  httpCaptureHeaders:
    request:
    - maxLength: 256
      name: Connection
    - maxLength: 128
      name: User-Agent
    - maxLength: 128
      name: X-Forwarded-For
    response:
    - maxLength: 256
      name: Content-Type
    - maxLength: 256
      name: Content-Length

Perquisites

  • Create a custom/sharded ingress controller
  • Configure the top-level loadbalancer in L7 mode to forward traffic to the custom-ingress controller.
  • Configure the top-level loadbalancer to pass the original client IP address in the X-Forwarded-For header when forwarding requests to the OpenShift Ingress router.

IngressController configuration

  • Configure the sharded/custom IngressController to capture X-Forwarded-For in the request section of httpCaptureHeaders
$ oc edit ingresscontrollers.operator.openshift.io shard1 
  httpCaptureHeaders:
    request:
    - maxLength: 128
      name: X-Forwarded-For

  • Define a custom httpLogFormat to show the original client IP(X-Forwarded-For) using %[capture.req.hdr(n)], where n represents the position of the captured header (0 for the first, 1 for the second, and so on).

  • Response headers from the backend PODs can be listed using %[capture.res.hdr(n)] where n represents the position of the captured header (0 for the first, 1 for the second, and so on).

httpLogFormat: '%ci:%cp org-client-xff:%[capture.req.hdr(0)] [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %r'

Here: 
    %ci = client_ip
    %cp = client_port 
    org-client-xff:%[capture.req.hdr(0)] = original client IP
    [%tr] = Request date
    %ft = frontend_name_transport
    %b/%s = backend_name/server_name
    %TR/%Tw/%Tc/%Tr/%Ta = time to receive the full request from 1st byte/%[req.timer.queue]/%[bc.timer.connect]/response time/Active time of the request (from TR to end)
    %ST = status_code  
    %B = bytes_read
    %CC = captured_request_cookie
    %CS = captured_response_cookie
    %tsc = termination_state with cookie status 
    %ac/%fc/%bc/%sc/%rc = active_conn/frontend concurrent connections/backend concurrent connections/retries
    %sq/%bq = srv_queue/backend_queue
    %r = http_request

IngressController custom log format configuration example

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  finalizers:
  - ingresscontroller.operator.openshift.io/finalizer-ingresscontroller
  name: shard1
  namespace: openshift-ingress-operator
spec:
  domain: <apps-sharded.basedomain.example.net>
  clientTLS:
    clientCA:
      name: ""
    clientCertificatePolicy: ""
  httpCompression: {}
  httpEmptyRequestsPolicy: Respond
  httpErrorCodePages:
    name: ""
  logging:
    access:
      destination:
        type: Container
      httpCaptureHeaders:
        request:
        - maxLength: 128
          name: X-Forwarded-For
      httpLogFormat: '%ci:%cp org-client-xff:%[capture.req.hdr(0)] [%tr]
        %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %r'
      logEmptyRequests: Log
  replicas: 2

Sample Application configuration(nginx)

  • Configure the application to log the original client IP address from the X-Forwarded-For.

  • Each application employs its own unique method for configuring access log formats, making it unfeasible to provide a one-size-fits-all guide for log formatting across all applications

  • The following is the sample log formatting configuration for a nginx weserver

# cat /etc/nginx/nginx.conf 

## Nginx XFF header setup
    set_real_ip_from 0.0.0.0/0;
    real_ip_recursive on;
    real_ip_header X-Forwarded-For;


http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

Sample ingress access logs

  • In the following logs, the load balancer IP is 192.168.100.2 and original client IP is 192.168.0.5. 
2024-11-15T07:18:44.003632+00:00 master-2 master-2 haproxy[274]: 192.168.100.2:54678 org-client-xff:192.168.0.5 [15/Nov/2024:07:18:43.998] fe_sni~ be_edge_http:test:nginx/pod:nginx-86b659894b-96jkg:nginx:8080-tcp:10.131.0.9:8080 0/0/3/2/5 200 203 - - --NI 2/1/0/0/0 0/0 HEAD / HTTP/1.1
  • When the top level loadbalancer doesn't forward original client IP in X-Forward-For header, value of the org-client-xff will be empty(-). 
2024-11-15T07:18:44.003632+00:00 master-2 master-2 haproxy[274]: 192.168.100.2:54678 org-client-xff:- [15/Nov/2024:07:18:43.998] fe_sni~ be_edge_http:test:nginx/pod:nginx-86b659894b-96jkg:nginx:8080-tcp:10.131.0.9:8080 0/0/3/2/5 200 203 - - --NI 2/1/0/0/0 0/0 HEAD / HTTP/1.1

Sample application access logs from a nginx POD

192.168.0.5 - - [15/Nov/2024:07:18:44 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.61.1" "192.168.0.5, 192.168.100.2"

Limitations

Note that this option only applies to cleartext HTTP connections and secure HTTP connections for which the ingress controller terminates encryption (that is, edge-terminated or reencrypt connections). Headers cannot be captured for TLS passthrough connections.

Root Cause

By default, OpenShift Ingress router pods do not capture the original client IP address in their access logs via the X-Forwarded-For header. To enable this crucial functionality, a custom access log format must be configured, along with setting the external or top-level load balancer to forward the original client IP in the X-Forwarded-For header to the OpenShift Ingress router pods. This setup ensures accurate logging of client source IPs, providing enhanced visibility and traceability for all incoming requests.

Diagnostic Steps

Sample nginx application POD image

In order to test this feature quickly on a sample application POD, use the following nginx image, the create a route and access the application. 

oc new-app --image=quay.io/rhn_support_rupatel/nginx:main

Quick test using curl command

Execute the following curl command on a test system for a route URL managed by the Openshift IngressController

$ curl -kvI --header "X-Forwarded-For: original client IP" https://URL

Sample ingress access logs

  • In the following ingress router access log, the load balancer IP is 192.168.100.2 and original client IP is 192.168.0.5. 
2024-11-15T07:18:44.003632+00:00 master-2 master-2 haproxy[274]: 192.168.100.2:54678 org-client-xff:192.168.0.5 [15/Nov/2024:07:18:43.998] fe_sni~ be_edge_http:test:nginx/pod:nginx-86b659894b-96jkg:nginx:8080-tcp:10.131.0.9:8080 0/0/3/2/5 200 203 - - --NI 2/1/0/0/0 0/0 HEAD / HTTP/1.1

Sample nginx webserver access logs deployed on Openshift

192.168.0.5 - - [15/Nov/2024:07:18:44 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.61.1" "192.168.0.5, 192.168.100.2"
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.