Procedure

1. Set the following environment by running the following command:

   $ export KUADRANT_GATEWAY_NS=api-gateway \
     export KUADRANT_GATEWAY_NAME=ingress-gateway \
     export KUADRANT_DEVELOPER_NS=toystore \
     export KUADRANT_AWS_ACCESS_KEY_ID=xxxx \
     export KUADRANT_AWS_SECRET_ACCESS_KEY=xxxx \
     export KUADRANT_ZONE_ROOT_DOMAIN=example.com \
     export KUADRANT_CLUSTER_ISSUER_NAME=self-signed

   Copy code to clipboard
   • KUADRANT_GATEWAY_NS: Namespace for your gateway in OpenShift Container Platform.
   • KUADRANT_GATEWAY_NAME: Name of your gateway in OpenShift Container Platform.
   • KUADRANT_DEVELOPER_NS: Namespace for the example Toystore app in OpenShift Container Platform. You can replace this value with the name of the application you want to use.
   • KUADRANT_AWS_ACCESS_KEY_ID: DNS provider access key ID. In this example, AWS is used. You can replace this value with your DNS provider information.
   • KUADRANT_AWS_SECRET_ACCESS_KEY: DNS provider secret access key with permissions to manage your DNS zone. In this example, AWS is used. You can replace this value with your DNS provider information.
   • KUADRANT_ZONE_ROOT_DOMAIN: The root domain associated with your DNS zone ID. In this example, the OpenShift Container Platform secret containing the credentials for the DNS provider is AWS Route53. You can replace this value with your DNS provider information.
   • KUADRANT_CLUSTER_ISSUER_NAME: Name of the certificate authority or issuer TLS certificates.

2. Create the namespace for the application by running the following command:

   $ oc create ns ${KUADRANT_DEVELOPER_NS}

   Copy code to clipboard

3. Deploy your application to the namespace you specified by running the following command:

   $ oc apply -f https://raw.githubusercontent.com/Kuadrant/Kuadrant-operator/main/examples/toystore/toystore.yaml -n ${KUADRANT_DEVELOPER_NS}

   Copy code to clipboard

   You can replace the Toystore application with the path to the one you want to use.

Procedure

1. Create the namespace you want your Gateway CR deployed in by running the following command:

   $ oc create ns ${KUADRANT_GATEWAY_NS}

   Copy code to clipboard

2. Create the secret credentials in the same namespace as the gateway namespace by running the following command:

   $ oc -n ${KUADRANT_GATEWAY_NS} create secret generic <aws-credentials> \
     --type=kuadrant.io/aws \
     --from-literal=AWS_ACCESS_KEY_ID=$KUADRANT_AWS_ACCESS_KEY_ID \
     --from-literal=AWS_SECRET_ACCESS_KEY=$KUADRANT_AWS_SECRET_ACCESS_KEY

   Copy code to clipboard

   Replace <aws-credentials> with the secret you want to use.

Procedure

1. Create a Gateway custom resource (CR), such as {KUADRANT_GATEWAY_NAME}.yaml, that has the following information:

   Example Gateway CR

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}
     namespace: ${KUADRANT_GATEWAY_NS}
     labels:
       kuadrant.io/gateway: "true"
   spec:
       gatewayClassName: openshift-default
       listeners:
       - allowedRoutes:
           namespaces:
             from: All
         hostname: "api.${KUADRANT_ZONE_ROOT_DOMAIN}"
         name: api
         port: 443
         protocol: HTTPS
         tls:
           certificateRefs:
           - group: ""
             kind: Secret
             name: api-${KUADRANT_GATEWAY_NAME}-tls
           mode: Terminate

   Copy code to clipboard

   Important

   In a multicluster environment, for Connectivity Link to balance traffic by using DNS across clusters, you must specify a gateway with a shared hostname. You can define this by using an HTTPS listener with a wildcard hostname based on the root domain.

2. Apply the Gateway CR by running the following command:

   $ oc apply -f {KUADRANT_GATEWAY_NAME}.yaml

   Copy code to clipboard

Verification

1. Check the status of your Gateway object by running the following command:

   $ oc get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Programmed")].message}'

   Copy code to clipboard

   Example output

   Resource accepted
   Resource programmed, assigned to service(s) ${KUADRANT_GATEWAY_NAME}.${KUADRANT_GATEWAY_NS}.svc.cluster.local:443

   Copy code to clipboard

2. Check the status of your HTTPS listener by running the following command:

   $ oc get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.listeners[0].conditions[?(@.type=="Programmed")].message}'

   Copy code to clipboard

   The HTTPS listener exists, but is not programmed or ready to accept traffic because you do not have valid certificates available.

Procedure

1. Before adding a TLS certificate issuer, create the secret credentials in the cert-manager namespace by running the following command:

   $ oc -n cert-manager create secret generic <dns-provider-credentials> \
     --type=kuadrant.io/aws \
     --from-literal=AWS_ACCESS_KEY_ID=$KUADRANT_AWS_ACCESS_KEY_ID \
     --from-literal=AWS_SECRET_ACCESS_KEY=$KUADRANT_AWS_SECRET_ACCESS_KEY

   Copy code to clipboard

   Replace <dns-provider-credentials> with the secret you want to use. This example uses Amazon Web Services (AWS).

2. Create a TLS certificate issuer resource, such as {KUADRANT_CLUSTER_ISSUER_NAME}.yaml, that has the following information:

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: ${KUADRANT_CLUSTER_ISSUER_NAME}
   spec:
     selfSigned: {}

   Copy code to clipboard

3. Apply the ClusterIssuer CR by running the following command:

   $ oc apply -f {KUADRANT_CLUSTER_ISSUER_NAME}.yaml

   Copy code to clipboard

Procedure

1. Create a TLSPolicy custom resource (CR) such as <toystore_tls.yaml>, that has the following information:

   Example TLSPolicy CR

   apiVersion: kuadrant.io/v1
   kind: TLSPolicy
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}-tls
     namespace: ${KUADRANT_GATEWAY_NS}
   spec:
     targetRef:
       name: ${KUADRANT_GATEWAY_NAME}
       group: gateway.networking.k8s.io
       kind: Gateway
     issuerRef:
       group: cert-manager.io
       kind: ClusterIssuer
       name: ${KUADRANT_CLUSTER_ISSUER_NAME}

   Copy code to clipboard

2. Apply your TLSPolicy CR by running the following command:

   $ oc apply -f <toystore_tls.yaml>

   Copy code to clipboard

   Replace <toystore> with the name of your TLSPolicy object.

Procedure

1. Create a DNSPolicy custom resource (CR), for example, {KUADRANT_GATEWAY_NAME}-dnspolicy.yaml, that includes the following information:

   apiVersion: kuadrant.io/v1
   kind: DNSPolicy
   metadata:
     name: <${KUADRANT_GATEWAY_NAME}-dnspolicy>
     namespace: <${KUADRANT_GATEWAY_NS}>
   spec:
     healthCheck:
       failureThreshold: 3
       interval: 1m
       path: /health
     loadBalancing:
       defaultGeo: true
       geo: GEO-NA
       weight: 120
     targetRef:
       name: <${KUADRANT_GATEWAY_NAME}>
       group: gateway.networking.k8s.io
       kind: Gateway
     providerRefs:
     - name: <dns_provider_credentials>

   Copy code to clipboard
   • Replace <{KUADRANT_GATEWAY_NAME}-dnspolicy.yaml> with the environment variable you defined.
   • Replace <{KUADRANT_GATEWAY_NS}> with the environment variable you defined.
   • spec.loadBalancing.geo: Defines a geographically relevant load balancer. In this example, GEO-NA is used. Change this to match your requirements.
   • spec.providerRefs: Replace <dns_provider_credentials> with a reference to the OpenShift Container Platform secret containing the credentials for your DNS provider.

2. Apply the DNSPolicy CR by running the following command:

   $ oc apply -f <{KUADRANT_GATEWAY_NAME}-dnspolicy.yaml> -n <gateway-namespace>

   Copy code to clipboard
   • Replace <{KUADRANT_GATEWAY_NAME}-dnspolicy.yaml> with the filename you used.
   • Replace <gateway-namespace> with name of the OpenShift Container Platform namespace that contains the gateway.

Verification

1. Verify that your DNSPolicy object has a status of Accepted and Enforced by running the following command:

   $ oc get dnspolicy <${KUADRANT_GATEWAY_NAME}-dnspolicy> -n <${KUADRANT_GATEWAY_NS}> -o=jsonpath='{.status.conditions[?(@.type=="SubResourcesHealthy")].message}'

   Copy code to clipboard
   • Replace <{KUADRANT_GATEWAY_NAME}-dnspolicy.yaml> with the filename you used.
   • Replace <{KUADRANT_GATEWAY_NS}> with the environment variable you used.
   • This process might take a few minutes.

2. Check the status of the DNS health checks that are enabled on your DNS policy by running the following command:

   $ oc get dnspolicy <${KUADRANT_GATEWAY_NAME}-dnspolicy> -n <${KUADRANT_GATEWAY_NS}> -o=jsonpath='{.status.conditions[?(@.type=="SubResourcesHealthy")].message}'

   Copy code to clipboard
   • Replace <{KUADRANT_GATEWAY_NAME}-dnspolicy.yaml> with the filename you used.
   • Replace <{KUADRANT_GATEWAY_NS}> with the environment variable you used.
   • These health checks flag a published endpoint as healthy or unhealthy based on the defined configuration. When unhealthy, an endpoint is not published if it has not already been published to the DNS provider. An endpoint is only unpublished if it is part of an A record that has multiple values. You can see endpoint status in all cases in the DNSPolicy object status.

Procedure

1. Create a default AuthPolicy CR, such as gateway_name-auth.yaml, with a deny-all setting for your Gateway object:

   Example AuthPolicy CR

   apiVersion: kuadrant.io/v1
   kind: AuthPolicy
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}-auth
     namespace: ${KUADRANT_GATEWAY_NS}
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: ${KUADRANT_GATEWAY_NAME}
     defaults:
      when:
        - predicate: "request.path != '/health'"
      rules:
       authorization:
         deny-all:
           opa:
             rego: "allow = false"
       response:
         unauthorized:
           headers:
             "content-type":
               value: application/json
           body:
             value: |
               {
                 "error": "Forbidden",
                 "message": "Access denied by default by the gateway operator. If you are the administrator of the service, create a specific auth policy for the route."
               }

   Copy code to clipboard

2. Apply your HTTPRoute CR by running the following command:

   $ oc apply -f <gateway_name-auth.yaml>

   Copy code to clipboard
   • Replace <gateway_name-auth.yaml> with the name of your HTTPRoute CR.

Verification

1. Check that your AuthPolicy has Accepted and Enforced status by running the following command:

   $ oc get authpolicy ${KUADRANT_GATEWAY_NAME}-auth -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Enforced")].message}'

   Copy code to clipboard

   Example output

   AuthPolicy has been accepted
   AuthPolicy has been successfully enforced

   Copy code to clipboard

2. Test your AuthPolicy by running the following curl command:

   $ curl -H 'Authorization: APIKEY ndyBzreUzF4zqDQsqSPMHkRhriEOtcRx' http://talker-api.127.0.0.1.nip.io:8000/hello

   Copy code to clipboard

   Example output

   HTTP/1.1 200 OK

   Copy code to clipboard

Procedure

1. Create a default RateLimitPolicy custom resource (CR), for example, gateway_name-rlp.yaml, that has the following information:

   Example RateLimitPolicy CR

   apiVersion: kuadrant.io/v1
   kind: RateLimitPolicy
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}-rlp
     namespace: ${KUADRANT_GATEWAY_NS}
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: ${KUADRANT_GATEWAY_NAME}
     defaults:
       limits:
         "low-limit":
           rates:
           - limit: 1
             window: 10s

   Copy code to clipboard

   • A low-limit value is used in this example for the ease of testing the CR. Configure the spec.defaults.limits: values that makes in your use case.

2. Apply your RateLimitPolicy CR by running the following command:

   $ oc apply -f <gateway_name-rlp.yaml>

   Copy code to clipboard
   • Replace <gateway_name-rlp.yaml> with the name of your RateLimitPolicy YAML.

Verification

1. Check that your RateLimitPolicy has Accepted and Enforced status by running the following command:

   $ oc get ratelimitpolicy ${KUADRANT_GATEWAY_NAME}-rlp -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Enforced")].message}'

   Copy code to clipboard

   Example output

   RateLimitPolicy has been accepted
   RateLimitPolicy has been successfully enforced

   Copy code to clipboard

2. Test your rate-limiting by running the following curl command:

   $ while :; do curl -k --write-out '%{http_code}\n' --silent --output /dev/null  "https://api.$KUADRANT_ZONE_ROOT_DOMAIN/cars" | grep -E --color "\b(429)\b|$"; sleep 1; done

   Copy code to clipboard

   HTTP 403 responses are expected.

1. On an incoming request, the gateway evaluates the matching rules and predicates from the TokenRateLimitPolicy resources.
2. If the request matches, the gateway prepares the necessary rate limit descriptors and monitors the response.
3. After receiving the response, the gateway extracts the usage.total_tokens field from the JSON response body.
4. The gateway then sends a RateLimitRequest to Limitador, including the actual token count as a hits_addend.
5. Limitador tracks the cumulative token usage and responds to the gateway with OK or OVER_LIMIT.

Procedure

1. Create a TokenRateLimitPolicy YAML, for example, tokenratelimitpolicy.yaml that includes the following information:

   Example TokenRateLimitPolicy YAML

   apiVersion: kuadrant.io/v1alpha1
   kind: TokenRateLimitPolicy
   metadata:
     name: llm-protection
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: ai-gateway
     limits:
       free-users:
         rates:
           - limit: 10000
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions"
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "free")
         counters:
           - expression: auth.identity.userid
       pro-users:
         rates:
           - limit: 100000
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions"
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "pro")
         counters:
           - expression: auth.identity.userid

   Copy code to clipboard

   • Choose a filename for the policy that makes sense in your environment.
   • spec.limits.free-users.rates.limit: 10,000 tokens per day for free tier.
   • spec.limits.free-users.when.predicate: Set to inference traffic only.
   • pro-users.rates.limit: 100,000 per day for the pro user.
   • spec.limits.pro-users.when.predicate: Set to inference traffic only.

2. Apply the TokenRateLimitPolicy policy by running the following command:

   $ oc apply -f <tokenratelimitpolicy.yaml> -n <gateway_namespace>

   Copy code to clipboard

   Replace <tokenratelimitpolicy.yaml> with the filename you used. Replace <gateway-namespace> with your gateway namespace.

3. Check the status of the policy to ensure it was accepted and enforced on the target HTTPRoute. Look for conditions with type: Accepted and type: Enforced with status: "True".

   $ oc get <tokenratelimitpolicy.yaml> llm-protection -n <gateway-namespace> -o jsonpath='{.status.conditions}'

   Copy code to clipboard

   Replace <tokenratelimitpolicy.yaml> with the filename you used. Replace <gateway_namespace> with your gateway namespace.

4. Send requests to your API endpoint, including the required authentication details, by running the following command:

   $ curl -H "Authorization: <auth_policy>" \
        -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}' \
        <api_endpoint>

   Copy code to clipboard

   Replace <auth_policy> and <api_endpoint> with your values.

Procedure

1. Create an HTTPRoute custom resource (CR), such as <toystore>-route.yaml, that has the following information:

   Example HTTPRoute CR

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: <toystore>
     namespace: <${KUADRANT_DEVELOPER_NS}>
     labels:
       deployment: <toystore>
       service: <toystore>
   spec:
     parentRefs:
       - name: <${KUADRANT_GATEWAY_NAME}>
         namespace: <${KUADRANT_GATEWAY_NS}>
     hostnames:
       - api.${KUADRANT_ZONE_ROOT_DOMAIN}
     rules:
       - matches:
           - method: GET
             path:
               type: PathPrefix
               value: /cars
           - method: GET
             path:
               type: PathPrefix
               value: /health
         backendRefs:
           - name: <toystore>
             port: 80

   Copy code to clipboard

   • Replace <toystore> with the name of your application.
   • metadata.namespace: The namespace in which you are deploying your application. Replace <${KUADRANT_DEVELOPER_NS}> with the environment variable you used during installation.
   • spec.parentRefs.name and spec.parentRefs.namespace: The values must match the Gateway object.
   • spec.hostnames: The hostname must match the one specified in the Gateway object.
   • hostname.rules.matches.backendRefs.name: The name of the Service for your application.

2. Apply your HTTPRoute CR by running the following command:

   $ oc apply -f <toystore>-route.yaml

   Copy code to clipboard

   Replace <toystore> with the name of your application.

   Example output

   httproute.gateway.networking.k8s.io/toystore-route created

   Copy code to clipboard

   The output indicates that the route to the application exists.

Verification

1. Verify that the HTTPRoute is created by running the following command:

   $ oc get httproute <toystore> -n kuadrant -o=jsonpath='{.status.parents[].conditions[?(@.type=="Accepted")].message}{"\n"}{.status.parents[].conditions[?(@.type=="ResolvedRefs")].message}'

   Copy code to clipboard

   Replace <toystore> with the name of your application.

   Example output

   Route was valid

   Copy code to clipboard

2. If you have DNSPolicy and TLSPolicy objects applied, you can validate that your backend is reachable by running the following command:

   $ curl -k  https://api.${KUADRANT_ZONE_ROOT_DOMAIN}:443/cars

   Copy code to clipboard

   Note that the example TLSPolicy CR uses a self-signed ClusterIssuer object.

Procedure

1. Create a new RateLimitPolicy custom resource (CR) in a different namespace to override the default low-limit policy and set rate limits for specific users by using the following example:

   Example RateLimitPolicy CR for specific users

   apiVersion: kuadrant.io/v1
   kind: RateLimitPolicy
   metadata:
     name: toystore-rlp
     namespace: ${KUADRANT_DEVELOPER_NS}
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: HTTPRoute
       name: toystore
     limits:
       "general-user":
         rates:
         - limit: 5
           window: 10s
         counters:
         - expression: auth.identity.userid
         when:
         - predicate: "auth.identity.userid != 'bob'"
       "bob-limit":
         rates:
         - limit: 2
           window: 10s
         when:
         - predicate: "auth.identity.userid == 'bob'"

   Copy code to clipboard

2. Apply the CR by running the following command:

   $ oc apply -f <toystore-rlp>

   Copy code to clipboard
   • Replace <toystore-rlp> with the name of your YAML.
   • Wait a few minutes for the RateLimitPolicy CR to be applied.

3. Check that the RateLimitPolicy has a status of Accepted and Enforced by running the following command:

   $ oc get ratelimitpolicy -n ${KUADRANT_DEVELOPER_NS} toystore-rlp -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Enforced")].message}'

   Copy code to clipboard

4. Check that the status of the HTTPRoute is now affected by the RateLimitPolicy CR in the same namespace:

   $ oc get httproute toystore -n ${KUADRANT_DEVELOPER_NS} -o=jsonpath='{.status.parents[0].conditions[?(@.type=="kuadrant.io/RateLimitPolicyAffected")].message}'

   Copy code to clipboard

Verification

1. Send requests as user alice by running the following command:

   $ while :; do curl -k --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMALICE' "https://api.$KUADRANT_ZONE_ROOT_DOMAIN/cars" | grep -E --color "\b(429)\b|$"; sleep 1; done

   Copy code to clipboard

   The expected outcome is an HTTP status 200 every second for 5 seconds, followed by HTTP status 429 every second for 5 seconds.

2. Send requests as user bob by running the following command:

   $ while :; do curl -k --write-out '%{http_code}\n' --silent --output /dev/null -H 'Authorization: APIKEY IAMBOB' "https://api.$KUADRANT_ZONE_ROOT_DOMAIN/cars" | grep -E --color "\b(429)\b|$"; sleep 1; done

   Copy code to clipboard

   The expected outcome is an HTTP status 200 every second for 2 seconds, followed by HTTP status 429 every second for 8 seconds.