F5 AI Gateway Quickstart¶

This document provides the steps to help you get started with F5 AI Gateway. We will configure AI Gateway to route OpenAI chat completion requests to the OpenAI.

Before you begin¶

Follow the Install with Helm documentation.

The guide assumes the chart was installed using the following command:

helm upgrade aigw oci://private-registry.f5.com/aigw/aigw \
  -n ai-gateway \
  --set "imagePullSecrets[0].name=f5-registry-secret"

Configure a service¶

To run F5 AI Gateway, you need one or more LLM Services.

In this guide we use the OpenAI API.

services:
  - name: openai/public
    type: gpt-4o
    executor: openai
    config:
      endpoint: "https://api.openai.com/v1/chat/completions"
      secrets:
        - source: EnvVar
          targets:
            apiKey: OPENAI_API_KEY

Services like OpenAI require an API key. The API key can be configured with a Kubernetes secret.

Create a Kubernetes secret that contains a OpenAI API key, in place of my-api-key.

kubectl create secret -n ai-gateway generic openai-secret --from-literal="OPENAI_API_KEY=<my-api-key>"

Configure the Helm chart my-values.yaml to expose the secret to the aigw deployment.

# my-values.yaml
aigw:
  env:
    - name: OPENAI_API_KEY
      valueFrom:
        secretKeyRef:
          name: openai-secret
          key: OPENAI_API_KEY

Upgrade the Helm deployment with an updated my-values.yaml file.

helm upgrade aigw oci://private-registry.f5.com/aigw/aigw \
  -n ai-gateway \
  --set "imagePullSecrets[0].name=f5-registry-secret" \
  -f my-values.yaml

Create the config file¶

Create a aigw.yaml configuration that adds a route to the OpenAI service:

version: 1

server:
  address: :4141

routes:
  - path: /demo-endpoint
    policy: demo-policy     # Maps a route to policy
    schema: v1/chat_completions # Use the AI Gateway common schema for requests

policies:
  - name: demo-policy
    profiles:
      - name: demo-profile  # Maps a policy to a profile

profiles:
  - name: demo-profile
    services:
      - name: openai/public # Sends traffic to the OpenAI API

services:
  - name: openai/public
    type: gpt-4o
    executor: openai
    config:
      endpoint: "https://api.openai.com/v1/chat/completions"
      secrets:
        - source: EnvVar
          targets:
            apiKey: OPENAI_API_KEY

Apply the aigw.yaml configuration, by upgrading the Helm deployment.

Using the aigw.yaml file with the Helm CLI¶

Apply the aigw.yaml configuration by using the --set-file flag, which will consume the contents of the file and update the deployment.

helm upgrade aigw oci://private-registry.f5.com/aigw/aigw \
  -n ai-gateway \
  --set "imagePullSecrets[0].name=f5-registry-secret" \
  --set-file "config.contents=aigw.yaml"

Using values.yaml¶

Specify the contents of the aigw.yaml file in my-values.yaml:

# my-values.yaml
aigw:
  env:
    - name: OPENAI_API_KEY
      valueFrom:
        secretKeyRef:
          name: openai-secret
          key: OPENAI_API_KEY

config:
  contents: |
    version: 1

    server:
      address: :4141

    routes:
      - path: /demo-endpoint
        policy: demo-policy     # Maps a route to policy
        schema: openai          # Uses the OpenAI API JSON schema for requests

    policies:
      - name: demo-policy
        profiles:
          - name: demo-profile  # Maps a policy to a profile

    profiles:
      - name: demo-profile
        services:
          - name: openai/public # Sends traffic to the OpenAI API

    services:
      - name: openai/public
        type: gpt-4o
        executor: openai
        config:
          endpoint: "https://api.openai.com/v1/chat/completions"
          secrets:
            - source: EnvVar
              targets:
                apiKey: OPENAI_API_KEY

Upgrade the Helm deployment with an updated my-values.yaml file.

helm upgrade aigw oci://private-registry.f5.com/aigw/aigw \
  -n ai-gateway \
  --set "imagePullSecrets[0].name=f5-registry-secret" \
  -f my-values.yaml

Test the AI Gateway¶

Use kubectl port-forward to proxy to the aigw Kubernetes service to send requests.

kubectl port-forward -n ai-gateway svc/aigw 4141:80  # Port 80 is the default port for svc/aigw

Alternatively, configure a LoadBalancer or Ingress in front of the AI Gateway to make these requests.

Confirm that a prompt is proxied to the OpenAI service.

curl -X POST localhost:4141/demo-endpoint \
  --header 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'

If the request fails or returns an empty string, review the aigw deployment logs for any error messages. Access the aigw deployment logs with kubectl.

kubectl logs -n ai-gateway deployments/aigw

Unsuccessful error log, due to the API key being invalid:

2024/12/04 02:13:32 INFO AI Gateway successfully started
2024/12/04 02:13:32 INFO aigw main server (HTTP) starting addr=:4141
2024/12/04 02:13:32 INFO successfully reported entitlement data
2024/12/04 02:30:41 INFO service selected name=openai/gpt-4o
2024/12/04 02:30:41 INFO executing openai service type=gpt-4o
2024/12/04 02:08:55 INFO sending to service endpoint=https://api.openai.com/v1/chat/completions
2024/12/04 02:08:56 ERROR failed to selectAndExecuteService: failed to serviceHandler.Process: service returned status code 401,
  message "{\"message\":\"Incorrect API key provided: my-api-key. You can find your API key at
  https://platform.openai.com/account/api-keys.\",\"type\":\"invalid_request_error\",\"param\":null,\"code\":\"invalid_api_key\"}"

Successful log:

2024/12/04 02:13:32 INFO AI Gateway successfully started
2024/12/04 02:13:32 INFO aigw main server (HTTP) starting addr=:4141
2024/12/04 02:13:32 INFO successfully reported entitlement data
2024/12/04 02:30:41 INFO service selected name=openai/gpt-4o
2024/12/04 02:30:41 INFO executing openai service type=gpt-4o
2024/12/04 02:30:41 INFO sending to service endpoint=https://api.openai.com/v1/chat/completions
2024/12/04 02:30:58 INFO service response name=openai/gpt-4o result="map[status:200 OK]"

Previous Next