Running AI Gateway in a container¶

Before you begin¶

To run F5 AI Gateway in a container, you will need Docker installed on your machine.

To run F5 AI Gateway, you need one or more processors and one or more pre-configured services to run.

Install and configure the processors and services that you plan to use before running AI Gateway.

Creating an AI Gateway Docker image¶

To create an AI Gateway Docker image, you will need a Dockerfile and an AI Gateway configuration file.

Note

To create the Docker image used in the example, you need to have access to the F5 AI Gateway ECR repository. Use the instructions provided by F5 to authorize your connection to the container registry.

Configure your environment¶

Once your F5 partner has provided the required credentials, you can configure your environment to access the F5 AWS account.

Install the AWS CLI v2 on your machine. For more information, see the AWS CLI v2 installation guide.
On your terminal, run the following command:
```
aws configure sso
```

Add a SSO session name, SSO start URL, and SSO region when prompted. For example:

SSO session name (Recommended): your-session-name
SSO start URL [none]: https://<the URL provided by your F5 contact>
SSO region [None]: <the region provided by your F5 contact>

Leave the scope empty.

Confirm that the verification code in your browser matches the code displayed in the terminal.

Log in to the ECR repository¶

To log in to the ECR repository, run the following command:
```
aws configure
```
Note

When you run aws configure sso there is an option to set a profile name. If you leave it empty it will be set as default , requiring no additional CLI flags. If you add a profile name when configuring, like my-example, you will need to prefix all aws calls with --profile my-example.
Enter the AWS access key ID, secret access key, and region provided by F5, when prompted.
Verify that you are logged in to the ECR repository:
```
aws ecr get-login-password --region us-east-1
```
which should return the Docker token.

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 616653012569.dkr.ecr.us-east-1.amazonaws.com

Dockerfile¶

Create a Dockerfile file with the following content:

FROM 616653012569.dkr.ecr.us-east-1.amazonaws.com/ai-gateway/aigw:v0.0.2

COPY aigw.yaml ./

ENTRYPOINT ["aigw", "start", "./aigw.yaml"]

Configuration file¶

The configuration file is a YAML file that defines the AI Gateway configuration.

To learn more about the configuration file, see the Configuration file documentation. You can find an example configuration file in the root of the repository, (aigw.yaml).

Create your configuration file and save it as aigw.yaml in the same directory as your Dockerfile.

Build the Docker image¶

Build the Docker image using the following command:

   docker build . -t containerdemo:dev

Example output:

   [+] Building 7.3s (8/8) FINISHED
    => [internal] load build definition from Dockerfile                        0.1s
    => => transferring dockerfile: 133B                                        0.0s
    => [internal] load metadata for 616653012569.dkr.ecr.us-east-1.amazonaws.com/ai-gateway/aigw:v0.0.2  3.0s
    => [auth] nginxinc/aigw:pull token for 616653012569.dkr.ecr.us-east-1.amazonaws.com          0.0s
    => [internal] load .dockerignore                                           0.0s
    => => transferring context: 2B                                             0.0s
    => [internal] load build context                                           0.0s
    => => transferring context: 362B                                           0.0s
    => [1/2] FROM 616653012569.dkr.ecr.us-east-1.amazonaws.com/ai-gateway/aigw:v0.0.2@sha256:424a0dab[...]464c3280      3.5s
    => => resolve 616653012569.dkr.ecr.us-east-1.amazonaws.com/ai-gateway/aigw:v0.0.2@sha256:42d7741a[...]464c3280      0.0s
    => => sha256:34382245fdf29174c880[...]4ae56490 674B / 674B                 0.0s
    => => sha256:1de76bdedb75aefbcab9[...]07e4be0f1439 1.40kB / 1.40kB         0.0s
    => => sha256:09f376ebb190216b0459[...]eae 29.15MB / 29.15MB                1.8s
    => => sha256:837578bc343b11b9a80c[...]f593f1a 5.30MB / 5.30MB              0.7s
    => => sha256:424a0dab2e7d29c65caa[...]64c3280 1.61kB / 1.61kB              0.0s
    => => extracting sha256:09f376ebb[...]6b0459f470e71e                       1.3s
    => => extracting sha256:837578bc3[...]b95b32e0cc90f593f1a                  0.1s
    => [2/2] COPY aigw.yaml ./                                                 0.6s
    => exporting to image                                                      0.1s
    => => exporting layers                                                     0.0s
    => => writing image sha256:b7bc73dca4e3389032[...]cdf7977c56f673a8d        0.0s
    => => naming to docker.io/library/containerdemo:dev

Run the AI Gateway Docker container¶

Run AI Gateway in a container using the following command:

   docker run -it -p 4141:4141 containerdemo:dev

Example output:

   2024/06/06 14:35:16 INFO -- Starting: {{product}} --
   2024/06/06 14:35:16 INFO Initializing no-op tracing provider
   2024/06/06 14:35:16 INFO loading service name=ollama/phi executor=ollama type=phi3
   2024/06/06 14:35:16 INFO mounting route path=/ollama
   2024/06/06 14:35:16 INFO successfully applied config
   2024/06/06 14:35:16 INFO aigw main server (HTTP) starting addr=:4141
   2024/06/06 14:35:16 INFO aigw admin server started addr=localhost:8080

Expose the AI Gateway admin server¶

You can expose the AI Gateway admin server to allow external access to the admin API. This will allow you to apply and reload configurations using the admin API.

To expose the admin server, remove the localhost address from the adminServer configuration in your aigw.yaml file before building the Docker image:

   adminServer:
     address: :8080

Follow the steps from the previous section to build the Docker image.

Run the AI Gateway Docker container with the following command:

   docker run -it -p 4141:4141 -p 8080:8080 containerdemo:dev

You can now update and reload the configuration using the admin API.

Previous Next