Configure selectors¶

Selectors allow you to add conditional logic to your LLM workflows.

A basic selector in the services section of the profile looks like this:

services:
  - name: openai/public
    selector:
      operand: not
      tags:
        - language:en
  - name: english-only-llm

This means that the service called openai/public, which we will call the “configuration object”, is chosen if the request has not been tagged as an English language request.

If the condition in the selector does not match (it has tag language:en), then the configuration object english-only-llm will be used.

          graph LR
      A[request] --> B{"`has tag <code>language:en</code>`"}
      B -- no --- C[openai/public]
      B --> D[english-only-llm]

Selectors always attach to a “configuration object”. The valid objects are:

Configuration Object	Description	Examples
`profile`	In the policies, selectors are used to select the `profile` to which the policy is routed.	With: request headers JSON Web Tokens
`stage`	In the profiles section, selectors are used to choose how request flows through stages.	Example in the profiles documentation.
`service`	In the profiles, selectors are used to select the `service` (LLM) to which the request will be routed.	Example in the profiles documentation.

Configuration Object

Description

Examples

profile

In the policies, selectors are used to select the profile to which the policy is routed.

With:

stage

In the profiles section, selectors are used to choose how request flows through stages.

Example in the profiles documentation.

service

In the profiles, selectors are used to select the service (LLM) to which the request will be routed.

Example in the profiles documentation.

Fields¶

Valid selector fields vary depending on the type of the selector. Additionally, certain type values are only valid in certain contexts. To learn more about the different rules selector fields and values, see the specific sections on each selector type:

Header selectors
JWT selectors
Tag selectors
Model selectors

A general description of available fields:

Setting	Description	Required	Examples
`type`	Describes the a category of values to be considered by this selector. If `type` is not specified, then `tags` is expected. Allowable values: `header` `jwt` `input.model`	no	`type: jwt`
`operand`	The logic that should be applied to the `values` or `tags`. See operands. Defaults to `any`.	No	`operand: and`
`key`	The `key` for the selected `type`. For `header` type selectors it is the header name, for `jwt` type selectors it is the name of the claim.	If `type: header` or `type: jwt`	`key: User-Role`
`values`	An array of values that will be compared using `operand`. The source of the value differs by `type`: `type: header`: A list of header values for the header name in `key` `type: jwt`: A list of values for the `aud` claim `type: input.model`: A list of model names from the set defined in the `models` key of the profile	If `type` is specified	`values:` `- admin`
`tags`	If `type` is not given, `tags` should be specified as an array of tags added to the request by processors on the input or response stage of the request. See Tag selectors	If `type` is not specified	`tags:` `- language:en`

Operands¶

An operand describes how values or tags will be evaluated to determine whether the parent configuration object is “selected” or not. The following operands are supported:

or/any, and/all, not/none

If no operand is provided, AI Gateway defaults to any.

In the below example, the service llama-de-ja will be used if the tags language:ja or language:de are present:

services:
  - name: llama-de-ja
    selector:
      operand: or
      tags:
        - language:ja
        - language:de

Selector logic and ordering¶

In the areas of the configuration where selectors can be used, each configuration object will be evaluated in the order specified. The first configuration object with no selector or with a matching selector will be chosen. For example:

services:
  - name: en-llm
    selector:
      operand: not
      tags:
        - language:ja
        - language:de
  - name: multilingual-llm

The behavior of the above configuration is:

If the request has not been tagged as a Japanese or German language conversation, send it to the service called en-llm. Otherwise, send it to the service called multilingual-llm.

However, look at what happens if the order is changed to move multilingual-llm, a service reference without a selector, above the en-llm service with a selector:

services:
  - name: multilingual-llm
  - name: en-llm
    selector:
      operand: not
      tags:
        - language:ja
        - language:de

In this case, multilingual-llm will always be used, regardless of which tags are added to the request.

Note

When using selectors, follow these guidelines for predictable operation:

If possible, make sure there is a “default” case. A configuration object with no selectors that comes last.
Make sure all of your configuration objects with selectors come first in the list.
Place selectors that are more specific first. For example, place a service that uses the and operand with many conditions before services with less restrictive selectors.

Header selectors¶

Note

Header selectors can only be used in the policies section.

A selector with type: header allows selection of a profile using the HTTP request headers.

This type of selector requires that key and values are also specified:

key: The name of the header. This can be any value - custom headers like Role or standard headers like Accept-Language are both acceptable. Key selection is case insensitive: key: role will get values from the header Role.
values: The potential values of the header. The values will be compared using operand.

selector:
  type: header
  operand: any
  key: Role
  values:
    - admin
    - superuser

On header value parsing¶

A header specified multiple times in the request can satisfy the and condition. Given the following selector:

selector:
  type: header
  operand: and
  key: Accept-Language
  values:
    - ja
    - de

Headers given like this will satisfy the condition:

Accept-Language: ja
Accept-Language: de

However, a single header with comma or newline separated values will not be evaluated as having multiple values and may fail and conditions. Both of the following will not satisfy the condition:

Accept-Language: ja, de

Accept-Language: ja\nde

JWT selectors¶

Note

JWT selectors can only be used in the policies section.

The type: jwt selector uses the claims in a JWT token sent in the Authorization header of your HTTP request (with or without bearer) to select the appropriate profile based on the JWT claims. See JWT Authentication Configuration in the policies documentation for more detail.

Currently the only key that is supported is the aud claim in the JWT.

Example:

selector:
  type: jwt
  key: aud
  values:
    - admin.aud

Tag selectors¶

Note

Tag selectors can only be used to select a stage or a service in the profile context.

Selectors with no type specified allow the selection of processing stages or services based on tags added to the request as it moves through the request or response cycle in the AI Gateway.

If the type key is not specified, the tags key must then be specified. Processors add tags to the request. For example, the language-id processor will add the language tag. That tag can then be referenced in selectors coming after the invocation of the language-id processor.

To understand which tags will be added by a given processor, see the “tags” section in that processor’s documentation.

Example:

selector:
  operand: not
  tags:
    - language:ja

Model selectors¶

Note

Model selectors can only be used to select a stage or a service in the profile context.

The type: input.model selector requires that the values key is specified.

Valid values are those specified under the models key in a profile routed to by a route using the schema v1/models. For more details, see Selecting a service based on a model, although the input.model type selector can also be used to select input and output stages.

Example:

selector:
  type: input.model
  values:
    - best

Errors¶

If any selector fields are incorrectly specified, an error will appear in the logs and AI Gateway will fail to apply the configuration.

Although not an error case, be aware that it is valid to create a configuration in which all configuration objects have selectors and a request does not match any of the selectors.

In such cases, you may see an error message like:

{
  "error":
  {
    "type": "resource_not_found",
    "message": "no service selected"
  }
}

Previous Next