Design considerations¶

This page outlines a set of recommendations designed to help developers create processors which behave consistently for users of F5 AI Gateway.

Single responsibility¶

We recommend designing processors to have a single responsibility. This allows owners of the AI Gateway configuration file to easily understand and control how an input or response will be impacted as it flows through the different stages of the request. When designing processors which connect with a multipurpose system, consider creating a processor for each individual use case of the system.

For example, if you have an existing application which can handle both language translation and sentiment analysis, exposing these capabilities as individual processors allows for flexibility of composition. These individual processors could be hosted in the same processor server to avoid running multiple instances of the system.

Handling the conversation history¶

Processors are designed to receive the complete conversation history that is sent by the client. Consider the following example:

User: What is the capital of France?
Assistant: The capital of France is Paris.
User: How many people live there?

When a processor runs in an input stage:

During the first request, it processes message 1.
On the second request, the processor receives all preceding messages. This includes two messages with role: user and one with role: assistant. The client typically sends the list of messages in chronological order.

This behavior ensures that processors have access to the necessary context to handle conversations accurately. A few reasons why this approach is used include:

Context Awareness:
Certain processors will require the full conversation’s context to correctly interpret a user’s message. For instance, the user might refer to a previously benign instruction and request the opposite action in a later message. Without context, such nuances could be missed.
Security:
Past messages in the conversation could potentially be maliciously modified before reaching AI Gateway. Providing the entire message history allows processors to verify the integrity of the conversation.

Conversation processing optimizations¶

The way message history is processed depends on the processor developer and the specific use case. Here are some strategies:

Selective Examination:

Depending on your use case, it can be sufficient to focus only on the most recent message or specific messages with a particular role.
Performance Optimization in Rejection Scenarios:

For processors designed to reject messages, it can improve performance to process messages in reverse chronological order. Since the most recent message is often the most likely to trigger a rejection, this allows for early termination of processing in such scenarios, avoiding unnecessary examination of earlier messages.

Threshold parameters for non-deterministic processors¶

When developing non-deterministic processors it is recommended to expose a threshold parameter which can be used to evaluate if the confidence of a prediction or classification made by a model meets the required level. This will allow the user to make the processor more or less sensitive based on their use case.

Annotating requests with tags¶

Each tag can have multiple values. An example of this is the language identification setting multiple values for language, such as en and fr.

It is also important to note that values can be added to the same tag by multiple processors. This means that if you want to develop a dedicated language processor which detects a language not covered by the existing language identification processor, both could add values to the language tag. Here is an example showing an hypothetical processor which adds language:ga if it detects the Irish language (Gaeilge), and a stage that runs if either English or Irish is detected.

profiles:
  - name: demo
    services:
      - name: echo
    inputStages:
      - name: check-langauge
        steps:
          - name: language-id
          - name: language-id-irish
      - name: en-or-ga-stage
        selector:
          operand: or
          tags:
            - language:en
            - language:ga
        steps:
          - name: other-processor

Common tags¶

This table covers some common tags currently in use by processors.

Tag	Processors
`attacks-detected`	prompt injection repetition detect
`language`	language identification

Handling unsupported parameters¶

It is possible that certain combinations of parameter settings will result in your processor not being able to operate. For example, in the case of a processor which removes an offensive word, it will not be able to take any action when run with modify: false. In cases like this you will need to decide if you can still provide value by adding annotations about actions you would have taken. In the word replacement example, there can still be value in adding a tag such as validation-failures: offensive, which will allow the AI Gateway administrator to make routing decisions based on your processors output instead of blocking the request.

If a combination of parameters are provided with which it is not possible to execute your processor, consider raising a validation error on in the parameters initialization, such as this example with the Python SDK.

Previous Next