Processors overview¶

Processors are components that a gateway interacts with in order to change the flow of data between an inbound request and an outbound response. Processors are steps (or commands) in a chain. Processors evaluate requests and responses and return a status to the gateway indicating if the requested prompt should proceed. In addition to gating flow, processors may also change the request or response data so that the next item in a processor chain has a different state. For example, an implementation could change the word “cat” to “dog” for every request.

Processor server¶

A processor server is an application which serves one or more processors.

Processor parameters¶

Processors define configuration options which are provided by F5 AI Gateway on each request. For example a processor which uses a machine learning model to detect if a user-provided prompt is malicious may accept a threshold parameter, which sets the confidence threshold that the processor needs to reach before taking action.

Common parameters¶

All processors support the following parameters.

Parameters	Description	Type	Required	Defaults
`annotate`	Whether the processor can annotate the input with tags. It is recommended to leave this option as `true`.	boolean	No	`true`
`modify`	Whether the processor can modify the input. Set this to `true` when the processor should modify the input.	boolean	No	`false`
`reject`	Whether the processor can reject requests. Set this to `true` when the processor should reject certain inputs.	boolean	No	`false`

If any of these parameters are set to false, the processor is not allowed to carry out that action. This allows AI Gateway to make decisions about scheduling requests to processors based on the actions the processor is allowed to take.

This also ensures that processors cannot take actions which have not been approved by the administrator. The modify and reject parameters are mutually exclusive.

As a result, some processors will fail to start if they require a specific parameter to be enabled. For example if a processor’s only function is to add disclaimer to output from an LLM, it will fail to start without the modify parameter being set to true.

Processors and tags¶

If a processor is configured with annotate: true, it may add tags to the request or response.

As a request or response moves through the processor pipeline it can collect multiple tags from the processors that are applied to it.

Tags are used by the AI Gateway core for the following purposes:

To determine which stages should be applied to a request or response
To determine which AI model should receive a request

Tags may also be used when performing data analysis on audit logs exported by AI Gateway.

The tags that a processor may produce are defined in the documentation for that processor.

Tags are strings and generally follow the format key:value. For example:

language:en
attacks-detected:prompt-injection

A processor may produce multiple values for a single key. For example a prompt containing multiple languages may result in both language:en and language:ja.

See Tag selectors for detailed documentation about the usage of tags in conditionals.

Processor configuration optimization¶

When configuring the inputStages and responseStages, the order that processors are placed in steps can have a significant impact. The following recommendations can help to ensure that your processors behave as intended.

Reject early¶

Processors that can potentially reject a request should be placed in in early steps. This prevents wasted processing time by ensuring processors which modify the request do not run unless they need to.

Modify late¶

Processors which modify the request should be placed towards the end of the stage steps. This ensures that there is no confusion between user input and processor input when reject processors are inspecting prompts. For example, if the system prompt processor was placed in an earlier stage than the prompt injection processor, the system prompts may be misidentified as prompt injection attacks.

Deterministic vs non-deterministic processors¶

Processors can be classified as deterministic or non-deterministic.

Deterministic processors always produce the same result, the underlying hardware or environment have no bearing on the output.

Non-deterministic processors can yield different model outputs from the same input. In exchange, they allow us to gain insights on the provided input that are not possible or very difficult using deterministic methods. A non-deterministic processor’s outputs may vary as a result of:

GPU batch inference due to thread scheduling or GPU operations
Changes to the environment or underlying hardware

These small numerical differences in model output from run to run should not be significant enough to change classification results of the processor.

Previous Next