Agentic Data Plane

Configure an LLM Provider

Create an LLM provider to give your applications a managed proxy URL. Redpanda handles the upstream API keys, forwards requests to the provider, and records usage for you. Create a provider for each upstream you use, whether that’s OpenAI, Anthropic, Google Gemini API, Amazon Bedrock, Gemini Enterprise Agent Platform (formerly Vertex AI), Fireworks, or an OpenAI-compatible endpoint.

After reading this page, you will be able to:

  • Create an LLM provider for OpenAI, Anthropic, Google Gemini API, Amazon Bedrock, Gemini Enterprise Agent Platform, Fireworks, or an OpenAI-compatible endpoint

  • Select the models you want to expose through the provider

  • Verify the provider is reachable using the built-in Test connection control

Prerequisites

  • An API key for the upstream provider you want to configure, AWS credentials for Bedrock, or a Google Cloud Platform (GCP) service account key for Gemini Enterprise Agent Platform.

  • Optional: Secrets already created in the Redpanda Agentic Data Plane secret store for the provider’s credentials. The create form can also store a new API key inline, so you don’t have to prepare a secret first. Secret identifiers use UPPER_SNAKE_CASE. For example: OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID.

Open the Create LLM provider page

  1. Open LLM providers in the sidebar.

  2. Click Add provider.

LLM providers list with search, filters, an Add provider button, and rows of status, models, tags, requests, and spend.
Figure 1. The LLM providers list, with the Add provider button

Fill in the identity fields

The Identity section collects the provider’s name. Enter a Display name: a human-readable label shown in dashboards and model selectors, up to 253 characters.

Agentic Data Plane derives the provider’s resource ID from the display name in the background. The create form doesn’t show it. The resource ID is the machine identifier that rpk ai commands use. It starts with a lowercase letter, contains only lowercase letters, numbers, and hyphens, ends with a letter or number (^[a-z]([a-z0-9-]{0,61}[a-z0-9])?$), is at most 63 characters, and is immutable after creation. It appears in the proxy URL (/llm/v1/providers/<resource-id>/…​).

Choose a provider type

Select the provider type that matches your upstream service. Each tile carries the type’s name and icon and a one-line summary of what it reaches. Hover over a tile, or select it, to show a Docs link to the upstream’s own API documentation.

Type Use when

OpenAI

Proxy GPT, o-series, and embeddings through the OpenAI API. Best when you already hold an OpenAI API key or want the broadest GPT model catalog. With passthrough on, forwards each client’s own Authorization header to OpenAI instead of using a shared server-side key.

Anthropic

Call Claude Opus, Sonnet, and Haiku directly. Strong at coding, long-context reasoning, and tool use. With passthrough on, forwards each client’s own Authorization header to Anthropic instead of using a shared server-side key.

Google Gemini API

Reach Gemini models through the Gemini API, with an API key from Google AI Studio. Ideal for long-context workloads and image and video inputs.

Amazon Bedrock

Invoke foundation models (Claude, Llama, Titan, Nova, Mistral, AI21 Jamba, Gemma) hosted inside your AWS account. Requires an AWS region and credentials (static, STS-assumed role, or the default credential chain). Supports the native Bedrock APIs (InvokeModel, Converse) and an OpenAI-compatible Chat Completions endpoint for gpt-oss and Gemma models. See Set up Amazon Bedrock as an LLM provider for a step-by-step IAM and access-key walkthrough, and its Inference profiles and IAM section for picking the right model identifier.

OpenAI-compatible

Point at any OpenAI-compatible endpoint that ships /v1/chat/completions (vLLM, Ollama, LM Studio, LocalAI, Together, Groq, OpenRouter). Useful for self-hosted models and aggregator gateways. Requires a Base URL. Authentication is optional. Use a server-side key, turn on passthrough so each client sends its own, or leave both unset for an endpoint that needs no credential.

Gemini Enterprise Agent Platform

Run Gemini and Claude models through your own GCP project, which Google bills for them. Requires a GCP project ID, one location for all the provider’s models, and a service account key. Claude models also need setup in the project and organization first. See Set up Gemini Enterprise Agent Platform as an LLM provider for the GCP setup and a step-by-step walkthrough.

Fireworks

Call the serverless models that Fireworks hosts, such as GLM, MiniMax, DeepSeek, and Kimi models, with a Fireworks API key. Models accept requests on the Anthropic Messages, OpenAI Chat Completions, and OpenAI Responses APIs, so one provider works with Claude Code, Codex, and your own code. AI Gateway always sends the stored key to a fixed Fireworks endpoint.

Selecting a type reveals the type-specific configuration fields.

Fill in the type-specific configuration

Each API key reference and credential field points at a secret-store entry, not the secret value itself. On the create form, the Credential section lists the keys already in the Agentic Data Plane secret store. Pick one, or select Bring a new API key reference to store a new key inline. Enter a Key name and the API key value, then click Save key. Agentic Data Plane encrypts the key and stores it in the secret store under an identifier derived from the key name, which appears under the field. You can’t view the value again after you save it.

The picker lists only the secrets AI Gateway can read, which are the ones held in the Agentic Data Plane secret store. A secret that lives in Redpanda Cloud for another product, such as a Redpanda Connect pipeline, isn’t listed, and a new Agentic Data Plane secret can’t reuse its name.

To change the key later, open the provider’s Settings tab. Its Credential section uses the same picker as the create form. Select another stored key, or bring a new one.

Save-time validation checks the credential reference, not the secret behind it. AI Gateway looks up the secret on the first request, so a reference to a missing secret saves and then fails on that request. Some types also have their own rules:

  • OpenAI and Anthropic providers require exactly one of API key reference and Authorization passthrough. AI Gateway rejects a provider that sets neither or both.

  • A Google Gemini API provider requires a non-empty API key reference.

  • A Gemini Enterprise Agent Platform provider requires a service account key reference, a project ID, and a location. AI Gateway rejects the provider when a selected model isn’t served at that location, whether you create the provider or change its models or location.

  • An OpenAI-compatible provider requires no credential reference, so you can create a provider with no authentication for local runtimes such as Ollama or vLLM. AI Gateway rejects a provider that sets both API key reference and Authorization passthrough.

  • A Fireworks provider requires an API key reference. AI Gateway rejects the provider when a newly added model isn’t in Redpanda’s Fireworks catalog or the catalog marks it as retired, whether you create the provider or change its model list. AI Gateway doesn’t recheck models already on the provider.

Each of the following sections covers one provider type. Go to the section for the type you selected. To script the setup, see Manage LLM Providers with rpk ai for a complete command-line example of each type.

OpenAI

Set these fields for an OpenAI provider:

Field Notes

Base URL

Optional. Defaults to the standard OpenAI API (https://api.openai.com/v1). Override for Azure OpenAI or other hosted OpenAI-compatible endpoints. To let clients authenticate with their own Codex subscription, set it to https://chatgpt.com/backend-api/codex and turn on Authorization passthrough. A Codex subscription requires this base URL.

API key reference

Required unless Authorization passthrough is on. Secret-store reference for the OpenAI API key. Must be UPPER_SNAKE_CASE, for example OPENAI_API_KEY.

Authorization passthrough

Optional toggle. When on, AI Gateway forwards each client’s own Authorization header to the upstream provider unchanged instead of sending a server-side API key, so every client authenticates with its own OpenAI credentials. It also forwards ChatGPT-Account-ID when the client sends one, which selects a Codex workspace. No other caller header reaches the upstream. Turning the toggle on clears the API key reference and hides the credential picker. The form requires one or the other, and a provider that has both is rejected. With passthrough on, each client must also authenticate to AI Gateway with an API key in the X-Redpanda-Api-Key header, because its Authorization header now carries the upstream credential. AI Gateway rejects a passthrough request that doesn’t authenticate with an API key with HTTP 400 before contacting the upstream. It also rejects an upstream redirect with HTTP 502 instead of following it, so a redirected response never carries a client’s credentials to another host.

Anthropic

Set these fields for an Anthropic provider:

Field Notes

Base URL

Optional. Defaults to the standard Anthropic API (https://api.anthropic.com).

API key reference

Required unless Authorization passthrough is on. UPPER_SNAKE_CASE, for example ANTHROPIC_API_KEY.

Authorization passthrough

Optional toggle. When on, AI Gateway forwards each client’s own Authorization header to Anthropic unchanged instead of sending a server-side API key, so every client authenticates with its own Anthropic credentials. Leave the API key reference empty when passthrough is on. The form requires one or the other. With passthrough on, each client authenticates to AI Gateway with an API key in the X-Redpanda-Api-Key header, and AI Gateway rejects a request that authenticates with a gateway token, or that has no Authorization header, with HTTP 400. Test connection has no server-side credential to check, so it reports that the first real request verifies upstream access. The saved provider’s Overview tab shows the passthrough state on the Connection card.

Google Gemini API

Set these fields for a Google Gemini API provider:

Field Notes

Base URL

Optional. Defaults to the standard Google Gemini API endpoint (https://generativelanguage.googleapis.com).

API key reference

Required. Secret-store reference for the Google Gemini API key. UPPER_SNAKE_CASE, for example GOOGLE_AI_API_KEY.

Google’s SDKs send their API key as x-goog-api-key, and AI Gateway doesn’t accept a Google API key in that header as gateway authentication. Clients authenticate with an API key, which they can send in x-goog-api-key or X-Redpanda-Api-Key, or with a gateway token in the X-Redpanda-Cloud-Token header. AI Gateway sets x-goog-api-key to the provider’s stored key when it calls Google. See Connect your app to AI Gateway.

Amazon Bedrock

Set these fields for an Amazon Bedrock provider:

Field Notes

Region

Required. The AWS region name where the Bedrock endpoint is deployed, such as us-east-1, us-gov-west-1, or cn-north-1. AI Gateway builds the upstream hostname from this value, so it rejects anything that isn’t shaped like a region name.

Base URL

Optional. Override the default regional Bedrock endpoint with another Amazon Bedrock runtime endpoint, such as a FIPS or VPC endpoint. Leave it empty to use the standard regional endpoint. The URL has to use https:// on port 443, carry no user information, query string, or fragment, and name an Amazon Bedrock runtime host: bedrock-runtime.<region>.amazonaws.com, its FIPS form bedrock-runtime-fips.<region>.amazonaws.com, bedrock-mantle.<region>.api.aws, or a VPC interface endpoint for one of those services. Each amazonaws.com host is also accepted under amazonaws.com.cn. AI Gateway refuses any other host, because the base URL receives the credentials it signs each request with. With a custom base URL, AI Gateway sends every request to that URL, so models that Bedrock serves only on its separate bedrock-mantle endpoint, such as the Gemma 4 family and OpenAI gpt-5.x models, aren’t reachable through this provider unless the base URL is itself a bedrock-mantle endpoint.

Credential type

How AI Gateway authenticates to Bedrock: Bedrock API key, Default chain, Static keys, or Assume IAM role. Rows that name a mode apply only to that mode.

Bedrock API key reference

Bedrock API key only. Required. Secret-store reference for an Amazon Bedrock API key, UPPER_SNAKE_CASE (for example, BEDROCK_API_KEY). The secret holds the key on its own, with no Bearer prefix and no quotes.

Access key ID reference

Static keys only. Secret-store reference for the AWS access key ID, UPPER_SNAKE_CASE (typically AWS_ACCESS_KEY_ID).

Secret access key reference

Static keys only. Secret-store reference for the AWS secret access key, UPPER_SNAKE_CASE (typically AWS_SECRET_ACCESS_KEY).

Role ARN

Assume IAM role only. Required. ARN of the IAM role AI Gateway assumes through AWS STS, for example arn:aws:iam::123456789012:role/BedrockRole. The role’s trust policy should require the external ID that this provider sends. A role in another AWS account, or one whose trust policy names a whole account as the principal, also has to sit under the /redpanda-adp/ path.

External ID

Assume IAM role only. Read-only. The external ID AI Gateway sends when it assumes the role, built from your environment ID and the provider’s name as <environment-id>/llm-provider/<provider-name>. You can’t set or change it. It appears, with a copy button, after you create the provider. Before that, the form says where to find it.

Session name

Assume IAM role only. Optional. Session name that appears in AWS CloudTrail audit logs, for example redpanda-adp.

Select a Credential type to control how AI Gateway authenticates to Bedrock:

  • Bedrock API key: An Amazon Bedrock API key stored in the secret store, which AI Gateway sends as a bearer token instead of signing requests with SigV4. Use this when you want a single credential and no IAM key pair. Generate the key in the Amazon Bedrock console, and store the key value on its own. The identity behind the key needs the bedrock:CallWithBearerToken permission, plus bedrock:ListFoundationModels for Test connection and the invoke permissions for the models you call. AI Gateway never generates or renews a Bedrock API key, so an expired or revoked key fails every call until you update the secret. A short-term key expires within 12 hours and works only in the region it was generated for, so a provider that uses one stops working when the key expires unless you update the secret first. See Generate Amazon Bedrock API keys for how to create a key and which permissions it carries.

  • Default chain (default): Leave the credentials unset to use the AWS SDK’s default provider chain (environment variables, shared config, EKS Pod Identity, IRSA, or instance profile). Use this when the gateway already runs with an AWS identity. An Agentic Data Plane environment hosted on a cloud other than AWS has no such identity, so this mode can’t obtain credentials there.

  • Static keys: An IAM access key pair stored in the secret store. Use this when no ambient AWS identity is available, including on an environment hosted on a cloud other than AWS. This is the path the Bedrock setup guide walks through. This mode takes an IAM access key pair only. A Bedrock API key belongs in Bedrock API key, and a temporary access key that needs a session token belongs in Assume IAM role or Default chain.

  • Assume IAM role: AI Gateway assumes an IAM role through AWS STS. Use this for cross-account access or when your security policy requires short-lived credentials. The role ARN names the role to assume. AI Gateway, not you, sets the external ID it sends with the call. The role’s trust policy should require that external ID. Otherwise any LLM provider or guardrail in the environment can assume the same role. The form shows the external ID only after you create the provider. So create the provider, copy the external ID from its External ID field, add the condition to the trust policy, and then check the connection from the Settings tab. Test connection on the create form is unavailable in this mode for the same reason. AI Gateway still authenticates the AssumeRole call through the default chain, so this mode also needs an ambient AWS identity. A provider carries one credential type, so you can’t pair a role ARN with an access key pair. See Bedrock troubleshooting for the error a provider reports when it has no reachable AWS identity.

For how Bedrock model identifiers and IAM ARNs map to these fields, see Inference profiles and IAM.

Gemini Enterprise Agent Platform

Set these fields for a Gemini Enterprise Agent Platform provider:

Field Notes

Service account key

Required. Secret-store reference for a GCP service account key in JSON format, for example VERTEX_SERVICE_ACCOUNT_KEY. The form checks that the value is a service account key: a JSON object whose type is service_account, with a client_email and an RSA private_key. A service account key is the only credential this type accepts.

Project ID

Required. The ID of the GCP project that hosts the models and is billed for them, for example my-gcp-project. Use the project ID, not the numeric project number.

Location

Required. The location that serves every model on this provider, such as global, us, eu, or a named region such as us-east5. The list shows how many models each location serves. To use models from two locations, create a provider for each. See Choose a location.

Base URL

Optional. Leave it empty to use the location’s endpoint. If you set it, it must start with https://.

The service account needs the roles/aiplatform.user role, and Claude models need setup in your GCP project and organization before they answer. See Set up Gemini Enterprise Agent Platform as an LLM provider.

OpenAI-compatible

Set these fields for an OpenAI-compatible provider:

Field Notes

Base URL

Required. URL of your OpenAI-compatible endpoint, for example http://vllm.internal:8000/v1, http://ollama.local:11434/v1, or an aggregator like Together / Groq / OpenRouter.

API key reference

Optional. Leave empty for endpoints with no authentication (common for local runtimes), or when Authorization passthrough is on. UPPER_SNAKE_CASE if set.

Authorization passthrough

Optional toggle. Behaves as it does on an OpenAI provider. AI Gateway forwards each client’s own Authorization header, and its ChatGPT-Account-ID header when present, to the upstream endpoint instead of sending a server-side API key. Unlike the OpenAI type, this type also accepts having neither, which is the configuration for an endpoint that needs no credential. It rejects both together.

OpenAI-compatible endpoints can serve any model. Enter the exact model identifiers your upstream server exposes (for example, meta-llama/Llama-3.3-70B-Instruct or qwen3:8b).

Fireworks

Set these fields for a Fireworks provider:

Field Notes

API key reference

Required. Secret-store reference for the Fireworks API key. UPPER_SNAKE_CASE, for example FIREWORKS_API_KEY. Store the key value on its own, with no Bearer prefix and no surrounding whitespace or line breaks. AI Gateway sends it as a bearer token on every request.

This type has no Base URL field and no Authorization passthrough toggle. AI Gateway sends every request to the Fireworks inference API with the stored key, and it refuses an upstream redirect rather than following it.

Clients call each model on the API they already speak, under the provider’s proxy URL. Use Anthropic Messages at /v1/messages, OpenAI Chat Completions at /chat/completions, or OpenAI Responses at /responses. AI Gateway forwards the request to the same API on Fireworks without translating it, so the response comes from Fireworks unchanged. Requests run on the Fireworks standard service tier. AI Gateway rejects service_tier as a query parameter, and in the body it accepts only standard. Either one returns HTTP 400 with the error type unsupported_service_tier.

Attach a guardrail (optional)

Attach a guardrail to inspect and control the prompts and responses this provider proxies. In the create form’s Traffic controls section, select a guardrail from the Guardrail list, or leave it empty for none. Each provider references at most one guardrail.

AI Gateway validates the name when you save. It rejects a guardrail that doesn’t exist or is being deleted. To create one first, see Create a guardrail.

To attach or change a guardrail later, use the Traffic controls section on the provider’s Settings tab. From the CLI, use the --guardrail flag on rpk ai llm-provider create or rpk ai llm-provider update. See Update a provider.

Select models

Models you select on this form become the catalog the provider exposes. The same list appears in the Models section of a saved provider’s Settings tab, so you change the selection the same way later.

For OpenAI, Anthropic, Google Gemini API, and Amazon Bedrock, the form shows one row per model from the provider’s catalog, with the full catalog selected by default. Keep the default, or clear a model’s checkbox to narrow what the provider exposes. Each row names the model, and where the catalog has a display name for it, the identifier sits beside that name. The list dims a row you can’t enable, and the row’s tooltip says why. For OpenAI-compatible, the form takes a freeform list. Type the exact identifiers your upstream serves.

With more than 10 rows, the list gains a Filter models box, a count of how many models are enabled out of the total, and Enable all and Disable all. Filtering narrows both controls to Enable shown and Disable shown. A filter that matches nothing reports No models match the filter. and offers Clear filter.

Unless the catalog groups models by publisher, the list separates Current models from Previous generations, where each row names the model that supersedes it, and from Other saved models. Other saved models holds the identifiers this provider can’t serve, both the ones the catalog doesn’t recognize and the ones it knows but doesn’t offer at the provider’s region or location. Saving keeps those identifiers rather than dropping them, so a model you typed stays on the provider. A model the provider has retired appears in Previous generations instead, and you can’t enable it again after you clear its checkbox.

For Gemini Enterprise Agent Platform, the list shows only the models served at the provider’s location, so choose the location first. The form preselects the Gemini models there, and Claude models start unselected. Select only the Claude models you enabled in your GCP project. If you change the location, the selection resets to that location’s Gemini models.

For Fireworks, the list shows the serverless models in Redpanda’s Fireworks catalog, grouped under the vendor that publishes each one, with every model selected by default except one the catalog has marked for retirement. Identifiers use the form accounts/fireworks/models/<model>, as Fireworks publishes them. You can add a model only by selecting it from the list. AI Gateway rejects a save that adds an identifier that isn’t in the catalog or a model that the catalog marks as retired. A retired model that is already on the provider stays until you clear it. Select at least one model. The provider serves only the models you select, and its connection check sends a request to each of them, up to 100.

Redpanda maintains the catalog of available models in the list. When an upstream provider publishes a new model, it usually appears in the list within a day or two. Admins don’t have to wait for a Redpanda release. New models aren’t enabled automatically. An admin still selects the model in the catalog to make it callable through this provider. A model the upstream provider has retired drops out of the list. A provider that already has it selected keeps serving it, its recorded spend keeps pricing at the catalog rates, and agents using it keep working.

For Bedrock, the list shows inference-profile IDs for models that Bedrock serves only through a profile, such as Claude, and the bare model ID for models invoked in-region, such as Mistral Large 3. See Inference profiles and IAM.

The Bedrock list groups models under the vendor that publishes them, such as Anthropic, Amazon, Google, Meta, Mistral, and OpenAI. Because Bedrock reaches a model through one inference geography at a time, each model gets a row with a column per geography, and each column is a model ID of its own. Select as many geographies for a model as you want to serve. The list says what each geography means, including that Global can run inference in any commercial AWS region, so your IAM role and any service control policies have to allow it. Enable all becomes a menu that turns on one geography across the list.

Redpanda stores models as structured ProviderModel entries (one entry per model, with the model name as the only required field). Each model can carry custom pricing overrides that replace the catalog rates for that model in cost reporting. See Override per-model pricing. The legacy flat models field still works on writes for backward compatibility.

Beyond pricing, the catalog carries each model’s capabilities and context-window limits: max_input_tokens (the largest context the model accepts) and max_output_tokens (the most it can generate in a single response). Both are read-only catalog metadata. A limit the catalog doesn’t declare stays unset rather than reported as zero. To see a model’s limits, run rpk ai model get <model>. The context-window indicator on an agent’s Playground tab uses them to show how full the agent’s context window is.

Override per-model pricing

Cost reporting prices each call at the catalog rates for the model. If your organization negotiates non-standard rates, or you track spend against an internal chargeback rate, override the rates per model on this provider.

Click Edit rates to open the rates editor. It covers the models that are enabled on this provider, one row each, so enable a model before you price it. When an enabled model has neither a list price nor a rate of your own, the control reads Set prices instead, and a warning says how many models count as $0, so budgets miss their spend.

The editor shows one column per billing bucket, in the same order as the provider’s published rate card:

Bucket What it bills

Input

Per 1M input tokens. Tool-use input also bills at this rate.

Output

Per 1M output tokens. Reasoning tokens also bill at this rate.

Cache read

Per 1M tokens read from prompt cache.

Cache write, 5 min

Per 1M tokens written to a 5-minute prompt cache.

Cache write, 1 hour

Per 1M tokens written to a 1-hour prompt cache.

Enter rates in dollars per million tokens. Each field is independent:

  • Leave a field blank to keep the catalog rate for that bucket. The catalog rate shows as the field’s placeholder.

  • Enter a positive value to replace the catalog rate for that bucket only.

  • Enter 0 to make that bucket explicitly free, which is different from leaving it blank.

The editor hides the two cache write buckets unless a model already carries a cache write rate of your own. Turn on Show cache write rates to show them. Cache writes with an unknown TTL always bill at the catalog rate. They have no override field.

Click Reset on a row to clear that model’s rates and return it to the catalog rates. To price a whole rate card at a negotiated discount, enter the percentage in Discount off list price and click Fill all rates, which fills every bucket that carries a list price, on every model that has one. The editor stores the result as fixed rates, so a later change to the list price doesn’t move them. A rate below zero or above $10,000 blocks Apply rates, and the editor says what it accepts.

Click Apply rates to put your rates on the form, then save the provider to store them. Overrides are scoped to this provider and model, they apply to new requests while past spend keeps the cost it was recorded at, and they change what Agentic Data Plane’s cost reporting computes, not what the upstream provider actually charges you.

A rate you enter is the model’s standard, default-speed rate, which is what most calls bill at. Agentic Data Plane derives the model’s other rates from it:

  • A call that ran at a faster or cheaper service tier, or in a provider’s fast mode, bills at the same multiple of your rate that the catalog tier is of the catalog standard rate. A discounted tier stays discounted and a premium tier stays at a premium, both relative to the rate you negotiated. See Per-request pricing variations.

  • A region-specific rate is replaced by the rate you enter, with no regional premium added, because a negotiated rate is already region-specific.

  • A rate that applies only above a context-length threshold keeps its catalog value, as does any bucket you leave blank.

To see at a glance what a model costs, read the cost tier at the end of its row, which reports Free, Low cost, Medium cost, or High cost, or Rates for a model the catalog puts in no tier. A model you priced yourself reads Custom price instead, so the rows you negotiated stand out from the rest. Hover over the tier to open the model’s rate card, which gives its input, output, and cache read rates per million tokens, with any override already applied, and the largest context it accepts. A bucket with no rate reads Not priced, and a model with no input or output rate at all reads No list price.

You can set the same overrides from the command line with the repeatable --pricing flag on rpk ai llm-provider create and rpk ai llm-provider update. Each value names the model with model, which is required, then sets one rate key per billing bucket, in dollars per million tokens: input, output, cached, cache_write_5m, and cache_write_1h. As in the dialog, an omitted rate keeps the catalog rate and 0 sets an explicit free rate. Set at least one rate per model. For an example, and for how --pricing replaces the model list on update, see Update a provider.

Tag the provider (optional)

The Tags section, last on the create form, takes key/value pairs that organize and filter providers and that access policies can read in their conditions. Enter a tag key and a tag value, then press Enter or click the add (+) button.

To change tags after you create the provider, use the Tags section on its Settings tab. A tagged provider shows its tags on the connection card of its detail page and in the Tags column of the LLM providers list.

For the limits and rules that apply to tags on every resource, see Tag Resources.

Configure message recording

The Transcripts section controls whether AI Gateway records the message bodies this provider proxies. It appears on the create form and on the provider’s Settings tab, and it has two independent toggles:

Toggle What it captures

Record inputs

Captures the full request body (prompt content and tool-call arguments) on observability traces.

Record outputs

Captures the full response body (completion content and tool-call results) on observability traces.

When you create a provider, recording starts on, so a new provider records message bodies from its first request. This holds on every create path. The Create provider form starts with both toggles on, and a create with rpk ai llm-provider create that doesn’t mention transcripts gets the same default. The bodies land on the AI Gateway’s own trace data for the calls it proxies. They don’t feed agent transcripts, which a managed agent records under its own transcript recording mode. Turn either toggle off, on the create form or later on the Settings tab, for workloads where the message body must not be retained, such as regulated PII or customer secrets. To keep that content out of agent transcripts too, set each agent’s recording mode to Metadata only or Off.

Two cases don’t get that default:

  • An existing provider keeps whatever it was saved with. The default applies at create time and doesn’t change any provider you already had.

  • A create that explicitly disables recording stays off. From the CLI, pass --transcripts.record-input-messages=false --transcripts.record-output-messages=false. Set both flags together, or neither. If a create sets only one of them, the other starts off instead of taking the default.

These are per-provider settings, not per-request. Applications can’t opt in or out at call time. To split sensitive from non-sensitive traffic, create one provider with recording on and another with it off, and route each application to whichever proxy URL matches its data class.

These toggles govern only the message bodies the AI Gateway records for the calls it proxies. An agent’s Transcripts tab is assembled from spans the agent exports itself. A managed agent’s transcript recording mode, or a self-managed agent’s own instrumentation capture settings, controls the text those transcripts show, not these toggles. See Transcript recording and Message content capture.

Recording settings do not affect cost and usage telemetry. Token counts, latency, and provider/model attribution are always recorded, so the Cost and usage page reports spend for traffic on the provider regardless of these toggles. Only the message bodies are withheld when the toggles are off.

Changing a toggle takes effect for new requests. Message bodies already recorded under the previous setting are not retroactively redacted. Delete or rotate the provider if you need to purge historical content.

Save and verify

Before you click Create provider, check the Transcripts section. Both toggles arrive on, so the provider starts recording prompts and responses with its first request, and what it captures is not retroactively redacted. See Configure message recording.

  1. Optional: Click Test connection in the Verify connection box in the Models section to check the credential and endpoint you entered before you commit to them. The check creates nothing, so a failure costs you only the retry. It needs a provider key (for the types that require one) and a base URL that starts with http:// or https://. For an OpenAI-compatible provider, the base URL is required. For other provider types, it’s checked only if you set one. A Bedrock provider set to Assume IAM role can’t be checked here, because the form shows the external ID for its role’s trust policy only after you create the provider. Create it, then check it from the Settings tab. A Fireworks provider needs between one and 100 selected models, and the check sends a request to each one.

  2. Click Create provider. Until the form is complete, the button stays disabled and its hint tells you what’s still needed (for example, saving the new key).

  3. After create, you land on the provider’s Connect tab, with client setup instructions and code examples that already include the proxy URL. To copy the proxy URL on its own, open the Overview tab and use the copy control on the Connection card. The proxy URL is where your applications point.

  4. To confirm the saved provider is reachable, open the Settings tab and click Test connection in the Test saved configuration box of the Connection section. The result appears as an alert in the same box. Save any pending changes first, because the test stays disabled until you do.

  5. To wire up an application, use the Connect tab or follow Connect your app to AI Gateway.

What a successful result proves depends on the provider type, and the panel says so before you run it:

  • For most types, the check lists the models the upstream service offers, using the credential and endpoint you configured. It sends no model request, so it confirms authentication and the network path rather than access to any one model.

  • For Gemini Enterprise Agent Platform, the check sends one request to each selected model. For a Gemini model it counts the tokens of a short prompt, which Google doesn’t bill. For a Claude model it sends a request capped at one output token, which Google bills to your GCP project. It can send a model more than one request when a request fails or is slow to answer. If no model is selected, it checks one model served at the location. The result lists the models that didn’t answer, grouped under the cause they share, so it confirms access to each model, not only the credential. A Gemini model that passes the check can still be slow or busy on a real request, because counting tokens doesn’t reach the servers that generate the answer. See Causes a connection check reports.

  • For Amazon Bedrock, the check confirms the AWS credentials in the configured region. It doesn’t verify access to a model, which IAM grants separately (see Inference profiles and IAM). A Bedrock provider with a custom Base URL can’t run the check, and the result says so.

  • For a Bedrock provider that authenticates with a Bedrock API key, the check lists the foundation models in the region, so the key’s identity needs bedrock:ListFoundationModels as well as bedrock:CallWithBearerToken. AWS answers a rejected key and a key that lacks a permission the same way, so the check reports both as an authentication failure, and it names the cause when AWS’s response does, such as an expired key. As with the other Bedrock credential types, a check that passes doesn’t confirm that the key can invoke a model or use a guardrail.

  • For Fireworks, the check sends one request to each selected model, capped at one output token, on the API the catalog lists first for that model. It runs up to four requests at a time, with a single attempt per model. Fireworks bills each request to your account, and the Test connection control warns you about this cost before you run it. The result reports how many of the selected models responded, so it confirms access to each model rather than only the key. The check shows Connected with its latency when every model responds, a warning with the number of models that responded when some fail, and an error with a count of zero when none responds.

  • For any provider with authorization passthrough on, there’s no server-side credential to test. The first real request is what verifies upstream access, because each client authenticates with its own key.

If the check fails, see Troubleshooting.

To monitor, connect to, and manage the provider after you create it, see Monitor and Manage an LLM Provider.

Troubleshooting

Symptom What to check

secret "<NAME>" not found

Confirm the secret exists in the Agentic Data Plane secret store and the reference in the provider configuration is spelled identically (UPPER_SNAKE_CASE, no typos).

Test connection fails with connection refused, DNS lookup failed for <host>, or no answer within 10s

The check reached no HTTP response at all, so the credential was never tried. Check the Base URL for a typo, and check that your network allows egress to that host. A check that reports no answer within 10s timed out, so retry it before you change anything.

Bedrock returns AccessDenied, ValidationException, an on-demand throughput error, or a geography restriction error

These come from the provider’s AWS region, its IAM policy, or the model identifier you called. See Bedrock troubleshooting.

A Gemini Enterprise Agent Platform provider returns 404 NOT_FOUND, 403 PERMISSION_DENIED, 400 FAILED_PRECONDITION, or 429

These come from the GCP project’s setup: the service account role, the Claude models enabled in Model Garden, the organization policy, or the location. See Gemini Enterprise Agent Platform troubleshooting.

The upstream provider returns 401 when passthrough is enabled

Confirm the client is sending its own Authorization header and the API key reference field on the provider is empty. On an OpenAI provider, also confirm the base URL matches the credential your clients send. A Codex subscription requires the Codex base URL.

A passthrough request fails with HTTP 400 and authorization passthrough requires verified ADP API-key authentication

With passthrough on, the Authorization header carries the client’s upstream credential, so the client must authenticate to AI Gateway with an API key in the X-Redpanda-Api-Key header. A gateway token doesn’t work with passthrough. Create a key, send it in that header, and remove any X-Redpanda-Cloud-Token header. The passthrough guide on the provider’s Connect tab shows the two headers together.

A passthrough request to an Anthropic provider fails with HTTP 400 and authorization passthrough requires separate upstream authorization

The request carried no Authorization header, so there’s no upstream credential to forward. Send your Anthropic credential in Authorization and your API key in X-Redpanda-Api-Key.

A passthrough request fails with HTTP 502 and upstream redirects are not allowed with authorization passthrough

The upstream endpoint answered with a redirect, which AI Gateway refuses to follow when passthrough is on so that a client’s credentials never travel to another host. Point the provider’s Base URL at the endpoint’s final address. For a Codex subscription, that is https://chatgpt.com/backend-api/codex.

Gemini returns 401

The client didn’t send its gateway token in a header AI Gateway reads. Send it in X-Redpanda-Cloud-Token. The Google SDKs put their API key in x-goog-api-key, which AI Gateway reads only for an Agentic Data Plane API key, not a Google API key. See Connect your app to AI Gateway.

Requests fail with HTTP 403 and model_not_allowed

The requested model isn’t enabled on the provider. Enable it in the Models section of the provider’s Settings tab, or request a model that is enabled.

Saving a Fireworks provider fails with model "<id>" is not in the gateway’s Fireworks catalog; custom pricing does not declare a wire API contract or model "<id>" is retired and cannot be newly selected

The Fireworks type accepts only identifiers from Redpanda’s Fireworks catalog, and it refuses to add a model that the catalog marks as retired. Select models from the list on the form, and remove the identifier the message names. When more than one model fails, the message lists each one. A retired model that was already on the provider stays until you clear it.

Test connection on a Fireworks provider reports <n> of <m> models answered on their selected API

One or more of the selected models didn’t respond successfully to the single request each received. The result shows how many models responded, not which ones. The common causes are:

  • Fireworks rejects the key, which fails every model.

  • The account behind the key can’t use a selected model.

  • Fireworks doesn’t serve a selected model on the API the catalog lists for it.

  • Fireworks rate-limits the check.

  • Fireworks is unreachable, times out, or returns an error.

Check the key first. To find a failing model, clear models from the selection one at a time and run the check again. Each run sends one request per selected model.

A Fireworks request fails with HTTP 400 and unsupported_service_tier

AI Gateway sends Fireworks requests on the Fireworks standard service tier only. Remove service_tier from the query string. In the request body, remove it or set it to the string standard.

A managed agent on a Fireworks provider fails to save with model "<id>" is not enabled on llm_provider "<provider>"; enable it on the provider or pick one of its models

The agent’s model isn’t selected on the provider. Enable the model in the Models section of the provider’s Settings tab, or select a model the provider already serves. The identifier must match exactly, including the accounts/fireworks/models/ prefix.

Provider list empty or 403

Confirm your account can act on LLM providers. Listing them needs Action::"LLMProvider.list" and creating one needs Action::"LLMProvider.create". Outside the Admin role, both come from an access policy. See LLM provider permissions.

Limitations

AI Gateway does not provide these capabilities. For current status, see the Agentic Data Plane release notes.

  • Multi-provider routing, failover, and retries across providers. A synthetic provider that fans requests to multiple upstreams is not part of AI Gateway.

  • Rate limits. Requests-per-second, per-minute, or per-day limits are not available. To cap spend rather than request rate, use budgets, which enforce a per-agent hard cap.

  • Token counting on Fireworks. The Anthropic count_tokens endpoint returns HTTP 501 on a Fireworks provider, because Fireworks doesn’t serve it. Clients that count tokens before a request must handle that response.