# Streaming - Full Markdown Export > This file contains all Streaming documentation pages in markdown format for AI agent consumption. > Generated from 552 pages on 2026-08-28T16:28:36.301Z > Component: streaming | Version: 26.2 > Site: https://docs.redpanda.com ## About This Export This export includes the **latest version** (26.2) of the Streaming documentation. ### AI-Friendly Documentation Formats We provide multiple formats optimized for AI consumption: - **https://docs.redpanda.com/llms.txt**: Curated overview of all Redpanda documentation - **https://docs.redpanda.com/llms-full.txt**: Complete documentation export with all components - **https://docs.redpanda.com/streaming-full.txt**: This file - Streaming documentation only - **Individual markdown pages**: Each HTML page has a corresponding .md file ### Accessing Older Versions This component has versioned documentation. Older versions can be accessed by replacing the version segment in the URL: - Latest: `https://docs.redpanda.com/current/page-path` - Specific version: `https://docs.redpanda.com/24.3/page-path`, `https://docs.redpanda.com/25.1/page-path`, etc. --- # Page 1: Introduction to Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console.md --- # Introduction to Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Introduction to Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/index.adoc description: "Learn about Redpanda Console: a web interface for managing and interacting with Redpanda clusters." page-git-created-date: "2023-05-17" page-git-modified-date: "2026-08-13" support-status: supported --- Redpanda Console is a web interface for managing and interacting with Redpanda clusters. Built to provide a seamless experience for developers working with streaming data, Redpanda Console simplifies tasks associated with managing data streams, offering a UI that helps you monitor, troubleshoot, and optimize your streaming workloads. ![overview](_images/overview.png) ## [](#cluster-management)Cluster management Explore a comprehensive overview of your cluster, including: - **Broker monitoring**: View and manage the health, status, and configurations of your brokers. - **Topic management**: Create, configure, and monitor topics, including detailed information on partitions, replicas, and message counts. - **Consumer group insights**: Track the activity and performance of your consumer groups, manage offsets, and identify potential bottlenecks. Consumers that manage offsets outside of Kafka consumer groups, such as Apache Spark Structured Streaming, do not appear here. Flink consumers appear only when Flink is configured to commit offsets on checkpoints. For details, see [Consumer Offsets](https://docs.redpanda.com/streaming/current/develop/consume-data/consumer-offsets/). - [**Shadow link management**](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/): Create shadow links for asynchronous, offset-preserving replication between distinct Redpanda clusters. The shadow cluster operates in read-only mode while continuously receiving updates from the source cluster. During a disaster, you can failover individual topics or an entire shadow link to make resources fully writable for production traffic. ![broker overview](_images/broker-overview.png) ## [](#data-observability-and-debugging)Data observability and debugging Observe and debug your streaming data: - **Message inspection**: Browse and filter messages within your topics, with options to search by key, timestamp, or custom filters. See [Paginate Messages in Redpanda Console](ui/paginate-messages-events/) to inspect large result sets. - [**Programmable push filters**](ui/programmable-push-filters/): Write custom JavaScript filters to isolate specific messages, enabling deep inspection and debugging. - **Rewind and Replay**: Roll back consumer offsets to reprocess messages, allowing you to correct issues or replay data as needed. ![topic](_images/topic.png) ## [](#access-control)Access control Manage Redpanda users and control who has access to Redpanda Console: - **Visual ACL management**: Create, view, and edit individual ACLs, with role-inherited permissions shown inline so you can see exactly what each principal is authorized to do. - **Role-based access control**: Group permissions into roles and assign roles to users or OIDC groups; manage role membership from a dedicated role page. - **Rotate credentials**: Update user and service account passwords to maintain security without downtime. - **Identity provider integration**: For enterprise users, Redpanda Console integrates with identity providers (IdPs) for single sign-on (SSO), making user management straightforward and secure. ![user](_images/user.png) ## [](#schema-management)Schema management [Manage and browse your schemas](ui/schema-reg/), ensuring your data is correctly structured and validated across your streams. ![schema reg](_images/schema-reg.png) ## [](#connectivity-and-integrations)Connectivity and integrations View and manage Kafka Connect clusters and connectors, simplifying the integration of external systems with your streaming data. > 📝 **NOTE: Community** > > **Kafka Connect is community-supported on [Redpanda Community Slack](https://redpanda.com/slack)**. Redpanda Data does not provide enterprise support for Kafka Connect with Redpanda Console. For a supported and scalable Kafka Connect alternative, try [Redpanda Connect](https://docs.redpanda.com/connect/get-started/). ## [](#who-should-use-redpanda-console)Who should use Redpanda Console? Redpanda Console is designed for: - **Developers** who need to manage, monitor, and debug streaming data without the overhead of complex CLI tools. - **Data engineers** who require a robust interface to manage Redpanda clusters and ensure data pipelines are running smoothly. - **DevOps engineers** who want a single place to monitor the health and performance of streaming data infrastructure. - **Security teams** who need to configure and audit access controls within the data streaming environment. ## [](#suggested-videos)Suggested videos - [Demo: Redpanda Console](https://www.youtube.com/watch?v=ezDYSpC7JcU) ## [](#next-steps)Next steps [Get started](https://docs.redpanda.com/streaming/current/get-started/quick-start/) --- # Page 2: Redpanda Console Telemetry **URL**: https://docs.redpanda.com/streaming/current/console/config/analytics.md --- # Redpanda Console Telemetry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Console Telemetry latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/analytics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/analytics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/analytics.adoc description: Learn what telemetry Redpanda Console collects by default, how it is handled, and how to disable it. page-git-created-date: "2025-08-04" page-git-modified-date: "2025-12-04" support-status: supported --- Redpanda Console collects telemetry (analytics) data for purposes including product improvements and user experience optimization. This document explains what data is collected, how it is processed, and how you can disable telemetry if desired. ## [](#what-is-tracked-by-default)What is tracked by default When telemetry is enabled, which is the default behavior, Redpanda Console collects metadata and sends it securely to Redpanda. ### [](#redpanda-console-metadata)Redpanda Console metadata The following information is collected from the running instance of Redpanda Console: - Startup timestamp (when the Redpanda Console process starts) - Runtime UUID (a unique ID generated per instance) - Redpanda Console version and build timestamp - License hash, license type, and license organization - Kafka configuration hash (anonymized) - Whether Kafka is connected as `localhost` - Behavioral user analytics ### [](#cluster-metadata)Cluster metadata The following information is collected from the connected Kafka cluster: - Cluster configuration details (excluding sensitive values) - Broker log dir statistics such as the size - Topic configurations (settings only, no message contents) - Consumer group metadata (total count, and counts by protocol and state) - General cluster metadata: - Number of brokers - Number of topics - Number of partitions - Number of racks - Cluster ID ## [](#how-telemetry-data-is-handled)How telemetry data is handled Telemetry data is processed with the following features: - All telemetry payloads are signed and encoded as JWTs - Data is sent using HTTPS POST requests to Redpanda’s telemetry endpoint - Configuration data is anonymized using secure hashes - No message contents or credentials are collected or transmitted ## [](#disable-telemetry)Disable telemetry To turn off telemetry and user tracking, set the following in your Redpanda Console configuration: ### Standalone ```yaml analytics: enabled: false ``` ### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: #### Operator `redpanda-console`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: analytics: enabled: false ``` #### Helm `redpanda-values.yaml` ```yaml console: enabled: true console: config: analytics: enabled: false ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: analytics: enabled: false ``` Restart the Redpanda Console service to apply the change and stop all telemetry and user tracking. --- # Page 3: Configure Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/configure-console.md --- # Configure Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/configure-console page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/configure-console.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/configure-console.adoc description: Learn how to configure Redpanda Console using environment variables, YAML files, or command-line arguments. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- Redpanda Console loads configuration properties from three sources, in the following order of precedence: 1. Environment variables 2. YAML file configuration 3. Command-line arguments Environment variables and YAML configurations can overwrite input that is set on the command line. ## [](#environment-variable-mapping)Environment variable mapping Configuration options can be set using environment variables. The key for the environment variable is auto-generated by converting the [YAML equivalent](#config-yaml) to uppercase and adding an underscore for each indentation level. For example: | YAML | Environment variable | | --- | --- | | kafka.rackId | KAFKA_RACKID | | kafka.tls.caFilepath | KAFKA_TLS_CAFILEPATH | For configuration properties that expect a list of values, use commas between each value. For example: ```bash KAFKA_BROKERS=redpanda-0:9092,redpanda-1:9092,redpanda-2:9092 ``` > 📝 **NOTE** > > You cannot use environment variables to configure object arrays, such as the configuration for Kafka Connect clusters. In this case, use a YAML file, and provide secrets using environment variables or command line arguments. ## [](#platform-specific-configuration)Platform-specific configuration Redpanda Console can be deployed in several ways. The configuration method and file structure you use depends on your deployment scenario. Use the tabs on this page and throughout the docs to find the instructions and examples for your environment. - **Standalone (binary, Docker, or systemd):** For users running Redpanda Console as a separate service, either on a VM, bare metal, or in a container (not managed by Kubernetes or the Redpanda Operator/Helm chart). - **Kubernetes embedded:** For users running Redpanda Console as part of a Redpanda cluster managed by the Redpanda Operator or the Redpanda Helm chart. - **Kubernetes standalone:** For users deploying Redpanda Console in Kubernetes using the dedicated Redpanda Console Helm chart (not as part of a Redpanda cluster). ### Standalone Use this method when deploying Redpanda Console as a standalone service (binary, Docker, or systemd). The recommended configuration source is a YAML file. You can specify the path to the configuration file by setting either the `-config.filepath` flag or the `CONFIG_FILEPATH` environment variable. In Linux package installations, this file is located in `/etc/redpanda/redpanda-console-config.yaml` by default and Redpanda Console is configured to read from this file path. In containerized environments, ensure that the configuration file is mounted to a directory accessible by the Redpanda Console container. When the file is mounted, you can specify its file path using the `-config.filepath` flag or the `CONFIG_FILEPATH` environment variable. ### Kubernetes embedded Use this method when Redpanda Console is deployed as part of the Redpanda Helm chart or Redpanda Operator. > 📝 **NOTE** > > When Redpanda Console is embedded in the Redpanda deployment, the Kafka broker configuration is automatically set to connect to the Redpanda cluster in the same deployment. #### Operator Configure Redpanda Console in the `console` section of your Redpanda custom resource: `redpanda-console`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda # Reference to your Redpanda cluster config: server: listenPort: 8080 auditLog: enabled: false additionalHeaders: - "X-Forwarded-For" - "User-Agent" # Add any other Redpanda Console configuration here ``` Apply the configuration: ```bash kubectl apply -f redpanda-console.yaml --namespace ``` #### Helm Configure Redpanda Console in the `console` section of your Redpanda Helm values file: `redpanda-values.yaml` ```yaml console: enabled: true console: config: #kafka: #brokers: [] # You can leave this empty. The chart is already auto-configured to connect to your Redpanda cluster server: listenPort: 8080 auditLog: enabled: false # default; set to true to write audit logs to stdout additionalHeaders: # specifies HTTP headers to include in audit logs # Add any other Redpanda Console configuration here ``` Apply the configuration: ```bash helm upgrade --install redpanda redpanda/redpanda \ --namespace \ --values redpanda-values.yaml ``` ### Kubernetes standalone Use this method when deploying Redpanda Console as a standalone service in Kubernetes using the dedicated Redpanda Console Helm chart. When using the standalone Redpanda Console Helm chart, configure Redpanda Console through the `config` section of your values file: `console-values.yaml` ```yaml config: kafka: brokers: - redpanda-0.redpanda.svc.cluster.local:9092 - redpanda-1.redpanda.svc.cluster.local:9092 server: listenPort: 8080 auditLog: enabled: false # default; set to true to write audit logs to stdout additionalHeaders: # specifies HTTP headers to include in audit logs - "X-Forwarded-For" - "User-Agent" # Add any other Redpanda Console configuration here ``` Apply the configuration: ```bash helm upgrade --install console redpanda/console \ --namespace redpanda \ --values console-values.yaml ``` ## [](#docker-compose-example)Docker Compose example If you are using Docker Compose, you can mount the configuration file and set the environment variable in your `docker-compose.yml` file: ```yaml console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:v3.11.0 entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml' volumes: - ./config:/tmp/config/ environment: CONFIG_FILEPATH: ${CONFIG_FILEPATH:-/tmp/config.yml} CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092","redpanda-1:9092","redpanda-2:9092"] auditLog: enabled: false # default; set to true to write audit logs to stdout additionalHeaders: # specifies HTTP headers to include in audit logs - "X-Forwarded-For" - "User-Agent" ``` ## [](#config-yaml)Complete configuration file example The following YAML file contains a complete list of all Redpanda Console configuration properties and their descriptions. All values are default values. > ⚠️ **CAUTION** > > - Where necessary, ensure that values are enclosed in quotes and escaped. For example, put passwords with special characters in single quotes. > > - This configuration file contains both Redpanda Enterprise and Redpanda Community Edition configurations. If you don’t provide an enterprise license, Redpanda Console ignores configurations for enterprise features. [Download the sample file](https://docs.redpanda.com/streaming/current/shared/_attachments/redpanda-console-config.yaml). redpanda-console-config.yaml ```yaml # This is an example configuration file for Redpanda Console v3.x.x #---------------------------------------------------------------------------- # Kafka configuration #---------------------------------------------------------------------------- kafka: # Brokers is a list of bootstrap servers with ports. brokers: - "broker-0.mycompany.com:19092" - "broker-1.mycompany.com:19092" - "broker-2.mycompany.com:19092" # Optional: Client ID used to identify Redpanda Console to the Kafka cluster. # clientId: "console" # Optional: Rack identifier to optimize message consumption in multi-zone clusters. # rackId: "zone-a" # sasl: # enabled: true # Supported mechanisms include: # - OAUTHBEARER (OIDC) # - SCRAM-SHA-256 or SCRAM-SHA-512 (basic authentication) # - GSSAPI (Kerberos); if using Kerberos, ensure impersonateUser is false. # - AWS_MSK_IAM (AWS MSK IAM) # mechanism: SCRAM-SHA-256 # impersonateUser: false # oauth: # token: "example-oauth-token" # clientId: "example-client-id" # clientSecret: "example-client-secret" # tokenEndpoint: "https://accounts.google.com/token" # tokenFilepath: "/var/run/secrets/kafka/serviceaccount/token" # scope: "openid" # Example for basic authentication (uncomment to use): # username: "your-username" # password: "your-password" # Example for GSSAPI (Kerberos) - impersonateUser must be false: # gssapi: # authType: KEYTAB_AUTH # keyTabPath: "/path/to/keytab" # kerberosConfigPath: "/path/to/krb5.conf" # serviceName: "kafka" # username: "your-username" # password: "your-password" # realm: "MY.REALM" # enableFast: true # tls: # enabled: false # Uncomment and set the following paths if TLS is required: # caFilepath: "/path/to/ca-cert.pem" # certFilepath: "/path/to/client-cert.pem" # keyFilepath: "/path/to/client-key.pem" # insecureSkipTlsVerify: false # Startup is a configuration block to specify how often and with what delays # we should try to connect to the Kafka service. If all attempts fail the # application exits with code 1. # startup: # maxRetries: 5 # retryInterval: 1s # maxRetryInterval 60s # backoffMultiplier: 2 #---------------------------------------------------------------------------- # Schema Registry configuration (top-level) #---------------------------------------------------------------------------- schemaRegistry: enabled: true urls: - "http://schema-registry.mycompany.com:8081" # Optional: Authentication for Schema Registry. # authentication: # basic: # username: "example-user" # password: "example-password" # bearerToken: "example-bearer-token" tls: enabled: false # Uncomment and configure if TLS is required: # caFilepath: "/path/to/ca-cert.pem" # certFilepath: "/path/to/client-cert.pem" # keyFilepath: "/path/to/client-key.pem" # insecureSkipTlsVerify: false #---------------------------------------------------------------------------- # Redpanda Console authentication #---------------------------------------------------------------------------- authentication: jwtSigningKey: "secret-value" useSecureCookies: true # Maximum browser session age (Enterprise). Accepts duration strings (for example, "90d", "24h", "30m"). Default: 1 year. # maximumSessionAge: "90d" # Optionally enable cookie chunking if cookie size is an issue. # useCookieChunking: false # OIDC configuration (if using OIDC): # oidc: # enabled: true # issuerUrl: "https://accounts.google.com" # clientId: "your-oidc-client-id" # clientSecret: "your-oidc-client-secret" # redirectUrl: "http://localhost:9090/auth/callbacks/oidc" # successfulLoginRedirectUrl: "http://localhost:3000" # accessType: "offline" # prompt: "consent" # issuerTls: # enabled: true # caFilepath: "/path/to/ca.pem" # certFilepath: "/path/to/issuer-cert.pem" # keyFilepath: "/path/to/issuer-key.pem" # insecureSkipTlsVerify: false # Basic authentication is supported by default. #---------------------------------------------------------------------------- # Redpanda Console authorization and role bindings #---------------------------------------------------------------------------- authorization: roleBindings: - roleName: admin users: - loginType: oidc name: "admin@mycompany.com" - roleName: viewer users: - loginType: basic name: "user@mycompany.com" #---------------------------------------------------------------------------- # Redpanda Admin API configuration #---------------------------------------------------------------------------- redpanda: adminApi: enabled: true urls: - "admin-0.mycompany.com:9644" - "admin-1.mycompany.com:9644" authentication: impersonateUser: true # If impersonateUser is false, configure static credentials here: # authentication: # basic: # username: "example-user" # password: "example-password" startup: establishConnectionEagerly: true maxRetries: 5 retryInterval: 1s maxRetryInterval: 60s backoffMultiplier: 2 tls: enabled: true caFilepath: "/path/to/ca-cert.pem" certFilepath: "/path/to/client-cert.pem" keyFilepath: "/path/to/client-key.pem" insecureSkipTlsVerify: false #---------------------------------------------------------------------------- # Kafka Connect configuration (optional) #---------------------------------------------------------------------------- kafkaConnect: enabled: false # connectTimeout: 15s # readTimeout: 60s # requestTimeout: 6s clusters: [] # Example: # clusters: # - name: my-connect-cluster # url: "http://connect.mycompany.com:8083" # tls: # enabled: false # username: "connect-user" # password: "connect-password" # token: "optional-token" #---------------------------------------------------------------------------- # Enterprise License configuration (optional) #---------------------------------------------------------------------------- # To mount an enterprise license, set either license or licenseFilepath. # This is only required if you want to use an enterprise feature # such as SSO or RBAC. # Filepath to your redpanda.license file # licenseFilepath: "" # License string. # license: "" #---------------------------------------------------------------------------- # Serde settings #---------------------------------------------------------------------------- serde: maxDeserializationPayloadSize: 20480 # protobuf: # enabled: false # mappings: [] # Map the Proto type names for each of your topics. # These Proto types will be used for deserialization. # - topicName: xy # You can specify the Proto type for the record key # and/or value (just one will work too) # valueProtoType: fake_model.Order # keyProtoType: package.Type # Configure the fileSystem if you want Redpanda Console to # search the local file system for the Proto files # fileSystem: # enabled: false # paths: [] # refreshInterval: 5m # importPaths is a list of paths from which to import Proto files into Redpanda Console. # Paths are relative to the root directory. # The `git` configuration must be enabled to use this feature. #importPaths: [] # Git is where the Proto files come from. # git: # enabled: false # repository: # url: # branch: (defaults to primary/default branch) # baseDirectory: (defaults to the root directory of the repo/branch above) # How often Redpanda Console pulls the repository to look for new files. # Set to 0 to disable periodic pulls. # refreshInterval: 5m # To use GitHub's personal access tokens, use `token` # as username and pass the same token as password. # basicAuth: # enabled: true # username: # Password can also be set using the --serde.protobuf.git.basic-auth.password flag. # password: # You can pass the private key file directly using a flag on the command line, or you can specify it in the # yaml configuration file. Another alternative is to provide the filepath to a mounted key # file in this configuration block. # ssh: # enabled: false # username: # privateKey can also be set using the --serde.protobuf.git.ssh.private-key flag. # privateKey: # privateKeyFilepath: # Passphrase can also be set using the --serde.protobuf.git.ssh.passphrase flag. # passphrase: # messagePack: # enabled: false # List of topic name regexes, defaults to /.*/ # topicNames: ["/.*/"] #---------------------------------------------------------------------------- # Redpanda Console settings #---------------------------------------------------------------------------- console: topicDocumentation: enabled: false # git: # enabled: false # repository: # url: # branch: (defaults to primary/default branch) # baseDirectory: . # # How often Redpanda Console pulls the repository to look for new files. # # Set to 0 to disable periodic pulls. # # refreshInterval: 1m # # To use GitHub's personal access tokens, use `token` as username and pass the actual token as the password. # basicAuth: # enabled: true # username: token # password: # # You can pass the private key file directly using a flag on the command line, or you can specify it in the yaml configuration file. Another alternative is to provide the filepath to a mounted key file in this configuration block. # ssh: # enabled: false # username: git # privateKey: | # -----BEGIN PRIVATE KEY----- # ... # -----END PRIVATE KEY----- # privateKeyFilepath: /path/to/private/key # passphrase: auditLog: enabled: false # default; set to true to write audit logs to stdout additionalHeaders: # specifies HTTP headers to include in audit logs - "X-Forwarded-For" - "User-Agent" #---------------------------------------------------------------------------- # Server settings #---------------------------------------------------------------------------- server: listenAddress: "0.0.0.0" listenPort: 8080 httpsListenPort: 8081 advertisedHttpsListenPort: 443 gracefulShutdownTimeout: 30s readTimeout: 30s writeTimeout: 30s idleTimeout: 30s compressionLevel: 4 basePath: "" setBasePathFromXForwardedPrefix: true stripPrefix: true tls: enabled: false # Uncomment and configure if HTTPS is required: # certFilepath: "/path/to/https-cert.pem" # keyFilepath: "/path/to/https-key.pem" allowedOrigins: [] #---------------------------------------------------------------------------- # Logger settings #---------------------------------------------------------------------------- logger: level: info #---------------------------------------------------------------------------- # Developer settings #---------------------------------------------------------------------------- # Only relevant for developers who want to run the frontend separately. # Uncomment the following line to serve the frontend separately. # serveFrontend: true #---------------------------------------------------------------------------- # Metrics settings #---------------------------------------------------------------------------- # Prefix for all exported Prometheus metrics. # Uncomment and set your metrics namespace. # metricsNamespace: "console" #---------------------------------------------------------------------------- # Analytics / telemetry (optional) #---------------------------------------------------------------------------- analytics: enabled: true ``` --- # Page 4: Configure Redpanda Console to Connect to a Redpanda Cluster **URL**: https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda.md --- # Configure Redpanda Console to Connect to a Redpanda Cluster > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Redpanda Console to Connect to a Redpanda Cluster latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/connect-to-redpanda page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/connect-to-redpanda.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/connect-to-redpanda.adoc description: Learn how to configure Redpanda Console to connect to a Redpanda cluster and ensure communication with your Redpanda brokers. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- Redpanda Console connects to your Redpanda cluster using dedicated configuration blocks for the Kafka API, Schema Registry API, and Admin API. Each connection serves a different purpose: - **Kafka API:** Enables core messaging operations and authenticates requests to your Kafka cluster. - **Schema Registry API:** Allows Redpanda Console to manage and display schema information. - **Admin API:** Unlocks management features such as viewing cluster details, managing users, and generating debug bundles. To ensure secure and reliable communication, each connection must be configured with the appropriate authentication, TLS, and startup settings. The authentication mechanism you configure in Redpanda Console must match the one set up in your Redpanda cluster, ensuring that the user’s credentials are properly verified. This guide provides detailed instructions and examples for configuring each of these connections. ## [](#prerequisites)Prerequisites Ensure that you have: - Access to your Redpanda cluster’s broker URLs. - The necessary configuration details if your cluster requires authentication, TLS, or other security measures. ## [](#configure-a-connection-to-the-kafka-api)Configure a connection to the Kafka API Redpanda Console must be configured to communicate with the Kafka API on your Redpanda brokers. This configuration involves specifying the broker endpoints and, if needed, configuring client identification, SASL, and TLS settings. Here is an example configuration for the Kafka API using OAuth for SASL authentication: ### Standalone ```yaml kafka: brokers: - "broker1.example.com:9092" - "broker2.example.com:9092" # Optional client identification: # clientId: "console" # rackId: "zone-a" sasl: enabled: true mechanism: OAUTHBEARER impersonateUser: false oauth: # token: "example-oauth-token" (for static token) clientId: "example-client-id" clientSecret: "example-client-secret" tokenEndpoint: "https://accounts.google.com/token" scope: "openid" ``` ### Kubernetes embedded Operator When deploying Redpanda Console with the Redpanda Operator, you can connect Console to your Redpanda cluster in two ways: - **clusterRef:** Reference the Redpanda cluster by name for automatic connection. This is the recommended approach for most users. - **staticConfiguration:** Manually specify connection details for Kafka, Admin API, and Schema Registry if you need custom settings. The Operator will automatically configure Console to connect to the referenced cluster when using `clusterRef`. Use `staticConfiguration` only if you need to override the default connection or connect to a different cluster. Helm chart Redpanda Console is automatically configured to connect to the Redpanda cluster managed by the Helm chart. No additional connection configuration is required. Only configure the `kafka` block if you need to override the default connection settings (for example, to connect to a different cluster or use custom authentication). ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: kafka: brokers: - "broker1.example.com:9092" - "broker2.example.com:9092" # Optional client identification: # clientId: "console" # rackId: "zone-a" sasl: enabled: true mechanism: OAUTHBEARER impersonateUser: false oauth: # token: "example-oauth-token" (for static token) clientId: "example-client-id" clientSecret: "example-client-secret" tokenEndpoint: "https://accounts.google.com/token" ``` > 📝 **NOTE** > > When Redpanda Console is embedded in the Redpanda deployment, you can usually omit the broker addresses as they are automatically configured to connect to the Redpanda cluster in the same deployment. For clusters using Kerberos, the configuration may resemble the following: ```yaml kafka: brokers: - "broker1.example.com:9092" sasl: enabled: true mechanism: GSSAPI impersonateUser: false # Must be false when using Kerberos with Redpanda Console gssapi: authType: KEYTAB_AUTH keyTabPath: "/path/to/keytab" kerberosConfigPath: "/path/to/krb5.conf" serviceName: "kafka" username: "example-user" realm: "EXAMPLE.REALM" enableFast: true ``` > 📝 **NOTE** > > Kerberos (GSSAPI) configurations cannot be used with user impersonation enabled, because Redpanda Console supports only OIDC and basic authentication for its own login authentication. When using Kerberos, ensure `impersonateUser` is set to `false`. | Kafka API Configuration Option | Description | | --- | --- | | brokers | A list of Kafka broker endpoints including the URL scheme (http:// or https://), hostname, and port. Include all brokers in your cluster for high availability. | | clientId (optional) | An identifier used by Redpanda Console to identify itself to the cluster. | | rackId (optional) | Specifies the rack for multi-zone clusters to optimize local message consumption. | | sasl (optional) | Contains settings for SASL-based authentication. Configure this block if your cluster requires authentication through one of the following mechanisms:Basic authentication: Provide username and password.OAuth 2.0: Provide token to use a static token or provide clientId, clientSecret, tokenEndpoint, and scope to acquire new tokens at runtime.Kerberos (GSSAPI): Provide parameters such as authType, keyTabPath, kerberosConfigPath, serviceName, username, password, realm, and enableFast.Supported mechanisms:PLAINSCRAM-SHA-256 or SCRAM-SHA-512GSSAPIOAUTHBEARERFor more detailed instructions on configuring authentication, see Authentication in Redpanda Console. | | startup (optional) | Controls connection behavior at startup:establishConnectionEagerly: Tests the connection immediately.maxRetries, retryInterval, maxRetryInterval, backoffMultiplier: Define the retry logic for establishing a connection. | | tls (optional) | Contains settings to secure the connection using TLS. Specify paths for the CA certificate, client certificate, and client key. Optionally configure insecureSkipTlsVerify for testing purposes. | ## [](#configure-access-to-the-schema-registry)Configure access to the Schema Registry To enable schema management features, you must configure Redpanda Console to connect to the Schema Registry API. This includes specifying the service endpoints and, if needed, setting up authentication. Example Schema Registry configuration: ```yaml schemaRegistry: enabled: true urls: - "broker1.example.com:8081" - "broker2.example.com:8081" # Optional authentication settings authentication: impersonateUser: true #basic: #username: "example-user" #password: "example-password" #bearerToken: "example-bearer-token" ``` | Schema Registry Configuration Option | Description | | --- | --- | | urls | A list of Schema Registry endpoints including the URL scheme (http:// or https://), hostname, and port. Include all endpoints for redundancy. | | authentication (optional) | Configure authentication for the Schema Registry. Options include basic authentication or bearer tokens. For more detailed instructions on configuring authentication, see Authentication in Redpanda Console. | ## [](#admin)Configure access to the Redpanda Admin API Configuring a connection to the Redpanda Admin API unlocks additional features in Redpanda Console, including viewing the current Redpanda version, managing data transforms, administering SASL-SCRAM users, and generating debug bundles. This section details the configuration options and how they interact. Example configuration template: ```yaml redpanda: adminApi: enabled: true # Enable connection to the Admin API. urls: - "broker1.example.com:9644" # Provide all endpoints (host:port) for high availability. - "broker2.example.com:9644" # Optional authentication settings authentication: impersonateUser: true # Use the logged-in user's credentials for authentication. # For basic authentication. #basic: #username: "example-user" #password: "example-password" # For OIDC, use the `bearerToken` field instead of `basic`. #bearerToken: "example-bearer-token" startup: establishConnectionEagerly: true # Test the connection at startup. maxRetries: 5 # Maximum number of retry attempts. retryInterval: 1s # Initial wait time between retries. maxRetryInterval: 60s # Maximum wait time between retries. backoffMultiplier: 2 # Multiplier for increasing retry intervals. ``` > 📝 **NOTE** > > Include the URLs of _all_ endpoints in the `redpanda.adminApi.urls` array. For some requests such as collecting debug bundles, Redpanda Console must be able to communicate with all brokers individually. | Redpanda Admin API Configuration Option | Description | | --- | --- | | enabled | Activates the connection to the Admin API. Set to true to enable the integration. | | urls | A list of Admin API endpoints including the URL scheme (http:// or https://), hostname, and port. Providing all URLs enhances reliability. | | authentication (optional) | Provides credentials using basic authentication or bearer tokens when impersonation is disabled. For more detailed instructions on configuring authentication, see Authentication in Redpanda Console. | | startup (optional) | Controls connection behavior at startup:establishConnectionEagerly: Tests the connection immediately.maxRetries, retryInterval, maxRetryInterval, backoffMultiplier: Define the retry logic for establishing a connection. | ## [](#suggested-reading)Suggested reading - [Redpanda Console Security](https://docs.redpanda.com/streaming/current/console/config/security/) - [Configure Message Deserialization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/deserialization/) --- # Page 5: Configure Message Deserialization in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/deserialization.md --- # Configure Message Deserialization in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Message Deserialization in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/deserialization page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/deserialization.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/deserialization.adoc description: Learn how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- Redpanda Console provides tools for deserializing and inspecting messages in Kafka topics. This topic explains how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed. ## [](#sr)Use Schema Registry The Schema Registry allows Redpanda Console to dynamically retrieve schemas for deserializing Avro, Protobuf, and JSON messages. This setup is important to ensure that messages are correctly interpreted based on your schema definitions. See [Configure access to the Schema Registry](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/#sr). ## [](#protobuf-configuration)Protobuf configuration Redpanda Console supports several methods for providing Protobuf schemas, including the Schema Registry, local file system, and GitHub repositories. > 📝 **NOTE** > > You don’t need to provide standard types, such as Google’s timestamp, in your schemas. These standard types are included by default. Most Kafka clients that serialize Protobuf messages put the serialized byte array into a binary wrapper that contains meta information, like the schema ID or the used prototypes, so the application that deserializes the Kafka records must recognize the format. The deserialization process requires Redpanda Console to be aware of the used Protobuf files as well as a mapping of what prototype should be used for each topic. This information can either be sourced from the Schema Registry or it can be provided with additional configuration so the files can be pulled from the local file system or a GitHub repository. ### [](#sr-protobuf)Use Schema Registry for Protobuf If you use a Schema Registry for Protobuf deserialization, Redpanda Console can automatically fetch and use the required schemas without the need for manual topic mappings or additional configuration. When using Schema Registry for Protobuf, you must not configure `serde.protobuf`. Redpanda Console detects and uses the Protobuf schemas from the Schema Registry automatically. If you configure `serde.protobuf`, it enables manual deserialization mode, which requires you to specify topic mappings and source providers. Without those, Redpanda Console fails to start due to validation errors. When using Schema Registry for Protobuf, you do not need to provide specific topic mappings, as the schemas will be fetched dynamically. ### [](#topic-mapping)Topic mapping If you’re not [using a Schema Registry for Protobuf deserialization](#sr-protobuf), you must manually provide mappings between Kafka topics and their corresponding Protobuf types. This is necessary to inform Redpanda Console of the correct types to use for deserialization. Consider a Kafka topic called `address-v1` and a corresponding `address.proto` file with the following structure: `address.proto` ```proto syntax = "proto3"; package fake_models; option go_package = "pkg/protobuf"; message Address { int32 version = 1; string id = 2; message Customer { string customer_id = 1; string customer_type = 2; } } ``` To map this topic to the Protobuf schema, use the following configuration: #### Standalone ```yaml serde: protobuf: enabled: true mappings: - topicName: address-v1 valueProtoType: fake_models.Address # The full Protobuf type name # keyProtoType: Not specified because the key is a plain string ``` #### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: ##### Operator `redpanda-cluster`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: serde: protobuf: enabled: true mappings: - topicName: address-v1 valueProtoType: fake_models.Address # keyProtoType: Not specified because the key is a plain string ``` ##### Helm `redpanda-values.yaml` ```yaml console: enabled: true console: config: serde: protobuf: enabled: true mappings: - topicName: address-v1 valueProtoType: fake_models.Address # The full Protobuf type name # keyProtoType: Not specified because the key is a plain string ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: serde: protobuf: enabled: true mappings: - topicName: address-v1 valueProtoType: fake_models.Address # The full Protobuf type name # keyProtoType: Not specified because the key is a plain string ``` - `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization. - `serde.protobuf.mappings.topicName`: The name of the Kafka topic. - `serde.protobuf.mappings.valueProtoType`: The fully-qualified Protobuf type for the message value. - `serde.protobuf.mappings.keyProtoType`: Specify the key Protobuf type if the key is not a plain string. ### [](#local-file-system)Local file system You can mount Protobuf files directly from your local file system. Redpanda Console will search the specified paths for Protobuf files and build a registry with all the available types. Configuration example: #### Standalone ```yaml serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey fileSystem: enabled: true # How often to refresh the Protobuf files from the filesystem refreshInterval: 5m # Directories containing the Protobuf files paths: - /etc/protos ``` #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Protobuf deserialization through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey fileSystem: enabled: true refreshInterval: 5m paths: - /etc/protos ``` ##### Helm ```yaml console: enabled: true console: config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey fileSystem: enabled: true refreshInterval: 5m paths: - /etc/protos ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey fileSystem: enabled: true refreshInterval: 5m paths: - /etc/protos ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` - `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization. - `serde.protobuf.fileSystem.paths`: Paths to directories where Protobuf files are stored. - `serde.protobuf.fileSystem.refreshInterval`: The frequency at which Redpanda Console checks for updates to these files. ### [](#github-repository)GitHub repository If your Protobuf files are stored in a GitHub repository, Redpanda Console can fetch and use them directly. This is particularly useful if your organization maintains Protobuf definitions in version control. Configuration example: #### Standalone ```yaml serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey git: enabled: true repository: url: https://github.com/myorg/kafka-proto-files.git branch: master # How often to pull the git repository to refresh the schema files refreshInterval: 10m # Where all .proto files are stored in the git repository paths: - ./ ``` #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure GitHub repository access through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey git: enabled: true repository: url: https://github.com/myorg/kafka-proto-files.git branch: master refreshInterval: 10m paths: - ./ ``` ##### Helm ```yaml console: enabled: true console: config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey git: enabled: true repository: url: https://github.com/myorg/kafka-proto-files.git branch: master refreshInterval: 10m paths: - ./ ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: serde: protobuf: enabled: true mappings: - topicName: orders valueProtoType: fake_models.Order keyProtoType: fake_models.OrderKey git: enabled: true repository: url: https://github.com/myorg/kafka-proto-files.git branch: master refreshInterval: 10m paths: - ./ ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` - `serde.protobuf.enabled`: Set to `true` to enable Protobuf deserialization. - `serde.protobuf.git.repository.url`: The URL of the GitHub repository containing your Protobuf files. - `serde.protobuf.git.basicAuth`: Basic authentication credentials, often an API token for private repositories. - `serde.protobuf.git.refreshInterval`: Frequency at which the repository is polled for updates. #### [](#private-git-repositories)Private Git repositories If Protobuf files are stored in a private GitHub repository, Redpanda Console must authenticate using one of the following methods: - A [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) (PAT) over HTTPS (basic auth) - An [SSH private key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh) ##### [](#authenticate-using-a-github-personal-access-token-pat)Authenticate using a GitHub Personal Access Token (PAT) Use this method to authenticate to GitHub over HTTPS using a personal access token. ###### Standalone 1. Set environment variables: ```bash SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME=token SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD= ``` 2. Configure Redpanda Console: ```yaml serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: https://github.com//.git branch: refreshInterval: 10m paths: - ./ basicAuth: enabled: true ``` ###### Kubernetes embedded 1. Create a secret: ```yaml apiVersion: v1 kind: Secret metadata: name: protobuf-git-auth namespace: redpanda type: Opaque stringData: SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: ``` 2. Reference the secret and set the username: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda extraEnv: - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: protobuf-git-auth config: serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: https://github.com//.git branch: refreshInterval: 10m paths: - ./ basicAuth: enabled: true ``` ###### Helm ```yaml console: enabled: true extraEnv: - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: protobuf-git-auth console: config: serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: https://github.com//.git branch: refreshInterval: 10m paths: - ./ basicAuth: enabled: true ``` ###### Kubernetes standalone 1. Create a secret: ```yaml apiVersion: v1 kind: Secret metadata: name: protobuf-git-auth namespace: redpanda type: Opaque stringData: SERDE_PROTOBUF_GIT_BASICAUTH_PASSWORD: ``` 2. Update Helm values: ```yaml config: serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: https://github.com//.git branch: refreshInterval: 10m paths: - ./ basicAuth: enabled: true extraEnv: - name: SERDE_PROTOBUF_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: protobuf-git-auth ``` Replace the following values: - ``: A GitHub personal access token that has `repo` scope - ``: Kafka topic to be deserialized - ``: Fully qualified Protobuf message type (for example, `com.example.Order`) - ``: GitHub organization or user that owns the repository - ``: Name of the repository containing `.proto` files - ``: Git branch to clone (for example, `main`) #### [](#authenticate-using-ssh)Authenticate using SSH Use this method to authenticate with GitHub over SSH using a private key. ##### Standalone 1. Save the SSH private key on the local filesystem (for example, `/etc/redpanda/ssh/id_rsa`). 2. Set environment variables: ```bash SERDE_PROTOBUF_GIT_SSH_ENABLED=true SERDE_PROTOBUF_GIT_SSH_USERNAME=git SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH=/etc/redpanda/ssh/id_rsa SERDE_PROTOBUF_GIT_SSH_PASSPHRASE= ``` 3. Configure Redpanda Console: ```yaml serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: git@github.com:/.git branch: refreshInterval: 10m paths: - ./ ssh: enabled: true ``` ##### Kubernetes embedded 1. Create a secret with the SSH key: ```yaml apiVersion: v1 kind: Secret metadata: name: protobuf-git-ssh namespace: redpanda type: Opaque stringData: privateKey: | -----BEGIN OPENSSH PRIVATE KEY----- -----END OPENSSH PRIVATE KEY----- passphrase: ``` 2. Mount the secret and configure environment variables: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: protobuf-git-ssh extraEnv: - name: SERDE_PROTOBUF_GIT_SSH_ENABLED value: "true" - name: SERDE_PROTOBUF_GIT_SSH_USERNAME value: git - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE value: ``` ###### Helm ```yaml console: enabled: true extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: protobuf-git-ssh extraEnv: - name: SERDE_PROTOBUF_GIT_SSH_ENABLED value: "true" - name: SERDE_PROTOBUF_GIT_SSH_USERNAME value: git - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE value: ``` ##### Kubernetes standalone 1. Create a secret: ```yaml apiVersion: v1 kind: Secret metadata: name: protobuf-git-ssh namespace: redpanda type: Opaque stringData: privateKey: | -----BEGIN OPENSSH PRIVATE KEY----- -----END OPENSSH PRIVATE KEY----- passphrase: ``` 2. Update Helm values: ```yaml config: serde: protobuf: enabled: true mappings: - topicName: valueProtoType: git: enabled: true repository: url: git@github.com:/.git branch: refreshInterval: 10m paths: - ./ ssh: enabled: true extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: protobuf-git-ssh extraEnv: - name: SERDE_PROTOBUF_GIT_SSH_ENABLED value: "true" - name: SERDE_PROTOBUF_GIT_SSH_USERNAME value: git - name: SERDE_PROTOBUF_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: SERDE_PROTOBUF_GIT_SSH_PASSPHRASE value: ``` Replace the following values: - ``: Kafka topic to be deserialized - ``: Fully qualified Protobuf message type (for example, `com.example.Order`) - ``: GitHub organization or user that owns the repository - ``: Name of the repository containing `.proto` files - ``: Git branch to clone (for example, `main`) - ``: SSH private key content used to authenticate to GitHub (must be base64-safe if stored in secrets) - ``: Passphrase used to decrypt the SSH private key, if applicable ## [](#messagepack-deserialization)MessagePack deserialization If your data is serialized using MessagePack, Redpanda Console can be configured to deserialize it. ### Standalone ```yaml serde: messagePack: enabled: true # Define which topics use MessagePack serialization # Regex to match all topics by default topicNames: ["/.*/"] ``` ### Kubernetes embedded #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: serde: messagePack: enabled: true topicNames: - "/.*/" ``` #### Helm ```yaml console: enabled: true console: config: serde: messagePack: enabled: true topicNames: ["/.*/"] ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: serde: messagePack: enabled: true topicNames: ["/.*/"] ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` - `serde.messagePack.enabled`: Enables MessagePack deserialization. - `serde.messagePack.topicNames`: A list of topic name regex patterns that specify which topics use MessagePack serialization. The default pattern (`/.*/`) matches all topics. ## [](#best-practices)Best practices - Use Schema Registry when possible. Schema Registry simplifies schema management and ensures that all messages are serialized and deserialized consistently across your Kafka ecosystem. - Organize Protobuf files. Whether using a local file system or a GitHub repository, keep your Protobuf files organized and use consistent naming conventions to avoid confusion. - Monitor deserialization performance. Regularly check the performance impact of deserialization, especially when using complex Protobuf schemas or large numbers of messages. Adjust refresh intervals and schema caching as needed. - Secure access. Ensure that credentials for accessing the Schema Registry or GitHub repositories are securely managed and rotated regularly. ## [](#troubleshooting)Troubleshooting If you encounter issues with deserialization: - Ensure that the Schema Registry URL and credentials are correctly configured and accessible. - Check your topic mappings and Protobuf type names for accuracy. - Review the Redpanda Console for insights into any errors occurring during deserialization. --- # Page 6: Add a License Key to Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/enterprise-license.md --- # Add a License Key to Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Add a License Key to Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/enterprise-license page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/enterprise-license.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/enterprise-license.adoc description: Learn how to apply or update a license key to Redpanda Console. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- To enable [enterprise features for Redpanda Console](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/#console), you must have an Enterprise Edition license to load at startup. This guide explains how to configure Redpanda Console to load the license key from its local configuration. > 💡 **TIP** > > This option is best for deployments that are not connected to a Redpanda cluster. If you plan to connect Redpanda Console to a Redpanda cluster, consider uploading the license to the Redpanda cluster. See [Add an Enterprise Edition License to Redpanda Console](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-console/). ## [](#prerequisites)Prerequisites You must have an Enterprise Edition license. To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). If Redpanda Console has enterprise features enabled and cannot find a valid license locally or in the connected Redpanda cluster, it shuts down. See [Redpanda Licenses and Enterprise Features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/). ## [](#add-a-new-license-to-redpanda-console)Add a new license to Redpanda Console To add a new license to Redpanda Console, you have two options: - [Provide the path to the license file](#file). - [Provide the license key contents directly](#inline). ### [](#file)Use a license file #### Standalone Set the `licenseFilepath` property in the `/etc/redpanda/redpanda-console-config.yaml` configuration file: ```yaml licenseFilepath: ``` Or set the `REDPANDA_LICENSE_FILEPATH` environment variable: ```bash export REDPANDA_LICENSE_FILEPATH= ``` #### Kubernetes embedded By default, when deploying Redpanda Console with the Redpanda Operator or Redpanda Helm chart, Redpanda Console inherits the license provided to Redpanda in `enterprise.license` or `enterprise.licenseSecretRef`. See [Add an Enterprise Edition License to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/kubernetes/) for details. You cannot override the inherited license. #### Kubernetes standalone For production deployments, the safest and most secure way to provide your license is to store it in a Kubernetes Secret and reference it using `enterprise.licenseSecretRef` in your `values.yaml`. This keeps sensitive license data out of your Helm values and version control. Example: ```yaml enterprise: licenseSecretRef: name: key: ``` Alternatively, you can set the license directly in your Helm values using `console.config.license` or `console.config.licenseFilepath`, or with environment variables, but these methods are less secure for production environments. ```yaml console: extraEnv: - name: REDPANDA_LICENSE value: ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` ### [](#inline)Use the license key contents directly If you don’t want to provide a path to the license file, you can use the contents of the license key directly. #### Standalone Set the `license` property in the `/etc/redpanda/redpanda-console-config.yaml` configuration file: ```yaml license: ``` Or set the `REDPANDA_LICENSE` environment variable: ```bash export REDPANDA_LICENSE= ``` #### Kubernetes embedded By default, when deploying Redpanda Console with the Redpanda Operator or Redpanda Helm chart, Redpanda Console inherits the license provided to Redpanda in `enterprise.license` or `enterprise.licenseSecretRef`. See [Add an Enterprise Edition License to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/kubernetes/) for details. You cannot override the inherited license. #### Kubernetes standalone For production deployments, the safest and most secure way to provide your license is to store it in a Kubernetes Secret and reference it using `enterprise.licenseSecretRef` in your `values.yaml`. This keeps sensitive license data out of your Helm values and version control. Example: ```yaml enterprise: licenseSecretRef: name: key: ``` Alternatively, you can set the license directly in your Helm values using `console.config.license` or `console.config.licenseFilepath`, or with environment variables, but these methods are less secure for production environments. ```yaml console: extraEnv: - name: REDPANDA_LICENSE value: ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` ## [](#update-an-existing-license)Update an existing license To update an existing license: 1. Update your configuration file or environment variables with one of the following: - [The path to your new license file](#file) - [The contents of your new license key](#inline) 2. Restart Redpanda Console to make the changes take effect. ## [](#next-steps)Next steps [Check the Status of Licenses](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/). ## [](#suggested-reading)Suggested reading - [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/) - [Manage Enterprise Edition Licenses through Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/) --- # Page 7: HTTP Path Rewrites in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/http-path-rewrites.md --- # HTTP Path Rewrites in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: HTTP Path Rewrites in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/http-path-rewrites page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/http-path-rewrites.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/http-path-rewrites.adoc description: Learn how to configure Redpanda Console to work with your URL path rewrites, particularly when hosted under a subpath. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- If you want to host Redpanda Console under a subpath rather than the root path, you need to configure HTTP path rewrites. This allows you to serve Redpanda Console under a subpath of your domain, such as `https://my-company.com/redpanda/console`, instead of directly from `https://my-company.com`. This type of configuration is often necessary when: - You have multiple services and applications running under the same domain. - Redpanda Console is behind a reverse proxy. The proxy might add a path prefix based on routing rules, and Redpanda Console needs to know about this prefix to handle requests correctly. > 📝 **NOTE** > > If you host Redpanda Console at a root path, for example under a sub-domain such as `https://console.redpanda.my-company.com`, you don’t need to configure HTTP path rewrites. ## [](#configuration)Configuration To configure HTTP path rewrites, set the following properties within the `server` object in your Redpanda Console configuration file: | Configuration | Description | Default | | --- | --- | --- | | basePath | The subpath under which Redpanda Console is hosted.If you have a proxy in front of Redpanda Console that sets the X-Forwarded-Prefix header and setBasePathFromXForwardedPrefix is enabled, you do not need to set basePath manually.See Custom subpath scenario. | '' | | setBasePathFromXForwardedPrefix | Tells Redpanda Console to use the X-Forwarded-Prefix header on incoming requests for determining the path prefix.If this header is present, its value will be used as the path prefix, and the value set in basePath will be ignored.See Custom subpath scenario. | true | | stripPrefix | Removes the specified prefix from the request path before routing it to Redpanda Console.If your proxy is configured to remove the prefix, you should disable stripPrefix in Redpanda Console.See Prefix removal scenario. | true | ## [](#custom-subpath-scenario)Custom subpath scenario Consider the following setup where you want to host Redpanda Console at `https://my-company.com/redpanda/console/` and are using a reverse proxy to manage the routing: 1. Nginx is configured to route requests from `/redpanda/console` to Redpanda Console, running on `http://localhost:8080`. ### Standalone Nginx configuration ```nginx location /redpanda/console/ { proxy_pass http://localhost:8080/; proxy_set_header X-Forwarded-Prefix /redpanda/console; } ``` ### Kubernetes ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: redpanda-console annotations: # Pass the base-path header through to Redpanda Console nginx.ingress.kubernetes.io/configuration-snippet: | proxy_set_header X-Forwarded-Prefix /redpanda/console; spec: ingressClassName: nginx # or your ingress class name tls: - hosts: - my-company.com secretName: my-company-tls # optional rules: - host: my-company.com http: paths: - path: /redpanda/console pathType: Prefix backend: service: name: redpanda-console # your Service name port: number: 8080 # your Service port ``` 2. Redpanda Console is configured to use the `X-Forwarded-Prefix` header for determining the base path. ### Standalone ```yaml server: setBasePathFromXForwardedPrefix: true stripPrefix: true ``` ### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: server: setBasePathFromXForwardedPrefix: true stripPrefix: true ``` #### Helm ```yaml console: enabled: true console: config: server: setBasePathFromXForwardedPrefix: true stripPrefix: true ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: server: setBasePathFromXForwardedPrefix: true stripPrefix: true ``` 3. A user navigates to `https://my-company.com/redpanda/console/topics` in their browser. 4. Nginx receives the request and recognizes the `/redpanda/console` subpath. It forwards the request to Redpanda Console at `http://localhost:8080/topics`, while adding the `X-Forwarded-Prefix: /redpanda/console` header. 5. Because `setBasePathFromXForwardedPrefix` is set to `true`, Redpanda Console checks the `X-Forwarded-Prefix` header. It identifies that the base path is `/redpanda/console` and removes this prefix from the incoming request path. The request path `/topics` is now correctly routed in Redpanda Console. If Nginx were configured without the `X-Forwarded-Prefix` header, or if `setBasePathFromXForwardedPrefix` was set to `false`, Redpanda Console would not correctly recognize the subpath, leading to routing issues. This configuration is particularly useful when multiple environments or proxies might route to the same Redpanda Console instance under different subpaths. ## [](#prefix-removal-scenario)Prefix removal scenario Some proxies, such as Traefik, can remove a prefix from the URL path before forwarding it. If both the proxy and Redpanda Console attempt to remove the prefix, Redpanda Console may fail to route the request correctly. Only one part of the stack (either the proxy or Redpanda Console) should remove the prefix. To better understand this problem, consider the following scenario: 1. Traefik is configured to route `/topics` to Redpanda Console with the "StripPrefix" middleware enabled. 2. Redpanda Console is configured with the default settings, which includes `stripPrefix: true`. 3. A user enters the following address in their browser: `example.com/topics/topics/example-topic`. 4. Traefik removes the first `/topics` from the path, leaving `/topics/example-topic`, and sets `/topics` in the `X-Forwarded-Prefix` header. 5. Redpanda Console sees the `X-Forwarded-Prefix` and attempts to remove what it thinks is the prefix. The path becomes `/example-topic`. 6. Redpanda Console then tries to find a handler for `/example-topic`, but no such route exists, leading to a failure. To avoid this issue, either disable `stripPrefix` in Redpanda Console or ensure that the proxy does not modify the request path in a conflicting manner. ## [](#proxy-rewrites)Proxy rewrites If you have a reverse proxy between Redpanda Console and Kafka Connect, ensure that any rewrite rules retain the necessary expand parameters in the query string. These parameters are crucial for Kafka Connect to return the correct details about the connectors: `://:8083/connectors?expand=info&expand=status` This ensures that Redpanda Console can correctly communicate with Kafka Connect even when hosted behind a proxy that rewrites URLs. --- # Page 8: Connect Redpanda Console to Kafka Connect Clusters **URL**: https://docs.redpanda.com/streaming/current/console/config/kafka-connect.md --- # Connect Redpanda Console to Kafka Connect Clusters > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Connect Redpanda Console to Kafka Connect Clusters latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/kafka-connect page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/kafka-connect.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/kafka-connect.adoc description: Learn how to connect one or more Kafka Connect clusters with Redpanda Console. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- > 📝 **NOTE: Community** > > **Kafka Connect is community-supported on [Redpanda Community Slack](https://redpanda.com/slack)**. Redpanda Data does not provide enterprise support for Kafka Connect with Redpanda Console. For a supported and scalable Kafka Connect alternative, try [Redpanda Connect](https://docs.redpanda.com/connect/get-started/). Redpanda Console provides a user interface that lets you manage multiple Kafka Connect clusters. You can inspect or patch connectors; restart, pause, and resume connector tasks; and delete connectors. Redpanda Console queries all configured Kafka Connect clusters for their status, so you have an overview of all your Kafka Connect clusters. ## [](#prerequisites)Prerequisites You must [deploy a Kafka Connect cluster separately](https://docs.redpanda.com/streaming/current/deploy/kafka-connect/deploy-kafka-connect/) before configuring Redpanda Console to connect to it. ## [](#configure-a-connection-to-kafka-connect)Configure a connection to Kafka Connect For each cluster, provide a unique name, the HTTP address of the cluster, and the authentication settings, if required. The name can be any unique string that helps you to identify the Kafka Connect cluster. See all available configuration options in the [Redpanda Console Configuration](https://docs.redpanda.com/streaming/current/console/config/configure-console/). ### Standalone ```yaml kafkaConnect: enabled: true clusters: - name: datawarehouse # Required field, will be used as identifier in the frontend url: http://dwh-connect.mycompany.com:8083 tls: enabled: false # Trusted certs are still allowed by default username: admin # password: # Set using flag --kafkaConnect.clusters.0.password=secret - name: analytics # Required field, will be used as identifier in the frontend url: http://analytics.mycompany.com:8083 # No auth configured on that cluster, hence no username/password set ``` ### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: #### Operator `redpanda-console`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: kafkaConnect: enabled: true clusters: - name: datawarehouse url: http://dwh-connect.mycompany.com:8083 tls: enabled: false username: admin # password: (add if needed) - name: analytics url: http://analytics.mycompany.com:8083 # No auth configured on this cluster ``` #### Helm `redpanda-values.yaml` ```yaml console: enabled: true console: config: kafkaConnect: enabled: true clusters: - name: datawarehouse # Required field, will be used as identifier in the frontend url: http://dwh-connect.mycompany.com:8083 tls: enabled: false # Trusted certs are still allowed by default username: admin # password: - name: analytics # Required field, will be used as identifier in the frontend url: http://analytics.mycompany.com:8083 # No auth configured on that cluster, hence no username/password set ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: kafkaConnect: enabled: true clusters: - name: datawarehouse # Required field, will be used as identifier in the frontend url: http://dwh-connect.mycompany.com:8083 tls: enabled: false # Trusted certs are still allowed by default username: admin # password: - name: analytics # Required field, will be used as identifier in the frontend url: http://analytics.mycompany.com:8083 # No auth configured on that cluster, hence no username/password set ``` --- # Page 9: Redpanda Console Security **URL**: https://docs.redpanda.com/streaming/current/console/config/security.md --- # Redpanda Console Security > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Console Security latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/security/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/security/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/security/index.adoc description: Learn about security topics for Redpanda Console. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- - [Authentication in Redpanda Console](authentication/) Learn how authentication in Redpanda Console enables users to log in and optionally forward their credentials to the connected Redpanda cluster, ensuring all API requests are executed under the user's identity. - [Authorization in Redpanda Console](authorization/) Learn how to configure role-based access control (RBAC) in Redpanda Console, binding roles to users or OIDC groups to restrict access to authorized users. - [TLS Termination in Redpanda Console](tls-termination/) Learn how to secure Redpanda Console using TLS, either by letting Redpanda Console handle TLS termination or by offloading it to an upstream component, such as a reverse proxy or a Cloud HTTPS LoadBalancer. --- # Page 10: Authentication in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/security/authentication.md --- # Authentication in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Authentication in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/security/authentication page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/security/authentication.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/security/authentication.adoc description: Learn how authentication in Redpanda Console enables users to log in and optionally forward their credentials to the connected Redpanda cluster, ensuring all API requests are executed under the user's identity. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-08-20" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda Console has enterprise features enabled and it cannot find a valid license, it redirects you to the license expiration landing page, and all other access is restricted. This topic describes how to enable authentication in Redpanda Console and how it integrates with the Kafka, Admin, and Schema Registry APIs in Redpanda. Authentication in Redpanda Console lets users log in and optionally forward their credentials to the connected cluster. This ensures API requests run under the user’s identity. Redpanda Console supports the following authentication methods: - **OIDC (OpenID Connect):** Integrates with external identity providers (IdPs) for single sign-on (SSO). - **Basic authentication:** Uses traditional username and password credentials. ![login](https://docs.redpanda.com/streaming/current/console/_images/login.png) ## [](#how-authentication-works)How authentication works Redpanda Console can authenticate to Redpanda APIs in two ways: - **User impersonation:** Uses the same credentials you log in with to authenticate API requests. This ensures accurate audit logs and unified identity enforcement. - **Static service account credentials:** Uses preconfigured credentials defined in the Redpanda Console configuration file. Useful when impersonation is disabled or [RBAC](https://docs.redpanda.com/streaming/current/reference/glossary/#rbac) needs to be separated from Redpanda identities. Upon login, Redpanda Console generates a secure session with a JSON Web Token (JWT), signed by the `authentication.jwtSigningKey`. The JWT can be stored as a secure cookie and is used to authenticate API requests. For OIDC-based login flows, Redpanda Console reuses the OAuth 2.0 access token from the identity provider (IdP) to authenticate to Redpanda’s Kafka and HTTP APIs. These access tokens must be in JWT format to be compatible with Redpanda’s SASL/OAUTHBEARER authentication. Some IdPs, such as Google, issue opaque access tokens that are not JWTs. While these tokens work for logging in to Redpanda Console (the ID token is a JWT), they cannot be used for impersonation with the Kafka API. In such cases, impersonation must be disabled, and Redpanda Console must be configured to use static service account credentials instead. flowchart TD A((User authenticates with OIDC or SASL credentials)) --> B\[Redpanda Console\] B --> C\["Re-use OIDC token or basic credentials (if configured)"\] B --> D\[Get credentials from Redpanda Console config\] C --> E\[Kafka API\] C --> F\[Admin API\] C --> G\[Schema API\] E & F & G --> H((Redpanda)) D --> J\[Kafka Connect API\] D --> L\[Git\] Figure 1. Redpanda Console authenticates users and then authorizes their access based on the impersonation mode and configured RBAC or ACLs. ## [](#prerequisites)Prerequisites - You must have at least one superuser in Redpanda. - The authentication method used in Redpanda Console must match the configuration of the Kafka API: - **If using OIDC:** You must have SASL/OAUTHBEARER authentication configured for the Kafka API. - **If using basic authentication:** You must have SASL/SCRAM authentication configured for the Kafka API. [Learn how to configure authentication for Redpanda](https://docs.redpanda.com/streaming/current/manage/security/authentication/). ## [](#enable-authentication)Enable authentication Redpanda Console supports enabling both OIDC and basic authentication simultaneously. If both are enabled, users can choose how to log in. ### [](#enable-oidc-authentication)Enable OIDC authentication When you enable OIDC authentication, Redpanda Console uses an external IdP to authenticate users. This allows for single sign-on (SSO) and centralized user management. Redpanda and Redpanda Console require OAuth 2.0-compliant JWT tokens for user authentication. When using OIDC, your IdP must issue JWTs. Redpanda Console uses these tokens to authenticate to Redpanda APIs through SASL/OAUTHBEARER or Bearer headers. #### Standalone ```yaml authentication: jwtSigningKey: "" (1) useSecureCookies: false (2) oidc: enabled: true (3) issuerUrl: "https://login.microsoftonline.com/a5da3be7-35c1-44ff-b6e8-b3b755686ae2/v2.0" (4) clientId: "" (5) clientSecret: "" (6) additionalScopes: - "" (7) issuerTls: enabled: true (8) caFilepath: "/path/to/ca.pem" certFilepath: "/path/to/issuer-cert.pem" keyFilepath: "/path/to/issuer-key.pem" insecureSkipTlsVerify: false redirectUrl: "http://localhost:8080/auth/callbacks/oidc" (9) accessType: "offline" (10) prompt: "consent" (11) principalMapping: "$.sub" (12) ``` #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Redpanda Console authentication through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authentication: jwtSigningKey: "" (1) useSecureCookies: false (2) oidc: enabled: true (3) issuerUrl: "https://login.microsoftonline.com/a5da3be7-35c1-44ff-b6e8-b3b755686ae2/v2.0" (4) clientId: "" (5) clientSecret: "" (6) additionalScopes: - "" (7) issuerTls: enabled: true (8) caFilepath: "/path/to/ca.pem" certFilepath: "/path/to/issuer-cert.pem" keyFilepath: "/path/to/issuer-key.pem" insecureSkipTlsVerify: false redirectUrl: "http://localhost:8080/auth/callbacks/oidc" (9) accessType: "offline" (10) prompt: "consent" (11) principalMapping: "$.sub" (12) ``` ##### Helm ```yaml console: enabled: true console: config: authentication: jwtSigningKey: "" (1) useSecureCookies: false (2) oidc: enabled: true (3) issuerUrl: "https://login.microsoftonline.com/a5da3be7-35c1-44ff-b6e8-b3b755686ae2/v2.0" (4) clientId: "" (5) clientSecret: "" (6) additionalScopes: - "" (7) issuerTls: enabled: true (8) caFilepath: "/path/to/ca.pem" certFilepath: "/path/to/issuer-cert.pem" keyFilepath: "/path/to/issuer-key.pem" insecureSkipTlsVerify: false redirectUrl: "http://localhost:8080/auth/callbacks/oidc" (9) accessType: "offline" (10) prompt: "consent" (11) principalMapping: "$.sub" (12) ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart, configure OIDC in your Helm values: ```yaml config: authentication: jwtSigningKey: "" (1) useSecureCookies: false (2) oidc: enabled: true (3) issuerUrl: "https://login.microsoftonline.com/a5da3be7-35c1-44ff-b6e8-b3b755686ae2/v2.0" (4) clientId: "" (5) clientSecret: "" (6) additionalScopes: - "" (7) issuerTls: enabled: true (8) caFilepath: "/path/to/ca.pem" certFilepath: "/path/to/issuer-cert.pem" keyFilepath: "/path/to/issuer-key.pem" insecureSkipTlsVerify: false redirectUrl: "http://localhost:8080/auth/callbacks/oidc" (9) accessType: "offline" (10) prompt: "consent" (11) principalMapping: "$.sub" (12) ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` | 1 | Required. Secret key for signing JWTs. Must be at least 32 characters. Store securely. You can also use the AUTHENTICATION_JWTSIGNINGKEY environment variable. | | --- | --- | | 2 | Recommended in production. Marks cookies as secure. | | 3 | Required. Enables OIDC authentication. | | 4 | Required. URL of the OIDC identity provider (IdP). | | 5 | Required. The client ID from your IdP. | | 6 | Required. The client secret from your IdP. You can also use the AUTHENTICATION_OIDC_CLIENTSECRET environment variable. | | 7 | Requested scopes. Some IdPs such as Azure Entra ID require additional scopes to request OAuth 2.0-compliant tokens. | | 8 | Optional. TLS configuration for secure connections to the IdP. Configure TLS only if you require mTLS or use a self-signed certificate. | | 9 | Optional. Redirect URI registered with the IdP. This URI must point to the /auth/callbacks/oidc path in Redpanda Console. If not set, Redpanda Console constructs the URL from the request. Configure this option explicitly if you’re using HTTP path rewrites. | | 10 | Optional. Controls whether a refresh token is requested. offline (default) requests a refresh token. Set to online to disable refresh token requests. | | 11 | Optional. Determines how the authorization prompt appears. Use consent (default) to force re-consent. Other options include none and select_account. Some IdPs require consent to issue a refresh token. | | 12 | Optional. Extracts and optionally transforms a claim from the OIDC token to use as the user’s identity in Redpanda Console role bindings. The default is "$.sub", which uses the sub (subject) claim from the token. This value is then compared to the name field in your authorization.roleBindings configuration. For full syntax and transformation examples, see Transform identities with principal mappings. | > ❗ **IMPORTANT** > > For any secret values, [use environment variables](https://docs.redpanda.com/streaming/current/console/config/configure-console/) instead of hardcoding them in the configuration file. For example, use `AUTHENTICATION_OIDC_CLIENTSECRET` for the client secret. #### [](#extract-group-claims)Extract group claims > 📝 **NOTE** > > These settings are available in Redpanda Console v3.7.0 and later. To assign Redpanda Console roles to groups instead of individual users, Redpanda Console must know where to find group memberships in the OIDC token. Two optional settings control this: ```yaml authentication: oidc: groupClaimPath: "$.groups" nestedGroupClaimBehavior: none ``` `groupClaimPath` A [JSON path](https://goessner.net/articles/JsonPath/) expression identifying the claim that holds group memberships. The default is `$.groups`. Nested paths such as `$.custom.groups` are supported. Identity providers differ here: Auth0 and Okta typically use a top-level `groups` claim, while Keycloak nests roles under `$.realm_access.roles`. This setting mirrors Redpanda’s [`oidc_group_claim_path`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#oidc_group_claim_path). `nestedGroupClaimBehavior` How to handle group names that use path-style notation, such as `/departments/eng/platform`. Set to `none` (the default) to keep the full value, or `flatten` to keep only the last segment. Redpanda Console rejects any other value at startup. > ⚠️ **CAUTION** > > With `nestedGroupClaimBehavior: flatten`, groups that share a final segment collapse to the same name. For example, `/departments/eng/platform` and `/departments/sales/platform` both become `platform`, so a group binding for `platform` applies to members of both. Choose group naming conventions that avoid these collisions. To bind these groups to Redpanda Console roles, see [Grant permissions to groups](https://docs.redpanda.com/streaming/current/console/config/security/authorization/#grant-permissions-to-groups). #### [](#oidc-limitations)OIDC limitations - Redpanda requires JWT-formatted access tokens (not ID tokens) for Kafka API authentication using SASL/OAUTHBEARER. Access tokens issued by some IdPs, such as Google, are opaque and not supported. - The `rpk` CLI does not support OIDC login. - Redpanda requires OIDC principals to be set as superusers to access the Admin API. Granular authorization is not supported. - The `rpk` CLI does not support the SASL/OAUTHBEARER mechanism for deploying data transforms. Use SASL/SCRAM instead. #### [](#supported-identity-providers)Supported identity providers You can use any OIDC-compliant IdP with Redpanda Console. Here are common providers: | Provider | Example issuerUrl | | --- | --- | | Okta | https:///oauth2/default | | Microsoft Entra ID (Azure AD) | https://login.microsoftonline.com//v2.0 | | Keycloak | https:///realms/ | Some IdPs, such as Google, issue opaque access tokens that are not JWTs. While these tokens work for logging in to Redpanda Console (the ID token is a JWT), they cannot be used for impersonation with the Kafka API. In such cases, impersonation must be disabled, and Redpanda Console must be configured to use static service account credentials instead. For example, this is how to configure Redpanda Console with Entra ID: ##### Standalone ```yaml authentication: jwtSigningKey: vazxnT+ZHtxKslK6QlDGovcYnSjTk/lKMmZ+mHrBVE+YdVDkLgSuP6AszAKe9Gvq basic: enabled: true oidc: enabled: true issuerUrl: "https://login.microsoftonline.com//v2.0" clientId: "" clientSecret: "" redirectUrl: "http://localhost:8080/auth/callbacks/oidc" accessType: "offline" prompt: "consent" additionalScopes: - "api:///entraid.v2-access-tokens" (1) ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Redpanda Console authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authentication: jwtSigningKey: vazxnT+ZHtxKslK6QlDGovcYnSjTk/lKMmZ+mHrBVE+YdVDkLgSuP6AszAKe9Gvq basic: enabled: true oidc: enabled: true issuerUrl: "https://login.microsoftonline.com//v2.0" clientId: "" clientSecret: "" redirectUrl: "http://localhost:8080/auth/callbacks/oidc" accessType: "offline" prompt: "consent" additionalScopes: - "api:///entraid.v2-access-tokens" (1) ``` ###### Helm ```yaml console: enabled: true console: config: authentication: jwtSigningKey: vazxnT+ZHtxKslK6QlDGovcYnSjTk/lKMmZ+mHrBVE+YdVDkLgSuP6AszAKe9Gvq basic: enabled: true oidc: enabled: true issuerUrl: "https://login.microsoftonline.com//v2.0" clientId: "" clientSecret: "" redirectUrl: "http://localhost:8080/auth/callbacks/oidc" accessType: "offline" prompt: "consent" additionalScopes: - "api:///entraid.v2-access-tokens" (1) ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart, configure authentication in your Helm values: ```yaml config: authentication: jwtSigningKey: vazxnT+ZHtxKslK6QlDGovcYnSjTk/lKMmZ+mHrBVE+YdVDkLgSuP6AszAKe9Gvq basic: enabled: true oidc: enabled: true issuerUrl: "https://login.microsoftonline.com//v2.0" clientId: "" clientSecret: "" redirectUrl: "http://localhost:8080/auth/callbacks/oidc" accessType: "offline" prompt: "consent" additionalScopes: - "api:///entraid.v2-access-tokens" (1) ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` | 1 | In Entra ID, scopes are required to explicitly request OAuth 2.0-compliant access tokens. See Microsoft documentation for more information. | | --- | --- | #### [](#connect-clients-to-redpanda)Connect clients to Redpanda When using OIDC, clients authenticate to Redpanda using OAuth 2.0 access tokens (JWTs). These tokens are issued by your identity provider (IdP) and must be refreshed before they expire. Token refresh can be handled in different ways depending on the Kafka client library. For example, with [KafkaJS](https://kafka.js.org/docs/configuration#oauthbearer-example), use the `oauthBearerProvider` option to provide a token refresh function. ### [](#enable-basic-authentication)Enable basic authentication To configure basic authentication, choose the configuration method that matches your deployment: #### Standalone ```yaml authentication: jwtSigningKey: "" (1) useSecureCookies: true (2) basic: enabled: true (3) ``` #### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: ##### Operator `redpanda-cluster`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authentication: jwtSigningKey: "" (1) useSecureCookies: true (2) basic: enabled: true (3) ``` ##### Helm `redpanda-values.yaml` ```yaml console: enabled: true console: config: authentication: jwtSigningKey: "" (1) useSecureCookies: true (2) basic: enabled: true (3) ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: authentication: jwtSigningKey: "" (1) useSecureCookies: true (2) basic: enabled: true (3) ``` | 1 | Required. Secret key for JWTs. Must be at least 32 characters. | | --- | --- | | 2 | Recommended in production. Marks cookies as secure. | | 3 | Required. Enables username/password login. | #### [](#connect-clients-to-redpanda-2)Connect clients to Redpanda When using basic authentication, clients authenticate to Redpanda using a SASL/SCRAM username and password. The credentials must match a user configured in the Redpanda cluster. Most Kafka client libraries support SASL/SCRAM out of the box. You must configure the client with: - `sasl.mechanism`: One of `SCRAM-SHA-256` or `SCRAM-SHA-512` - `sasl.username`: The Redpanda username - `sasl.password`: The corresponding password ## [](#configure-session-duration)Configure session duration By default, Redpanda Console sessions remain valid for one year. For enterprise deployments, you can limit the maximum session duration using the `maximumSessionAge` configuration parameter. When a session exceeds the configured maximum age, users must re-authenticate to continue using Redpanda Console. ### Standalone ```yaml authentication: jwtSigningKey: "" useSecureCookies: true maximumSessionAge: "90d" (1) basic: enabled: true ``` ### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart: #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authentication: jwtSigningKey: "" useSecureCookies: true maximumSessionAge: "90d" (1) basic: enabled: true ``` #### Helm ```yaml console: enabled: true console: config: authentication: jwtSigningKey: "" useSecureCookies: true maximumSessionAge: "90d" (1) basic: enabled: true ``` ### Kubernetes standalone ```yaml config: authentication: jwtSigningKey: "" useSecureCookies: true maximumSessionAge: "90d" (1) basic: enabled: true ``` | 1 | Maximum duration for browser sessions. Accepts duration strings such as 90d (90 days), 24h (24 hours), or 30m (30 minutes). If not specified, sessions remain valid for one year. | | --- | --- | ## [](#configure-api-authentication)Configure API authentication After enabling authentication, you must configure how Redpanda Console authenticates to each Redpanda API: Kafka, Admin, and Schema Registry. Choose one method per API: - **User impersonation:** Uses the login credentials of the current user. - **Static credentials:** Uses preconfigured credentials of a superuser to communicate with Redpanda, and role bindings to control access in Redpanda Console for logged in users. > 💡 **TIP** > > Redpanda Data recommends user impersonation so that access control is fine-grained and centralized within Redpanda. This way, audit logs are also more accurate, as they reflect the actual user identity. > 📝 **NOTE** > > When using OIDC with static credentials, Redpanda Console authenticates to Redpanda as the OIDC client itself (usually a service principal). In this case, Redpanda evaluates access based on the `sub` claim in the token. Ensure you grant ACLs for your principals. For help creating ACLs, see [Configure Access Control Lists](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/). ### [](#kafka-api-examples)Kafka API examples This section provides examples of how to configure authentication for communicating with the Kafka API from Redpanda Console. You can choose between user impersonation or static credentials. #### [](#user-impersonation)User impersonation This option is useful when you want to use the same login credentials to authenticate Kafka API requests in Redpanda. This ensures accurate audit logs and enforces unified identity. ##### Standalone ```yaml kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: true ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Kafka authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: true ``` ###### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: true ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: true ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-scram)Static credentials with SCRAM This option is useful when you want to use basic authentication. Redpanda Console uses the provided credentials for authentication. ##### Standalone ```yaml kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false username: "console-superuser" password: "superuser-password" mechanism: "SCRAM-SHA-256" authorization: roleBindings: - roleName: viewer users: - loginType: basic name: "matt" ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Kafka authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "superuser-password" mechanism: "SCRAM-SHA-256" authorization: roleBindings: - roleName: viewer users: - loginType: basic name: "matt" ``` ###### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "superuser-password" mechanism: "SCRAM-SHA-256" authorization: roleBindings: - roleName: viewer users: - loginType: basic name: "matt" ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false username: "console-superuser" password: "superuser-password" mechanism: "SCRAM-SHA-256" authorization: roleBindings: - roleName: viewer users: - loginType: basic name: "matt" ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-oidc-token-acquired-at-runtime)Static credentials with OIDC (token acquired at runtime) This option is useful when you want to use OIDC. This configuration instructs Redpanda Console to fetch an OAuth 2.0 access token at runtime using the client credentials grant flow. ##### Standalone ```yaml kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: clientId: "" (1) clientSecret: "" (2) tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" (3) scope: "api:///.default" (4) ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Kafka authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: clientId: "" (1) clientSecret: "" (2) tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" (3) scope: "api:///.default" (4) ``` ###### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: clientId: "" (1) clientSecret: "" (2) tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" (3) scope: "api:///.default" (4) ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: clientId: "" (1) clientSecret: "" (2) tokenEndpoint: "https://login.microsoftonline.com//oauth2/v2.0/token" (3) scope: "api:///.default" (4) ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` | 1 | Client ID registered with the identity provider (IdP). | | --- | --- | | 2 | Client secret associated with the client ID. Store securely using an environment variable, such as KAFKA_SASL_OAUTH_CLIENTSECRET. | | 3 | OAuth 2.0 token endpoint URL provided by the IdP. | | 4 | Requested scope to authorize access. Required by some IdPs, such as Azure Entra ID. | #### [](#static-credentials-with-oidc-pre-acquired-token)Static credentials with OIDC (pre-acquired token) This option is suitable when a token is issued externally and injected into the environment (for example, through CI/CD, Vault, or other automation workflows). Redpanda Console does not attempt to refresh or renew the token. ##### Standalone ```yaml kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: token: "" (1) ``` | 1 | A valid OAuth 2.0 JWT. Redpanda Console uses this token when authenticating to Kafka. To avoid hardcoding sensitive data, provide this value using the KAFKA_SASL_OAUTH_TOKEN environment variable. | | --- | --- | ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Kafka authentication through the cluster configuration: | 1 | A valid OAuth 2.0 JWT. Redpanda Console uses this token when authenticating to Kafka. To avoid hardcoding sensitive data, provide this value using the KAFKA_SASL_OAUTH_TOKEN environment variable. | | --- | --- | ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: token: "" (1) ``` ###### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: token: "" (1) ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: token: "" (1) ``` | 1 | A valid OAuth 2.0 JWT. Redpanda Console uses this token when authenticating to Kafka. To avoid hardcoding sensitive data, provide this value using the KAFKA_SASL_OAUTH_TOKEN environment variable. | | --- | --- | Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-oidc-token-from-file)Static credentials with OIDC (token from file) This option is useful when running Redpanda Console in Kubernetes, where a service account token is mounted to the Pod filesystem. Redpanda Console reads this token at startup and uses it for authentication. Redpanda Console does not monitor the token file for changes after startup. To ensure the token is refreshed, restart the Redpanda Console periodically or implement a sidecar that triggers restarts on token rotation. ##### Standalone ```yaml kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: tokenFilepath: "/var/run/secrets/kafka/serviceaccount/token" (1) ``` | 1 | Path to a file containing a valid OAuth 2.0 JWT token. Redpanda Console reads this file at startup and uses its contents as the access token. | | --- | --- | ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Kafka authentication through the cluster configuration: | 1 | Path to a file containing a valid OAuth 2.0 JWT token. Redpanda Console reads this file at startup and uses its contents as the access token. | | --- | --- | ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: tokenFilepath: "/var/run/secrets/kafka/serviceaccount/token" (1) ``` ###### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: tokenFilepath: "/var/run/secrets/kafka/serviceaccount/token" (1) ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1:9092"] sasl: enabled: true impersonateUser: false mechanism: OAUTHBEARER oauth: tokenFilepath: "/var/run/secrets/kafka/serviceaccount/token" (1) ``` | 1 | Path to a file containing a valid OAuth 2.0 JWT token. Redpanda Console reads this file at startup and uses its contents as the access token. | | --- | --- | Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` ### [](#schema-registry-api-examples)Schema Registry API examples This section provides examples of how to configure authentication for communicating with the Schema Registry API from Redpanda Console. You can choose between user impersonation or static credentials. #### [](#user-impersonation-2)User impersonation This option is useful when you want to use the same login credentials to authenticate API requests in the Schema Registry. This ensures accurate audit logs and enforces unified identity. ##### Standalone ```yaml schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: true ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Schema Registry authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: true ``` ###### Helm ```yaml console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: true ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: true ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-basic-auth)Static credentials with basic auth This option is useful when you want to use basic authentication. Redpanda Console uses the provided credentials for authentication. ##### Standalone ```yaml schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: editor users: - loginType: basic name: "matt" ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Schema Registry authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: editor users: - loginType: basic name: "matt" ``` ###### Helm ```yaml console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: editor users: - loginType: basic name: "matt" ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: editor users: - loginType: basic name: "matt" ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-oidc-bearer-token)Static credentials with OIDC bearer token This option is useful when you want to use OIDC but do not want to implement a custom token refresh mechanism. Redpanda Console uses a pre-fetched token for authentication. ##### Standalone ```yaml schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: editor users: - loginType: OIDC name: "" ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Schema Registry authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: editor users: - loginType: OIDC name: "" ``` ###### Helm ```yaml console: enabled: true console: config: schemaRegistry: authentication: enabled: true impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: editor users: - loginType: OIDC name: "" ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: schemaRegistry: urls: ["broker1:8081"] authentication: enabled: true impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: editor users: - loginType: OIDC name: "" ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` > 📝 **NOTE** > > You can supply a static bearer token here, but this token must be refreshed manually before it expires. For automatic token acquisition, configure a background token refresher or consider using impersonation where possible. ### [](#admin-api-examples)Admin API examples This section provides examples of how to configure authentication for communicating with the Admin API from Redpanda Console. You can choose between user impersonation or static credentials. #### [](#user-impersonation-3)User impersonation This option is useful when you want to use the same login credentials to authenticate API requests in the Admin API. This ensures accurate audit logs and enforces unified identity. ##### Standalone ```yaml redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: true ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Admin API authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: true ``` ###### Helm ```yaml console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: true ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: true ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-basic-auth-2)Static credentials with basic auth This option is useful when you want to use basic authentication. Redpanda Console uses the provided credentials for authentication. ##### Standalone ```yaml redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: admin users: - loginType: basic name: "matt" ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Admin API authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: admin users: - loginType: basic name: "matt" ``` ###### Helm ```yaml console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: admin users: - loginType: basic name: "matt" ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: false basic: username: "console-superuser" password: "superuser-password" authorization: roleBindings: - roleName: admin users: - loginType: basic name: "matt" ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#static-credentials-with-oidc-bearer-token-2)Static credentials with OIDC bearer token This option is useful when you want to use OIDC but do not want to implement a custom token refresh mechanism. Redpanda Console uses a pre-fetched token for authentication. ##### Standalone ```yaml redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: "" ``` ##### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Admin API authentication through the cluster configuration: ###### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: "" ``` ###### Helm ```yaml console: enabled: true console: config: redpanda: adminApi: enabled: true authentication: impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: "" ``` ##### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: redpanda: adminApi: enabled: true urls: ["broker1:9644"] authentication: impersonateUser: false bearerToken: "" authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: "" ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` ## [](#next-steps)Next steps - [Configure role-based access control (RBAC) in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authorization/) - [Configure authentication for Redpanda APIs](https://docs.redpanda.com/streaming/current/manage/security/authentication/) ## Suggested labs - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 11: Authorization in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/security/authorization.md --- # Authorization in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Authorization in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/security/authorization page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/security/authorization.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/security/authorization.adoc description: Learn how to configure role-based access control (RBAC) in Redpanda Console, binding roles to users or OIDC groups to restrict access to authorized users. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-08-20" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda Console has enterprise features enabled and it cannot find a valid license, it redirects you to the license expiration landing page, and all other access is restricted. Authentication allows users to log in, but authorization determines what they can do once authenticated. Redpanda Console decides authorization per API, not through a single global setting. You enable **user impersonation** separately for each upstream API. Redpanda authorizes requests to an API with impersonation enabled. Redpanda Console authorizes every other request itself, using the role bindings and group bindings in its configuration. | Request target | Who evaluates permissions? | | --- | --- | | An API with impersonateUser: true | Redpanda, using its internal RBAC and ACLs. Redpanda Console defers authorization for these endpoints, so its own role bindings and group bindings do not apply to them. | | An API with impersonateUser: false | Redpanda Console, using the role bindings and group bindings in its configuration. Redpanda sees these requests as coming from a static service account. | | An endpoint that Redpanda Console serves itself | Redpanda Console, using the role bindings and group bindings in its configuration. These endpoints have no upstream to defer to. | You set `impersonateUser` independently for the Kafka API (`kafka.sasl`), the Schema Registry (`schemaRegistry.authentication`), and the Redpanda Admin API (`redpanda.adminApi.authentication`). Enabling impersonation for one API does not change how the others are authorized. > ❗ **IMPORTANT** > > Enabling impersonation for every upstream API does not switch off Redpanda Console authorization. Redpanda Console continues to authorize the endpoints it serves itself, including Kafka Connect, secret management, and its own API. Role bindings and group bindings remain in effect for those endpoints, so keep them configured. For more information about authentication options, see the [Redpanda Console authentication](https://docs.redpanda.com/streaming/current/console/config/security/authentication/). ## [](#modes-of-service-account-usage)Modes of service account usage **When impersonation is disabled**, Redpanda Console uses a static service account to connect to Redpanda APIs. This service account must be listed as a superuser principal in the Redpanda `superusers` configuration. In Redpanda, a principal is the authenticated identity (such as a username or OIDC subject) used for access control. For details, see [Users, Principals, and Superusers](https://docs.redpanda.com/streaming/current/manage/security/authentication/#principals). > ❗ **IMPORTANT** > > For any secret values, [use environment variables](https://docs.redpanda.com/streaming/current/console/config/configure-console/) instead of hardcoding them in the configuration file. For example, use `KAFKA_SASL_PASSWORD` for the service account password. There are two ways to use this service account: ### [](#static-service-account-only-no-login)Static service account only (no login) Use this mode to connect Redpanda Console to Redpanda with a static service account and **no user login**. When using the static service account (no login) mode: - There is no login screen. - All users share the same access level. - Redpanda Console roles and RBAC are not enforced. - It can be useful for internal tools. Example ```yaml authentication: basic: enabled: false oidc: enabled: false kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "secret" ``` ### [](#static-service-account-with-user-login)Static service account with user login Use this mode to allow users to log in, while still having all API calls executed under a single service account. When using static service accounts with user login: - Users authenticate to the UI using basic or OIDC login. - All backend communication to Redpanda APIs uses the service account. - Redpanda Console roles (`roleBindings`) determine what users can see or do in the UI. - Redpanda authorization is evaluated against the service account. #### Standalone ```yaml authentication: basic: enabled: true oidc: enabled: true kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "secret" authorization: roleBindings: - roleName: admin users: - loginType: basic name: alice ``` #### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: ##### Operator `redpanda-cluster`.yaml ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authentication: basic: enabled: true oidc: enabled: true kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "secret" authorization: roleBindings: - roleName: admin users: - loginType: basic name: alice ``` ##### Helm `redpanda-values.yaml` ```yaml console: enabled: true console: config: authentication: basic: enabled: true oidc: enabled: true kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "secret" authorization: roleBindings: - roleName: admin users: - loginType: basic name: alice ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: `console-values.yaml` ```yaml config: authentication: basic: enabled: true oidc: enabled: true kafka: sasl: enabled: true impersonateUser: false username: "console-superuser" password: "secret" authorization: roleBindings: - roleName: admin users: - loginType: basic name: alice ``` This model is recommended if you want to manage RBAC in Redpanda Console, but still want to use a service account for API calls. ## [](#roles)Roles Roles in Redpanda Console determine which UI features users can access. You assign them in the `roleBindings` stanza (to individual users) or the `groupBindings` stanza (to groups from an OIDC token) of the Redpanda Console configuration file. Redpanda Console evaluates them for every request it authorizes itself: requests to APIs with impersonation disabled, and requests to endpoints it serves directly. Do you have user impersonation enabled? Redpanda Console roles are not used for requests to an API that has user impersonation enabled. For those requests, the user’s identity is forwarded to Redpanda, which then authorizes access based on RBAC and ACL rules. Roles still apply to the endpoints Redpanda Console authorizes itself. To successfully log in and access the UI, users must have valid Redpanda credentials and ACLs that permit access to the Kafka API. To access full Redpanda Console functionality (for example, downloading debug bundles or viewing broker configs), the user must also be a superuser in Redpanda. For information on assigning ACLs and superuser status, see [Configure Access Control Lists](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/). When Redpanda Console authorizes a request itself, it grants access to UI features based on the roles bound in `roleBindings` and `groupBindings`. However: - Redpanda Console roles are not shown in the **Roles** tab on the **Security** page. - Redpanda Console roles do not grant access to Redpanda APIs (such as the Kafka API). - A Redpanda Console Admin is not a Redpanda superuser by default. To perform protected actions (for example, creating topics, managing ACLs, or accessing the Admin API), users must be granted corresponding permissions in Redpanda. > ❗ **IMPORTANT** > > If you’re running Redpanda Console with impersonation disabled, ensure the Redpanda Console service account is listed as a superuser in Redpanda. Otherwise, Redpanda Console cannot perform administrative actions on behalf of logged in users. Redpanda Console provides the following predefined roles: | Role | Permissions | Limitations | | --- | --- | --- | | Viewer | View topic data (messages, configs, partitions with search filters)View cluster data (node configs, ACLs, service accounts, quotas)View consumer group data (consumer groups, group offsets, lags)View Schema Registry data (registered schemas with contents)View Kafka Connect data (configured clusters, connectors including status and configs) | Cannot view the list of users | | Editor | Inherits all Viewer permissions, plus:Manage topics (create, edit configurations, delete topics, publish and delete records)Manage cluster configurations (edit node or cluster settings)Manage consumer groups (edit or delete group offsets)Manage Kafka Connect (create, update, delete, start, pause, or stop connectors) | Cannot create or remove ACLs or service accounts | | Admin | Inherits all Editor permissions, plus:View and manage all users and ACLsGenerate debug bundles | No limitations | > 📝 **NOTE: Community** > > **Kafka Connect is community-supported on [Redpanda Community Slack](https://redpanda.com/slack)**. Redpanda Data does not provide enterprise support for Kafka Connect with Redpanda Console. For a supported and scalable Kafka Connect alternative, try [Redpanda Connect](https://docs.redpanda.com/connect/get-started/). ## [](#grant-permissions-through-role-bindings)Grant permissions through role bindings When impersonation is disabled, Redpanda Console connects to Redpanda APIs using the service account defined in: - `kafka.sasl`: For the Kafka API - `schemaRegistry.authentication`: For the Schema Registry API - `redpanda.adminApi.authentication`: For the Admin API The Redpanda Console service account must be a superuser principal in Redpanda. Without superuser privileges, Redpanda Console is unable to perform actions such as listing topics, retrieving cluster metrics, managing ACLs, or accessing administrative endpoints. To grant superuser permissions to the service account: ```bash rpk cluster config set superusers '["console-superuser"]' \ -X user= -X pass= ``` Before login, Redpanda Console authenticates the user by validating their credentials against the Kafka API. If the user does not exist in Redpanda (for example, if they have not been created or their identity is not mapped through OIDC), login fails. After login, all API communication is performed using the service account’s credentials, not the end-user’s. This ensures Redpanda Console can act on the user’s behalf. As such, the service account must have sufficient privileges to cover the actions users are allowed to perform in the UI. ### [](#example-assign-the-viewer-role)Example: Assign the Viewer role This example shows how to grant the Viewer role to a user named `matt`. The Redpanda Console service account (`console-superuser`) is used to authenticate to all Redpanda services. #### Standalone ```yaml kafka: brokers: ["broker1.example.com:9092"] sasl: enabled: true impersonateUser: false (1) username: "console-superuser" (1) password: "secret-password" mechanism: "SCRAM-SHA-256" schemaRegistry: enabled: true urls: - "broker1.example.com:8081" - "broker2.example.com:8081" authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" redpanda: adminApi: enabled: true urls: - "broker1.example.com:9644" authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" authorization: roleBindings: - roleName: viewer (2) users: - loginType: basic name: "matt" (2) ``` #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure authorization through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false (1) username: "console-superuser" (1) password: "secret-password" mechanism: "SCRAM-SHA-256" schemaRegistry: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" redpanda: adminApi: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" authorization: roleBindings: - roleName: viewer (2) users: - loginType: basic name: "matt" (2) ``` ##### Helm ```yaml console: enabled: true console: config: kafka: sasl: enabled: true impersonateUser: false (1) username: "console-superuser" (1) password: "secret-password" mechanism: "SCRAM-SHA-256" schemaRegistry: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" redpanda: adminApi: enabled: true authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" authorization: roleBindings: - roleName: viewer (2) users: - loginType: basic name: "matt" (2) ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: kafka: brokers: ["broker1.example.com:9092"] sasl: enabled: true impersonateUser: false (1) username: "console-superuser" (1) password: "secret-password" mechanism: "SCRAM-SHA-256" schemaRegistry: enabled: true urls: - "broker1.example.com:8081" - "broker2.example.com:8081" authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" redpanda: adminApi: enabled: true urls: - "broker1.example.com:9644" authentication: impersonateUser: false basic: username: "console-superuser" (1) password: "secret-password" authorization: roleBindings: - roleName: viewer (2) users: - loginType: basic name: "matt" (2) ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` | 1 | With impersonation disabled, Redpanda Console uses the static service account (console-superuser) to connect to the Redpanda APIs. For security, store private data such as passwords in environment variables. For example: KAFKA_SASL_PASSWORD. | | --- | --- | | 2 | Role bindings map UI users to Redpanda Console roles. This does not grant Redpanda-level permissions. The user (matt) must also exist in Redpanda. | All APIs must be accessible using a superuser principal. If the service account lacks superuser status, Redpanda Console may not be able to fetch cluster status, access the Admin API, or interact with the Schema Registry. ### [](#transform-identities-with-principal-mappings)Transform identities with principal mappings If you use OIDC login, the identity in the JWT token is extracted using a JSONPath expression. By default, this expression is `$.sub`, which means the value of the `sub` claim is used as the username. You can override this expression using the `principalMapping` option to transform or extract a different claim to match the `name` values in your `roleBindings` configuration. To keep authorization consistent between Redpanda Console and Redpanda, set `principalMapping` to match Redpanda’s cluster configuration value for [`oidc_principal_mapping`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#oidc_principal_mapping). Example: Map email to local username ```yaml authentication: oidc: enabled: true issuerUrl: https://auth.dev.cloud.redpanda.com/ clientId: R1iX7Pls9UMXiUoYBOn4NcIUTbaGX4JG clientSecret: redacted redirectUrl: http://localhost:9090/auth/callbacks/oidc successfulLoginRedirectUrl: http://localhost:3000 principalMapping: $.email/([^@]+)@example.com/$1/L ``` This example: - Extracts the `email` field from the OIDC token - Uses a regular expression to capture the username part - Replaces the value with the captured group `$1` - Converts the result to lowercase (`L` modifier) As a result, the user identity becomes `matt` when the email is `matt@example.com`. You can then define role bindings like this: #### Standalone ```yaml authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: matt ``` #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure role bindings through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: console: enabled: true console: config: authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: matt ``` ##### Helm ```yaml console: enabled: true console: config: authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: matt ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: authorization: roleBindings: - roleName: admin users: - loginType: OIDC name: matt ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` #### [](#match-syntax-and-modifiers)Match syntax and modifiers The `principalMapping` syntax uses this format: /// - `jsonpath`: Path to the claim field in the token (such as `$.email`) - `regex`: A regular expression to extract part of the claim - `replacement`: A replacement string using a captured group (such as `$1`) - `modifiers`: Optional flags, such as: - `L` — convert to lowercase - `U` — convert to uppercase > 📝 **NOTE** > > If `principalMapping` is not set, Redpanda Console defaults to using the `sub` claim as the user identity. ## [](#grant-permissions-to-groups)Grant permissions to groups > 📝 **NOTE** > > Group bindings are available in Redpanda Console v3.7.0 and later. Instead of binding a role to each user, you can bind a role to a group carried in the user’s OIDC token. Group bindings are useful when your identity provider already models teams, because your identity provider manages membership rather than the Redpanda Console configuration. Group names come from the OIDC token, so group bindings apply to users who log in through [OIDC](https://docs.redpanda.com/streaming/current/console/config/security/authentication/#enable-oidc-authentication). Example: Assign roles to groups ```yaml authorization: groupBindings: - roleName: admin groups: - platform-team - roleName: viewer groups: - data-analysts - support-engineers ``` Each entry requires: - `roleName`: A built-in role (`viewer`, `editor`, or `admin`), or a custom role if you define one in the `roles` stanza. Redpanda Console fails to start if the role does not exist. - `groups`: One or more group names. The list cannot be empty. Redpanda Console reads group names from the token using [`authentication.oidc.groupClaimPath`](https://docs.redpanda.com/streaming/current/console/config/security/authentication/#extract-group-claims) and compares them to the `groups` list using an exact, case-sensitive match. A group name that differs in case or surrounding whitespace does not match. A user receives the union of every role bound to them directly and every role bound to a group they belong to. See [Multiple roles and duplicate bindings](#multiple-roles-and-duplicate-bindings). > 📝 **NOTE** > > Redpanda Console does not apply group bindings in Redpanda Serverless deployments, which use a custom principal extractor. ### [](#group-bindings-compared-with-redpanda-group-based-access-control)Group bindings compared with Redpanda group-based access control Redpanda Console group bindings and Redpanda [group-based access control](https://docs.redpanda.com/streaming/current/manage/security/authorization/gbac/) are separate mechanisms. Each reads group names from the token using its own claim settings, so you must configure both independently. They resolve the same set of groups only when you set both claim paths to the same value. | Aspect | Redpanda Console group bindings | Redpanda group-based access control | | --- | --- | --- | | What it grants | A Redpanda Console role, controlling access to UI features and to the endpoints Redpanda Console authorizes | Redpanda permissions for Group: principals, controlling access to the Kafka API and other Redpanda resources. You can create ACLs directly for the group or assign the group to an RBAC role. | | Where you configure it | authorization.groupBindings in the Redpanda Console configuration | Redpanda cluster properties and ACLs | | Which claim path applies | authentication.oidc.groupClaimPath in Redpanda Console | oidc_group_claim_path in Redpanda | | How nested group names are handled | authentication.oidc.nestedGroupClaimBehavior: none (default) or flatten | nested_group_behavior: none (default) or suffix | The `flatten` and `suffix` values have the same effect, keeping only the last path segment, but the names differ. You cannot copy one setting into the other. Redpanda Console does not grant Redpanda permissions. A group binding that assigns the `admin` role does not make its members Redpanda superusers. To grant permissions on Redpanda resources, either create ACLs for the `Group:` principal or assign the group to a Redpanda RBAC role, as described in [Configure Group-Based Access Control](https://docs.redpanda.com/streaming/current/manage/security/authorization/gbac/). ## [](#multiple-roles-and-duplicate-bindings)Multiple roles and duplicate bindings You can assign multiple roles to the same user through role bindings, group bindings, or a combination of the two. In this case, the user receives the union of all permissions associated with those roles. For example, if a user is assigned both `viewer` and `editor`, they can perform all actions granted to each role. Example: User with multiple role assignments ```yaml authorization: roleBindings: - roleName: viewer users: - loginType: OIDC name: john.doe@redpanda.com - roleName: editor users: - loginType: OIDC name: john.doe@redpanda.com ``` Duplicate role assignments do not cause errors. If the same role is assigned more than once to a user, whether through repeated role bindings or through both a role binding and a group binding, the user receives the permissions only once. ## [](#suggested-reading)Suggested reading For details on how to assign Redpanda-level permissions through ACLs or RBAC, see: - [Configure Access Control Lists](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/) - [Configure Role-Based Access Control](https://docs.redpanda.com/streaming/current/manage/security/authorization/rbac/) - [Configure Group-Based Access Control](https://docs.redpanda.com/streaming/current/manage/security/authorization/gbac/) ## Suggested labs - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 12: TLS Termination in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/security/tls-termination.md --- # TLS Termination in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: TLS Termination in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/security/tls-termination page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/security/tls-termination.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/security/tls-termination.adoc description: Learn how to secure Redpanda Console using TLS, either by letting Redpanda Console handle TLS termination or by offloading it to an upstream component, such as a reverse proxy or a Cloud HTTPS LoadBalancer. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- To secure Redpanda Console using TLS (Transport Layer Security), you can either let Redpanda Console handle TLS termination or you can offload it to an upstream component, such as a reverse proxy or a cloud HTTPS load balancer. TLS termination refers to the process of decrypting incoming TLS-encrypted traffic. Choosing the right approach depends on various factors, such as your application’s traffic load, the complexity of your infrastructure, security requirements, and resource availability: - Redpanda Console handles TLS termination | Advantages | Drawbacks | | --- | --- | | Simplicity: You don’t need an additional component to handle the TLS termination. | Performance: TLS termination can be computationally expensive, especially for high-traffic applications. | | Control: Because the TLS termination happens within Redpanda Console, you have direct control over the process, which can be beneficial for troubleshooting and custom configurations. | | - An upstream component handles TLS termination | Advantages | Drawbacks | | --- | --- | | Performance: Offloading the task of TLS termination to another component can help improve the performance of Redpanda Console by reducing its computational load. | Complexity: Using another component for TLS termination can increase the overall complexity of your system. | | Flexibility: You can use different types of upstream components (like various reverse proxies or load balancers) depending on your infrastructure needs and preferences. | | | Simplicity: Reverse proxies like NGINX Ingress can integrate with other components such as cert-manager, which automatically renews certificates from LetsEncrypt. | | ## [](#use-redpanda-console-for-tls-termination)Use Redpanda Console for TLS termination When you use Redpanda Console to terminate the TLS connection, Redpanda Console starts two HTTP servers: - An HTTPS server on the configured HTTPS port. - An HTTP server on the configured HTTP port which redirects HTTP requests to the HTTPS port. #### Standalone Add the following configuration to your `/etc/redpanda/redpanda-console-config.yaml` file: ```yaml server: # httpsListenPort defines the port on which Redpanda Console is listening for TLS connections, while advertisedHttpsListenPort defines the port that is advertised to clients, which may be different due to network configurations such as load balancers or proxies. advertisedHttpsListenPort is needed when redirecting a HTTP request to an HTTPS URL. httpsListenPort: 443 advertisedHttpsListenPort: 443 listenPort: 8080 tls: enabled: true certFilepath: keyFilepath: # AllowedOrigins is a list of origins that can send requests from a browser to the Redpanda Console # API. By default, a same-site policy is enforced to prevent CSRF-attacks. # Only in very specific deployment models you may need to change the secure default. # For example, during development, it's common to have the API server and the client running on different ports of localhost, which are treated as different origins by browsers. In this case, you would need to set `allowedOrigins` to include the origin of your client's development server. # allowedOrigins: [] ``` Replace `` and `` with the paths of your TLS certificate and key, respectively. #### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, configure Redpanda Console TLS through the cluster configuration: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda-sample spec: chartRef: {} clusterSpec: console: config: server: httpsListenPort: 443 advertisedHttpsListenPort: 443 listenPort: 8080 tls: enabled: true certFilepath: keyFilepath: ``` ##### Helm ```yaml console: enabled: true console: config: server: httpsListenPort: 443 advertisedHttpsListenPort: 443 listenPort: 8080 tls: enabled: true certFilepath: keyFilepath: ``` #### Kubernetes standalone When using the standalone Redpanda Console Helm chart, configure TLS in your Helm values: ```yaml config: server: httpsListenPort: 443 advertisedHttpsListenPort: 443 listenPort: 8080 tls: enabled: true certFilepath: keyFilepath: ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` In this example, Redpanda Console is serving HTTPS traffic on port 443, where both `httpsListenPort` and `advertisedHttpsListenPort` are set to the same value. Any requests to the `listenPort` 8080 are redirected to the `advertisedHttpsListenPort`. If you want Redpanda Console to serve HTTPS on a non-standard port like 8081, but you want to present the URL to users as though it’s serving on the standard HTTPS port 443, you can set `httpsListenPort` to 8081 and `advertisedHttpsListenPort` to 443. This configuration might be useful in development or testing scenarios. For example, if Redpanda Console’s internal address is `https://192.168.1.100:8081` but externally it’s accessed through `https://public-address.com:443`, set `httpsListenPort` to 8081 and `advertisedHttpsListenPort` to 443. Despite listening internally on 8081, Redpanda Console will generate URLs for clients using port 443. > 📝 **NOTE** > > If you host Redpanda Console under a sub-path of your domain, such as `https://my-company.com/redpanda/console`, configure [HTTP path rewrites](https://docs.redpanda.com/streaming/current/console/config/http-path-rewrites/) in Redpanda Console. ### [](#http-strict-transport-security-hsts)HTTP Strict Transport Security (HSTS) When TLS is enabled, Redpanda Console server automatically adds the HTTP Strict Transport Security (HSTS) header to all responses: ```none Strict-Transport-Security: max-age=31536000 ``` The HSTS header instructs web browsers to: - Always connect to Redpanda Console using HTTPS, never HTTP - Automatically upgrade any HTTP requests to HTTPS for the next 365 days (31536000 seconds) - Refuse connections if there are certificate errors or warnings This behavior begins after the browser’s first successful HTTPS connection to Redpanda Console. HSTS provides protection against: - Protocol downgrade attacks: Prevents attackers from forcing connections to use insecure HTTP - Accidental insecure connections: Users typing `http://` in their browser are automatically redirected to HTTPS - Session hijacking: Eliminates the risk window where HTTP traffic could be intercepted before redirect You can verify that HSTS is enabled by checking the response headers: ```none curl -svk https://localhost:9091/ 2>&1 | grep -i strict-transport-security ``` Expected output: ```none < strict-transport-security: max-age=31536000 ``` ## [](#use-an-upstream-component-for-tls-termination)Use an upstream component for TLS termination When you use an upstream component for TLS termination, the upstream component handles the secure TLS connection, and Redpanda Console receives unencrypted HTTP traffic from this component. You can use various upstream components, including reverse proxies, such as NGINX and HAProxy, as well as cloud HTTPS load balancers. To use this option, you must: 1. Configure the upstream component to handle TLS termination. 2. Ensure that the upstream component routes traffic to the address and port of Redpanda Console. 3. Ensure that the upstream component is configured to pass along the original host header so that Redpanda Console can generate correct URLs, even when it’s behind a reverse proxy or load balancer. 4. Disable TLS in Redpanda Console: ### Standalone Add the following configuration to your `/etc/redpanda/redpanda-console-config.yaml` file: ```yaml server: listenPort: 8080 tls: enabled: false ``` ### Kubernetes embedded When using the Redpanda Operator or the Redpanda Helm chart, disable Redpanda Console TLS through the cluster configuration: #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda-sample spec: chartRef: {} clusterSpec: console: config: server: listenPort: 8080 tls: enabled: false ``` #### Helm ```yaml console: config: server: listenPort: 8080 tls: enabled: false ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart, disable TLS in your Helm values: ```yaml config: server: listenPort: 8080 tls: enabled: false ``` Apply with: ```bash helm upgrade --install redpanda-console redpanda/console -f console-values.yaml ``` > 📝 **NOTE** > > TLS is disabled by default. Although Redpanda Console isn’t using TLS, the traffic remains secure because the upstream component handles TLS. If you host Redpanda Console under a sub-path of your domain, such as `https://my-company.com/redpanda/console`, configure [HTTP path rewrites](https://docs.redpanda.com/streaming/current/console/config/http-path-rewrites/) in Redpanda Console. ## [](#suggested-reading)Suggested reading - [`server` configuration options](https://docs.redpanda.com/streaming/current/console/config/configure-console/) - NGINX - [NGINX Beginner’s Guide](http://nginx.org/en/docs/beginners_guide.html) - [NGINX SSL Termination](https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/) - [HAProxy documentation](https://www.haproxy.com/documentation/) - [AWS Elastic Load Balancing documentation](https://docs.aws.amazon.com/elasticloadbalancing/) - [Cloud Load Balancing documentation](https://cloud.google.com/load-balancing/docs) - [OpenSSL documentation](https://www.openssl.org/docs/) --- # Page 13: Enable Topic Documentation in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/config/topic-documentation.md --- # Enable Topic Documentation in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Enable Topic Documentation in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config/topic-documentation page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config/topic-documentation.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/config/topic-documentation.adoc description: Learn how to embed your Kafka topic documentation into the Redpanda Console UI by linking a Git repository that contains the topic documentation files. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- You can embed topic documentation into the Redpanda Console user interface by providing access to a public or private Git repository that hosts the documentation files in Markdown format. ![topic documentation](https://docs.redpanda.com/streaming/current/console/_images/topic-documentation.png) Redpanda Console clones the provided Git repository and stores all Markdown files it finds in memory. The **Documentation** tab in the frontend displays the content of the Markdown file that matches the name of the Kafka topic. | Path/Filename | Topic Name | Matches | | --- | --- | --- | | ordersv2.md | orders-v2 | ✗ | | Orders-v2.md | orders-v2 | ✗ | | orders-v2.md | orders-v2 | ✓ | | /orders/orders-v2.md | orders-v2 | ✓ | ## [](#repository-information)Repository information Start by specifying the Git repository that contains the Markdown documentation files. Provide the repository URL, the branch, and the base directory that contains the documentation. ### Standalone ```yaml console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: ``` ### Kubernetes embedded When Redpanda Console is part of the Redpanda Helm chart or Operator: #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: ``` #### Helm ```yaml console: enabled: true console: config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: ``` ### Kubernetes standalone When using the standalone Redpanda Console Helm chart: ```yaml config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: ``` ## [](#refresh-interval)Refresh interval Define how often Redpanda Console should refresh the documentation from the repository. ```yaml console: topicDocumentation: git: enabled: true refreshInterval: 10m ``` - `refreshInterval`: Set a duration like `10m` to refresh documentation every 10 minutes. Use `0` to disable automatic refresh. ## [](#authenticate-with-private-git-repositories)Authenticate with private Git repositories If the Git repository is private, Redpanda Console must authenticate using either: - A [GitHub Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) (PAT) over HTTPS (basic auth) - An [SSH private key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh) ### [](#authenticate-using-a-github-personal-access-token-pat)Authenticate using a GitHub Personal Access Token (PAT) #### Standalone 1. Set the environment variables: ```bash TOPICDOCUMENTATION_GIT_BASICAUTH_USERNAME=token TOPICDOCUMENTATION_GIT_BASICAUTH_PASSWORD= ``` Replace `` with a GitHub personal access token that has `repo` scope. 2. Configure Redpanda Console: ```yaml console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: refreshInterval: 10m basicAuth: enabled: true ``` #### Kubernetes embedded 1. Create the Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: topic-doc-git-auth namespace: redpanda type: Opaque stringData: TOPICDOCUMENTATION_GIT_BASICAUTH_PASSWORD: ``` 2. Configure the deployment: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda extraEnv: - name: TOPICDOCUMENTATION_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: topic-doc-git-auth config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: refreshInterval: 10m basicAuth: enabled: true ``` ##### Helm ```yaml console: enabled: true extraEnv: - name: TOPICDOCUMENTATION_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: topic-doc-git-auth console: config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: refreshInterval: 10m basicAuth: enabled: true ``` #### Kubernetes standalone 1. Create the Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: topic-doc-git-auth namespace: redpanda type: Opaque stringData: TOPICDOCUMENTATION_GIT_BASICAUTH_PASSWORD: ``` 2. Reference it in Helm values: ```yaml config: console: topicDocumentation: git: enabled: true repository: url: https://github.com//.git branch: baseDirectory: refreshInterval: 10m basicAuth: enabled: true extraEnv: - name: TOPICDOCUMENTATION_GIT_BASICAUTH_USERNAME value: token extraEnvFrom: - secretRef: name: topic-doc-git-auth ``` ### [](#authenticate-using-ssh)Authenticate using SSH #### Standalone 1. Save the SSH key to a secure path (for example, `/etc/redpanda/ssh/id_rsa`). 2. Set the environment variables: ```bash TOPICDOCUMENTATION_GIT_SSH_ENABLED=true TOPICDOCUMENTATION_GIT_SSH_USERNAME=git TOPICDOCUMENTATION_GIT_SSH_PRIVATEKEYFILEPATH=/etc/redpanda/ssh/id_rsa TOPICDOCUMENTATION_GIT_SSH_PASSPHRASE= ``` 3. Configure Redpanda Console: ```yaml console: topicDocumentation: git: enabled: true repository: url: git@github.com:/.git branch: baseDirectory: refreshInterval: 10m ssh: enabled: true ``` #### Kubernetes embedded 1. Create the Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: topic-doc-git-ssh namespace: redpanda type: Opaque stringData: privateKey: | -----BEGIN OPENSSH PRIVATE KEY----- -----END OPENSSH PRIVATE KEY----- passphrase: ``` 2. Mount the secret and set environment variables: ##### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console spec: clusterRef: name: redpanda extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: topic-doc-git-ssh extraEnv: - name: TOPICDOCUMENTATION_GIT_SSH_ENABLED value: "true" - name: TOPICDOCUMENTATION_GIT_SSH_USERNAME value: git - name: TOPICDOCUMENTATION_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: TOPICDOCUMENTATION_GIT_SSH_PASSPHRASE value: ``` ##### Helm ```yaml console: enabled: true extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: topic-doc-git-ssh extraEnv: - name: TOPICDOCUMENTATION_GIT_SSH_ENABLED value: "true" - name: TOPICDOCUMENTATION_GIT_SSH_USERNAME value: git - name: TOPICDOCUMENTATION_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: TOPICDOCUMENTATION_GIT_SSH_PASSPHRASE value: ``` #### Kubernetes standalone 1. Create the Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: topic-doc-git-ssh namespace: redpanda type: Opaque stringData: privateKey: | -----BEGIN OPENSSH PRIVATE KEY----- -----END OPENSSH PRIVATE KEY----- passphrase: ``` 2. Configure Helm values: ```yaml config: console: topicDocumentation: git: enabled: true repository: url: git@github.com:/.git branch: baseDirectory: refreshInterval: 10m ssh: enabled: true extraVolumeMounts: - name: git-ssh mountPath: /etc/git-ssh readOnly: true extraVolumes: - name: git-ssh secret: secretName: topic-doc-git-ssh extraEnv: - name: TOPICDOCUMENTATION_GIT_SSH_ENABLED value: "true" - name: TOPICDOCUMENTATION_GIT_SSH_USERNAME value: git - name: TOPICDOCUMENTATION_GIT_SSH_PRIVATEKEYFILEPATH value: /etc/git-ssh/privateKey - name: TOPICDOCUMENTATION_GIT_SSH_PASSPHRASE value: ``` Replace the following values throughout the examples: - ``: GitHub organization or user that owns the repository - ``: Name of the GitHub repository containing the topic documentation files - ``: Name of the Git branch to use (for example, `main` or `docs`) - ``: Relative path inside the repository where Markdown documentation files are stored - ``: GitHub personal access token with `repo` scope (used for basic authentication) - ``: SSH private key content used to authenticate to GitHub (must be base64-safe if stored in secrets) - ``: Passphrase used to decrypt the SSH private key, if applicable ## [](#example-configuration)Example configuration This example is for a setup where documentation needs frequent updates and is stored in a private repository accessed through SSH: ```yaml console: topicDocumentation: enabled: true git: enabled: true repository: url: https://github.com/example/redpanda-docs branch: main baseDirectory: path/to/documentation refreshInterval: 10m ssh: enabled: true username: git privateKeyFilepath: "/home/user/.ssh/redpanda_docs_key" passphrase: "passphrase" ``` This configuration is designed for a secure and automated integration of topic documentation into the Redpanda Console, using SSH for secure repository access and a refresh interval that keeps the documentation consistently updated without manual intervention. --- # Page 14: Manage Enterprise Edition Licenses through Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/add-license.md --- # Manage Enterprise Edition Licenses through Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Enterprise Edition Licenses through Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/add-license page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/add-license.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/add-license.adoc description: Learn how to manage Enterprise Edition licenses in Redpanda Console. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- You can add or update an [Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/#console) for both Redpanda and Redpanda Console directly through the Redpanda Console UI. ## [](#prerequisites)Prerequisites - You must have an Enterprise Edition license. [Request a license](https://www.redpanda.com/contact) if you don’t have one already. - Redpanda Console must be [connected to a Redpanda cluster](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/). - Redpanda Console must be [configured to connect to the Redpanda Admin API](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/#admin). > 💡 **TIP** > > You can also [configure Redpanda Console to load the license key from its local configuration](https://docs.redpanda.com/streaming/current/console/config/enterprise-license/). ## [](#upload-a-license)Upload a license When a license is uploaded through Redpanda Console, it is replicated across the connected Redpanda cluster and stored persistently in Redpanda’s internal metadata, ensuring it is retained across restarts. > ⚠️ **CAUTION** > > If you use Kubernetes to deploy Redpanda, do not use Redpanda Console to update the license if it’s already set in your Kubernetes resources. During upgrades or redeployments, license values in your Kubernetes resources will override the license set using Redpanda Console. For consistent license management, set the license using either Redpanda Console or Kubernetes resources, but not both. To upload a new license directly through the Redpanda Console UI: 1. Open the **Upload License** page, using one of the following methods: - **Cluster Overview** page: Navigate to the **Cluster Overview** page in Redpanda Console. Under the **Licensing** section, click on the **Upload new license** link to upload a new license key. - Expiration warning banner: If the existing license expires soon, you can click the **Upload license** button in the expiration warning banner. 2. Upload your license. You can drag and drop a license file into the box or copy and paste the license string into the text input. ![license](https://docs.redpanda.com/streaming/current/console/_images/license.png) When a new license is uploaded, enterprise features in Redpanda Streaming are unlocked immediately without requiring a cluster restart. However, to unlock enterprise features in Redpanda Console, you must restart the Redpanda Console instance. After restarting Redpanda Console, enterprise features such as RBAC are unlocked. However, to enable and use these features, you must configure them. See [Redpanda Console](https://docs.redpanda.com/streaming/current/manage/console/). ## [](#next-steps)Next steps [Check the Status of Licenses](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/). ## [](#suggested-reading)Suggested reading - [Check License Status in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/check-license/) - [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/) - [Redpanda Licenses and Enterprise Features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/) --- # Page 15: Check License Status in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/check-license.md --- # Check License Status in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Check License Status in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/check-license page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/check-license.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/check-license.adoc description: Learn how to check the status of your Redpanda Enterprise Edition license using the Redpanda Console. This topic includes steps to check license details and understand license warnings. page-git-created-date: "2024-12-17" page-git-modified-date: "2026-05-26" support-status: supported --- You can check the expiration date of a license on the **Cluster Overview** page in Redpanda Console, under the **Licensing** section. Redpanda Console tries to load a valid license at startup in the following order: 1. From the local configuration file or environment variables. 2. From the connected Redpanda cluster (if available). ## [](#prerequisites)Prerequisites - Redpanda Console must have an Enterprise Edition license. [Request a license](https://www.redpanda.com/try-enterprise) if you don’t have one already. ## [](#check-the-expiration-date-of-a-license)Check the expiration date of a license 1. Go to the **Cluster Overview** page. 2. Locate the **Licensing** section. 3. Review the license details. ## [](#license-warnings-in-redpanda-console)License warnings in Redpanda Console Redpanda Console displays warnings in the following scenarios: - **Upcoming license expiration**: Redpanda Console checks the license status at startup. If the license expires within 30 days, a warning is displayed in the UI. Redpanda Console also logs the license details at startup, including the expiration date. For example: ```json {"level":"info","msg":"successfully loaded Redpanda Enterprise license","license_org":"redpanda","license_type":"enterprise","expires_at":"Oct 12 2024"} ``` - **Redpanda Console enterprise features are in use without a valid license**: Redpanda Console fails to start if enterprise features for Redpanda Console are enabled without a valid license. ```json {"level":"fatal","ts":"2024-12-16T11:27:27.308Z","msg":"Looks like you've enabled a Redpanda Enterprise feature(s) without a valid license. Please enter an active Redpanda license key. If you don't have one, please request a new/trial license at https://redpanda.com/license-request"} ``` - **Redpanda enterprise features are in use without a valid license**: If the connected Redpanda cluster is using enterprise features without a valid license, Redpanda Console displays a warning specifying the features in use without proper licensing. ## [](#next-steps)Next steps For more detailed information about a license such as the type, organization, and a list of enterprise features that are in use, use one of the following tools: - [`rpk`](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/rpk/) - [Redpanda Operator (Kubernetes)](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/redpanda-operator/) --- # Page 16: Manage Data Transforms in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/data-transforms.md --- # Manage Data Transforms in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Data Transforms in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/data-transforms page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/data-transforms.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/data-transforms.adoc description: Use Redpanda Console to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- Use Redpanda Console to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. ## [](#prerequisites)Prerequisites Before you begin, ensure that you have the following: - Redpanda Console must be [connected to a Redpanda cluster](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/). - Redpanda Console must be [configured to connect to the Redpanda Admin API](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/#admin). - [Data transforms enabled](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#enable-transforms) in your Redpanda cluster. - At least one transform function deployed to your Redpanda cluster. ## [](#monitor)Monitor transform functions To monitor transform functions: 1. Navigate to the **Transforms** menu. 2. Click the name of a transform function to view detailed information: - The partitions that the function is running on - The broker (node) ID - Any lag (the amount of pending records on the input topic that have yet to be processed by the transform) ## [](#logs)View logs To view logs for a transform function: 1. Navigate to the **Transforms** menu. 2. Click on the name of a transform function. 3. Click the **Logs** tab to see the logs. Redpanda Console displays a limited number of logs for transform functions. To view the full history of logs, use the [`rpk` command-line tool](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/#logs). ## [](#delete)Delete transform functions To delete a transform function: 1. Navigate to the **Transforms** menu. 2. Find the transform function you want to delete from the list. 3. Click the delete icon at the end of the row. 4. Confirm the deletion when prompted. Deleting a transform function will remove it from the cluster and stop any further processing. ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) - [Deploy Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/deploy/) - [Monitor Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/) --- # Page 17: Edit Topic Configuration in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/edit-topic-configuration.md --- # Edit Topic Configuration in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Edit Topic Configuration in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/edit-topic-configuration page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/edit-topic-configuration.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/edit-topic-configuration.adoc description: Learn how to use Redpanda Console to edit the configuration of existing topics in a cluster. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- Learn how to use Redpanda Console to edit the configuration of existing topics in a cluster. 1. In the menu, go to **Topics**. 2. Select a topic, and open the **Configuration** tab. 3. Click the pencil icon in the row of the property that you want to edit. 4. Make your changes, and click **Save changes**. ## [](#suggested-reading)Suggested reading - [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/) - [Manage Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/) --- # Page 18: Manage Debug Bundles in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/generate-bundle.md --- # Manage Debug Bundles in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Debug Bundles in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/generate-bundle page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/generate-bundle.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/generate-bundle.adoc description: Learn how to generate, download, and delete debug bundles in Redpanda Console for comprehensive cluster diagnostics. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda Console has enterprise features enabled and it cannot find a valid license, it redirects you to the license expiration landing page, and all other access is restricted. Learn how to generate, download, and delete debug bundles in Redpanda Console for comprehensive cluster diagnostics. When your cluster is unhealthy, Redpanda Console displays debugging data on the **Overview** page. If you are logged in as an admin user, you also get a link to generate a debug bundle. ![degraded cluster](https://docs.redpanda.com/streaming/current/console/_images/degraded-cluster.png) ## [](#prerequisites)Prerequisites - Redpanda Console must be [connected to a Redpanda cluster](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/) and [configured to connect to the Redpanda Admin API](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/#admin). ## [](#generate-a-debug-bundle)Generate a debug bundle You can generate a debug bundle for all brokers in the cluster and download it onto your local computer for inspection. > 📝 **NOTE** > > Only one debug bundle can exist at a time. If you generate a new debug bundle, any existing bundle from a previous run will be automatically deleted. 1. Click **Debug bundle**. 2. Click **Generate default**. 3. Wait until the process is complete. 4. Click **debug-bundle.zip** to download the bundle on your local computer. 5. Unzip the file to inspect the contents. For guidance on reading the debug bundle, see [Inspect a Debug Bundle](https://docs.redpanda.com/streaming/current/troubleshoot/debug-bundle/inspect/). ## [](#delete-a-debug-bundle)Delete a debug bundle To manually delete a debug bundle: 1. Click **Debug bundle**. 2. Click the trash icon next to **debug-bundle.zip** to delete the bundle. ## [](#next-steps)Next steps - [Inspect a Debug Bundle](https://docs.redpanda.com/streaming/current/troubleshoot/debug-bundle/inspect/) - [Cluster Diagnostics](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/) - [Error Messages and Solutions](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/) --- # Page 19: Paginate Messages in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/paginate-messages-events.md --- # Paginate Messages in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Paginate Messages in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/paginate-messages-events page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/paginate-messages-events.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/paginate-messages-events.adoc description: Enable Continuous Pagination on a topic's Messages tab to scroll through all records instead of being capped by Max results. page-topic-type: how-to page-git-created-date: "2026-04-30" page-git-modified-date: "2026-04-30" support-status: supported --- By default, the **Messages** tab on a topic returns the number of records set in **Max results**. Enable **Continuous Pagination** when you need to inspect a topic beyond that cap. ## [](#browse-all-messages-in-a-topic)Browse all messages in a topic 1. Go to **Topics** and select a topic. 2. Open the **Messages** tab. 3. (Optional) Set **Start offset** and **Max results**, or apply filters, to narrow the records you want to inspect. See [Filter Messages with JavaScript in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters/) and [View Deserialized Messages in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/record-deserialization/). 4. Enable the **Continuous Pagination** toggle. 5. Scroll the message list. Redpanda Console keeps loading records until you reach the end of the topic. When continuous pagination is on, the max results cap no longer limits the browsing session. ## [](#performance-considerations)Performance considerations Retrieving large result sets increases load on the Redpanda Console backend and the cluster. To keep responses fast: - Narrow the result set with filters or a bounded offset range before enabling continuous pagination. - Use [JavaScript push filters](https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters/) to match only the records you need. - Leave continuous pagination off and rely on max results when you only need a sample. --- # Page 20: Filter Messages with JavaScript in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters.md --- # Filter Messages with JavaScript in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Filter Messages with JavaScript in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/programmable-push-filters page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/programmable-push-filters.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/programmable-push-filters.adoc description: Learn how to filter Kafka records using custom JavaScript code within Redpanda Console. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- You can use push-down filters in Redpanda Console to search through large Kafka topics that may contain millions of records. Filters are JavaScript functions executed on the backend, evaluating each record individually. Your function must return a boolean: - `true`: record is included in the frontend results. - `false`: record is skipped. Multiple filters combine logically with `AND` conditions. ## [](#add-a-javascript-filter)Add a JavaScript filter To add a JavaScript filter: 1. Navigate to the topic’s **Messages** page. 2. Click **Add filter > JavaScript Filter**. 3. Define your JavaScript filtering logic in the provided input area. ![JavaScript filter in Redpanda Console](https://docs.redpanda.com/streaming/current/console/_images/js-filter.png) ## [](#resource-usage-and-performance)Resource usage and performance JavaScript filters are executed on the backend, consuming CPU and network resources. The performance of your filter depends on the complexity of your JavaScript code and the volume of data being processed. Complex JavaScript logic or large data volumes may increase CPU load and network usage. ## [](#available-javascript-properties)Available JavaScript properties Redpanda Console injects these properties into your JavaScript context: | Property | Description | Type | | --- | --- | --- | | headers | Record headers as key-value pairs (ArrayBuffers) | Object | | key | Decoded record key | String | | keySchemaID | Schema Registry ID for key (if present) | Number | | partitionId | Partition ID of the record | Number | | offset | Record offset within partition | Number | | timestamp | Timestamp as JavaScript Date object | Date | | value | Decoded record value | Object/String | | valueSchemaID | Schema Registry ID for value (if present) | Number | > 📝 **NOTE** > > Values, keys, and headers are deserialized before being injected into your script. ## [](#javascript-filter-examples)JavaScript filter examples ### [](#filter-by-header-value)Filter by header value **Scenario:** Records tagged with headers specifying customer plan type. Sample header data (string value) ```json headers: { "plan_type": "premium" } ``` JavaScript filter ```javascript let headerValue = headers["plan_type"]; if (headerValue) { let stringValue = String.fromCharCode(...new Uint8Array(headerValue)); return stringValue === "premium"; } return false; ``` **Scenario:** Records include a header with JSON-encoded customer metadata. Sample header data (JSON value) ```json headers: { "customer": "{"orgID":"123-abc","name":"ACME Inc."}" } ``` JavaScript filter ```javascript let headerValue = headers["customer"]; if (headerValue) { let stringValue = String.fromCharCode(headerValue); let valueObj = JSON.parse(stringValue); return valueObj["orgID"] === "123-abc"; } return false; ``` ### [](#filter-by-timestamp)Filter by timestamp **Scenario:** Retrieve records from a promotional event. JavaScript filter ```javascript return timestamp.getMonth() === 10 && timestamp.getDate() === 24; ``` ### [](#filter-by-schema-id)Filter by schema ID **Scenario:** Filter customer activity records based on Avro schema version. JavaScript filter ```javascript return valueSchemaID === 204; ``` ### [](#filter-json-record-values)Filter JSON record values **Scenario:** Filter transactions by customer ID. Sample JSON record ```json { "transaction_id": "abc123", "customer_id": "cust789", "amount": 59.99 } ``` JavaScript filter (top-level property) ```javascript return value.customer_id === "cust789"; ``` **Scenario:** Filter orders by item availability. Sample JSON record ```json { "order_id": "ord456", "inventory": { "item_id": "itm001", "status": "in_stock" } } ``` JavaScript filter (nested property) ```javascript return value.inventory.status === "in_stock"; ``` **Scenario:** Filter products missing price information. JavaScript filter (property absence) ```javascript return !value.hasOwnProperty("price"); ``` ### [](#filter-string-keys)Filter string keys **Scenario:** Filter sensor data records by IoT device ID. JavaScript filter ```javascript return key === "sensor-device-1234"; ``` --- # Page 21: View Deserialized Messages in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/record-deserialization.md --- # View Deserialized Messages in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: View Deserialized Messages in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/record-deserialization page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/record-deserialization.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/record-deserialization.adoc description: Learn how Redpanda Console deserializes messages. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-05-26" support-status: supported --- In Redpanda, the messages exchanged between producers and consumers contain raw bytes. Schemas work as an agreed-upon format, like a contract, for producers and consumers to serialize and deserialize those messages. If a producer breaks this contract, consumers can fail. Redpanda Console automatically tries to deserialize incoming messages and displays them in human-readable format. It tests different deserialization strategies until it finds one with no errors. If no deserialization attempts are successful, Redpanda Console renders the byte array in a hex viewer. Sometimes, the payload is displayed in hex bytes because it’s encrypted or because it uses a serializer that Redpanda Console cannot deserialize. When this happens, Redpanda Console displays troubleshooting information. You can also download the raw bytes of the message to feed it directly to your client deserializer or share it with a support team. All deserialized messages are rendered as JSON objects and can be used as JavaScript objects in [JavaScript filters (push filters)](https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters/). ## [](#prerequisites)Prerequisites Ensure that Redpanda Console is configured to handle the specific deserialization formats you plan to use, such as Avro, Protobuf, or MessagePack. Encoding formats that rely on external schemas or metadata may require additional configuration. See [Configure Message Deserialization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/deserialization/). ## [](#display-messages-in-a-specific-format)Display messages in a specific format Redpanda Console tries to automatically identify the correct deserialization type by decoding the message’s key, value, or header with all available deserialization methods. To display your messages in another format: 1. Open your topic. 2. Click the cog icon. 3. Click **Deserialization**. 4. Choose a new deserializer for either the keys or values in your messages. Supported deserializers include: - Plain text - Kafka’s internal binary formats; for example, the `__consumer_offsets` topic - JSON - JSON with Schema Registry encoding - Smile - XML - Avro with Schema Registry encoding - Protobuf - Protobuf with Schema Registry encoding - Messagepack (for topics explicitly enabled to test MessagePack) - UTF-8 / strings - `uint8`, `uint16`, `uint32`, `uint64` ## [](#suggested-reading)Suggested reading - [Redpanda Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/) --- # Page 22: Use Schema Registry in Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/console/ui/schema-reg.md --- # Use Schema Registry in Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use Schema Registry in Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: ui/schema-reg page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: ui/schema-reg.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/console/pages/ui/schema-reg.adoc description: Learn how to perform common Schema Registry management operations in the Redpanda Console. page-git-created-date: "2024-09-11" page-git-modified-date: "2026-08-25" support-status: supported --- In Redpanda Console, the **Schema Registry** menu lists registered and verified schemas, including their serialization format and versions. Select an individual schema to see which topics it applies to. > 📝 **NOTE** > > The Schema Registry is built into Redpanda, and you can use it with the Schema Registry API or with the UI. This section describes Schema Registry operations available in the UI. ## [](#prerequisites)Prerequisites You must add a valid `schemaRegistry` configuration in Redpanda Console. For help configuring Redpanda Console to connect to Schema Registry, see [Configure Message Deserialization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/deserialization/). ## [](#create-or-edit-a-schema)Create or edit a schema A schema is registered in the registry with a _subject_, which is a name that is associated with the schema as it evolves. To register a schema, click **Create new schema**. 1. On the **Create schema** page, select the strategy type for how to derive the subject name. - **Topic** (default): The subject name is derived from the Redpanda topic name. See [Topic strategy use case](#topic-strategy-use-case). - **Record**: The subject name is derived from the Kafka record name. See [Record strategy use case](#record-strategy-use-case). - **TopicRecord**: The subject name is derived from both topic name and record name, allowing for finer-grained schema organization. See [TopicRecord strategy use case](#topicrecord-strategy-use-case). - **Custom**: The subject name is user-defined. 2. Depending on the strategy, select the **Topic name** (Topic and TopicRecord strategies) or enter the **Subject name** (Custom strategy). For the Record strategy, Console derives the subject name from the schema definition. 3. Select whether the schema applies to the message **Key** or **Value**. Console appends `-key` or `-value` to the subject name. This selection is required for the Topic strategy, and it is not available for the Custom strategy. 4. Select the serialization format with the schema definition. 5. (Optional) Enable **Normalize** to convert the schema to a canonical form before registering it. Normalization prevents duplicate schema versions caused by formatting differences, such as whitespace or field ordering. Normalization is supported for Avro, JSON, and Protobuf formats. 6. To build more complex schema definitions, add a reference to other schemas. For example, the two `import` statements are references to the `PhoneNumber` and `Address` schemas: ```json { syntax = "proto3"; import "PhoneNumber.proto"; import "Address.proto"; message Person { string name = 1; string email = 2; PhoneNumber phone = 3; repeated Address address = 4; } } ``` 7. After registering a schema, you can add a new version to it, change its compatibility, or delete it. ### [](#topic-strategy-use-case)Topic strategy use case The Topic strategy is suitable when all messages in a topic follow the same schema. Suppose you’re tracking product order information in a topic named `Transactions`. When you create a schema with the Topic strategy, you select the topic and whether the schema applies to the message key or value. The subject name is the topic name plus a `-key` or `-value` suffix: - `Transactions-key` - `Transactions-value` All producers writing to the `Transactions` topic use the same schemas, and the key and value schemas can evolve independently under their own subjects. ### [](#record-strategy-use-case)Record strategy use case The Record strategy is most useful when you have multiple types of records in a single topic and need more granular categorization influenced by the record name. Suppose there’s an `Events` topic with event types A and B. With the Record strategy, each event type gets its own subject and schema, named after the record rather than the topic. Console derives the record name from the schema definition: the top-level `name` field for Avro and JSON schemas, or the first `message` name for Protobuf schemas. For example, for Avro schemas named `EventTypeA` and `EventTypeB`, the subjects are: - `EventTypeA` - `EventTypeB` > 📝 **NOTE** > > Console does not include the Avro `namespace` or Protobuf `package` in the subject name. To register a fully qualified subject name such as `com.example.EventTypeA`, either use the full dotted name in the Avro `name` field or use the Custom strategy. Most JSON Schema documents have no top-level `name` field. For JSON schemas, either add a top-level `name` to the schema definition or use the Custom strategy. If you select **Key** or **Value** when creating the schema, Console appends `-key` or `-value` to the subject name (for example, `EventTypeA-value`). Because the subjects aren’t tied to a topic, producers can write these record types to any topic, and multiple topics can share the same schemas. > ⚠️ **CAUTION** > > This naming differs from the `RecordNameStrategy` that [server-side schema ID validation](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-id-validation/#set-subject-name-strategy-per-topic) enforces, which expects the subject to be the fully qualified record name with no `-key` or `-value` suffix. If a topic has schema ID validation enabled with `RecordNameStrategy`, use the Custom strategy so that the subject name matches exactly. ### [](#topicrecord-strategy-use-case)TopicRecord strategy use case The TopicRecord strategy is suitable when you want to organize schemas by both topic and record type. Suppose there’s a microservices architecture where different services produce their own event types to a shared topic named `SharedEvents`. With the TopicRecord strategy, each schema’s subject combines the topic name and the record name: - `SharedEvents-MicroserviceAEvent` - `SharedEvents-MicroserviceBEvent` If you select **Key** or **Value** when creating the schema, Console appends `-key` or `-value` to the subject name (for example, `SharedEvents-MicroserviceAEvent-value`). This lets multiple record types govern the same topic, with each record type’s schema evolving under its own subject. > ⚠️ **CAUTION** > > This naming differs from the `TopicRecordNameStrategy` that [server-side schema ID validation](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-id-validation/#set-subject-name-strategy-per-topic) enforces, which expects `-` with no `-key` or `-value` suffix. If a topic has schema ID validation enabled with `TopicRecordNameStrategy`, use the Custom strategy so that the subject name matches exactly. ## [](#manage-schema-metadata)Manage schema metadata Schema Registry can store arbitrary key-value [metadata properties](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/#metadata-properties), such as `owner` or `team`, alongside each schema version. In Redpanda Console, you can view the metadata on a schema version and set metadata when you register a schema or a new schema version. Schema metadata requires Redpanda version 26.1 or later and Redpanda Console v3.5.1 or later. If the cluster does not support metadata, the schema is registered without it. ### [](#view-metadata)View metadata To view the metadata on a schema version, select the subject from the **Schema Registry** menu. On the **Definition** tab, the **Metadata** section below the schema definition lists the properties of the selected version. If the version has no metadata, the section shows **No properties**. ### [](#set-metadata-when-creating-a-schema)Set metadata when creating a schema When you register a schema on the **Create schema** page, use the **Schema metadata** section to attach optional key-value properties: 1. Click **Add property**. 2. Enter a key (for example, `owner`) and a value (for example, `team-platform`). Keys must be unique within a schema version. Rows with an empty key or value are ignored. 3. Click **Save**. ### [](#edit-or-remove-metadata)Edit or remove metadata Metadata is stored per schema version, so you cannot change the metadata on an existing version. To change metadata, register a new version: 1. Select the subject and click **Add new version**. 2. In the **Schema metadata** section, the form is pre-filled with the properties of the latest active version. Edit values, remove properties, or add new ones. 3. Click **Save**. Registering the same schema definition with different metadata properties creates a new schema version with a new ID. The previous version and its metadata remain unchanged. ## [](#configure-schema-compatibility)Configure schema compatibility Applications are often modeled around a specific business object structure. As applications change and the shape of their data changes, producer schemas and consumer schemas may no longer be compatible. You can decide how a consumer handles data from a producer that uses an older or newer schema, and reduce the chance of consumers hitting deserialization errors. You can configure different types of schema compatibility, which are applied to a subject when a new schema is registered. The Schema Registry supports the following compatibility types: - `BACKWARD` (**default**) - Consumers using the new schema (for example, version 10) can read data from producers using the previous schema (for example, version 9). - `BACKWARD_TRANSITIVE` - Consumers using the new schema (for example, version 10) can read data from producers using all previous schemas (for example, versions 1-9). - `FORWARD` - Consumers using the previous schema (for example, version 9) can read data from producers using the new schema (for example, version 10). - `FORWARD_TRANSITIVE` - Consumers using any previous schema (for example, versions 1-9) can read data from producers using the new schema (for example, version 10). - `FULL` - A new schema and the previous schema (for example, versions 10 and 9) are both backward and forward compatible with each other. - `FULL_TRANSITIVE` - Each schema is both backward and forward compatible with all registered schemas. - `NONE` - No schema compatibility checks are done. ### [](#compatibility-uses-and-constraints)Compatibility uses and constraints - A consumer that wants to read a topic from the beginning (for example, an AI learning process) benefits from backward compatibility. It can process the whole topic using the latest schema. This allows producers to remove fields and add attributes. - A real-time consumer that doesn’t care about historical events but wants to keep up with the latest data (for example, a typical streaming application) benefits from forward compatibility. Even if producers change the schema, the consumer can carry on. - Full compatibility can process historical data and future data. This is the safest option, but it limits the changes that can be done. This only allows for the addition and removal of optional fields. If you make changes that are not inherently backward-compatible, you may need to change compatibility settings or plan a transitional period, updating producers and consumers to use the new schema while the old one is still accepted. | Schema format | Backward-compatible tasks | Not backward-compatible tasks | | --- | --- | --- | | Avro | Add fields with default valuesMake fields nullable | Remove fieldsChange data types of fieldsChange enum valuesChange field constraintsChange record of field names | | Protobuf | Add fieldsRemove fields | Remove required fieldsChange data types of fields | | JSON | Add optional propertiesRelax constraints, for example:Decrease a minimum value or increase a maximum valueDecrease minItems, minLength, or minProperties; increase maxItems, maxLength, maxPropertiesAdd more property types (for example, "type": "integer" to "type": ["integer", "string"])Add more enum valuesReduce multipleOf by an integral factorRelaxing additional properties if additionalProperties was not previously specified as falseRemoving a uniqueItems property that was false | Remove propertiesAdd required propertiesChange property names and typesTighten or add constraints | ## [](#delete-a-schema)Delete a schema Select a schema to soft-delete a version of it or all schemas of its subject. Schemas cannot be deleted if any other schemas reference it. A soft-deleted schema can be recovered, but a permanently-deleted schema cannot be recovered. Redpanda does not recommend permanently deleting schemas in a production environment. ## [](#schema-registry-contexts)Schema Registry contexts Schema Registry contexts are namespaces that group subjects and schemas within a single Schema Registry instance. For context configuration details and prerequisites, see [Schema Registry Contexts](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/). ### [](#how-console-uses-contexts)How Console uses contexts When Schema Registry contexts are enabled in your cluster, Console provides context-aware subject browsing and management. Console lists subjects according to their context, using context-capable APIs to ensure subjects and their versions come from the correct namespace. When you open a subject, Console fetches its schema versions using both the subject name and its context. This avoids ambiguity when the same subject name exists in multiple contexts. Console surfaces Schema Registry mode and compatibility settings and, where supported, lets you adjust them at: - Global level (entire registry) - Subject level (within a specific context) This lets you apply safe defaults globally while fine-tuning behavior for individual subjects in specific contexts. ### [](#automatic-feature-detection)Automatic feature detection Console automatically detects whether Schema Registry contexts are available: - If contexts are supported, Console shows context-aware UI and uses context-specific APIs. - If contexts are not supported, Console falls back to a standard non-context view, so you can continue working with schemas without errors. ## [](#suggested-reading)Suggested reading - [Configure Message Deserialization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/deserialization/) - [Redpanda Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/) - [Schema Registry Contexts](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/) --- # Page 23: Self-Managed Deployment **URL**: https://docs.redpanda.com/streaming/current/deploy.md --- # Self-Managed Deployment > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Self-Managed Deployment latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/index.adoc description: Learn about Redpanda Streaming deployments. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-05-26" support-status: supported --- - [Deploy Redpanda](redpanda/) Overview of Redpanda deployment options and links to platform-specific guides. - [Deploy Redpanda Console](console/) Overview of Redpanda Console deployment options and links to platform-specific guides. - [Redpanda Connect Documentation](https://docs.redpanda.com/connect/home/) A data streaming service for building scalable, high-performance pipelines that drive real-time analytics. Hundreds of prebuilt connectors, change-data-capture, and YAML-configurable pipelines. - [Deploy Kafka Connect](kafka-connect/) Overview of Kafka Connect deployment options and links to platform-specific guides. --- # Page 24: Deploy Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/deploy/console.md --- # Deploy Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/index.adoc description: Overview of Redpanda Console deployment options and links to platform-specific guides. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- Redpanda Console provides a web-based UI for managing and monitoring your Redpanda clusters. This section describes the available deployment methods and links to detailed, platform-specific instructions. See the platform-specific pages for prerequisites, configuration, and deployment steps. - [Deploy Redpanda Console on Kubernetes](kubernetes/) Learn about deployment options for Redpanda Console on Kubernetes, as well as requirements for installation. - [Deploy Redpanda Console on Linux](linux/) Learn about deployment options for Redpanda Console on Linux, as well as requirements for installation. --- # Page 25: Deploy Redpanda Console on Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/console/kubernetes.md --- # Deploy Redpanda Console on Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda Console on Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/kubernetes/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/kubernetes/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/kubernetes/index.adoc description: Learn about deployment options for Redpanda Console on Kubernetes, as well as requirements for installation. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- Redpanda Console provides a web-based UI for managing and monitoring your Redpanda clusters. This topic describes how to deploy Redpanda Console on Kubernetes and links to detailed, platform-specific instructions. - [Redpanda Console Kubernetes Requirements and Recommendations](k-requirements/) System requirements and recommendations for deploying Redpanda Console on Kubernetes in production. - [Deploy Redpanda Console on Kubernetes](deploy/) Deploy Redpanda Console on Kubernetes using the Redpanda Operator, Helm charts, or YAML manifests. --- # Page 26: Deploy Redpanda Console on Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy.md --- # Deploy Redpanda Console on Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda Console on Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/kubernetes/deploy page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/kubernetes/deploy.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/kubernetes/deploy.adoc description: Deploy Redpanda Console on Kubernetes using the Redpanda Operator, Helm charts, or YAML manifests. page-topic-type: how-to personas: platform_operator learning-objective-1: Deploy Redpanda Console on Kubernetes using the Redpanda Operator, Helm charts, or YAML manifests learning-objective-2: Configure TLS and SASL authentication for Redpanda Console learning-objective-3: Verify and scale a Redpanda Console deployment page-git-created-date: "2025-08-15" page-git-modified-date: "2026-07-28" support-status: supported --- This page shows you how to deploy Redpanda Console as a standalone service on Kubernetes using the Redpanda Operator (Console custom resource), Helm charts, or YAML manifests. > 📝 **NOTE** > > When you deploy a Redpanda cluster using the [Redpanda Operator or Redpanda Helm chart](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/), Redpanda Console is automatically deployed alongside your cluster. > > Use this standalone deployment guide only when you need to: > > - Connect to a Redpanda cluster running outside Kubernetes. > > - Deploy Redpanda Console independently from your Redpanda cluster. > > - Deploy multiple Redpanda Console instances for different environments. After reading this page, you will be able to: - Deploy Redpanda Console on Kubernetes using the Redpanda Operator, Helm charts, or YAML manifests - Configure TLS and SASL authentication for Redpanda Console - Verify and scale a Redpanda Console deployment ## [](#prerequisites)Prerequisites - You must have a running Redpanda or Kafka cluster available to connect to. Redpanda Console requires a cluster to function. For instructions on deploying a Redpanda cluster, see [Deploy on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/). - Review the [system requirements for Redpanda Console on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/k-requirements/). ## [](#install-redpanda-console)Install Redpanda Console Choose your deployment method. #### Operator The Redpanda Operator provides a `Console` custom resource (CR) that lets you deploy and manage Redpanda Console declaratively. The operator handles the lifecycle of the Console deployment, including creating the underlying Deployment, Service, and ConfigMap resources. 1. Create a Console custom resource: `console.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: cluster: clusterRef: (1) name: redpanda replicaCount: 2 (2) resources: (3) requests: cpu: 100m memory: 512Mi limits: cpu: 4000m memory: 2Gi service: (4) type: LoadBalancer port: 8080 ingress: (5) enabled: true annotations: cert-manager.io/cluster-issuer: letsencrypt-prod className: nginx hosts: - host: console.example.com paths: - path: / pathType: Prefix tls: - secretName: console-tls hosts: - console.example.com ``` | 1 | Reference to your Redpanda cluster CR. The operator automatically configures broker addresses, TLS, and authentication based on the referenced cluster. If your Redpanda cluster is not managed by the operator, use staticConfiguration instead of clusterRef. See the TLS section for staticConfiguration examples. | | --- | --- | | 2 | For production, run at least two replicas for high availability and rolling upgrades. | | 3 | Adjust resource requests and limits based on your expected workload and available node resources. | | 4 | Use LoadBalancer for cloud environments or when you want Redpanda Console to be accessible from outside the cluster. Use ClusterIP for internal-only access. | | 5 | Enable and configure Ingress if you want to expose Redpanda Console using a domain name and use TLS/HTTPS. Make sure your cluster has an Ingress controller installed. To use the Kubernetes Gateway API instead of Ingress, see Expose Redpanda Console with the Gateway API. | 2. Apply the Console CR: ```bash kubectl apply -f console.yaml --namespace redpanda ``` The operator reconciles the Console CR and creates the necessary Deployment, Service, and ConfigMap resources. #### Helm 1. Create a values file: The values file is where you configure how Redpanda Console connects to your Redpanda or Kafka cluster. You must specify the broker addresses in the `config.kafka.brokers` section. `console-values.yaml` ```yaml config: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9092 (1) - redpanda-1.redpanda.redpanda.svc.cluster.local:9092 - redpanda-2.redpanda.redpanda.svc.cluster.local:9092 # Resource configuration resources: (2) requests: cpu: 100m memory: 512Mi limits: cpu: 4000m memory: 2Gi # High availability configuration replicaCount: 2 (3) # Pod anti-affinity for node separation affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/name: console topologyKey: kubernetes.io/hostname # Service configuration service: (4) type: LoadBalancer port: 8080 # Ingress configuration (optional) ingress: (5) enabled: true annotations: cert-manager.io/cluster-issuer: letsencrypt-prod ingressClassName: nginx hosts: - host: console.example.com paths: - path: / pathType: Prefix tls: - secretName: console-tls hosts: - console.example.com ``` | 1 | Replace these addresses with the internal DNS names or external addresses of your Redpanda brokers. If you deployed Redpanda using the Redpanda Helm chart or Redpanda Operator, you can find the broker service names by running:kubectl get svc -n Look for services named like redpanda-0, redpanda-1, etc. The port is typically 9092 for Kafka traffic. If your brokers are outside the cluster, use their reachable addresses instead. | | --- | --- | | 2 | Adjust resource requests and limits based on your expected workload and available node resources. | | 3 | For production, run at least two replicas for high availability and rolling upgrades. | | 4 | Use LoadBalancer for cloud environments or when you want Redpanda Console to be accessible from outside the cluster. Use ClusterIP for internal-only access. | | 5 | Enable and configure Ingress if you want to expose Redpanda Console using a domain name and use TLS/HTTPS. Make sure your cluster has an Ingress controller installed. To use the Kubernetes Gateway API instead of Ingress, see Expose Redpanda Console with the Gateway API. | 2. Install the chart: ```bash helm install redpanda-console redpanda/console \ --namespace redpanda \ --create-namespace \ --values console-values.yaml ``` ### [](#connect-to-redpanda-clusters-with-tls)Connect to Redpanda clusters with TLS If your Redpanda cluster uses TLS encryption (the default for Helm deployments), you must configure Redpanda Console to connect securely. #### Operator When you use `clusterRef` to reference a Redpanda cluster managed by the operator, TLS is configured automatically. No additional steps are required. If you use `staticConfiguration` to connect to an external cluster with TLS: 1. Extract the CA certificate: ```bash kubectl get secret redpanda-default-root-certificate -n redpanda -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt ``` 2. Create a secret with the CA certificate: ```bash kubectl create secret generic redpanda-console-tls --from-file=ca.crt=ca.crt -n redpanda ``` 3. Configure the Console CR: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: cluster: staticConfiguration: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9093 tls: caCertSecretRef: name: redpanda-console-tls key: ca.crt secretMounts: - name: redpanda-console-tls secretName: redpanda-console-tls path: /etc/console/secrets ``` 4. Apply the updated Console CR: ```bash kubectl apply -f console.yaml --namespace redpanda ``` #### Helm 1. Run the following command to extract the CA certificate from the Redpanda Helm deployment: ```bash kubectl get secret redpanda-default-root-certificate -n redpanda -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt ``` 2. Create a secret named `redpanda-console` in the `redpanda` namespace with the CA certificate: ```bash kubectl create secret generic redpanda-console --from-file=ca.crt=ca.crt -n redpanda ``` 3. In your `console-values.yaml`: ```yaml config: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9093 tls: enabled: true caFilepath: /etc/console/secrets/ca.crt insecureSkipTlsVerify: true # For local/testing only secretMounts: - name: redpanda-console secretName: redpanda-console path: /etc/console/secrets ``` 4. Upgrade or install Redpanda Console: ```bash helm upgrade --install redpanda-console redpanda/console \ --namespace redpanda \ --values console-values.yaml ``` Redpanda Console now connects securely to your Redpanda cluster using TLS. For production, set `insecureSkipTlsVerify: false` and use a trusted CA. ## [](#expose-console-gateway-api)Expose Redpanda Console with the Gateway API As an alternative to Ingress, you can expose Redpanda Console through a Kubernetes [Gateway API](https://gateway-api.sigs.k8s.io/) HTTPRoute. This feature is in beta and requires Redpanda Operator 26.2 or later, or Redpanda Helm chart 26.2.0 or later. Beta features are not recommended for production environments. The Gateway API separates infrastructure from application routing: your platform team manages a shared Gateway, which is the entry point that terminates TLS, and Redpanda Console attaches to that Gateway with an HTTPRoute that the Redpanda Operator or Helm chart renders and manages for you. Choose the Gateway API over Ingress when your cluster already routes traffic through a Gateway controller, such as Envoy Gateway, Istio, Cilium, or NGINX Gateway Fabric, or when you want typed, role-based routing configuration instead of controller-specific Ingress annotations. Ingress and the Gateway API are mutually exclusive for Redpanda Console: enabling both fails validation with `ingress and gateway cannot both be enabled; use one or the other`. > 📝 **NOTE** > > With an HTTPRoute, TLS terminates at the Gateway’s listener, so you configure the certificate on the Gateway resource instead of in the Redpanda Console configuration. ### [](#prerequisites-2)Prerequisites Redpanda Console and the Redpanda Operator don’t bundle Gateway API resources: 1. Install the Gateway API CRDs and a compatible controller, such as [Envoy Gateway](https://gateway.envoyproxy.io/), [Istio](https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/), [Cilium](https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/), or [NGINX Gateway Fabric](https://github.com/nginx/nginx-gateway-fabric). For versions and installation steps, see the prerequisites in [Configure External Access through Gateway API](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-gateway-api/#prerequisites). HTTPRoute ships in the standard channel, so either release channel works for this feature. 2. Create a Gateway for the HTTPRoute to attach to, or identify an existing one: `gateway.yaml` ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: my-gateway namespace: gateway-system spec: gatewayClassName: eg (1) listeners: - name: https protocol: HTTPS port: 443 tls: mode: Terminate certificateRefs: - name: console-tls (2) allowedRoutes: namespaces: from: All (3) ``` | 1 | The GatewayClass name depends on your controller. For example, Envoy Gateway installs eg. | | --- | --- | | 2 | A Secret in the Gateway’s namespace that contains the TLS certificate for your Redpanda Console hostname. | | 3 | The listener must allow routes from the namespace where Redpanda Console runs. from: All accepts routes from any namespace. Use from: Selector to restrict which namespaces can attach. | 3. Apply the manifest and verify that the controller accepts the listener: ```bash kubectl apply -f gateway.yaml kubectl get gateway my-gateway --namespace gateway-system \ -o jsonpath='{range .status.listeners[*]}{.name}: {range .conditions[*]}{.type}={.status} {end}{"\n"}{end}' ``` The listener reports `Accepted=True` and `Programmed=True`. ### [](#enable-the-httproute)Enable the HTTPRoute Configure the `gateway` block instead of the `ingress` block: #### Operator 1. Add a `gateway` block to the Console resource: `console.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: cluster: clusterRef: name: redpanda gateway: enabled: true parentRefs: (1) - name: my-gateway namespace: gateway-system sectionName: https hostnames: (2) - console.example.com path: / (3) pathType: PathPrefix ``` | 1 | References the Gateway that the HTTPRoute attaches to. sectionName optionally pins the route to one listener on that Gateway. Without parentRefs, Redpanda creates the HTTPRoute but it doesn’t attach to any Gateway. | | --- | --- | | 2 | The hostnames that the route matches. Point DNS for these hostnames at the Gateway’s address. | | 3 | The path to match. pathType is one of PathPrefix, Exact, or RegularExpression. | 2. Apply the resource: ```bash kubectl apply -f console.yaml --namespace redpanda ``` The operator creates the HTTPRoute and keeps it in sync with the Console resource. #### Helm 1. Add a `gateway` block to your values file: `console-values.yaml` ```yaml gateway: (1) enabled: true parentRefs: - name: my-gateway namespace: gateway-system sectionName: https hostnames: - console.example.com path: / pathType: PathPrefix ``` | 1 | The fields have the same meaning as in the Console resource: parentRefs selects the Gateway (and optionally one listener through sectionName), hostnames lists the hostnames to match, and pathType is one of PathPrefix, Exact, or RegularExpression. | | --- | --- | 2. Upgrade the release: ```bash helm upgrade --install redpanda-console redpanda/console \ --namespace redpanda \ --values console-values.yaml ``` ### [](#verify-the-route)Verify the route 1. Check that the HTTPRoute exists and that the Gateway accepted it: ```bash kubectl get httproute --namespace redpanda kubectl describe httproute redpanda-console --namespace redpanda ``` In the route’s status, the `Accepted` and `ResolvedRefs` conditions are both `True` for the referenced Gateway. If `Accepted` is `False` with reason `NotAllowedByListeners`, update the Gateway listener’s `allowedRoutes` to include the Redpanda Console namespace. 2. Get the Gateway’s external address and point DNS for your Redpanda Console hostname at it: ```bash kubectl get gateway my-gateway --namespace gateway-system ``` 3. Confirm that Redpanda Console responds through the Gateway. Until DNS is in place, resolve the hostname to the Gateway address explicitly: ```bash curl --resolve console.example.com:443: https://console.example.com -I # Replace with the address from the previous step. ``` ### [](#switch-between-ingress-and-the-gateway-api)Switch between Ingress and the Gateway API To move an existing Redpanda Console deployment from Ingress to the Gateway API, remove or disable the `ingress` block and add the `gateway` block in the same update. The operator or chart deletes the Ingress and creates the HTTPRoute. Switching back works the same way in reverse: remove the `gateway` block, add the `ingress` block, and the operator or chart removes the HTTPRoute and creates an Ingress. > 💡 **TIP** > > To expose the Redpanda cluster itself through the Gateway API, see [Configure External Access through Gateway API](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-gateway-api/). ## [](#deploy-redpanda-console-as-standalone-service-with-yaml-manifests)Deploy Redpanda Console as standalone service with YAML manifests If you prefer to deploy using YAML manifests, you can create the following resources: console-deployment.yaml ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: redpanda-console namespace: redpanda labels: app.kubernetes.io/name: console app.kubernetes.io/component: console spec: replicas: 2 selector: matchLabels: app.kubernetes.io/name: console template: metadata: labels: app.kubernetes.io/name: console spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchLabels: app.kubernetes.io/name: console topologyKey: kubernetes.io/hostname containers: - name: console image: docker.redpanda.com/redpandadata/console:v3.11.0 ports: - containerPort: 8080 name: http resources: requests: cpu: 200m memory: 512Mi limits: cpu: 1000m memory: 2Gi env: - name: KAFKA_BROKERS value: "redpanda-0.redpanda.redpanda.svc.cluster.local:9092,redpanda-1.redpanda.redpanda.svc.cluster.local:9092,redpanda-2.redpanda.redpanda.svc.cluster.local:9092" livenessProbe: httpGet: path: /health port: http initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: httpGet: path: /health port: http initialDelaySeconds: 5 periodSeconds: 5 ``` console-service.yaml ```yaml apiVersion: v1 kind: Service metadata: name: redpanda-console namespace: redpanda labels: app.kubernetes.io/name: console spec: type: LoadBalancer ports: - port: 8080 targetPort: http protocol: TCP name: http selector: app.kubernetes.io/name: console ``` For more complex configurations, create a ConfigMap: console-config.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: name: redpanda-console-config namespace: redpanda data: config.yaml: | kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9092 - redpanda-1.redpanda.redpanda.svc.cluster.local:9092 - redpanda-2.redpanda.redpanda.svc.cluster.local:9092 server: listenPort: 8080 console: enabled: true ``` Apply the manifests: ```bash kubectl apply -f console-config.yaml kubectl apply -f console-deployment.yaml kubectl apply -f console-service.yaml ``` ## [](#configuration)Configuration Make sure to configure the following settings in your Console CR, values file, or ConfigMap: ### [](#connect-to-redpanda)Connect to Redpanda Configure the connection to your Redpanda cluster by setting the broker addresses in your Console CR or values file. See [Configure Redpanda Console to Connect to a Redpanda Cluster](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/). ### [](#authentication-and-security)Authentication and security For production deployments, configure: - **TLS encryption**: Enable TLS for secure communication - **SASL authentication**: Configure SASL if Redpanda uses authentication - **RBAC**: Set up role-based access control Configure authentication based on your deployment method. #### Operator When you use `clusterRef`, the operator automatically inherits SASL and TLS settings from the referenced Redpanda cluster. No additional Console configuration is needed. To configure SASL manually with `staticConfiguration`: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: cluster: staticConfiguration: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9092 sasl: enabled: true mechanism: SCRAM-SHA-256 secret: kafka: saslPassword: ``` You can also reference an existing Kubernetes Secret for credentials: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: cluster: staticConfiguration: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9092 sasl: enabled: true mechanism: SCRAM-SHA-256 username: console-user passwordFilepath: /etc/console/secrets/password secretMounts: - name: kafka-credentials secretName: console-kafka-credentials path: /etc/console/secrets ``` #### Helm Example with SASL authentication: ```yaml config: kafka: brokers: - redpanda-0.redpanda.redpanda.svc.cluster.local:9092 sasl: enabled: true mechanism: SCRAM-SHA-256 username: console-user password: console-password ``` See [Redpanda Console Security](https://docs.redpanda.com/streaming/current/console/config/security/). ## [](#verify-deployment)Verify deployment Use the following steps to confirm that Redpanda Console is running and accessible. ### Operator 1. Check the Console CR status: ```bash kubectl get console -n redpanda ``` The output shows the replica status of your Console deployment: ```bash NAME REPLICAS UPDATED READY AVAILABLE redpanda-console 2 2 2 2 ``` 2. Check pod status: ```bash kubectl get pods -n redpanda -l app.kubernetes.io/name=console ``` 3. Check service status: ```bash kubectl get svc -n redpanda redpanda-console ``` 4. Access the Redpanda Console: 1. If using LoadBalancer: ```bash kubectl get svc -n redpanda redpanda-console -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` 2. If using port-forward for testing: ```bash kubectl port-forward -n redpanda svc/redpanda-console 8080:8080 ``` Open [http://localhost:8080](http://localhost:8080) in your browser. ### Helm 1. Check pod status: ```bash kubectl get pods -n redpanda -l app.kubernetes.io/name=console ``` 2. Check service status: ```bash kubectl get svc -n redpanda redpanda-console ``` 3. Access the Redpanda Console: 1. If using LoadBalancer: ```bash kubectl get svc -n redpanda redpanda-console -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` 2. If using port-forward for testing: ```bash kubectl port-forward -n redpanda svc/redpanda-console 8080:8080 ``` Open [http://localhost:8080](http://localhost:8080) in your browser. ## [](#scaling)Scaling For production deployments, consider the following scaling strategies: ### [](#horizontal-scaling)Horizontal scaling Scale the deployment: ```bash kubectl scale deployment redpanda-console -n redpanda --replicas=3 ``` ### [](#auto-scaling)Auto-scaling Create an HPA for automatic scaling: console-hpa.yaml ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: redpanda-console-hpa namespace: redpanda spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: redpanda-console minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 80 ``` ## [](#monitoring)Monitoring Configure metrics exposure and Prometheus scraping for Redpanda Console. Enable monitoring for Redpanda Console: ```yaml config: server: metrics: enabled: true port: 9090 ``` ### [](#prometheus-servicemonitor)Prometheus ServiceMonitor If you use the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator), deploy a `ServiceMonitor` resource alongside Redpanda Console. Prometheus then discovers and scrapes Console metrics from the `/admin/metrics` endpoint. #### Operator To enable the ServiceMonitor in the Console custom resource, set `monitoring.enabled` to `true`: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Console metadata: name: redpanda-console namespace: redpanda spec: monitoring: enabled: true (1) scrapeInterval: "30s" (2) labels: (3) release: kube-prometheus-stack cluster: clusterRef: name: redpanda ``` | 1 | Set to true to create a ServiceMonitor resource. Default: false. | | --- | --- | | 2 | How often Prometheus scrapes the metrics endpoint. Default: 1m. | | 3 | Additional labels to apply to the ServiceMonitor. Match your Prometheus Operator’s serviceMonitorSelector by applying the same labels here. | Apply the Console CR: ```bash kubectl apply -f console.yaml --namespace redpanda ``` #### Helm To enable the ServiceMonitor in the Console Helm chart, add the following to your `console-values.yaml`: ```yaml monitoring: enabled: true (1) scrapeInterval: "30s" (2) labels: {} (3) ``` | 1 | Set to true to create a ServiceMonitor resource. Default: false. | | --- | --- | | 2 | How often Prometheus scrapes the metrics endpoint. Default: 1m. | | 3 | Additional labels to apply to the ServiceMonitor. Match your Prometheus Operator’s serviceMonitorSelector by applying the same labels here. For example:monitoring: enabled: true labels: release: kube-prometheus-stack | If you deploy Redpanda Console as a subchart of the Redpanda Helm chart, configure monitoring under the `console` key. All `monitoring` options are available under this key. ```yaml console: monitoring: enabled: true ``` When the Console server is configured with TLS (`config.server.tls.enabled: true`), the ServiceMonitor uses HTTPS and configures CA validation for scraping. ## [](#troubleshooting)Troubleshooting - **Connection refused**: Verify Redpanda broker addresses and network policies - **Authentication failed**: Check SASL credentials and configuration - **Resource limits**: Monitor CPU and memory usage, adjust limits as needed ### [](#logs)Logs Check Redpanda Console logs: ```bash kubectl logs -n redpanda -l app.kubernetes.io/name=console -f ``` ## [](#next-steps)Next steps - [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) - [Authentication in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authentication/) - [Authorization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authorization/) --- # Page 27: Redpanda Console Kubernetes Requirements and Recommendations **URL**: https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/k-requirements.md --- # Redpanda Console Kubernetes Requirements and Recommendations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Console Kubernetes Requirements and Recommendations latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/kubernetes/k-requirements page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/kubernetes/k-requirements.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/kubernetes/k-requirements.adoc description: System requirements and recommendations for deploying Redpanda Console on Kubernetes in production. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- This page provides the system requirements and recommendations for deploying Redpanda Console on Kubernetes in production environments. ## [](#operating-system)Operating system - **Linux**: All major distributions (Ubuntu, CentOS, RHEL, Debian) - **Container platforms**: Docker, Kubernetes, OpenShift - **Cloud platforms**: AWS, GCP, Azure ## [](#cpu-and-memory)CPU and memory - **CPU**: Minimum 2 cores. Recommended 4+ cores for production. - **Memory**: Minimum 1 GiB per replica. Recommended 1-2 GiB for most workloads, 2 GiB+ for high concurrency or large data sets. ### [](#resource-requests-and-limits)Resource requests and limits Set resource requests to ensure Redpanda Console always has enough CPU and memory to start, and set higher limits to allow for bursts. For production, use conservative requests and higher limits. ```yaml resources: requests: cpu: 100m memory: 512Mi limits: cpu: 4000m memory: 2Gi ``` - **Requests**: Minimum guaranteed resources. Set conservatively to ensure scheduling on most nodes. - **Limits**: Maximum allowed resources. Set higher to allow for bursts. HPA can scale replicas based on CPU/memory utilization. ### [](#scheduling-constraints)Scheduling constraints Use node affinity and tolerations to control where Redpanda Console Pods are scheduled. Example: Node affinity to prefer nodes with a specific label ```yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/console operator: In values: - "true" ``` Example: Tolerate a taint so Redpanda Console can run on tainted nodes ```yaml tolerations: - key: "console-only" operator: "Exists" effect: "NoSchedule" ``` - **Node affinity**: Ensures Pods are scheduled only on nodes with matching labels. - **Tolerations**: Allow Pods to be scheduled on nodes with specific taints. - Combine affinity and tolerations for advanced scheduling and isolation patterns. ### [](#auto-scaling)Auto-scaling Use the Horizontal Pod Autoscaler (HPA) to automatically scale based on CPU and memory utilization. ## [](#storage)Storage - **Minimum**: 1 GiB available disk space ## [](#network-requirements)Network requirements - **Redpanda connectivity**: TCP access to Redpanda brokers on configured ports - **Web interface**: HTTP/HTTPS access for users (typically port 8080 or 443) - **Schema Registry**: TCP access to Schema Registry, if used ## [](#security-considerations)Security considerations - **TLS encryption**: Configure TLS for all network communications - **Authentication**: Set up appropriate authentication mechanisms - **RBAC**: Implement role-based access control where supported - **Network policies**: Use Kubernetes Network Policies to restrict access ## [](#scaling-and-high-availability)Scaling and high availability - Deploy multiple Redpanda Console replicas for high availability and rolling updates. - Use pod anti-affinity to spread replicas across nodes. - For production, run at least two replicas. ## [](#separate-node-deployment)Separate node deployment Run Redpanda Console on separate nodes from Redpanda brokers for resource isolation and operational independence. ## [](#schema-registry-integration)Schema Registry integration - Allocate extra memory for schema caching if using Schema Registry. - Ensure reliable network connectivity and configure authentication as needed. ## [](#next-steps)Next steps - [Deploy Redpanda Console on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy/) - [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) ## [](#suggested-reading)Suggested reading - [Redpanda Console Security](https://docs.redpanda.com/streaming/current/console/config/security/) - [Redpanda Console Helm Chart Specification](https://docs.redpanda.com/streaming/current/reference/k-console-helm-spec/) --- # Page 28: Deploy Redpanda Console on Linux **URL**: https://docs.redpanda.com/streaming/current/deploy/console/linux.md --- # Deploy Redpanda Console on Linux > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda Console on Linux latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/linux/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/linux/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/linux/index.adoc description: Learn about deployment options for Redpanda Console on Linux, as well as requirements for installation. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- Redpanda Console provides a web-based UI for managing and monitoring your Redpanda clusters. This topic describes how to deploy Redpanda Console on Linux and links to detailed, platform-specific instructions. - [Redpanda Console Linux Requirements and Recommendations](requirements/) System requirements and recommendations for deploying Redpanda Console on Linux in production. - [Deploy Redpanda Console on Linux](deploy/) Deploy Redpanda Console using Docker or the Linux packages. --- # Page 29: Deploy Redpanda Console on Linux **URL**: https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy.md --- # Deploy Redpanda Console on Linux > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda Console on Linux latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/linux/deploy page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/linux/deploy.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/linux/deploy.adoc description: Deploy Redpanda Console using Docker or the Linux packages. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-07-17" support-status: supported --- This page shows you how to deploy Redpanda Console on Linux using Docker or the packages. ## [](#prerequisites)Prerequisites - You must have a running Redpanda or Kafka cluster available to connect to. Redpanda Console requires a cluster to function. For instructions on deploying a Redpanda cluster, see [Linux Deployment Options](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/). - Review the [system requirements for Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/requirements/). ## [](#deploy-with-docker)Deploy with Docker Run Redpanda Console as a Docker container: ```bash docker run -d \ --name redpanda-console \ -p 8080:8080 \ -e KAFKA_BROKERS=localhost:19092 \ docker.redpanda.com/redpandadata/console:v3.11.0 ``` ## [](#packaged)Deploy the packaged Redpanda Console Redpanda Console is available as OS packages (`deb`/`rpm`) for Linux distributions. The following instructions install the `redpanda-console` package from the official repositories and enable the systemd service for Redpanda Console. > ❗ **IMPORTANT** > > Redpanda’s Linux package repositories are moving from `dl.redpanda.com` to `linux.pkg.redpanda.com`, with a new GPG signing key. The legacy `dl.redpanda.com` repository is scheduled to shut down on **September 28, 2026**. If you currently install or upgrade Redpanda from `dl.redpanda.com`, re-point your package manager before then. For steps, including a manual configuration option for environments that cannot pipe a script to `bash`, see [Migrate the Linux Package Repository](https://docs.redpanda.com/streaming/current/migrate/linux-package-repository/). 1. Install Redpanda Console: ### Fedora/RedHat ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash && sudo yum install redpanda-console -y ``` ### Debian/Ubuntu ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | \ sudo -E bash && sudo apt-get install redpanda-console -y ``` 2. Start Redpanda Console: ```bash sudo systemctl enable --now redpanda-console ``` 3. Make sure that Redpanda Console is active and running: ```bash sudo systemctl status redpanda-console ``` ## [](#configuration)Configuration You can configure Redpanda Console using environment variables, command-line flags, or a YAML config file. See [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) for details. ## [](#verify-deployment)Verify deployment Check that Redpanda Console is running and accessible at [http://localhost:8080](http://localhost:8080). ## [](#next-steps)Next steps - [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) - [Authentication in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authentication/) - [Authorization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authorization/) --- # Page 30: Redpanda Console Linux Requirements and Recommendations **URL**: https://docs.redpanda.com/streaming/current/deploy/console/linux/requirements.md --- # Redpanda Console Linux Requirements and Recommendations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Console Linux Requirements and Recommendations latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/linux/requirements page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/linux/requirements.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/console/linux/requirements.adoc description: System requirements and recommendations for deploying Redpanda Console on Linux in production. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- This page provides the system requirements and recommendations for deploying Redpanda Console on Linux in production environments. ## [](#operating-system)Operating system - **Linux**: All major distributions (Ubuntu, CentOS, RHEL, Debian) - **Container platforms**: Docker, Kubernetes, OpenShift - **Cloud platforms**: AWS, GCP, Azure ## [](#cpu-and-memory)CPU and memory - **CPU**: Minimum 2 cores. Recommended 4+ cores for production. - **Memory**: Minimum 1 GiB per replica. Recommended 1-2 GiB for most workloads, 2 GiB+ for high concurrency or large data sets. ## [](#storage)Storage - **Minimum**: 1 GiB available disk space ## [](#network-requirements)Network requirements - **Redpanda connectivity**: TCP access to Redpanda brokers on configured ports - **Web interface**: HTTP/HTTPS access for users (typically port 8080 or 443) - **Schema Registry**: TCP access to Schema Registry, if used ## [](#security-considerations)Security considerations - **TLS encryption**: Configure TLS for all network communications - **Authentication**: Set up appropriate authentication mechanisms - **RBAC**: Implement role-based access control where supported - **Firewall rules**: Configure appropriate firewall rules to restrict access ## [](#scaling-and-high-availability)Scaling and high availability - Deploy multiple Redpanda Console replicas for high availability and rolling updates. - Use pod anti-affinity to spread replicas across nodes. - For production, run at least two replicas. ## [](#separate-node-deployment)Separate node deployment Run Redpanda Console on separate nodes from Redpanda brokers for resource isolation and operational independence. ## [](#schema-registry-integration)Schema Registry integration - Allocate extra memory for schema caching if using Schema Registry. - Ensure reliable network connectivity and configure authentication as needed. ## [](#next-steps)Next steps - [Deploy Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy/) - [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) ## [](#suggested-reading)Suggested reading - [Redpanda Console Security](https://docs.redpanda.com/streaming/current/console/config/security/) --- # Page 31: Deploy Kafka Connect **URL**: https://docs.redpanda.com/streaming/current/deploy/kafka-connect.md --- # Deploy Kafka Connect > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Kafka Connect latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: kafka-connect/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: kafka-connect/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/kafka-connect/index.adoc description: Overview of Kafka Connect deployment options and links to platform-specific guides. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-05-26" support-status: supported --- Kafka Connect enables scalable and reliable streaming data integration with Redpanda. This section provides an overview of deployment options and links to platform-specific instructions. > 💡 **TIP** > > Try [Redpanda Connect](https://docs.redpanda.com/connect/home/) for a faster way to build streaming data pipelines. It’s fully compatible with the Kafka API but eliminates the complex setup and maintenance of Kafka Connect. Redpanda Connect also comes with built-in connectors to support AI integrations. Refer to the platform-specific pages for prerequisites, configuration, and deployment steps. - [Deploy Kafka Connect in Kubernetes](k-deploy-kafka-connect/) Learn how to deploy and configure Kafka Connect using the standalone `connectors` Helm chart. - [Deploy Kafka Connect in Docker](deploy-kafka-connect/) Learn how to use the Docker image to configure connectors for Redpanda. --- # Page 32: Deploy Kafka Connect in Docker **URL**: https://docs.redpanda.com/streaming/current/deploy/kafka-connect/deploy-kafka-connect.md --- # Deploy Kafka Connect in Docker > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Kafka Connect in Docker latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: kafka-connect/deploy-kafka-connect page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: kafka-connect/deploy-kafka-connect.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/kafka-connect/deploy-kafka-connect.adoc description: Learn how to use the Docker image to configure connectors for Redpanda. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- > 📝 **NOTE: Community** > > **The Redpanda Connectors Docker image is a community-supported artifact**. Redpanda Data does not provide enterprise support for this image. For support, reach out to the Redpanda team in [Redpanda Community Slack](https://redpanda.com/slack). > 💡 **TIP** > > Try [Redpanda Connect](https://docs.redpanda.com/connect/home/) for a faster way to build streaming data pipelines. It’s fully compatible with the Kafka API but eliminates the complex setup and maintenance of Kafka Connect. Redpanda Connect also comes with built-in connectors to support AI integrations. The [Redpanda Connectors Docker image](https://hub.docker.com/r/redpandadata/connectors/tags) includes a pre-configured instance of [Kafka Connect](https://redpanda.com/guides/kafka-tutorial/what-is-kafka-connect) that works with Redpanda. This image contains _only_ the MirrorMaker2 connector but you can build a custom image to install additional connectors. The latest Docker image contains: - Red Hat Enterprise Linux 8.9 - OpenJDK 21 LTS - Kafka Connect - JMX-Exporter JMX Prometheus JavaAgent The image also includes the following connectors as plugins: - MirrorSourceConnector - MirrorCheckpointConnector - MirrorHeartbeatConnector ## [](#docker-image-configuration-properties)Docker image configuration properties The following table lists the available Docker image properties. | Property | Description | | --- | --- | | CONNECT_BOOTSTRAP_SERVERS | Comma-separated list of host and port pairs that are the addresses of the Redpanda brokers. | | CONNECT_CONFIGURATION | Properties-based Kafka Connect configuration. | | CONNECT_ADDITIONAL_CONFIGURATION | Comma-separated Kafka Connect properties. This can be used as an alternative to the CONNECT_CONFIGURATION property. If the same Kafka Connect property is defined in CONNECT_CONFIGURATION and CONNECT_ADDITIONAL_CONFIGURATION, the one from CONNECT_ADDITIONAL_CONFIGURATION is used.Example: offset.flush.interval.ms=1000,producer.linger.ms=1 | | CONNECT_SASL_MECHANISM | SASL mechanism. Allowed values: "plain","scram-sha-256", or "scram-sha-512". Do not set if SASL is not used.Default: not set | | CONNECT_SASL_USERNAME | SASL username used to authenticate connecting to a Redpanda broker.Default: not set | | CONNECT_SASL_PASSWORD_FILE | Relative path to a file containing the SASL password, relative to the /opt/kafka/connect-password directory. If the file is in /opt/kafka/connect-password/pass-dir/password, then set pass-dir/password. The SASL password is given in plain text.Default: not set | | CONNECT_TLS_ENABLED | Set to "true" if TLS enabled, and "false" if not.Default: "false" | | CONNECT_TLS_AUTH_CERT | TLS authentication certificate location (relative path from /opt/kafka/connect-certs/).For example: "user-secret/user.crt" when file is in /opt/kafka/connect-certs/user-secret/user.crt | | CONNECT_TLS_AUTH_KEY | TLS authentication key location (relative path from /opt/kafka/connect-certs/).For example: "user-secret/user.key" when file is in /opt/kafka/connect-certs/user-secret/user.key | | CONNECT_TRUSTED_CERTS | Truststore locations (relative path from /opt/kafka/connect-certs/).For example: "my-secret/ca.crt;my-secret/new-cert.crt" when file is in /opt/kafka/connect-certs/my-secret/ca.crt and /opt/kafka/connect-certs/my-secret/new-cert.crt | | CONNECT_ADDITIONAL_TLS_AUTH_CERT | Additional TLS authentication certificate location, used, for example, to connect with the source MM2 cluster, (relative path from /opt/kafka/connect-certs/).For example: "user-secret/user.crt" when file is in /opt/kafka/connect-certs/user-secret/user.crt | | CONNECT_ADDITIONAL_TLS_AUTH_KEY | Additional TLS authentication key location, used, for example, to connect with the source MM2 cluster (relative path from /opt/kafka/connect-certs/).For example: "user-secret/user.key" when file is in /opt/kafka/connect-certs/user-secret/user.key | | CONNECT_ADDITIONAL_TRUSTED_CERTS | Additional truststore locations, used, for example, to connect with the source MM2 cluster (relative path from /opt/kafka/connect-certs/).For example: "my-secret/cert.crt;my-secret/new-cert.crt" when file is in /opt/kafka/connect-certs/my-secret/cert.crt and /opt/kafka/connect-certs/my-secret/new-cert.crt | | CONNECT_METRICS_ENABLED | Set to "true" to enable Prometheus metrics, port 9404. Set to "false" to disable. Default: "true" | | CONNECT_PLUGIN_PATH | Comma-separated list of directories with plugins to load by Kafka Connect.Default: /opt/kafka/redpanda-plugins | | CONNECT_GC_LOG_ENABLED | Set to "true" to enable GC logging. Set to "false" to disable GC logs.Default: "false" | | CONNECT_HEAP_OPTS | JVM heap options. For example -Xms2G -Xmx2G.Default: -Xms256M -Xmx256M | | CONNECT_LOG4J_CONFIGURATION | By default, Kafka Connect logs at "info" info level using Redpanda Console appender. Use this property to pass custom log4j properties-based configuration. | | CONNECT_LOG_LEVEL | By default, Kafka Connect logs at "warn" info level using Redpanda Console appender. Use "info" to change the log level to info or "debug" for debug log level. | ## [](#install-new-connector-type)Install new connector type To install a new connector type: 1. Prepare a new connector jar. Place a fat-jar or a jar with all dependent jars in a dedicated directory. For example: `./connect-plugins/snowflake-sink/snowflake-sink-fat.jar` 2. Mount a volume to bind the directory with a container. For example, make the `./connect-plugins` directory content visible in `/opt/kafka/connect-plugins` in a container: ```yaml volumes: - ./connect-plugins:/opt/kafka/connect-plugins ``` 3. Use the `CONNECT_PLUGIN_PATH` image property to configure a directory with the new connector. Use Kafka Connect to discover new connectors. For example: ```yaml CONNECT_PLUGIN_PATH: "/opt/kafka/connect-plugins" ``` 4. The new connector type should be discovered by Kafka Connect automatically on startup. Use the `/connector-plugins` Kafka Connect REST endpoint to check available connector types. For example: `curl localhost:8083/connector-plugins` > 💡 **TIP** > > Create a separate child directory for each connector, and place the connector’s jar files and other resource files in that child directory. ## [](#configure-sasl)Configure SASL To configure SASL: 1. Prepare the SASL user and password, making sure the user has necessary permissions. - Required: Write access for internal topics and access to consumer groups (so all workers in the cluster can communicate with each other). - ACLs depend on used connector type (source/sink) and topics used by the connectors. 2. Create a file containing the plain text password in a dedicated directory. For example, `./connect-password/redpanda-password/password` where the `password` file contains just the password 3. Mount a volume to bind the directory with a container. For example, make the `./connect-password` directory content visible in `/opt/kafka/connect-password` in a container: ```yaml volumes: - ./connect-password:/opt/kafka/connect-password ``` 4. Use `CONNECT_SASL_USERNAME` to set the SASL username, and use `CONNECT_SASL_PASSWORD_FILE` to set the relative path to a password file. For example, if the file is in `/opt/kafka/connect-password/redpanda-password/password`, use the `redpanda-password/password` value. ```yaml CONNECT_SASL_USERNAME: "connect-user" CONNECT_SASL_PASSWORD_FILE: "redpanda-password/password" ``` ## [](#configure-tls)Configure TLS To configure TLS: 1. Prepare Redpanda cluster certificate and key, and place them in a dedicated directory. For example: ./connect-certs/ca.crt ./connect-certs/client.crt ./connect-certs/client.key 2. Mount a volume to bind the directory with a container. For example, make the `./connect-certs` directory content visible in `/opt/kafka/connect-certs/user-secret` in a container: ```yaml volumes: - ./connect-certs:/opt/kafka/connect-certs/user-secret ``` 3. Set the `CONNECT_TLS_ENABLED` property to `"true"`. 4. Use the `CONNECT_TLS_AUTH_CERT`, `CONNECT_TRUSTED_CERTS`, and `CONNECT_TLS_AUTH_KEY` image properties to configure the relative path to the certificate and key. For example, if the files are in `/opt/kafka/connect-certs/user-secret`, use: ```yaml CONNECT_TRUSTED_CERTS: "user-secret/ca.crt" CONNECT_TLS_AUTH_CERT: "user-secret/client.crt" CONNECT_TLS_AUTH_KEY: "user-secret/client.key" ``` ## [](#connect-with-docker-compose)Connect with Docker Compose You can use the following Docker Compose sample file to connect: docker-compose.yml ```yaml version: '3.8' services: connect: image: docker.redpanda.com/redpandadata/connectors:latest volumes: - ./connect-password:/opt/kafka/connect-password - ./connect-plugins:/opt/kafka/connect-plugins - ./connect-certs:/opt/kafka/connect-certs/user-secret hostname: connect ports: - "8083:8083" environment: CONNECT_CONFIGURATION: | key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter group.id=connectors-group offset.storage.topic=_connectors_offsets config.storage.topic=_connectors_configs status.storage.topic=_connectors_status config.storage.replication.factor=-1 offset.storage.replication.factor=-1 status.storage.replication.factor=-1 CONNECT_BOOTSTRAP_SERVERS: ...data.redpanda:30499,...data.redpanda:30499,...data.redpanda:30499 CONNECT_GC_LOG_ENABLED: "false" CONNECT_HEAP_OPTS: -Xms1G -Xmx1G CONNECT_METRICS_ENABLED: "false" CONNECT_SASL_MECHANISM: "scram-sha-256" CONNECT_SASL_USERNAME: "connect-user" CONNECT_SASL_PASSWORD_FILE: "redpanda-password/password" CONNECT_TLS_ENABLED: "true" CONNECT_TRUSTED_CERTS: "user-secret/ca.crt" CONNECT_TLS_AUTH_CERT: "user-secret/client.crt" CONNECT_TLS_AUTH_KEY: "user-secret/client.key" CONNECT_PLUGIN_PATH: "/opt/kafka/connect-plugins" ``` ├── ... ├── connect-certs │ ├── ca.crt # A file with Redpanda cluster CA cert │ ├── client.crt # A file with Redpanda cluster cert │ └── client.key # A file with Redpanda cluster key ├── connect-password │ └── redpanda-password │ └──password # A file with SASL password ├── connect-plugins │ └── custom-connector │ └── custom-sink-connector-fat.jar # Connector fat jar or jar and dependencies jars └── docker-compose.yaml # A docker-compose file To connect with Docker Compose: 1. From a directory containing the `docker-compose.yaml` file, run: ```bash docker-compose up ``` 2. To list installed plugins, run: ```bash curl localhost:8083/connector-plugins ``` 3. To get Kafka Connect basic information, run: ```bash curl localhost:8083/ ``` 4. Metrics are available at `localhost:9404/`. 5. Use the Redpanda Console or Kafka Connect REST API to manage connectors. ## [](#connect-to-a-redpanda-cloud-cluster)Connect to a Redpanda Cloud cluster To connect to a Redpanda Cloud cluster with Docker Compose: 1. Use `rpk` or Redpanda Console (**Security** > **Users**) to create a Redpanda user. 2. Create ACLs for the user. 3. Set the username in the `CONNECT_SASL_USERNAME` property. 4. Create a file containing the user password (for example, in the path `passwords/redpanda-password/password`). Specify this path in the `CONNECT_SASL_PASSWORD_FILE` property. 5. Specify a value in the `CONNECT_BOOTSTRAP_SERVERS` property. You can view this value in Redpanda Console > **Overview** > **Kafka API**, in the `Bootstrap server URL` option. 6. Set the `CONNECT_SASL_MECHANISM` property value to `"scram-sha-256"`. 7. Set the `CONNECT_TLS_ENABLED` property value to `"true"`. docker-compose.yml ```yaml version: '3.8' connect: image: docker.redpanda.com/redpandadata/connectors:latest volumes: - ./passwords:/opt/kafka/connect-password/passwords hostname: connect ports: - "8083:8083" environment: CONNECT_CONFIGURATION: | key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter group.id=connectors-group offset.storage.topic=_connectors_offsets config.storage.topic=_connectors_configs status.storage.topic=_connectors_status config.storage.replication.factor=-1 offset.storage.replication.factor=-1 status.storage.replication.factor=-1 CONNECT_BOOTSTRAP_SERVERS: seed-....redpanda.com:9092 CONNECT_GC_LOG_ENABLED: "false" CONNECT_HEAP_OPTS: -Xms1G -Xmx1G CONNECT_SASL_MECHANISM: "scram-sha-256" CONNECT_SASL_USERNAME: "connectors-user" CONNECT_SASL_PASSWORD_FILE: "passwords/redpanda-password/password" CONNECT_TLS_ENABLED: "true" ``` ├── ... ├── passwords │ └── redpanda-password │ └──password # A file with SASL password └── docker-compose.yaml # A docker-compose file ### [](#redpanda-cloud-schema-registry)Redpanda Cloud Schema Registry For converters using Schema Registry (like AvroConverter, JsonConverter), use the following connector configuration properties to set up a connection with Schema Registry: | Property | Description | | --- | --- | | key.converter | Key converter class to use for the connector. | | key.converter.schema.registry.url | Key converter Schema Registry URL, which you can view in the cluster Overview > Schema Registry. | | key.converter.basic.auth.credentials.source | Key converter authentication method, should be USER_INFO. | | key.converter.basic.auth.user.info | Key converter user and password used for authentication, separated by a colon. | | value.converter | Value converter class to use for the connector. | | value.converter.schema.registry.url | Value converter Schema Registry URL, which you can view in the cluster Overview > Schema Registry. | | value.converter.basic.auth.credentials.source | Value converter authentication method, should be USER_INFO. | | value.converter.basic.auth.user.info | Value converter user and password used for authentication, separated by a colon. | Example: ```json { .... "value.converter.schema.registry.url": "https://schema-registry-....redpanda.com:30081", "value.converter.basic.auth.credentials.source": "USER_INFO", "value.converter.basic.auth.user.info": "connect-user:secret-password" } ``` ## [](#manage-connectors-with-kafka-connect)Manage connectors with Kafka Connect You can manage connectors using the Kafka Connect REST API. ### [](#view-version-of-kafka-connect-worker)View version of Kafka Connect worker To view the version of the Kafka Connect worker, run: ```bash curl localhost:8083 | jq ``` ### [](#view-list-of-connector-plugins)View list of connector plugins To view the list of available connector plugins, run: ```bash curl localhost:8083/connector-plugins | jq ``` ### [](#view-list-of-active-connectors)View list of active connectors To view the list of active connectors, run: ```bash curl 'http://localhost:8083/connectors?expand=status&expand=info' | jq ``` ### [](#create-connector)Create connector To create the connector, run: ```bash curl "localhost:8083/connectors" -H 'Content-Type: application/json' --data-raw '' ``` For example: ```bash curl "localhost:8083/connectors" \ -H 'Content-Type: application/json' \ --data-raw '{ "name": "heartbeat-connector", "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector", "heartbeats.topic.replication.factor": "1", "replication.factor": "1", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "redpanda:29092", "target.cluster.bootstrap.servers": "redpanda:29092"}}' ``` ### [](#view-connector-status)View connector status To view connector status, run: ```bash curl localhost:8083/connectors//status ``` For example: ```bash curl localhost:8083/connectors/heartbeat-connector/status ``` ### [](#delete-connector)Delete connector To delete the connector, run: ```bash curl "localhost:8083/connectors/" -X 'DELETE' ``` For example: ```bash curl "localhost:8083/connectors/heartbeat-connector" -X 'DELETE' ``` ## [](#manage-connectors-with-redpanda-console)Manage connectors with Redpanda Console Redpanda Console provides a user interface that lets you manage multiple Kafka Connect clusters. You can inspect or patch connectors; restart, pause, and resume connector tasks; and delete connectors. For details on how to set it up, see [Connect Redpanda Console to Kafka Connect Clusters](https://docs.redpanda.com/streaming/current/console/config/kafka-connect/). ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 33: Deploy Kafka Connect in Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/kafka-connect/k-deploy-kafka-connect.md --- # Deploy Kafka Connect in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Kafka Connect in Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: kafka-connect/k-deploy-kafka-connect page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: kafka-connect/k-deploy-kafka-connect.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/kafka-connect/k-deploy-kafka-connect.adoc description: Learn how to deploy and configure Kafka Connect using the standalone connectors Helm chart. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- This topic describes how to deploy Kafka Connect in Kubernetes using the standalone `connectors` Helm chart. > 📝 **NOTE: Community** > > **The Connectors Helm chart is a community-supported artifact**. Redpanda Data does not provide enterprise support for this chart. For support, reach out to the Redpanda team in [Redpanda Community Slack](https://redpanda.com/slack). > 💡 **TIP** > > Try [Redpanda Connect](https://docs.redpanda.com/connect/home/) for a faster way to build streaming data pipelines. It’s fully compatible with the Kafka API but eliminates the complex setup and maintenance of Kafka Connect. Redpanda Connect also comes with built-in connectors to support AI integrations. The `connectors` Helm chart is a standalone chart that deploys an instance of [Kafka Connect](https://redpanda.com/guides/kafka-tutorial/what-is-kafka-connect). The underlying Docker image contains _only_ the MirrorMaker2 connector but you can build a custom image to install additional connectors. | Built-In Connector | Description | | --- | --- | | MirrorSourceConnector | A source connector that replicates records between multiple Kafka clusters. It is part of Kafka’s MirrorMaker, which provides capabilities for mirroring data across Kafka clusters. | | MirrorCheckpointConnector | A source connector that ensures the mirroring process can resume from where it left off in case of failures. It tracks and emits checkpoints that mirror the offsets of the source and target clusters. | | MirrorHeartbeatConnector | A source connector that emits heartbeats to target topics at a defined interval, enabling MirrorMaker to track active topics on the source cluster and synchronize consumer groups across clusters. | > 📝 **NOTE** > > If you want to use other connectors, you must create a custom Docker image that includes them as plugins. See [Install a new connector](#install-a-new-connector). ## [](#prerequisites)Prerequisites - A Kubernetes cluster. You must have `kubectl` with at least version 1.27.0-0. To check if you have `kubectl` installed: ```bash kubectl version --client ``` - [Helm](https://helm.sh/docs/intro/install/) installed with at least version 3.10.0. To check if you have Helm installed: ```bash helm version ``` - You need [jq](https://stedolan.github.io/jq/download/) to parse JSON results when using the Kafka Connect REST API. - An understanding of [Kafka Connect](https://kafka.apache.org/20/documentation.html#connect_overview). ## [](#migrating-from-the-subchart)Migrating from the subchart If you’re currently using the connectors subchart (part of the Redpanda Helm chart), you need to migrate to the standalone connectors chart. Follow these steps: > 📝 **NOTE** > > The example values assume a Redpanda deployment named `redpanda` in the `default` namespace. Adjust the values according to your actual deployment. 1. Copy your existing connectors configuration from your Redpanda values file: Extract the `connectors` section from your current Redpanda Helm values and create a new values file for the standalone chart. Example migration ```yaml # Before (in Redpanda values.yaml) connectors: enabled: true auth: sasl: enabled: true brokerTLS: enabled: true # After (in new connectors-values.yaml) connectors: bootstrapServers: "redpanda-0.redpanda.default.svc.cluster.local:9093,redpanda-1.redpanda.default.svc.cluster.local:9093,redpanda-2.redpanda.default.svc.cluster.local:9093" auth: sasl: enabled: true brokerTLS: enabled: true ``` 2. Remove the connectors configuration from your Redpanda values file: ```yaml # Remove or comment out the entire connectors section # connectors: # enabled: true # ... ``` 3. Upgrade your Redpanda deployment: ```bash helm upgrade redpanda redpanda/redpanda \ --namespace \ --values redpanda-values.yaml ``` This will remove the connectors subchart deployment. 4. Deploy the standalone connectors chart: ```bash helm install redpanda-connectors redpanda/connectors \ --namespace \ --values connectors-values.yaml ``` 5. [Update your Redpanda Console configuration](#console) to point to the new service name if needed. ## [](#deploy-the-standalone-helm-chart)Deploy the standalone Helm chart The `connectors` Helm chart is a standalone chart that you deploy separately from your Redpanda cluster. The chart includes a Pod that runs Kafka Connect and the built-in connectors. The Pod is managed by a Deployment that you configure through Helm values. To connect Redpanda Console to your Kafka Connect deployment, you’ll need to configure Redpanda Console with the appropriate service endpoint. ![Redpanda Connectors deployed in a Kubernetes cluster with three worker nodes.](https://docs.redpanda.com/streaming/current/shared/_images/k-connectors-architecture.png) > 📝 **NOTE** > > Do not schedule Pods that run Kafka Connect on the same nodes as Redpanda brokers. Redpanda brokers require access to all node resources. See [Tolerations](#tolerations) and [Affinity rules](#affinity-rules). ### [](#deploy-kafka-connect)Deploy Kafka Connect To deploy Kafka Connect using the standalone chart, you need to configure connection settings to your Redpanda cluster. 1. Create a values file for the connectors chart: `connectors-values.yaml` ```yaml # Connection to Redpanda brokers connectors: bootstrapServers: "" # Configure TLS if your Redpanda cluster has TLS enabled brokerTLS: enabled: true ca: secretRef: "redpanda-default-cert" secretNameOverwrite: "ca.crt" # Configure SASL if your Redpanda cluster has SASL enabled auth: sasl: enabled: false mechanism: "SCRAM-SHA-512" userName: "" secretRef: "" ``` > 📝 **NOTE** > > To get the correct bootstrap servers for your Redpanda cluster, run: > > ```bash > kubectl run -it --restart=Never --rm --image busybox busybox -- ash -c 'nslookup -type=srv _kafka._tcp...svc.cluster.local | tail -n +4 | head -n -1 | awk '"'"'{print $7 ":" $6}'"'"'' > ``` > > Replace `` and `` with your actual release name and namespace. 2. Deploy the connectors chart: ```bash helm upgrade --install redpanda-connectors redpanda/connectors \ --namespace \ --create-namespace \ --values connectors-values.yaml ``` Replace `` with the namespace where you want to deploy Kafka Connect. 3. [Verify the deployment](#verify-the-deployment) using the Kafka Connect REST API or by configuring Redpanda Console. ### [](#example-values-file)Example values file Here’s a complete example values file that shows common configuration options: `connectors-values.yaml` ```yaml # Connection to Redpanda brokers connectors: bootstrapServers: "redpanda-0.redpanda.redpanda.svc.cluster.local:9093" # TLS configuration (disabled for local testing) brokerTLS: enabled: false # SASL authentication (disabled for local testing) auth: sasl: enabled: false # Resource configuration for local testing container: resources: requests: cpu: "0.5" memory: 1Gi limits: cpu: "1" memory: 1Gi javaMaxHeapSize: 512M # Single replica for local testing (testing scaling to 2) deployment: replicas: 2 # Monitoring disabled for local testing monitoring: enabled: false # Logging logging: level: "info" ``` ## [](#console)Configure Redpanda Console to connect to Kafka Connect To use Redpanda Console to manage your Kafka Connect deployment, you need to configure Redpanda Console to connect to the Kafka Connect service. ### [](#check-your-redpanda-console-version)Check your Redpanda Console version Redpanda Console configuration syntax varies by major version. Before configuring, determine which version you’re using: ```bash # Check console version from deployment kubectl get deployment -n redpanda-console -o jsonpath='{.spec.template.spec.containers[0].image}' # Or check from running pod kubectl get pod -n -l app.kubernetes.io/name=console -o jsonpath='{.items[0].spec.containers[0].image}' # Or check from console logs kubectl logs -n -l app.kubernetes.io/name=console | grep "started Redpanda Console" ``` If you see output like `redpandadata/console:v2.8.0`, you’re using Redpanda Console v2.x. If you see `redpandadata/console:v3.0.0`, you’re using Redpanda Console v3.x. ### [](#redpanda-console-deployed-as-part-of-redpanda-chart)Redpanda Console deployed as part of Redpanda chart If the Redpanda Console is deployed as part of the Redpanda Helm chart (the default), add the following configuration to your Redpanda values: #### Redpanda Console v2.x ```yaml console: enabled: true console: config: connect: enabled: true clusters: - name: "redpanda-connectors" url: "http://redpanda-connectors:8083" tls: enabled: false ``` #### Redpanda Console v3.x ```yaml console: enabled: true console: config: kafkaConnect: enabled: true clusters: - name: "redpanda-connectors" url: "http://redpanda-connectors:8083" tls: enabled: false ``` > 📝 **NOTE** > > If you deployed the connectors chart with a different release name, update the URL accordingly. The service name follows the pattern `:8083`. If Redpanda Console is deployed in a different namespace than Kafka Connect, use the fully qualified service name: #### Redpanda Console v2.x ```yaml console: enabled: true console: config: connect: enabled: true clusters: - name: "redpanda-connectors" url: "http://redpanda-connectors..svc.cluster.local:8083" tls: enabled: false ``` #### Redpanda Console v3.x ```yaml console: enabled: true console: config: kafkaConnect: enabled: true clusters: - name: "redpanda-connectors" url: "http://redpanda-connectors..svc.cluster.local:8083" tls: enabled: false ``` Update your Redpanda deployment: ```bash helm upgrade redpanda redpanda/redpanda \ --namespace \ --values redpanda-values.yaml ``` ### [](#troubleshooting-redpanda-console-connectivity)Troubleshooting Redpanda Console connectivity If you see "Kafka Connect is not configured in Redpanda Console" or cannot access connectors: 1. Ensure you’re using the correct configuration syntax for your Redpanda Console version (see [Check your Redpanda Console version](#check-your-redpanda-console-version)). 2. Check if Redpanda Console can connect to Kafka Connect: ```bash kubectl logs -n -l app.kubernetes.io/name=console --tail=20 ``` Look for these successful connection messages: "creating Kafka connect HTTP clients and testing connectivity to all clusters" "tested Kafka connect cluster connectivity","successful\_clusters":1,"failed\_clusters":0" "successfully create Kafka connect service" 3. Verify Redpanda Console can reach Kafka Connect service: ```bash kubectl exec -n deployment/redpanda-console -- curl -s http://redpanda-connectors..svc.cluster.local:8083 ``` This should return Kafka Connect version information. 4. Verify the connector service exists and is accessible: ```bash kubectl get svc -n | grep connectors ``` 5. If configuration changes aren’t taking effect: ```bash kubectl delete pod -n -l app.kubernetes.io/name=console ``` ### [](#verification-test)Verification test After you’ve deployed the connectors chart, you can verify everything is working with this test: 1. Get the connector Pod name: ```bash POD_NAME=$(kubectl get pod -l app.kubernetes.io/name=connectors --namespace -o jsonpath='{.items[0].metadata.name}') ``` 2. Test basic connectivity: ```bash echo "Testing Kafka Connect REST API..." kubectl exec $POD_NAME --namespace -- curl -s localhost:8083 | jq '.version' ``` 3. List available connector plugins: ```bash echo "Available connector plugins:" kubectl exec $POD_NAME --namespace -- curl -s localhost:8083/connector-plugins | jq '.[].class' ``` 4. Test cluster connectivity: ```bash echo "Testing Redpanda cluster connectivity..." kubectl exec $POD_NAME --namespace -- curl -s localhost:8083/connectors ``` If all commands completed without errors, Kafka Connect is working correctly. If any command fails, refer to the [Troubleshoot common issues](#troubleshoot-common-issues) section. ## [](#configuration-advice)Configuration advice This section provides advice for configuring the standalone `connectors` Helm chart. For all available settings, see [Redpanda Connectors Helm Chart Specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/). ### [](#security-configuration)Security configuration This section covers security-related configuration for the `connectors` Helm chart. #### [](#authentication)Authentication If your Redpanda cluster has SASL enabled, configure SASL authentication for secure communication with your Kafka connectors. ```yaml auth: sasl: enabled: true mechanism: "SCRAM-SHA-512" userName: "admin" secretRef: "sasl-password-secret" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#auth). #### [](#tls-configuration)TLS configuration If your Redpanda cluster has TLS enabled, configure TLS settings for secure communication: ```yaml brokerTLS: enabled: true ca: secretRef: "redpanda-default-cert" secretNameOverwrite: "ca.crt" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#brokertls). #### [](#service-account)Service account Restricting permissions is a best practice. Assign a dedicated service account for each deployment or app. ```yaml serviceAccount: create: true name: "redpanda-connector-service-account" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#serviceaccount). ### [](#scalability-and-reliability)Scalability and reliability This section covers configuration for scalable and reliable deployments. #### [](#number-of-replicas)Number of replicas You can scale the Kafka Connect Pods by modifying the `deployment.replicas` parameter in the Helm values. This parameter allows you to handle varying workloads by increasing or decreasing the number of running instances. ```yml deployment: replicas: 3 ``` The `replicas: 3` setting ensures that three instances of the Kafka Connect Pod will be running. You can adjust this number based on your needs. > 💡 **TIP** > > Redpanda Data recommends using an autoscaler such as [Keda](https://keda.sh/) to increase the number of Pod replicas automatically when certain conditions, such as high CPU or memory usage, are met. #### [](#container-resources)Container resources Specify resource requests and limits. Ensure that `javaMaxHeapSize` is not greater than `container.resources.limits.memory`. ```yaml container: resources: requests: cpu: 1 memory: 1Gi limits: cpu: 2 memory: 2Gi javaMaxHeapSize: 2G javaGCLogEnabled: false ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#container). #### [](#deployment-strategy)Deployment strategy For smooth and uninterrupted updates, use the default `RollingUpdate` strategy. Additionally, set a budget to ensure a certain number of Pod replicas remain available during the update. ```yaml deployment: strategy: type: "RollingUpdate" updateStrategy: type: "RollingUpdate" budget: maxUnavailable: 1 ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#deployment). #### [](#affinity-rules)Affinity rules Affinities control Pod placement in the cluster based on various conditions. Set these according to your high availability and infrastructure needs. ```yaml deployment: podAntiAffinity: topologyKey: kubernetes.io/hostname type: hard weight: 100 custom: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: "app" operator: "In" values: - "redpanda-connector" topologyKey: "kubernetes.io/hostname" preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: "app" operator: "In" values: - "redpanda-connector" topologyKey: "kubernetes.io/zone" ``` In this example: - The `requiredDuringSchedulingIgnoredDuringExecution` section ensures that the Kubernetes scheduler doesn’t place two Pods with the same `app: redpanda-connector` label on the same node due to the `topologyKey: kubernetes.io/hostname`. - The `preferredDuringSchedulingIgnoredDuringExecution` section is a soft rule that tries to ensure the Kubernetes scheduler doesn’t place two Pods with the same `app: redpanda-connector` label in the same zone. However, if it’s not possible, the scheduler can still place the Pods in the same zone. For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#deployment). #### [](#tolerations)Tolerations Tolerations and taints allow Pods to be scheduled onto nodes where they otherwise wouldn’t. If you have nodes dedicated to Kafka Connect with a taint `dedicated=redpanda-connectors:NoSchedule`, the following toleration allows the Pods to be scheduled on them. ```yaml tolerations: - key: "dedicated" operator: "Equal" value: "redpanda-connectors" effect: "NoSchedule" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#tolerations). #### [](#node-selection)Node selection Use node selectors to ensure connectors are scheduled on appropriate nodes and avoid scheduling on Redpanda broker nodes: ```yaml # Example: Schedule on nodes with specific labels nodeSelector: workload-type: "kafka-connect" # Or use node affinity for more complex selection deployment: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/hostname" operator: "NotIn" values: ["redpanda-node-1", "redpanda-node-2", "redpanda-node-3"] ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#nodeselector). #### [](#graceful-shutdown)Graceful shutdown If your connectors require additional time for a graceful shutdown, modify the `terminationGracePeriodSeconds`. ```yaml deployment: terminationGracePeriodSeconds: 30 ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#deployment). ### [](#monitoring-and-observability)Monitoring and observability This section covers monitoring, logging, and health check configuration. #### [](#monitoring)Monitoring If you have the [Prometheus Operator](https://prometheus-operator.dev/), enable monitoring to deploy a PodMonitor resource for Kafka Connect. ```yaml monitoring: enabled: true ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#monitoring). See also: [Monitor Kafka Connect in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-connectors/) #### [](#logging)Logging Use the `info` logging level to avoid overwhelming the storage. For debugging purposes, temporarily change the logging level to `debug`. ```yaml logging: level: "info" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#logging). #### [](#probes)Probes Probes determine the health and readiness of your Pods. Configure them based on the startup behavior of your connectors. ```yaml deployment: livenessProbe: initialDelaySeconds: 60 periodSeconds: 10 readinessProbe: initialDelaySeconds: 30 periodSeconds: 10 ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#deployment). ### [](#data-management)Data management This section covers configuration related to data handling and topic management. #### [](#topics)Topics Kafka Connect leverages internal topics to track processed data, enhancing its fault tolerance: - The offset topic logs the last processed position from the external data source. - In events like failures or restarts, the connector uses this logged position to resume operations, ensuring no data duplication or omission. ```yaml connectors: storage: topic: offset: _internal_connectors_offsets ``` Here, `_internal_connectors_offsets` is the dedicated Kafka topic where Kafka Connect persists the offsets of the source connector. For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#connectors). #### [](#producers)Producers When a source connector retrieves data from an external system for Redpanda, it assumes the role of a producer: - The source connector is responsible for transforming the external data into Kafka-compatible messages. - It then produces (writes) these messages to a specified Kafka topic. The `producerBatchSize` and `producerLingerMS` settings specify how Kafka Connect groups messages before producing them. ```yaml connectors: producerBatchSize: 131072 producerLingerMS: 1 ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#connectors). ### [](#general-configuration)General configuration This section covers other important configuration settings. #### [](#name-overrides)Name overrides Deploying multiple instances of the same Helm chart in a Kubernetes cluster can lead to naming conflicts. Using `nameOverride` and `fullnameOverride` helps differentiate between them. If you have a production and staging environment, different names help to avoid confusion. - Use `nameOverride` to customize: - The default labels `app.kubernetes.io/component=` and `app.kubernetes.io/name=` - The suffix in the name of the resources `redpanda-` - Use `fullnameOverride` to customize the full name of the resources such as the Deployment and Services. ```yaml nameOverride: 'redpanda-connector-production' fullnameOverride: 'redpanda-connector-instance-prod' ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#nameoverride). #### [](#labels)Labels Kubernetes labels help you to organize, query, and manage your resources. Use labels to categorize Kubernetes resources in different deployments by environment, purpose, or team. ```yaml commonLabels: env: 'production' ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#commonlabels). #### [](#docker-image)Docker image You can specify the image tag to deploy a known version of the Docker image. Avoid using the `latest` tag, which can lead to unexpected changes. If you’re using a private repository, always ensure your nodes have the necessary credentials to pull the image. ```yaml image: repository: "redpandadata/connectors" tag: "1.2.3" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#image). #### [](#kafka-connect-configuration)Kafka Connect configuration You can configure Kafka Connect connection settings. Change the default REST API port only if it conflicts with an existing port. The `bootstrapServers` setting should point to the Kafka API endpoints on your Redpanda brokers. If you want to use Schema Registry, ensure the URL is set to the IP address or domain name of a Redpanda broker and that it includes the Schema Registry port. ```yaml connectors: restPort: 8082 bootstrapServers: "redpanda-broker-0:9092" schemaRegistryURL: "http://schema-registry.default.svc.cluster.local:8081" ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#connectors). #### [](#deployment-history)Deployment history Keeping track of your deployment’s history is beneficial for rollback scenarios. Adjust the `revisionHistoryLimit` according to your storage considerations. ```yaml deployment: progressDeadlineSeconds: 600 revisionHistoryLimit: 10 ``` For all available settings, see the [Helm specification](https://docs.redpanda.com/streaming/current/reference/k-connector-helm-spec/#deployment). ## [](#verify-the-deployment)Verify the deployment To verify that the deployment was successful, you can use the Kafka Connect REST API or check the deployment in Redpanda Console (if configured). ### [](#verify-with-the-kafka-connect-rest-api)Verify with the Kafka Connect REST API 1. Get the name of the Pod that’s running Kafka Connect: ```bash kubectl get pod -l app.kubernetes.io/name=connectors --namespace ``` Expected output should show pods in `Running` status: ```bash NAME READY STATUS RESTARTS AGE redpanda-connectors-6d64b948f6-dk484 1/1 Running 0 5m ``` 2. Check if the Kafka Connect service is accessible: ```bash kubectl get svc -l app.kubernetes.io/name=connectors --namespace ``` Expected output: ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda-connectors ClusterIP 10.96.123.45 8083/TCP 5m ``` 3. View the version of Kafka Connect: ```bash kubectl exec --namespace -- curl localhost:8083 | jq ``` Example output ```json { "version": "3.8.0", "commit": "771b9576b00ecf5b", "kafka_cluster_id": "redpanda.3e2649b0-f84c-4c03-b5e3-d6d1643f65b2" } ``` 4. View the list of available connectors: ```bash kubectl exec --namespace -- curl localhost:8083/connector-plugins | jq ``` Example output ```json [ { "class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "type": "source", "version": "3.8.0" }, { "class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector", "type": "source", "version": "3.8.0" }, { "class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "type": "source", "version": "3.8.0" } ] ``` 5. Test connectivity to your Redpanda cluster: ```bash kubectl exec --namespace -- curl localhost:8083/connectors ``` This should return an empty array `[]` if no connectors are configured, indicating that Kafka Connect can communicate with your Redpanda cluster. ### [](#troubleshoot-common-issues)Troubleshoot common issues If the deployment isn’t working as expected, check these common issues: #### [](#pod-not-starting-or-crashing)Pod not starting or crashing 1. Check pod logs for error messages: ```bash kubectl logs -l app.kubernetes.io/name=connectors --namespace --tail=50 ``` 2. Check for resource constraints: ```bash kubectl describe pod -l app.kubernetes.io/name=connectors --namespace ``` Common issues and solutions: - **"You must set either bootstrap.servers or bootstrap.controllers"**: The `connectors.bootstrapServers` configuration is missing or incorrectly formatted. - **OutOfMemoryError**: Increase memory limits or reduce `javaMaxHeapSize`. - **Connection refused to Redpanda brokers**: Verify the bootstrap servers addresses and ensure Redpanda is running. #### [](#testing-network-connectivity)Testing network connectivity 1. Test if the connector can reach Redpanda brokers: ```bash kubectl exec --namespace -- nslookup ``` 2. Test port connectivity: ```bash kubectl exec --namespace -- nc -zv 9093 ``` #### [](#verify-with-redpanda-console)Verify with Redpanda Console If you have Redpanda Console configured to connect to Kafka Connect: 1. Access Redpanda Console through port-forward: ```bash kubectl port-forward svc/redpanda-console 8080:8080 --namespace ``` 2. Open [http://localhost:8080](http://localhost:8080) in your browser 3. Navigate to **Connect** If the Connectors page shows "No clusters configured" or connection errors, verify your Redpanda Console configuration includes the correct Kafka Connect service URL. ### [](#health-checks-and-monitoring)Health checks and monitoring The connectors chart includes built-in health checks that you can use to monitor the status: 1. **Liveness probe**: Checks if Kafka Connect is responsive ```bash kubectl exec --namespace -- curl -f localhost:8083/ ``` 2. **Readiness probe**: Checks if Kafka Connect is ready to accept requests ```bash kubectl exec --namespace -- curl -f localhost:8083/connectors ``` 3. **View connector worker information**: ```bash kubectl exec --namespace -- curl localhost:8083/admin/workers | jq ``` 4. **Check cluster information**: ```bash kubectl exec --namespace -- curl localhost:8083/admin/cluster | jq ``` These endpoints help you verify that Kafka Connect is not only running but also properly connected to your Redpanda cluster and ready to manage connectors. ## [](#install-a-new-connector)Install a new connector To install new connectors other than the ones included in the Redpanda Connectors Docker image, you must: 1. Prepare a JAR (Java archive) file for the connector. 2. Mount the JAR file into the plugin directory of the Redpanda Connectors Docker image. 3. Use that Docker image in the Helm chart. ### [](#prepare-a-jar-file)Prepare a JAR file Kafka Connect is written in Java. As such, connectors are also written in Java and packaged into JAR files. JAR files are used to distribute Java classes and associated metadata and resources in a single file. You can get JAR files for connectors in many ways, including: - **Build from source**: If you have the source code for a Java project, you can compile and package it into a JAR using build tools, such as: - Maven: Using the `mvn package` command. - Gradle: Using the `gradle jar` or `gradle build` command. - Java Development Kit (JDK): Using the `jar` command-line tool that comes with the JDK. - **Maven Central Repository**: If you’re looking for a specific Java library or framework, it may be available in the Maven Central Repository. From here, you can search for the library and download the JAR directly. - **Vendor websites**: If you are looking for commercial Java software or libraries, the vendor’s official website is a good place to check. > ⚠️ **CAUTION** > > To avoid security risks, always verify the source of the JAR files. Do not download JAR files from unknown websites. Malicious JAR files can present a security risk to your execution environment. ### [](#add-the-connector-to-the-docker-image)Add the connector to the Docker image The Redpanda Connectors Docker image is configured to find connectors in the `/opt/kafka/redpanda-plugins` directory. You must mount your connector’s JAR file to this directory in the Docker image. 1. Create a new Dockerfile: `Dockerfile` ```dockerfile FROM redpandadata/connectors: COPY /opt/kafka/connect-plugins// ``` Replace the following placeholders: - ``: The version of the Redpanda Connectors Docker image that you want to use. For all available versions, see [DockerHub](https://hub.docker.com/r/redpandadata/connectors/tags). - ``: The path to the JAR file on your local system. - ``: A unique directory name in which to mount your JAR files. - ``: The name of your JAR file, including the `.jar` file extension. 2. Change into the directory where you created the Dockerfile and run: ```bash docker build -t /connectors: . ``` - Replace `` with the name of your Docker repository and `` with your desired version or tag for the image. 3. Push the image to your Docker repository: ```bash docker push /connectors: ``` ### [](#deploy-the-helm-chart-with-your-custom-docker-image)Deploy the Helm chart with your custom Docker image 1. Modify your values file to use your new Docker image: ```yaml image: repository: /connectors tag: pullPolicy: IfNotPresent ``` Kafka Connect should discover the new connector automatically on startup. 2. Update your deployment: ```bash helm upgrade redpanda-connectors redpanda/connectors \ --namespace \ --values connectors-values.yaml ``` 3. Get the name of the Pod that’s running Kafka Connect: ```bash kubectl get pod -l app.kubernetes.io/name=connectors --namespace ``` 4. View all available connectors: ```bash kubectl exec --namespace -- curl localhost:8083/connector-plugins | jq ``` You should see your new connector in the list. ## [](#next-steps)Next steps - [Create and Manage Kafka Connect Connectors in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-connectors/) - [Monitor Kafka Connect in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-connectors/) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 34: Deploy Redpanda **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda.md --- # Deploy Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/index.adoc description: Overview of Redpanda deployment options and links to platform-specific guides. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- Redpanda can be deployed on a variety of platforms to suit your infrastructure and operational needs. This section provides an overview of the available deployment methods and links to detailed, platform-specific instructions. - [Deploy on Kubernetes](kubernetes/) Learn about deployment options on Kubernetes. - [Deploy on Linux](manual/) Learn about deployment options on Linux, as well as considerations for high availability and sizing. --- # Page 35: Deploy on Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes.md --- # Deploy on Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy on Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/index.adoc description: Learn about deployment options on Kubernetes. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- Kubernetes provides a standardized way of achieving high availability, disaster recovery, and scalability. - [Redpanda in Kubernetes](k-deployment-overview/) Learn about Redpanda in Kubernetes and the tools that are available. - [Get Started with Redpanda in Kubernetes](get-started-dev/) Find guides for setting up a three-broker Redpanda cluster in different Kubernetes platforms. - [Production Deployment Workflow for Kubernetes](k-production-workflow/) Learn how to deploy Redpanda in Kubernetes for production. - [Kubernetes Cluster Requirements and Recommendations](k-requirements/) A list of requirements and recommendations for provisioning Kubernetes clusters and worker nodes for running Redpanda in production. - [Tune Kubernetes Worker Nodes for Production](k-tune-workers/) To get the best performance from your hardware, set Redpanda to production mode and run the autotuner tool. The autotuner identifies your hardware configuration and tunes itself to give you the best performance. - [Choose Your Deployment](k-choose-deployment/) Compare Redpanda deployment topologies for Kubernetes and choose the one that matches the failures you must survive and your recovery objectives. - [Production Readiness Checklist](k-production-readiness/) Comprehensive checklist for validating Redpanda deployments in Kubernetes against production readiness standards. --- # Page 36: Deploy a Redpanda Cluster in Azure Kubernetes Service **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/aks-guide.md --- # Deploy a Redpanda Cluster in Azure Kubernetes Service > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy a Redpanda Cluster in Azure Kubernetes Service latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/aks-guide page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/aks-guide.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/aks-guide.adoc description: Deploy a secure Redpanda cluster and Redpanda Console in Azure Kubernetes Service (AKS). page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- Deploy a secure Redpanda cluster and Redpanda Console in Azure Kubernetes Service (AKS). After you deploy, use `rpk` both as an internal client and an external client to interact with your Redpanda cluster from the command line. Your Redpanda cluster has the following security features: - SASL for authenticating users' connections. - TLS with self-signed certificates for secure communication between the cluster and clients. ## [](#prerequisites)Prerequisites - Satisfy the prerequisites listed in the [AKS quickstart](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli#prerequisites) to get access to the Azure CLI. - Install [`kubectl`](https://kubernetes.io/docs/tasks/tools/). Minimum required Kubernetes version: 1.27.0-0. ```bash kubectl version --client ``` - Install [Helm](https://helm.sh/docs/intro/install/). Minimum required Helm version: 3.10.0 ```bash helm version ``` ## [](#create-an-aks-cluster)Create an AKS cluster Your AKS cluster must have one worker node available for each Redpanda broker that you plan to deploy in your Redpanda cluster. You also need to run the worker nodes on a machine type that supports the [requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) for production deployments. In this step, you create an AKS cluster with three nodes on [Standard\_L8s\_v3 Azure Virtual Machines (Azure VMs)](https://learn.microsoft.com/en-us/azure/virtual-machines/lsv3-series). Deploying three nodes allows your AKS cluster to support a Redpanda cluster with three brokers. The Standard\_L8s\_v3 Azure VMs come with: - 2 cores per worker node, which is a requirement for production. - Local NVMe disks, which is recommended for best performance. 1. Create a resource group for Redpanda: ```bash az group create --name redpandaResourceGroup --location eastus ``` 2. Create an AKS cluster: ```bash az aks create -g redpandaResourceGroup -n \ --node-count 3 \ --generate-ssh-keys \ --enable-node-public-ip \ --node-vm-size Standard_L8s_v3 \ --disable-file-driver \ --node-os-upgrade-channel Unmanaged (1) ``` | 1 | Set the OS upgrade channel to Unmanaged to prevent AKS from automatically rebooting or upgrading nodes.For more details, see the requirements and recommendations for deploying Redpanda in Kubernetes. | | --- | --- | For all available options, see the [AKS documentation](https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-create). ### [](#create-sc)Create a StorageClass for your local NVMe disks When you provisioned the Kubernetes cluster, you selected an instance type that comes with local NVMe disks. However, these disks are not automatically mounted or formatted upon creation. To use these local NVMe disks, you must mount and format them, and you must create the necessary PersistentVolumes (PVs). To automate this process, you can use a Container Storage Interface (CSI) driver. In this step, you install the recommended [local volume manager (LVM) CSI driver](https://github.com/metal-stack/csi-driver-lvm). Then, you create a StorageClass that references the LVM CSI driver and specifies the recommended XFS file system. 1. Install the LVM CSI driver: ```yaml helm repo add metal-stack https://helm.metal-stack.io helm repo update helm install csi-driver-lvm metal-stack/csi-driver-lvm \ --version 0.6.0 \ --namespace csi-driver-lvm \ --create-namespace \ --set lvm.devicePattern='/dev/nvme[0-9]n[0-9]' ``` The `lvm.devicePattern` property specifies the pattern that the CSI driver uses to identify available NVMe volumes on your worker nodes. > 📝 **NOTE** > > Version 0.6.0 is required to avoid volume-mounting issues caused by recent `mkfs.xfs` updates. Newer versions enable the `-i nrext64=1` option, triggering the following error on default AKS kernels: > > XFS (dm-0): Superblock has unknown incompatible features (0x20) enabled. 2. Create the StorageClass: `csi-driver-lvm-striped-xfs.yaml` ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-driver-lvm-striped-xfs provisioner: lvm.csi.metal-stack.io reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: "striped" csi.storage.k8s.io/fstype: xfs mkfsParams: "-i nrext64=0" ``` - `provisioner`: The LVM CSI driver responsible for provisioning the volume. - `reclaimPolicy`: The `Retain` policy ensures that the underlying volume is not deleted when the corresponding PVC is deleted. - `volumeBindingMode`: The `WaitForFirstConsumer` mode delays the binding and provisioning of a PersistentVolume until a Pod that uses the PVC is created. This mode is important for ensuring that the PV is created on the same node where the Pod will run because the PV will use the node’s local NVMe volumes. - `allowVolumeExpansion`: Allows the volume to be expanded after it has been provisioned. - `parameters.type`: Combines multiple physical volumes to create a single logical volume. In a striped setup, data is spread across the physical volumes in a way that distributes the I/O load evenly, improving performance by allowing parallel disk I/O operations. - `parameters.csi.storage.k8s.io/fstype`: Formats the volumes with the XFS file system. Redpanda Data recommends XFS for its enhanced performance with Redpanda workloads. - `parameters.mkfsParams`: Disables the nrext64 feature to ensure compatibility with older kernels. 3. Apply the StorageClass: ```bash kubectl apply -f csi-driver-lvm-striped-xfs.yaml ``` After applying this StorageClass, any PVC that references it will attempt to provision storage using the LVM CSI driver and the provided parameters. ### [](#configure-external-access)Configure external access In this step, you configure your AKS cluster to allow external access to the node ports on which the Redpanda deployment will be exposed. You use these node ports in later steps to configure external access to your Redpanda cluster. 1. Get your subscription ID: ```bash export SUBSCRIPTION_ID=$(az account show --query id --output tsv) ``` 2. Set up a connection to your AKS cluster: ```bash az account set --subscription $SUBSCRIPTION_ID az aks get-credentials --resource-group redpandaResourceGroup --name ``` 3. Open the [Azure Portal](https://portal.azure.com/), search for 'Network security groups', and click the name of the network security group in the **MC\_redpandaResourceGroup\_redpanda\_eastus** resource group. 4. Add an inbound security rule with the following values: - **Destination port ranges**: 31644,31092,30082,30081 - **Name** AllowRedpandaNodePorts ## [](#deploy-redpanda-and-redpanda-console)Deploy Redpanda and Redpanda Console In this step, you deploy Redpanda with SASL authentication and self-signed TLS certificates. Redpanda Console is included as a subchart in the Redpanda Helm chart. ### Operator 1. Make sure that you have permission to install custom resource definitions (CRDs): ```bash kubectl auth can-i create CustomResourceDefinition --all-namespaces ``` You should see `yes` in the output. You need these cluster-level permissions to install [cert-manager](https://cert-manager.io/docs/) and Redpanda Operator CRDs in the next steps. 2. Install [cert-manager](https://cert-manager.io/docs/installation/helm/) using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` The Redpanda Helm chart uses cert-manager to enable TLS and manage TLS certificates by default. 3. Deploy the Redpanda Operator: 1. To deploy in cluster scope, use: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ (1) --set crds.enabled=true (2) ``` | 1 | This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment.This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces. | 2. To deploy in namespace scope (managing only resources within its deployment namespace), use: ```bash helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace="]' (1) ``` | 1 | This flag restricts the Redpanda Operator to manage resources only within the specified namespace. | | --- | --- | 4. Ensure that the Deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` deployment "redpanda-controller-operator" successfully rolled out 5. Install a [Redpanda custom resource](https://docs.redpanda.com/streaming/current/reference/k-crd/) in the same namespace as the Redpanda Operator: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: image: tag: v26.2.2 external: domain: customredpandadomain.local auth: sasl: enabled: true users: - name: superuser password: secretpassword storage: persistentVolume: enabled: true storageClass: csi-driver-lvm-striped-xfs ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker will advertise to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. 6. Wait for the Redpanda Operator to deploy Redpanda using the Helm chart: ```bash kubectl get redpanda --namespace --watch ``` NAME READY STATUS redpanda True Redpanda reconciliation succeeded This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing: ```bash kubectl get pod --namespace ``` If it’s taking too long, see [Troubleshoot](#troubleshoot). ### Helm 1. Install cert-manager using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default. 2. Install Redpanda with SASL enabled: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ --namespace --create-namespace \ --set image.tag=v26.2.2 \ --set auth.sasl.enabled=true \ --set "auth.sasl.users[0].name=superuser" \ --set "auth.sasl.users[0].password=secretpassword" \ --set external.domain=customredpandadomain.local \ --set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \ --wait \ --timeout 1h ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker advertises to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. The installation displays some tips for getting started. If the installation is taking a long time, see [Troubleshoot](#troubleshoot). ## [](#verify-the-deployment)Verify the deployment When the Redpanda Helm chart is deployed, you should have: - Three Redpanda brokers. Each Redpanda broker runs inside a separate Pod and is scheduled on a separate worker node. - One PVC bound to a PV for each Redpanda broker. These PVs are what the Redpanda brokers use to store the Redpanda data directory with all your topics and metadata. 1. Verify that each Redpanda broker is scheduled on only one Kubernetes node: ```bash kubectl get pod --namespace \ -o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset ``` Example output: NODE POD\_NAME example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2 2. Verify that each Redpanda broker has a bound PVC: ```bash kubectl get persistentvolumeclaim \ --namespace \ -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName ``` Example output: NAME STATUS STORAGECLASS datadir-redpanda-0 Bound csi-driver-lvm-striped-xfs datadir-redpanda-1 Bound csi-driver-lvm-striped-xfs datadir-redpanda-2 Bound csi-driver-lvm-striped-xfs ## [](#create-a-user)Create a user In this step, you use `rpk` to create a new user. Then, you authenticate to Redpanda with the superuser to grant permissions to the new user. You’ll authenticate to Redpanda with this new user to create a topic in the next steps. > 💡 **TIP** > > As a security best practice, you should use the superuser only to grant permissions to new users through ACLs. Never delete the superuser. You need the superuser to grant permissions to new users. 1. Create a new user called `redpanda-twitch-account` with the password `changethispassword`: ```bash kubectl --namespace exec -ti redpanda-0 -c redpanda -- \ rpk security user create redpanda-twitch-account \ -p changethispassword ``` Example output: Created user "redpanda-twitch-account". 2. Use the superuser to grant the `redpanda-twitch-account` user permission to execute all operations only for a topic called `twitch-chat`. ```bash kubectl exec --namespace -c redpanda redpanda-0 -- \ rpk security acl create --allow-principal User:redpanda-twitch-account \ --operation all \ --topic twitch-chat \ -X user=superuser -X pass=secretpassword -X sasl.mechanism=SCRAM-SHA-512 ``` Example output: PRINCIPAL RESOURCE-TYPE RESOURCE-NAME OPERATION PERMISSION User:redpanda TOPIC twitch-chat ALL ALLOW ## [](#start-streaming)Start streaming In this step, you authenticate to Redpanda with the `redpanda-twitch-account` user to create a topic called `twitch-chat`. This topic is the only one that the `redpanda-twitch-account` user has permission to access. Then, you produce messages to the topic, and consume messages from it. 1. Create an alias to simplify the `rpk` commands: ```bash alias internal-rpk="kubectl --namespace exec -i -t redpanda-0 -c redpanda -- rpk -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256" ``` 2. Create a topic called `twitch-chat`: ### Operator 1. Create a Secret in which to store your user’s password: ```bash kubectl create secret generic redpanda-secret --from-literal=password='changethispassword' --namespace ``` 2. Create a [Topic resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/): `topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Topic metadata: name: twitch-chat spec: kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" sasl: username: redpanda-twitch-account mechanism: SCRAM-SHA-256 passwordSecretRef: name: redpanda-secret key: password ``` 3. Apply the Topic resource in the same namespace as your Redpanda cluster: ```bash kubectl apply -f topic.yaml --namespace ``` 4. Check the logs of the Redpanda Operator to confirm that the topic was created: ```bash kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace ``` You should see that the Redpanda Operator reconciled the Topic resource. For example: Example output ```json { "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } } ``` ### Helm ```bash internal-rpk topic create twitch-chat ``` Example output: TOPIC STATUS twitch-chat OK 3. Describe the topic: ```bash internal-rpk topic describe twitch-chat ``` Expected output: ```none SUMMARY ======= NAME twitch-chat PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG partition_count 1 DYNAMIC_TOPIC_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG replication_factor 1 DYNAMIC_TOPIC_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG ``` 4. Produce a message to the topic: ```bash internal-rpk topic produce twitch-chat ``` 5. Type a message, then press Enter: Pandas are fabulous! Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 6. Press Ctrl+C to finish producing messages to the topic. 7. Consume one message from the topic: ```bash internal-rpk topic consume twitch-chat --num 1 ``` Expected output: ```none { "topic": "twitch-chat", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ## [](#explore-your-topic-in-redpanda-console)Explore your topic in Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. In this step, you use port-forwarding to access Redpanda Console on your local network. > 💡 **TIP** > > Because you’re using the Community Edition of Redpanda Console, you should not expose Redpanda Console outside your local network. The Community Edition of Redpanda Console does not provide authentication, and it connects to the Redpanda cluster as superuser. To use the Enterprise Edition, you need a license key. See [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). 1. Expose Redpanda Console to your localhost: ```bash kubectl --namespace port-forward svc/redpanda-console 8080:8080 ``` The `kubectl port-forward` command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window. 2. Open Redpanda Console on [http://localhost:8080](http://localhost:8080). All your Redpanda brokers are listed along with their IP addresses and IDs. 3. Go to **Topics** > **twitch-chat**. The message that you produced to the topic is displayed along with some other details about the topic. 4. Press Ctrl+C in the command-line to stop the port-forwarding process. ## [](#configure-external-access-to-redpanda)Configure external access to Redpanda If you want to connect to the Redpanda cluster with external clients, Redpanda brokers must advertise an externally accessible address that external clients can connect to. External clients are common in Internet of Things (IoT) environments, or if you use external services that do not implement VPC peering in your network. When you created the cluster, you set the `external.domain` configuration to `customredpandadomain.local`, which means that your Redpanda brokers are advertising the following addresses: - `redpanda-0.customredpandadomain.local` - `redpanda-1.customredpandadomain.local` - `redpanda-2.customredpandadomain.local` To access your Redpanda brokers externally, you can map your worker nodes' IP addresses to these domains. > ⚠️ **CAUTION** > > IP addresses can change. If the IP addresses of your worker nodes change, you must update your `/etc/hosts` file with the new mappings. > > In a production environment, it’s a best practice to use ExternalDNS to manage DNS records for your brokers. See [Use ExternalDNS for external access](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#externaldns). 1. Add mappings in your `/etc/hosts` file between your worker nodes' IP addresses and their custom domain names: ```bash sudo true && kubectl --namespace get endpoints,node -A -o go-template='{{ range $_ := .items }}{{ if and (eq .kind "Endpoints") (eq .metadata.name "redpanda-external") }}{{ range $_ := (index .subsets 0).addresses }}{{ $nodeName := .nodeName }}{{ $podName := .targetRef.name }}{{ range $node := $.items }}{{ if and (eq .kind "Node") (eq .metadata.name $nodeName) }}{{ range $_ := .status.addresses }}{{ if eq .type "ExternalIP" }}{{ .address }} {{ $podName }}.customredpandadomain.local{{ "\n" }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}' | envsubst | sudo tee -a /etc/hosts ``` `/etc/hosts` 203.0.113.3 redpanda-0.customredpandadomain.local 203.0.113.5 redpanda-1.customredpandadomain.local 203.0.113.7 redpanda-2.customredpandadomain.local 2. Save the root certificate authority (CA) to your local file system outside Kubernetes: ```bash kubectl --namespace get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 3. Install `rpk` on your local machine, not on a Pod: ### Linux > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` ### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` 4. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` Replace `` with the name that you want to give this `rpk` profile. 5. Test the connection: ```bash rpk cluster info -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256 ``` ## [](#explore-the-default-kubernetes-components)Explore the default Kubernetes components By default, the Redpanda Helm chart deploys the following Kubernetes components: - [A StatefulSet](#statefulset) with three Pods. - [One PersistentVolumeClaim](#persistentvolumeclaim) for each Pod, each with a capacity of 20Gi. - [A headless ClusterIP Service and a NodePort Service](#service) for each Kubernetes node that runs a Redpanda broker. - [Self-Signed TLS Certificates](#tls-certificates). ### [](#statefulset)StatefulSet Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running. ```bash kubectl get statefulset --namespace ``` Example output: NAME READY AGE redpanda 3/3 3m11s StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as `redpanda-0`. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced. ```bash kubectl get pod --namespace ``` Expected output: ```none NAME READY STATUS RESTARTS AGE redpanda-0 1/1 Running 0 6m9s redpanda-1 1/1 Running 0 6m9s redpanda-2 1/1 Running 0 6m9s redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m redpanda-configuration-smqv7 0/1 Completed 0 6m9s ``` > 📝 **NOTE** > > The `redpanda-configuration` job updates the Redpanda runtime configuration. ### [](#persistentvolumeclaim)PersistentVolumeClaim Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using. ```bash kubectl get persistentvolumeclaims --namespace ``` Expected output: ```none NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s ``` ### [](#service)Service The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services: - Internal using the [Headless ClusterIP](#headless-clusterip-service) - External using the [NodePort](#nodeport-service) ```bash kubectl get service --namespace ``` Expected output: ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda ClusterIP None 5m37s redpanda-console ClusterIP 10.0.251.204 8080 5m redpanda-external NodePort 10.96.137.220 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s ``` #### [](#headless-clusterip-service)Headless ClusterIP Service The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other. An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other. ```bash kubectl --namespace exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml ``` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: redpanda-0.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-1.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-2.redpanda..svc.cluster.local. port: 33145 ``` #### [](#nodeport-service)NodePort Service External access is made available by a NodePort service that opens the following ports by default: | Listener | Node Port | Container Port | | --- | --- | --- | | Schema Registry | 30081 | 8081 | | HTTP Proxy | 30082 | 8083 | | Kafka API | 31092 | 9094 | | Admin API | 31644 | 9644 | To learn more, see [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/). ### [](#tls-certificates)TLS Certificates By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses [cert-manager](https://cert-manager.io/docs/) to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections. Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True - `redpanda-default-cert`: Self-signed certificate for internal communications. - `redpanda-default-root-certificate`: Root certificate authority for the internal certificate. - `redpanda-external-cert`: Self-signed certificate for external communications. - `redpanda-external-root-certificate`: Root certificate authority for the external certificate. By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/). > 📝 **NOTE** > > The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. ## [](#uninstall-redpanda)Uninstall Redpanda When you finish testing Redpanda, you can uninstall it from your Kubernetes cluster. The steps depend on how you installed Redpanda: using the Redpanda Operator or the Redpanda Helm chart. ### Operator Follow the steps in **exact order** to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection. 1. Delete all Redpanda-related custom resources: ```bash kubectl delete users --namespace --all kubectl delete topics --namespace --all kubectl delete schemas --namespace --all kubectl delete redpanda --namespace --all kubectl delete consoles --namespace --all ``` The Redpanda Operator creates a Console resource for each Redpanda resource, so you must delete Console resources too. The Redpanda Operator must still be running when you delete these resources. It removes their finalizers as part of its cleanup. If you uninstall the Redpanda Operator first, any remaining resource keeps its finalizer forever and the CRD deletion step below hangs. 2. Make sure requests for those resources return no results. Do not continue until every command returns `No resources found`: ```bash kubectl get users,topics,schemas,redpandas,consoles --namespace ``` 3. Uninstall the Redpanda Operator Helm release: ```bash helm uninstall redpanda-controller --namespace ``` Helm does not uninstall CRDs by default when using `helm uninstall` to avoid accidentally deleting existing custom resources. 4. Remove the CRDs. 1. List all Redpanda CRDs installed by the operator: ```bash kubectl api-resources --api-group='cluster.redpanda.com' ``` This command displays all CRDs defined by the Redpanda Operator. For example: ```bash NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User ``` 2. Delete the CRDs: ```bash kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl delete ``` This command lists all CRDs with the `cluster.redpanda.com` domain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually. 5. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ### Helm If you deployed Redpanda with the Redpanda Helm chart, follow these steps to uninstall it: 1. Uninstall the Helm release: ```bash helm uninstall redpanda --namespace ``` 2. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ## [](#delete-the-cluster)Delete the cluster To delete your Kubernetes cluster: ```bash az aks delete --name --resource-group redpandaResourceGroup ``` To remove the convenience alias created during the quickstart: ```bash unalias internal-rpk ``` ## [](#troubleshoot)Troubleshoot Before troubleshooting your cluster, make sure that you have all the [prerequisites](#prerequisites). ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ### [](#statefulset-never-rolls-out)StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). ## [](#next-steps)Next steps - [Try an example in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about Redpanda Console](https://docs.redpanda.com/streaming/current/manage/console/) - [Learn more about rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) > 💡 **TIP** > > When you’re ready to use a registered domain, make sure to remove your entries from the `/etc/hosts` file, and see [Configure External Access through a NodePort Service](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-nodeport/#use-the-default-redpanda-subdomains). ## [](#suggested-reading)Suggested reading - [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/) - [Configure TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) - [Configure SASL for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) - [Redpanda Console README](https://github.com/redpanda-data/console) on GitHub ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 37: Deploy a Redpanda Cluster in Amazon Elastic Kubernetes Service **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/eks-guide.md --- # Deploy a Redpanda Cluster in Amazon Elastic Kubernetes Service > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy a Redpanda Cluster in Amazon Elastic Kubernetes Service latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/eks-guide page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/eks-guide.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/eks-guide.adoc description: Deploy a secure Redpanda cluster and Redpanda Console in Amazon Elastic Kubernetes Service (EKS). page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- Deploy a secure Redpanda cluster and Redpanda Console in Amazon Elastic Kubernetes Service (EKS). Then, use `rpk` both as an internal client and an external client to interact with your Redpanda cluster from the command line. Your Redpanda cluster has the following security features: - SASL for authenticating users' connections. - TLS with self-signed certificates for secure communication between the cluster and clients. ## [](#prerequisites)Prerequisites Before you begin, you must meet the following prerequisites. ### [](#iam-user)IAM user You need an IAM user with at least the following policies. See the AWS documentation for help [creating IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) or for help [troubleshooting IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot.html). Policies Replace `` with your own [account ID](https://console.aws.amazon.com/iamv2/home). AmazonEC2FullAccess ```json { "Version": "2012-10-17", "Statement": [ { "Action": "ec2:*", "Effect": "Allow", "Resource": "*" }, { "Effect": "Allow", "Action": "elasticloadbalancing:*", "Resource": "*" }, { "Effect": "Allow", "Action": "cloudwatch:*", "Resource": "*" }, { "Effect": "Allow", "Action": "autoscaling:*", "Resource": "*" }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "autoscaling.amazonaws.com", "ec2scheduled.amazonaws.com", "elasticloadbalancing.amazonaws.com", "spot.amazonaws.com", "spotfleet.amazonaws.com", "transitgateway.amazonaws.com" ] } } } ] } ``` AWSCloudFormationFullAccess ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudformation:*" ], "Resource": "*" } ] } ``` EksAllAccess ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "eks:*", "Resource": "*" }, { "Action": [ "ssm:GetParameter", "ssm:GetParameters" ], "Resource": [ "arn:aws:ssm:*::parameter/aws/*", "arn:aws:ssm:*::parameter/aws/*" ], "Effect": "Allow" }, { "Action": [ "kms:CreateGrant", "kms:DescribeKey" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "logs:PutRetentionPolicy" ], "Resource": "*", "Effect": "Allow" } ] } ``` IamLimitedAccess ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetInstanceProfile", "iam:RemoveRoleFromInstanceProfile", "iam:GetRole", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:PutRolePolicy", "iam:ListInstanceProfiles", "iam:AddRoleToInstanceProfile", "iam:ListInstanceProfilesForRole", "iam:PassRole", "iam:DetachRolePolicy", "iam:DeleteRolePolicy", "iam:GetRolePolicy", "iam:GetOpenIDConnectProvider", "iam:CreateOpenIDConnectProvider", "iam:DeleteOpenIDConnectProvider", "iam:TagOpenIDConnectProvider", "iam:ListAttachedRolePolicies", "iam:TagRole", "iam:GetPolicy", "iam:CreatePolicy", "iam:DeletePolicy", "iam:ListPolicyVersions" ], "Resource": [ "arn:aws:iam:::instance-profile/eksctl-*", "arn:aws:iam:::role/eksctl-*", "arn:aws:iam:::policy/eksctl-*", "arn:aws:iam:::oidc-provider/*", "arn:aws:iam:::role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup", "arn:aws:iam:::role/eksctl-managed-*", "arn:aws:iam:::role/AmazonEKS_EBS_CSI_DriverRole" ] }, { "Effect": "Allow", "Action": [ "iam:GetRole" ], "Resource": [ "arn:aws:iam:::role/*" ] }, { "Effect": "Allow", "Action": [ "iam:CreateServiceLinkedRole" ], "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "eks.amazonaws.com", "eks-nodegroup.amazonaws.com", "eks-fargate.amazonaws.com" ] } } } ] } ``` ### [](#aws-cli)AWS CLI You need the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) to configure `kubeconfig` and get information about your EC2 instances. After you’ve installed the AWS CLI, make sure to [configure it](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html) with credentials for your IAM user. > 📝 **NOTE** > > If your account uses an identity provider in the IAM Identity Center (previously AWS SSO, [authenticate with the IAM Identity Center](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) (`aws sso login`). For troubleshooting, see the [AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#install-tshoot). ### [](#eksctl)eksctl You need [`eksctl`](https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html) to create an EKS cluster from the command line. ### [](#jq)jq You need [jq](https://stedolan.github.io/jq/download/) to parse JSON results and store the value in environment variables. ### [](#kubectl)kubectl You must have [`kubectl`](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html) with the following minimum required Kubernetes version: 1.27.0-0 To check if you have `kubectl` installed: ```bash kubectl version --client ``` ### [](#helm)Helm You must have the following minimum required version of [Helm](https://helm.sh/docs/intro/install/): 3.10.0 To check if you have Helm installed: ```bash helm version ``` ## [](#create-an-eks-cluster)Create an EKS cluster Your EKS cluster must have one worker node available for each Redpanda broker that you plan to deploy in your Redpanda cluster. You also need to run the worker nodes on an EC2 instance type that supports the [requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) for production deployments. In this step, you create an EKS cluster with three nodes on [`c5d.2xlarge` instance types](https://aws.amazon.com/ec2/instance-types/c5/). Deploying three nodes allows your EKS cluster to support a Redpanda cluster with three brokers. The `c5d.2xlarge` instance type comes with: - Sufficient CPU: Redpanda requires at least 2 full CPU cores per broker. The c5d.2xlarge instance provides 8 vCPUs, which comfortably meets this requirement. - Local NVMe disks, which is recommended for best performance. 1. Create an EKS cluster and give it a unique name. If your account is configured with OIDC, add the `--with-oidc` flag to the `create cluster` command. ```bash eksctl create cluster \ --name \ --nodegroup-name nvme-workers \ --node-type c5d.2xlarge \ --nodes 3 \ --external-dns-access ``` > ❗ **IMPORTANT** > > Do not enable [auto mode](https://docs.aws.amazon.com/eks/latest/userguide/automode.html) (`--enable-auto-mode`) on Amazon EKS clusters running Redpanda. > > Auto mode can trigger automatic reboots or node upgrades that disrupt Redpanda brokers, risking data loss or cluster instability. Redpanda requires manual control over node lifecycle events. > > For more details, see the [requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#node-updates) for deploying Redpanda in Kubernetes. To see all options: ```bash eksctl create cluster --help ``` Or, for help creating an EKS cluster, see the [Creating and managing clusters](https://eksctl.io/usage/creating-and-managing-clusters/) in the `eksctl` documentation. 2. Make sure that your local `kubeconfig` file points to your EKS cluster: ```bash kubectl get service ``` You should see a ClusterIP Service called `kubernetes`. If the `kubectl` command cannot connect to your cluster, update your local `kubeconfig` file to point to your EKS cluster. Your default region is in the `~/.aws/credentials` file. ```bash aws eks update-kubeconfig --region --name ``` ### [](#create-sc)Create a StorageClass for your local NVMe disks When you provisioned the Kubernetes cluster, you selected an instance type that comes with local NVMe disks. However, these disks are not automatically mounted or formatted upon creation. To use these local NVMe disks, you must mount and format them, and you must create the necessary PersistentVolumes (PVs). To automate this process, you can use a Container Storage Interface (CSI) driver. In this step, you install the recommended [local volume manager (LVM) CSI driver](https://github.com/metal-stack/csi-driver-lvm). Then, you create a StorageClass that references the LVM CSI driver and specifies the recommended XFS file system. 1. Install the LVM CSI driver: ```yaml helm repo add metal-stack https://helm.metal-stack.io helm repo update helm install csi-driver-lvm metal-stack/csi-driver-lvm \ --version 0.6.0 \ --namespace csi-driver-lvm \ --create-namespace \ --set lvm.devicePattern='/dev/nvme[1-9]n[0-9]' ``` The `lvm.devicePattern` property specifies the pattern that the CSI driver uses to identify available NVMe volumes on your worker nodes. > 📝 **NOTE** > > Version 0.6.0 is required to avoid volume-mounting issues caused by recent `mkfs.xfs` updates. Newer versions enable the `-i nrext64=1` option, triggering the following error on default EKS kernels: > > XFS (dm-0): Superblock has unknown incompatible features (0x20) enabled. 2. Create the StorageClass: `csi-driver-lvm-striped-xfs.yaml` ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-driver-lvm-striped-xfs provisioner: lvm.csi.metal-stack.io reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: "striped" csi.storage.k8s.io/fstype: xfs mkfsParams: "-i nrext64=0" ``` - `provisioner`: The LVM CSI driver responsible for provisioning the volume. - `reclaimPolicy`: The `Retain` policy ensures that the underlying volume is not deleted when the corresponding PVC is deleted. - `volumeBindingMode`: The `WaitForFirstConsumer` mode delays the binding and provisioning of a PersistentVolume until a Pod that uses the PVC is created. This mode is important for ensuring that the PV is created on the same node where the Pod will run because the PV will use the node’s local NVMe volumes. - `allowVolumeExpansion`: Allows the volume to be expanded after it has been provisioned. - `parameters.type`: Combines multiple physical volumes to create a single logical volume. In a striped setup, data is spread across the physical volumes in a way that distributes the I/O load evenly, improving performance by allowing parallel disk I/O operations. - `parameters.csi.storage.k8s.io/fstype`: Formats the volumes with the XFS file system. Redpanda Data recommends XFS for its enhanced performance with Redpanda workloads. - `parameters.mkfsParams`: Disables the nrext64 feature to ensure compatibility with older kernels. 3. Apply the StorageClass: ```bash kubectl apply -f csi-driver-lvm-striped-xfs.yaml ``` After applying this StorageClass, any PVC that references it will attempt to provision storage using the LVM CSI driver and the provided parameters. ### [](#configure-external-access)Configure external access In this step, you configure your EKS cluster to allow external access to the node ports on which the Redpanda deployment will be exposed. You use these node ports in later steps to configure external access to your Redpanda cluster. 1. Get the ID of the security group that’s associated with the nodes in your EKS cluster: ```bash AWS_SECURITY_GROUP_ID=`aws eks describe-cluster --name | jq -r '.cluster.resourcesVpcConfig.clusterSecurityGroupId'` ``` 2. Add inbound firewall rules to your EC2 instances so that external traffic can reach the node ports exposed on all Kubernetes worker nodes in the cluster: ```bash aws ec2 authorize-security-group-ingress \ --group-id ${AWS_SECURITY_GROUP_ID} \ --ip-permissions '[ { "IpProtocol": "tcp", "FromPort": 30081, "ToPort": 30081, "IpRanges": [{"CidrIp": "0.0.0.0/0"}] }, { "IpProtocol": "tcp", "FromPort": 30082, "ToPort": 30082, "IpRanges": [{"CidrIp": "0.0.0.0/0"}] }, { "IpProtocol": "tcp", "FromPort": 31644, "ToPort": 31644, "IpRanges": [{"CidrIp": "0.0.0.0/0"}] }, { "IpProtocol": "tcp", "FromPort": 31092, "ToPort": 31092, "IpRanges": [{"CidrIp": "0.0.0.0/0"}] } ]' ``` > ⚠️ **CAUTION** > > If you use `0.0.0.0/0`, you enable all IPv4 addresses to access your instances on those node ports. In production, you should authorize only a specific IP address or range of addresses to access your instances. For help creating firewall rules, see the [Amazon EC2 documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html). ## [](#deploy-redpanda-and-redpanda-console)Deploy Redpanda and Redpanda Console In this step, you deploy Redpanda with SASL authentication and self-signed TLS certificates. Redpanda Console is included as a subchart in the Redpanda Helm chart. ### Operator 1. Make sure that you have permission to install custom resource definitions (CRDs): ```bash kubectl auth can-i create CustomResourceDefinition --all-namespaces ``` You should see `yes` in the output. You need these cluster-level permissions to install [cert-manager](https://cert-manager.io/docs/) and Redpanda Operator CRDs in the next steps. 2. Install [cert-manager](https://cert-manager.io/docs/installation/helm/) using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` The Redpanda Helm chart uses cert-manager to enable TLS and manage TLS certificates by default. 3. Deploy the Redpanda Operator: 1. To deploy in cluster scope, use: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ (1) --set crds.enabled=true (2) ``` | 1 | This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment.This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces. | 2. To deploy in namespace scope (managing only resources within its deployment namespace), use: ```bash helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace="]' (1) ``` | 1 | This flag restricts the Redpanda Operator to manage resources only within the specified namespace. | | --- | --- | 4. Ensure that the Deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` deployment "redpanda-controller-operator" successfully rolled out 5. Install a [Redpanda custom resource](https://docs.redpanda.com/streaming/current/reference/k-crd/) in the same namespace as the Redpanda Operator: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: image: tag: v26.2.2 external: domain: customredpandadomain.local auth: sasl: enabled: true users: - name: superuser password: secretpassword storage: persistentVolume: enabled: true storageClass: csi-driver-lvm-striped-xfs ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker will advertise to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. 6. Wait for the Redpanda Operator to deploy Redpanda using the Helm chart: ```bash kubectl get redpanda --namespace --watch ``` NAME READY STATUS redpanda True Redpanda reconciliation succeeded This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing: ```bash kubectl get pod --namespace ``` If it’s taking too long, see [Troubleshoot](#troubleshoot). ### Helm 1. Install cert-manager using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default. 2. Install Redpanda with SASL enabled: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ --namespace --create-namespace \ --set image.tag=v26.2.2 \ --set auth.sasl.enabled=true \ --set "auth.sasl.users[0].name=superuser" \ --set "auth.sasl.users[0].password=secretpassword" \ --set external.domain=customredpandadomain.local \ --set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \ --wait \ --timeout 1h ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker advertises to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. The installation displays some tips for getting started. If the installation is taking a long time, see [Troubleshoot](#troubleshoot). ## [](#verify-the-deployment)Verify the deployment When the Redpanda Helm chart is deployed, you should have: - Three Redpanda brokers. Each Redpanda broker runs inside a separate Pod and is scheduled on a separate worker node. - One PVC bound to a PV for each Redpanda broker. These PVs are what the Redpanda brokers use to store the Redpanda data directory with all your topics and metadata. 1. Verify that each Redpanda broker is scheduled on only one Kubernetes node: ```bash kubectl get pod --namespace \ -o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset ``` Example output: NODE POD\_NAME example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2 2. Verify that each Redpanda broker has a bound PVC: ```bash kubectl get persistentvolumeclaim \ --namespace \ -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName ``` Example output: NAME STATUS STORAGECLASS datadir-redpanda-0 Bound csi-driver-lvm-striped-xfs datadir-redpanda-1 Bound csi-driver-lvm-striped-xfs datadir-redpanda-2 Bound csi-driver-lvm-striped-xfs ## [](#create-a-user)Create a user In this step, you use `rpk` to create a new user. Then, you authenticate to Redpanda with the superuser to grant permissions to the new user. You’ll authenticate to Redpanda with this new user to create a topic in the next steps. > 💡 **TIP** > > As a security best practice, you should use the superuser only to grant permissions to new users through ACLs. Never delete the superuser. You need the superuser to grant permissions to new users. 1. Create a new user called `redpanda-twitch-account` with the password `changethispassword`: ```bash kubectl --namespace exec -ti redpanda-0 -c redpanda -- \ rpk security user create redpanda-twitch-account \ -p changethispassword ``` Example output: Created user "redpanda-twitch-account". 2. Use the superuser to grant the `redpanda-twitch-account` user permission to execute all operations only for a topic called `twitch-chat`. ```bash kubectl exec --namespace -c redpanda redpanda-0 -- \ rpk security acl create --allow-principal User:redpanda-twitch-account \ --operation all \ --topic twitch-chat \ -X user=superuser -X pass=secretpassword -X sasl.mechanism=SCRAM-SHA-512 ``` Example output: PRINCIPAL RESOURCE-TYPE RESOURCE-NAME OPERATION PERMISSION User:redpanda TOPIC twitch-chat ALL ALLOW ## [](#start-streaming)Start streaming In this step, you authenticate to Redpanda with the `redpanda-twitch-account` user to create a topic called `twitch-chat`. This topic is the only one that the `redpanda-twitch-account` user has permission to access. Then, you produce messages to the topic, and consume messages from it. 1. Create an alias to simplify the `rpk` commands: ```bash alias internal-rpk="kubectl --namespace exec -i -t redpanda-0 -c redpanda -- rpk -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256" ``` 2. Create a topic called `twitch-chat`: ### Operator 1. Create a Secret in which to store your user’s password: ```bash kubectl create secret generic redpanda-secret --from-literal=password='changethispassword' --namespace ``` 2. Create a [Topic resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/): `topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Topic metadata: name: twitch-chat spec: kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" sasl: username: redpanda-twitch-account mechanism: SCRAM-SHA-256 passwordSecretRef: name: redpanda-secret key: password ``` 3. Apply the Topic resource in the same namespace as your Redpanda cluster: ```bash kubectl apply -f topic.yaml --namespace ``` 4. Check the logs of the Redpanda Operator to confirm that the topic was created: ```bash kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace ``` You should see that the Redpanda Operator reconciled the Topic resource. For example: Example output ```json { "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } } ``` ### Helm ```bash internal-rpk topic create twitch-chat ``` Example output: TOPIC STATUS twitch-chat OK 3. Describe the topic: ```bash internal-rpk topic describe twitch-chat ``` Expected output: ```none SUMMARY ======= NAME twitch-chat PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG partition_count 1 DYNAMIC_TOPIC_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG replication_factor 1 DYNAMIC_TOPIC_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG ``` 4. Produce a message to the topic: ```bash internal-rpk topic produce twitch-chat ``` 5. Type a message, then press Enter: Pandas are fabulous! Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 6. Press Ctrl+C to finish producing messages to the topic. 7. Consume one message from the topic: ```bash internal-rpk topic consume twitch-chat --num 1 ``` Expected output: ```none { "topic": "twitch-chat", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ## [](#explore-your-topic-in-redpanda-console)Explore your topic in Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. In this step, you use port-forwarding to access Redpanda Console on your local network. > 💡 **TIP** > > Because you’re using the Community Edition of Redpanda Console, you should not expose Redpanda Console outside your local network. The Community Edition of Redpanda Console does not provide authentication, and it connects to the Redpanda cluster as superuser. To use the Enterprise Edition, you need a license key. See [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). 1. Expose Redpanda Console to your localhost: ```bash kubectl --namespace port-forward svc/redpanda-console 8080:8080 ``` The `kubectl port-forward` command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window. 2. Open Redpanda Console on [http://localhost:8080](http://localhost:8080). All your Redpanda brokers are listed along with their IP addresses and IDs. 3. Go to **Topics** > **twitch-chat**. The message that you produced to the topic is displayed along with some other details about the topic. 4. Press Ctrl+C in the command-line to stop the port-forwarding process. ## [](#configure-external-access-to-redpanda)Configure external access to Redpanda If you want to connect to the Redpanda cluster with external clients, Redpanda brokers must advertise an externally accessible address that external clients can connect to. External clients are common in Internet of Things (IoT) environments, or if you use external services that do not implement VPC peering in your network. When you created the cluster, you set the `external.domain` configuration to `customredpandadomain.local`, which means that your Redpanda brokers are advertising the following addresses: - `redpanda-0.customredpandadomain.local` - `redpanda-1.customredpandadomain.local` - `redpanda-2.customredpandadomain.local` To access your Redpanda brokers externally, you can map your worker nodes' IP addresses to these domains. > ⚠️ **CAUTION** > > IP addresses can change. If the IP addresses of your worker nodes change, you must update your `/etc/hosts` file with the new mappings. > > In a production environment, it’s a best practice to use ExternalDNS to manage DNS records for your brokers. See [Use ExternalDNS for external access](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#externaldns). 1. Add mappings in your `/etc/hosts` file between your worker nodes' IP addresses and their custom domain names: ```bash sudo true && kubectl --namespace get endpoints,node -A -o go-template='{{ range $_ := .items }}{{ if and (eq .kind "Endpoints") (eq .metadata.name "redpanda-external") }}{{ range $_ := (index .subsets 0).addresses }}{{ $nodeName := .nodeName }}{{ $podName := .targetRef.name }}{{ range $node := $.items }}{{ if and (eq .kind "Node") (eq .metadata.name $nodeName) }}{{ range $_ := .status.addresses }}{{ if eq .type "ExternalIP" }}{{ .address }} {{ $podName }}.customredpandadomain.local{{ "\n" }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}' | envsubst | sudo tee -a /etc/hosts ``` `/etc/hosts` 203.0.113.3 redpanda-0.customredpandadomain.local 203.0.113.5 redpanda-1.customredpandadomain.local 203.0.113.7 redpanda-2.customredpandadomain.local 2. Save the root certificate authority (CA) to your local file system outside Kubernetes: ```bash kubectl --namespace get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 3. Install `rpk` on your local machine, not on a Pod: ### Linux > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` ### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` 4. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` Replace `` with the name that you want to give this `rpk` profile. 5. Test the connection: ```bash rpk cluster info -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256 ``` ## [](#explore-the-default-kubernetes-components)Explore the default Kubernetes components By default, the Redpanda Helm chart deploys the following Kubernetes components: - [A StatefulSet](#statefulset) with three Pods. - [One PersistentVolumeClaim](#persistentvolumeclaim) for each Pod, each with a capacity of 20Gi. - [A headless ClusterIP Service and a NodePort Service](#service) for each Kubernetes node that runs a Redpanda broker. - [Self-Signed TLS Certificates](#tls-certificates). ### [](#statefulset)StatefulSet Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running. ```bash kubectl get statefulset --namespace ``` Example output: NAME READY AGE redpanda 3/3 3m11s StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as `redpanda-0`. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced. ```bash kubectl get pod --namespace ``` Expected output: ```none NAME READY STATUS RESTARTS AGE redpanda-0 1/1 Running 0 6m9s redpanda-1 1/1 Running 0 6m9s redpanda-2 1/1 Running 0 6m9s redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m redpanda-configuration-smqv7 0/1 Completed 0 6m9s ``` > 📝 **NOTE** > > The `redpanda-configuration` job updates the Redpanda runtime configuration. ### [](#persistentvolumeclaim)PersistentVolumeClaim Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using. ```bash kubectl get persistentvolumeclaims --namespace ``` Expected output: ```none NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s ``` ### [](#service)Service The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services: - Internal using the [Headless ClusterIP](#headless-clusterip-service) - External using the [NodePort](#nodeport-service) ```bash kubectl get service --namespace ``` Expected output: ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda ClusterIP None 5m37s redpanda-console ClusterIP 10.0.251.204 8080 5m redpanda-external NodePort 10.96.137.220 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s ``` #### [](#headless-clusterip-service)Headless ClusterIP Service The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other. An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other. ```bash kubectl --namespace exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml ``` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: redpanda-0.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-1.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-2.redpanda..svc.cluster.local. port: 33145 ``` #### [](#nodeport-service)NodePort Service External access is made available by a NodePort service that opens the following ports by default: | Listener | Node Port | Container Port | | --- | --- | --- | | Schema Registry | 30081 | 8081 | | HTTP Proxy | 30082 | 8083 | | Kafka API | 31092 | 9094 | | Admin API | 31644 | 9644 | To learn more, see [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/). ### [](#tls-certificates)TLS Certificates By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses [cert-manager](https://cert-manager.io/docs/) to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections. Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True - `redpanda-default-cert`: Self-signed certificate for internal communications. - `redpanda-default-root-certificate`: Root certificate authority for the internal certificate. - `redpanda-external-cert`: Self-signed certificate for external communications. - `redpanda-external-root-certificate`: Root certificate authority for the external certificate. By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/). > 📝 **NOTE** > > The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. ## [](#uninstall-redpanda)Uninstall Redpanda When you finish testing Redpanda, you can uninstall it from your Kubernetes cluster. The steps depend on how you installed Redpanda: using the Redpanda Operator or the Redpanda Helm chart. ### Operator Follow the steps in **exact order** to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection. 1. Delete all Redpanda-related custom resources: ```bash kubectl delete users --namespace --all kubectl delete topics --namespace --all kubectl delete schemas --namespace --all kubectl delete redpanda --namespace --all kubectl delete consoles --namespace --all ``` The Redpanda Operator creates a Console resource for each Redpanda resource, so you must delete Console resources too. The Redpanda Operator must still be running when you delete these resources. It removes their finalizers as part of its cleanup. If you uninstall the Redpanda Operator first, any remaining resource keeps its finalizer forever and the CRD deletion step below hangs. 2. Make sure requests for those resources return no results. Do not continue until every command returns `No resources found`: ```bash kubectl get users,topics,schemas,redpandas,consoles --namespace ``` 3. Uninstall the Redpanda Operator Helm release: ```bash helm uninstall redpanda-controller --namespace ``` Helm does not uninstall CRDs by default when using `helm uninstall` to avoid accidentally deleting existing custom resources. 4. Remove the CRDs. 1. List all Redpanda CRDs installed by the operator: ```bash kubectl api-resources --api-group='cluster.redpanda.com' ``` This command displays all CRDs defined by the Redpanda Operator. For example: ```bash NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User ``` 2. Delete the CRDs: ```bash kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl delete ``` This command lists all CRDs with the `cluster.redpanda.com` domain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually. 5. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ### Helm If you deployed Redpanda with the Redpanda Helm chart, follow these steps to uninstall it: 1. Uninstall the Helm release: ```bash helm uninstall redpanda --namespace ``` 2. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ## [](#delete-the-cluster)Delete the cluster To delete your Kubernetes cluster: ```bash eksctl delete cluster --name ``` To remove the convenience alias created during the quickstart: ```bash unalias internal-rpk ``` ## [](#troubleshoot)Troubleshoot Before troubleshooting your cluster, make sure that you have all the [prerequisites](#prerequisites). ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ### [](#statefulset-never-rolls-out)StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). ## [](#next-steps)Next steps - [Try an example in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about Redpanda Console](https://docs.redpanda.com/streaming/current/manage/console/) - [Learn more about rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) > 💡 **TIP** > > When you’re ready to use a registered domain, make sure to remove your entries from the `/etc/hosts` file, and see [Configure External Access through a NodePort Service](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-nodeport/#use-the-default-redpanda-subdomains). ## [](#suggested-reading)Suggested reading - [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/) - [Configure TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) - [Configure SASL for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) - [Redpanda Console README](https://github.com/redpanda-data/console) on GitHub ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 38: Get Started with Redpanda in Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/get-started-dev.md --- # Get Started with Redpanda in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Get Started with Redpanda in Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/get-started-dev page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/get-started-dev.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/get-started-dev.adoc description: Find guides for setting up a three-broker Redpanda cluster in different Kubernetes platforms. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- For end-to-end guides on secure deployments of Redpanda on managed Kubernetes services, see: - [Azure Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/aks-guide/) (AKS) - [Elastic Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/eks-guide/) (EKS) - [Google Kubernetes Engine](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/gke-guide/) (GKE) - [Local (kind or minikube)](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/local-guide/) Or, if you’re ready to go into production, see the [production deployment workflow](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-workflow/). ## [](#suggested-reading)Suggested reading - [Getting started with Redpanda in Kubernetes](https://redpanda.com/blog/manage-clusters-k8s-streaming-data) ## Suggested labs - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 39: Deploy a Redpanda Cluster in Google Kubernetes Engine **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/gke-guide.md --- # Deploy a Redpanda Cluster in Google Kubernetes Engine > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy a Redpanda Cluster in Google Kubernetes Engine latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/gke-guide page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/gke-guide.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/gke-guide.adoc description: Deploy a secure Redpanda cluster and Redpanda Console in Google Kubernetes Engine (GKE). page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- Deploy a secure Redpanda cluster and Redpanda Console in Google Kubernetes Engine (GKE). Then, use `rpk` both as an internal client and an external client to interact with your Redpanda cluster from the command line. Your Redpanda cluster has the following security features: - SASL for authenticating users' connections. - TLS with self-signed certificates for secure communication between the cluster and clients. ## [](#prerequisites)Prerequisites - Complete the 'Before you begin' steps and the 'Launch Cloud Shell' steps of the [GKE quickstart](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#before-you-begin). Cloud Shell comes preinstalled with the Google Cloud CLI, the `kubectl` command-line tool, and the Helm package manager. - Ensure [`kubectl`](https://kubernetes.io/docs/tasks/tools/) is installed. Minimum required Kubernetes version: 1.27.0-0. ```bash kubectl version --client ``` - Ensure [Helm](https://helm.sh/docs/intro/install/) is installed. Minimum required Helm version: 3.10.0 ```bash helm version ``` ## [](#create-a-gke-cluster)Create a GKE cluster Your GKE cluster must have one worker node available for each Redpanda broker that you plan to deploy in your Redpanda cluster. You also need to run the worker nodes on a machine type that supports the [requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) for production deployments. In this step, you create a GKE cluster with three nodes on [`c2d-standard-8` machine types](https://cloud.google.com/compute/docs/compute-optimized-machines#c2d_series). Deploying three nodes allows your GKE cluster to support a Redpanda cluster with three brokers. The `c2d-standard-8` instance type comes with: - 2 cores per worker node, which is a requirement for production. - Local NVMe disks, which is recommended for best performance. Create a GKE cluster. Replace the `` placeholder with your own region. ```bash gcloud container clusters create \ --machine-type c2d-standard-8 \ --num-nodes=3 \ --local-nvme-ssd-block count=2 \ --region= ``` > ❗ **IMPORTANT** > > Do not enable [node auto-upgrades](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) (`--enable-autoupgrade`) on Google GKE clusters running Redpanda. > > Node auto-upgrades can trigger automatic reboots or node upgrades that disrupt Redpanda brokers, risking data loss or cluster instability. Redpanda requires manual control over node lifecycle events. > > For more details, see the [requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#node-updates) for deploying Redpanda in Kubernetes. To see all options that you can specify when creating a cluster, see the [Cloud SDK reference](https://cloud.google.com/sdk/gcloud/reference/container/clusters/create). Or, for help creating a GKE cluster, see the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster). ### [](#create-sc)Create a StorageClass for your local NVMe disks When you provisioned the Kubernetes cluster, you selected an instance type that comes with local NVMe disks. However, these disks are not automatically mounted or formatted upon creation. To use these local NVMe disks, you must mount and format them, and you must create the necessary PersistentVolumes (PVs). To automate this process, you can use a Container Storage Interface (CSI) driver. In this step, you install the recommended [local volume manager (LVM) CSI driver](https://github.com/metal-stack/csi-driver-lvm). Then, you create a StorageClass that references the LVM CSI driver and specifies the recommended XFS file system. 1. Install the LVM CSI driver: ```yaml helm repo add metal-stack https://helm.metal-stack.io helm repo update helm install csi-driver-lvm metal-stack/csi-driver-lvm \ --version 0.6.0 \ --namespace csi-driver-lvm \ --create-namespace \ --set lvm.devicePattern='/dev/nvme[0-9]n[0-9]' ``` The `lvm.devicePattern` property specifies the pattern that the CSI driver uses to identify available NVMe volumes on your worker nodes. > 📝 **NOTE** > > Version 0.6.0 is required to avoid volume-mounting issues caused by recent `mkfs.xfs` updates. Newer versions enable the `-i nrext64=1` option, triggering the following error on default GKE kernels: > > XFS (dm-0): Superblock has unknown incompatible features (0x20) enabled. 2. Create the StorageClass: `csi-driver-lvm-striped-xfs.yaml` ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-driver-lvm-striped-xfs provisioner: lvm.csi.metal-stack.io reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: "striped" csi.storage.k8s.io/fstype: xfs mkfsParams: "-i nrext64=0" ``` - `provisioner`: The LVM CSI driver responsible for provisioning the volume. - `reclaimPolicy`: The `Retain` policy ensures that the underlying volume is not deleted when the corresponding PVC is deleted. - `volumeBindingMode`: The `WaitForFirstConsumer` mode delays the binding and provisioning of a PersistentVolume until a Pod that uses the PVC is created. This mode is important for ensuring that the PV is created on the same node where the Pod will run because the PV will use the node’s local NVMe volumes. - `allowVolumeExpansion`: Allows the volume to be expanded after it has been provisioned. - `parameters.type`: Combines multiple physical volumes to create a single logical volume. In a striped setup, data is spread across the physical volumes in a way that distributes the I/O load evenly, improving performance by allowing parallel disk I/O operations. - `parameters.csi.storage.k8s.io/fstype`: Formats the volumes with the XFS file system. Redpanda Data recommends XFS for its enhanced performance with Redpanda workloads. - `parameters.mkfsParams`: Disables the nrext64 feature to ensure compatibility with older kernels. 3. Apply the StorageClass: ```bash kubectl apply -f csi-driver-lvm-striped-xfs.yaml ``` After applying this StorageClass, any PVC that references it will attempt to provision storage using the LVM CSI driver and the provided parameters. ### [](#configure-external-access)Configure external access Add inbound firewall rules to your instances so that external traffic can reach the following node ports on all Kubernetes worker nodes in the cluster: - 31644 - 31092 - 30082 - 30081 For help creating firewall rules, see the [Google VPC documentation](https://cloud.google.com/vpc/docs/using-firewalls). ## [](#deploy-redpanda-and-redpanda-console)Deploy Redpanda and Redpanda Console In this step, you deploy Redpanda with SASL authentication and self-signed TLS certificates. Redpanda Console is included as a subchart in the Redpanda Helm chart. ### Operator 1. Make sure that you have permission to install custom resource definitions (CRDs): ```bash kubectl auth can-i create CustomResourceDefinition --all-namespaces ``` You should see `yes` in the output. You need these cluster-level permissions to install [cert-manager](https://cert-manager.io/docs/) and Redpanda Operator CRDs in the next steps. 2. Install [cert-manager](https://cert-manager.io/docs/installation/helm/) using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` The Redpanda Helm chart uses cert-manager to enable TLS and manage TLS certificates by default. 3. Deploy the Redpanda Operator: 1. To deploy in cluster scope, use: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ (1) --set crds.enabled=true (2) ``` | 1 | This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment.This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces. | 2. To deploy in namespace scope (managing only resources within its deployment namespace), use: ```bash helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace="]' (1) ``` | 1 | This flag restricts the Redpanda Operator to manage resources only within the specified namespace. | | --- | --- | 4. Ensure that the Deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` deployment "redpanda-controller-operator" successfully rolled out 5. Install a [Redpanda custom resource](https://docs.redpanda.com/streaming/current/reference/k-crd/) in the same namespace as the Redpanda Operator: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: image: tag: v26.2.2 external: domain: customredpandadomain.local auth: sasl: enabled: true users: - name: superuser password: secretpassword storage: persistentVolume: enabled: true storageClass: csi-driver-lvm-striped-xfs ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker will advertise to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. 6. Wait for the Redpanda Operator to deploy Redpanda using the Helm chart: ```bash kubectl get redpanda --namespace --watch ``` NAME READY STATUS redpanda True Redpanda reconciliation succeeded This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing: ```bash kubectl get pod --namespace ``` If it’s taking too long, see [Troubleshoot](#troubleshoot). ### Helm 1. Install cert-manager using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default. 2. Install Redpanda with SASL enabled: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ --namespace --create-namespace \ --set image.tag=v26.2.2 \ --set auth.sasl.enabled=true \ --set "auth.sasl.users[0].name=superuser" \ --set "auth.sasl.users[0].password=secretpassword" \ --set external.domain=customredpandadomain.local \ --set "storage.persistentVolume.storageClass=csi-driver-lvm-striped-xfs" \ --wait \ --timeout 1h ``` - `image.tag`: Deploys the latest version of Redpanda. - `external.domain`: The custom domain that each broker advertises to clients externally. This domain is added to the internal and external TLS certificates so that you can connect to the cluster using this domain. - `auth.sasl.name`: Creates a superuser called `superuser` that can grant permissions to new users in your cluster using access control lists (ACLs). - `storage.persistentVolume.storageClass`: Points each PVC associated with the Redpanda brokers to the `csi-driver-lvm-striped-xfs` StorageClass. This StorageClass allows the LVM CSI driver to provision the appropriate local PersistentVolumes backed by NVMe disks for each Redpanda broker. The installation displays some tips for getting started. If the installation is taking a long time, see [Troubleshoot](#troubleshoot). ## [](#verify-the-deployment)Verify the deployment When the Redpanda Helm chart is deployed, you should have: - Three Redpanda brokers. Each Redpanda broker runs inside a separate Pod and is scheduled on a separate worker node. - One PVC bound to a PV for each Redpanda broker. These PVs are what the Redpanda brokers use to store the Redpanda data directory with all your topics and metadata. 1. Verify that each Redpanda broker is scheduled on only one Kubernetes node: ```bash kubectl get pod --namespace \ -o=custom-columns=NODE:.spec.nodeName,POD_NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset ``` Example output: NODE POD\_NAME example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2 2. Verify that each Redpanda broker has a bound PVC: ```bash kubectl get persistentvolumeclaim \ --namespace \ -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,STORAGECLASS:.spec.storageClassName ``` Example output: NAME STATUS STORAGECLASS datadir-redpanda-0 Bound csi-driver-lvm-striped-xfs datadir-redpanda-1 Bound csi-driver-lvm-striped-xfs datadir-redpanda-2 Bound csi-driver-lvm-striped-xfs ## [](#create-a-user)Create a user In this step, you use `rpk` to create a new user. Then, you authenticate to Redpanda with the superuser to grant permissions to the new user. You’ll authenticate to Redpanda with this new user to create a topic in the next steps. > 💡 **TIP** > > As a security best practice, you should use the superuser only to grant permissions to new users through ACLs. Never delete the superuser. You need the superuser to grant permissions to new users. 1. Create a new user called `redpanda-twitch-account` with the password `changethispassword`: ```bash kubectl --namespace exec -ti redpanda-0 -c redpanda -- \ rpk security user create redpanda-twitch-account \ -p changethispassword ``` Example output: Created user "redpanda-twitch-account". 2. Use the superuser to grant the `redpanda-twitch-account` user permission to execute all operations only for a topic called `twitch-chat`. ```bash kubectl exec --namespace -c redpanda redpanda-0 -- \ rpk security acl create --allow-principal User:redpanda-twitch-account \ --operation all \ --topic twitch-chat \ -X user=superuser -X pass=secretpassword -X sasl.mechanism=SCRAM-SHA-512 ``` Example output: PRINCIPAL RESOURCE-TYPE RESOURCE-NAME OPERATION PERMISSION User:redpanda TOPIC twitch-chat ALL ALLOW ## [](#start-streaming)Start streaming In this step, you authenticate to Redpanda with the `redpanda-twitch-account` user to create a topic called `twitch-chat`. This topic is the only one that the `redpanda-twitch-account` user has permission to access. Then, you produce messages to the topic, and consume messages from it. 1. Create an alias to simplify the `rpk` commands: ```bash alias internal-rpk="kubectl --namespace exec -i -t redpanda-0 -c redpanda -- rpk -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256" ``` 2. Create a topic called `twitch-chat`: ### Operator 1. Create a Secret in which to store your user’s password: ```bash kubectl create secret generic redpanda-secret --from-literal=password='changethispassword' --namespace ``` 2. Create a [Topic resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/): `topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Topic metadata: name: twitch-chat spec: kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" sasl: username: redpanda-twitch-account mechanism: SCRAM-SHA-256 passwordSecretRef: name: redpanda-secret key: password ``` 3. Apply the Topic resource in the same namespace as your Redpanda cluster: ```bash kubectl apply -f topic.yaml --namespace ``` 4. Check the logs of the Redpanda Operator to confirm that the topic was created: ```bash kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace ``` You should see that the Redpanda Operator reconciled the Topic resource. For example: Example output ```json { "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } } ``` ### Helm ```bash internal-rpk topic create twitch-chat ``` Example output: TOPIC STATUS twitch-chat OK 3. Describe the topic: ```bash internal-rpk topic describe twitch-chat ``` Expected output: ```none SUMMARY ======= NAME twitch-chat PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG partition_count 1 DYNAMIC_TOPIC_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG replication_factor 1 DYNAMIC_TOPIC_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG ``` 4. Produce a message to the topic: ```bash internal-rpk topic produce twitch-chat ``` 5. Type a message, then press Enter: Pandas are fabulous! Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 6. Press Ctrl+C to finish producing messages to the topic. 7. Consume one message from the topic: ```bash internal-rpk topic consume twitch-chat --num 1 ``` Expected output: ```none { "topic": "twitch-chat", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ## [](#explore-your-topic-in-redpanda-console)Explore your topic in Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. In this step, you use port-forwarding to access Redpanda Console on your local network. > 💡 **TIP** > > Because you’re using the Community Edition of Redpanda Console, you should not expose Redpanda Console outside your local network. The Community Edition of Redpanda Console does not provide authentication, and it connects to the Redpanda cluster as superuser. To use the Enterprise Edition, you need a license key. See [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). 1. Expose Redpanda Console to your localhost: ```bash kubectl --namespace port-forward svc/redpanda-console 8080:8080 ``` The `kubectl port-forward` command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window. 2. Open Redpanda Console on [http://localhost:8080](http://localhost:8080). All your Redpanda brokers are listed along with their IP addresses and IDs. 3. Go to **Topics** > **twitch-chat**. The message that you produced to the topic is displayed along with some other details about the topic. 4. Press Ctrl+C in the command-line to stop the port-forwarding process. ## [](#configure-external-access-to-redpanda)Configure external access to Redpanda If you want to connect to the Redpanda cluster with external clients, Redpanda brokers must advertise an externally accessible address that external clients can connect to. External clients are common in Internet of Things (IoT) environments, or if you use external services that do not implement VPC peering in your network. When you created the cluster, you set the `external.domain` configuration to `customredpandadomain.local`, which means that your Redpanda brokers are advertising the following addresses: - `redpanda-0.customredpandadomain.local` - `redpanda-1.customredpandadomain.local` - `redpanda-2.customredpandadomain.local` To access your Redpanda brokers externally, you can map your worker nodes' IP addresses to these domains. > ⚠️ **CAUTION** > > IP addresses can change. If the IP addresses of your worker nodes change, you must update your `/etc/hosts` file with the new mappings. > > In a production environment, it’s a best practice to use ExternalDNS to manage DNS records for your brokers. See [Use ExternalDNS for external access](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#externaldns). 1. Add mappings in your `/etc/hosts` file between your worker nodes' IP addresses and their custom domain names: ```bash sudo true && kubectl --namespace get endpoints,node -A -o go-template='{{ range $_ := .items }}{{ if and (eq .kind "Endpoints") (eq .metadata.name "redpanda-external") }}{{ range $_ := (index .subsets 0).addresses }}{{ $nodeName := .nodeName }}{{ $podName := .targetRef.name }}{{ range $node := $.items }}{{ if and (eq .kind "Node") (eq .metadata.name $nodeName) }}{{ range $_ := .status.addresses }}{{ if eq .type "ExternalIP" }}{{ .address }} {{ $podName }}.customredpandadomain.local{{ "\n" }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}' | envsubst | sudo tee -a /etc/hosts ``` `/etc/hosts` 203.0.113.3 redpanda-0.customredpandadomain.local 203.0.113.5 redpanda-1.customredpandadomain.local 203.0.113.7 redpanda-2.customredpandadomain.local 2. Save the root certificate authority (CA) to your local file system outside Kubernetes: ```bash kubectl --namespace get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 3. Install `rpk` on your local machine, not on a Pod: ### Linux > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` ### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` 4. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` Replace `` with the name that you want to give this `rpk` profile. 5. Test the connection: ```bash rpk cluster info -X user=redpanda-twitch-account -X pass=changethispassword -X sasl.mechanism=SCRAM-SHA-256 ``` ## [](#explore-the-default-kubernetes-components)Explore the default Kubernetes components By default, the Redpanda Helm chart deploys the following Kubernetes components: - [A StatefulSet](#statefulset) with three Pods. - [One PersistentVolumeClaim](#persistentvolumeclaim) for each Pod, each with a capacity of 20Gi. - [A headless ClusterIP Service and a NodePort Service](#service) for each Kubernetes node that runs a Redpanda broker. - [Self-Signed TLS Certificates](#tls-certificates). ### [](#statefulset)StatefulSet Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running. ```bash kubectl get statefulset --namespace ``` Example output: NAME READY AGE redpanda 3/3 3m11s StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as `redpanda-0`. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced. ```bash kubectl get pod --namespace ``` Expected output: ```none NAME READY STATUS RESTARTS AGE redpanda-0 1/1 Running 0 6m9s redpanda-1 1/1 Running 0 6m9s redpanda-2 1/1 Running 0 6m9s redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m redpanda-configuration-smqv7 0/1 Completed 0 6m9s ``` > 📝 **NOTE** > > The `redpanda-configuration` job updates the Redpanda runtime configuration. ### [](#persistentvolumeclaim)PersistentVolumeClaim Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using. ```bash kubectl get persistentvolumeclaims --namespace ``` Expected output: ```none NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s ``` ### [](#service)Service The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services: - Internal using the [Headless ClusterIP](#headless-clusterip-service) - External using the [NodePort](#nodeport-service) ```bash kubectl get service --namespace ``` Expected output: ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda ClusterIP None 5m37s redpanda-console ClusterIP 10.0.251.204 8080 5m redpanda-external NodePort 10.96.137.220 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s ``` #### [](#headless-clusterip-service)Headless ClusterIP Service The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other. An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other. ```bash kubectl --namespace exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml ``` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: redpanda-0.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-1.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-2.redpanda..svc.cluster.local. port: 33145 ``` #### [](#nodeport-service)NodePort Service External access is made available by a NodePort service that opens the following ports by default: | Listener | Node Port | Container Port | | --- | --- | --- | | Schema Registry | 30081 | 8081 | | HTTP Proxy | 30082 | 8083 | | Kafka API | 31092 | 9094 | | Admin API | 31644 | 9644 | To learn more, see [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/). ### [](#tls-certificates)TLS Certificates By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses [cert-manager](https://cert-manager.io/docs/) to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections. Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True - `redpanda-default-cert`: Self-signed certificate for internal communications. - `redpanda-default-root-certificate`: Root certificate authority for the internal certificate. - `redpanda-external-cert`: Self-signed certificate for external communications. - `redpanda-external-root-certificate`: Root certificate authority for the external certificate. By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/). > 📝 **NOTE** > > The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. ## [](#uninstall-redpanda)Uninstall Redpanda When you finish testing Redpanda, you can uninstall it from your Kubernetes cluster. The steps depend on how you installed Redpanda: using the Redpanda Operator or the Redpanda Helm chart. ### Operator Follow the steps in **exact order** to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection. 1. Delete all Redpanda-related custom resources: ```bash kubectl delete users --namespace --all kubectl delete topics --namespace --all kubectl delete schemas --namespace --all kubectl delete redpanda --namespace --all kubectl delete consoles --namespace --all ``` The Redpanda Operator creates a Console resource for each Redpanda resource, so you must delete Console resources too. The Redpanda Operator must still be running when you delete these resources. It removes their finalizers as part of its cleanup. If you uninstall the Redpanda Operator first, any remaining resource keeps its finalizer forever and the CRD deletion step below hangs. 2. Make sure requests for those resources return no results. Do not continue until every command returns `No resources found`: ```bash kubectl get users,topics,schemas,redpandas,consoles --namespace ``` 3. Uninstall the Redpanda Operator Helm release: ```bash helm uninstall redpanda-controller --namespace ``` Helm does not uninstall CRDs by default when using `helm uninstall` to avoid accidentally deleting existing custom resources. 4. Remove the CRDs. 1. List all Redpanda CRDs installed by the operator: ```bash kubectl api-resources --api-group='cluster.redpanda.com' ``` This command displays all CRDs defined by the Redpanda Operator. For example: ```bash NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User ``` 2. Delete the CRDs: ```bash kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl delete ``` This command lists all CRDs with the `cluster.redpanda.com` domain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually. 5. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ### Helm If you deployed Redpanda with the Redpanda Helm chart, follow these steps to uninstall it: 1. Uninstall the Helm release: ```bash helm uninstall redpanda --namespace ``` 2. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ## [](#delete-the-cluster)Delete the cluster To delete your Kubernetes cluster: ```bash gcloud container clusters delete \ --region= ``` ## [](#troubleshoot)Troubleshoot Before troubleshooting your cluster, make sure that you have all the [prerequisites](#prerequisites). ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ### [](#statefulset-never-rolls-out)StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). ## [](#next-steps)Next steps - [Try an example in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about Redpanda Console](https://docs.redpanda.com/streaming/current/manage/console/) - [Learn more about rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) > 💡 **TIP** > > When you’re ready to use a registered domain, make sure to remove your entries from the `/etc/hosts` file, and see [Configure External Access through a NodePort Service](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-nodeport/#use-the-default-redpanda-subdomains). ## [](#suggested-reading)Suggested reading - [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/) - [Configure TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) - [Configure SASL for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) - [Redpanda Console README](https://github.com/redpanda-data/console) on GitHub ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 40: Choose Your Deployment **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-choose-deployment.md --- # Choose Your Deployment > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Choose Your Deployment latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-choose-deployment page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-choose-deployment.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc description: Compare Redpanda deployment topologies for Kubernetes and choose the one that matches the failures you must survive and your recovery objectives. page-topic-type: concept personas: platform_operator, platform_engineer learning-objective-1: Compare Redpanda deployment topologies by failure domain, recovery objectives, and operational cost learning-objective-2: Choose the deployment guide that matches your availability requirements page-git-created-date: "2026-07-22" page-git-modified-date: "2026-08-13" support-status: supported --- Before you deploy Redpanda for production, choose the cluster topology that matches the failures you must survive and your recovery objectives: how much data you can afford to lose (recovery point objective, or RPO) and how quickly you must recover (recovery time objective, or RTO). Each topology mitigates a different class of failure, from the loss of a single broker to the loss of an entire region, and makes different trade-offs in latency, license requirements, and operational complexity. This page compares the available topologies along those dimensions and routes you to the right deployment guide. After reading this page, you will be able to: - Compare Redpanda deployment topologies by failure domain, recovery objectives, and operational cost - Choose the deployment guide that matches your availability requirements ## [](#what-youre-protecting-against)What you’re protecting against Use this table to identify the failures you must plan for and the mechanisms that mitigate them: | Failure | Impact | Mitigation strategy | | --- | --- | --- | | Broker failure | Loss of function for an individual broker or for any virtual machine (VM) that hosts the broker | Multi-broker deployment | | Rack or switch failure | Loss of brokers/VMs hosted within that rack, or loss of connectivity to them | Multi-broker deployment spread across multiple racks or network failure domains | | Data center failure | Loss of brokers/VMs hosted within that data center, or loss of connectivity to them | Multi-AZ or replicated deployment | | Region failure | Loss of brokers/VMs hosted within that region, or loss of connectivity to them | Geo-stretch (latency dependent), such as a Stretch Cluster on Kubernetes, or replicated deployment | | Global, systemic outage (DNS failures, routing failures) | Complete outage for all systems and services impacting customers and staff | Offline backups, replicas in 3rd-party domains | | Data loss or corruption (accidental or malicious) | Corrupt or unavailable data that also affects synchronous replicas | Offline backups | ## [](#decision-dimensions)Decision dimensions Weigh the options against these dimensions: - **Failure domain**: The largest unit whose loss you must survive: a broker, a rack, an availability zone (AZ), a region, or an entire Kubernetes cluster. - **RPO**: Synchronous Raft replication within one cluster means zero data loss. Asynchronous replication between clusters means near-real-time copies with some replication lag. - **Failover**: A single cluster fails over automatically through Raft leader election. Independent clusters require a runbook-driven failover, including redirecting clients. - **Latency**: Synchronous replication across distant locations adds produce latency and requires tuning. Asynchronous replication has no distance ceiling. - **Client semantics**: A single cluster keeps one set of offsets. Shadowing preserves offsets on an independent cluster, so consumers can resume after failover. - **License and complexity**: Some topologies require an Enterprise license and more operational investment. ## [](#compare-topologies)Compare topologies graph TB subgraph single\["Single cluster, multi-AZ: zonal resilience"\] direction LR sa\["Broker AZ 1"\] <--> sb\["Broker AZ 2"\] <--> sc\["Broker AZ 3"\] end subgraph stretch\["Stretch Cluster: one cluster across Kubernetes clusters"\] direction LR subgraph k1\["Kubernetes cluster 1"\] t1\["2 brokers"\] end subgraph k2\["Kubernetes cluster 2"\] t2\["2 brokers"\] end subgraph k3\["Kubernetes cluster 3"\] t3\["1 broker"\] end t1 <-->|"synchronous Raft"| t2 t2 <--> t3 end subgraph shadow\["Independent clusters with Shadowing: disaster recovery"\] direction LR subgraph src\["Source cluster"\] p1\["Brokers"\] end subgraph dst\["Shadow cluster"\] p2\["Brokers"\] end p1 -->|"asynchronous shadow link"| p2 end single ~~~ stretch ~~~ shadow Figure 1. The three topologies at a glance A single cluster replicates synchronously between brokers, whether those brokers share one Kubernetes cluster (multi-AZ) or span several (Stretch Cluster). Shadowing instead copies data asynchronously between two independent clusters. | | Single cluster, multi-AZ | Stretch Cluster (one cluster across Kubernetes clusters) | Independent clusters with asynchronous replication (Shadowing) | | --- | --- | --- | --- | | Survives | Loss of one AZ (with three or more AZs) | Loss of one region, data center, or Kubernetes cluster (with three or more) | Loss of an entire region or cluster; also isolates the replica from many data-corruption events | | Replication and RPO | Synchronous (Raft), zero data loss | Synchronous (Raft), zero data loss, one set of offsets | Asynchronous, near real time, and offset-preserving, with an RPO greater than zero | | Failover | Automatic | Automatic | Runbook-driven: fail over shadow topics, then redirect clients | | Latency requirement | Low inter-AZ latency. Produce round trips stay in the single-digit milliseconds. | Low inter-cluster latency (under 50 ms recommended, which in practice means same-continent locations). Produce pays a cross-location round trip of tens of milliseconds. Requires WAN tuning. | None. Produce stays local at any distance because replication is out of band. | | Relative cost | $$: one cluster, intra-region traffic only | $$$: cross-location replication traffic on every write | $$: two clusters, but the shadow can be provisioned for recovery rather than full production load | | Tiered Storage | Supported | Not supported (including Iceberg topics and Cloud Topics) | Supported | | License | Community | Enterprise | Enterprise | | Deployment guide | Deploy Redpanda for Production in Kubernetes with rack awareness | Deploy a Stretch Cluster on Kubernetes | Shadowing Overview | For read-only offloading rather than disaster recovery, see [Remote Read Replicas](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-remote-read-replicas/). For the lowest-cost recovery option, [Whole Cluster Restore](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-whole-cluster-restore/) rebuilds a cluster from Tiered Storage in the same region, accepting minutes of data loss (RPO) and a high recovery time (RTO) in exchange for the lower cost of object storage. ## [](#quick-picks)Quick picks - **You need zonal resilience with the lowest latency and cost**: Deploy a single cluster across three or more AZs with rack awareness. - **You need zero data loss across regions and you run Kubernetes**: Deploy a [Stretch Cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). - **You need disaster recovery across any distance, or an isolated recovery copy**: Keep clusters independent and replicate with [Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/). - **You need to serve remote consumers without a second writable cluster**: Use [Remote Read Replicas](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-remote-read-replicas/). ## [](#stretch-clusters-and-shadowing)Stretch Clusters and Shadowing Stretch Clusters and Shadowing solve different problems and can complement each other. A Stretch Cluster is one logical cluster with synchronous replication and a single set of offsets; Shadowing asynchronously replicates data and metadata between two independent clusters for disaster recovery, protecting against events that affect a whole cluster, including some it cannot survive synchronously. If you are deciding between the two, compare them dimension by dimension: | | Stretch Cluster | Shadowing | | --- | --- | --- | | Replication | Synchronous: the Raft quorum spans locations. | Asynchronous: data is copied after the write is acknowledged. | | RPO | Zero: acknowledged writes survive the loss of a location. | Greater than zero: a replication-lag window always exists. | | RTO | Near zero: leaders re-elect automatically, with no cutover. | Minutes: you promote the shadow cluster and repoint clients. | | Client failover | No re-bootstrap: clients keep one broker list and reconnect on their own. | Re-bootstrap required: you repoint clients to the promoted cluster. Offsets are preserved, so consumers resume where they left off. | | Topology | One logical cluster with one set of offsets. | Two independent clusters: source and shadow. | | Produce latency | Cross-location round trip on every acknowledged write. | Local: replication happens out of band. | | Best for | Workloads that cannot lose an acknowledged write. | Disaster recovery where seconds of data loss is acceptable, or any deployment that spans distances too large for synchronous replication. | A Stretch Cluster can act as the shadow (target) cluster of a shadow link. Using a Stretch Cluster as the source cluster of a shadow link is not supported. ## [](#next-steps)Next steps - [High Availability in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-high-availability/) for the concepts behind these topologies, including replica placement and failure modes. - [Deploy Redpanda](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/) or [Deploy a Stretch Cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/) to get started. - [Production Readiness Checklist](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness/) to validate your deployment, whichever topology you choose. ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 41: Redpanda in Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-deployment-overview.md --- # Redpanda in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda in Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-deployment-overview page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-deployment-overview.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-deployment-overview.adoc description: Learn about Redpanda in Kubernetes and the tools that are available. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-12-04" support-status: supported --- Kubernetes is a container orchestration tool that helps you manage Redpanda deployments using declarative configuration files called _manifests_. Kubernetes provides a standardized way of achieving the following: - High availability - Disaster recovery - Scalability ## [](#deployment-tools)Deployment tools To deploy Redpanda in Kubernetes, you can choose between Helm for its simplicity or the Redpanda Operator for advanced lifecycle management. While Helm provides an easy way to install and upgrade Redpanda, it has limitations in managing complex, stateful workloads at scale. The Redpanda Operator is the recommended option for production deployments as it enables better upgrade management, dynamic configuration, and improved lifecycle automation. | Feature | Redpanda Operator | Helm | Description | | --- | --- | --- | --- | | Managed upgrade and rollback | ✅ | ⚠️ | Helm provides basic upgrades with rollback capabilities but requires manual intervention and increases operational risk. The Redpanda Operator automates safe, rolling upgrades with reconciliation, significantly reducing risk. | | Dynamic configuration | ✅ | ❌ | Helm configurations must be updated manually through Helm values and redeployment. The Redpanda Operator dynamically applies real-time configuration changes through custom resources (CRDs). | | Advanced health checks and metrics | ✅ | ⚠️ | Helm relies on standard Kubernetes-level health metrics, while the Redpanda Operator includes advanced, application-specific metrics and health checks. | | Lifecycle automation | ✅ | ⚠️ | Helm has limited automation and depends on manual management for scaling, failover, and cleanup. The Redpanda Operator automates scaling, failover, resource reconciliation, and cleanup tasks. | | Multi-tenancy management | ✅ | ⚠️ | Helm requires separate releases to manage multiple clusters, making management more complex. The Redpanda Operator simplifies multi-tenancy by managing clusters across different namespaces from a single operator instance. | > 💡 **TIP** > > If you are already using the Redpanda Helm chart and want to migrate to the latest Redpanda Operator for better lifecycle management, see [Migrate from the Redpanda Helm chart](https://docs.redpanda.com/streaming/current/migrate/kubernetes/helm-to-operator/). ### [](#helm-and-redpanda-operator)Redpanda Operator The Redpanda Operator is designed for production-grade Redpanda deployments, offering enhanced lifecycle management, automation, and GitOps compatibility. The Redpanda Operator defaults to cluster scope, enabling it to manage multiple Redpanda clusters across different namespaces from a single operator instance. The Redpanda Operator directly reconciles Redpanda resources, performing tasks such as installations, updates, and cleanup. ### [](#helm)Helm [Helm](https://helm.sh/docs) is a package manager for Kubernetes that simplifies defining, installing, and upgrading Kubernetes applications. Helm uses charts, a collection of files describing Kubernetes resources, to deploy applications in a Kubernetes cluster. The Redpanda Helm chart provides all the manifest files required to deploy Redpanda in Kubernetes, including: - A StatefulSet to manage Redpanda brokers - A Headless ClusterIP Service for internal communication with the Redpanda cluster - A NodePort Service for external communication with the Redpanda cluster ## [](#kubernetes-deployment-environments)Kubernetes deployment environments You can run Redpanda on managed Kubernetes services as well as in bare-metal environments. Managed Kubernetes services offer simpler deployment and maintenance, while bare-metal environments provide complete control and cost efficiencies. ### [](#managed-kubernetes)Managed Kubernetes Managed Kubernetes services, such as Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS), handle core components of a Kubernetes cluster, offering benefits such as: - **Ease of deployment**: Pre-configured instances join your Kubernetes cluster automatically. - **Control plane maintenance**: The provider maintains the control plane, ensuring security and reliability. - **Health monitoring and repairs**: The provider monitors master nodes and repairs them as needed. You remain responsible for deploying and maintaining Redpanda instances on worker nodes. > ❗ **IMPORTANT** > > Deploy Kubernetes clusters with **unmanaged (manual) node updates**. Managed (automatic) updates during cluster deployment can lead to service downtime, data loss, or quorum instability. Transitioning from managed updates to unmanaged updates after deployment may require downtime. To avoid these disruptions, plan for unmanaged node updates from the start. See [Kubernetes Cluster Requirements and Recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#node-updates). ### [](#bare-metal-kubernetes-environments)Bare-metal Kubernetes environments Bare-metal Kubernetes environments give you complete control over both the control plane and the worker nodes, which can be advantageous when you want the following: - **Complete control**: Bare-metal Kubernetes offers control over every aspect of deployment, suited for highly customized environments. - **Custom configurations**: You have granular control to fine-tune the Kubernetes setup. - **Cost efficiency**: Owning and operating your hardware may be more economical over time. ## [](#documentation-conventions)Documentation conventions This documentation follows conventions to help users easily identify Kubernetes resource types and Helm values: - **Resource names**: Kubernetes resource names, such as Service or PersistentVolume, are capitalized and in Pascal case to match the manifest files. - **Helm values**: Helm values, like `storage.persistentVolume.enabled`, are displayed in monospace font. ## [](#next-steps)Next steps - Get started - [Local Deployment Guide](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/local-guide/) (kind and minikube) - [Azure Kubernetes Service Guide](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/aks-guide/) (AKS) - [Elastic Kubernetes Service Guide](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/eks-guide/) (EKS) - [Google Kubernetes Engine Guide](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/gke-guide/) (GKE) - [Kubernetes Cluster Requirements and Recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) - [Production deployment workflow](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-workflow/) ## [](#suggested-reading)Suggested reading - [Kubernetes operator documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) - [Helm documentation](https://helm.sh/docs/intro/using_helm/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) ## Suggested labs - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 42: Deploy Redpanda for Production in Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment.md --- # Deploy Redpanda for Production in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Redpanda for Production in Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-production-deployment page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-production-deployment.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-production-deployment.adoc description: Deploy a Redpanda cluster in Kubernetes. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-27" support-status: supported --- This topic describes how to configure and deploy one or more Redpanda clusters and Redpanda Console in Kubernetes. Each Redpanda cluster runs within a single Kubernetes cluster. For multi-region topologies and other deployment shapes, see [Choose Your Deployment](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-choose-deployment/). > 💡 **TIP** > > To try Redpanda on a local Kubernetes cluster for development or testing, see the [guide for kind or minikube](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/local-guide/) instead. ## [](#prerequisites)Prerequisites Make sure that your Kubernetes cluster meets the [requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/). You must already have a ConfigMap that stores your `io-config.yaml` file. See [Generate optimal I/O configuration settings](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/#io). ## [](#deploy-a-redpanda-cluster)Deploy a Redpanda cluster To deploy Redpanda and Redpanda Console, you can use the following tools: - **Redpanda Operator**: The Redpanda Operator extends Kubernetes with custom resource definitions (CRDs), allowing you to define Redpanda clusters as native Kubernetes resources. The resource that the Redpanda Operator uses to represent a Redpanda cluster is the Redpanda resource. The Redpanda Operator can be deployed in either cluster scope (managing resources across all namespaces) or namespace scope (managing resources within a single namespace). - **Helm**: [Helm](https://helm.sh/docs) is a package manager for Kubernetes, which simplifies the process of defining, installing, and upgrading Kubernetes applications. Helm uses charts, a collection of files that describe a related set of Kubernetes resources, to deploy applications in a Kubernetes cluster. > 💡 **TIP** > > For more details about the differences between these two methods, see [Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-deployment-overview/). ### Operator The Redpanda Operator can be deployed in two different scopes: - **Cluster scope** (recommended): The Redpanda Operator manages Redpanda resources across all namespaces in your Kubernetes cluster. This provides centralized management and is ideal for production environments. - **Namespace scope**: The Redpanda Operator manages Redpanda resources only within a specific namespace. This provides better isolation and is suitable when you need strict namespace boundaries. > ⚠️ **WARNING** > > Do not run multiple Redpanda Operators in different scopes (cluster and namespace scope) in the same cluster as this can cause resource conflicts. 1. Make sure that you have permission to install custom resource definitions (CRDs): ```bash kubectl auth can-i create CustomResourceDefinition --all-namespaces ``` You should see `yes` in the output. You need these cluster-level permissions to install [cert-manager](https://cert-manager.io/docs/) and Redpanda Operator CRDs in the next steps. 2. Install [cert-manager](https://cert-manager.io/docs/installation/helm/): ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` The Redpanda Helm chart enables TLS by default and uses cert-manager to manage TLS certificates. 3. Deploy the Redpanda Operator in your chosen scope: 1. To deploy in cluster scope, use: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ (1) --set crds.enabled=true (2) ``` | 1 | This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment.This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces. | 2. To deploy in namespace scope (managing only resources within its deployment namespace), use: ```bash helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace="]' (1) ``` | 1 | This flag restricts the Redpanda Operator to manage resources only within the specified namespace. | | --- | --- | 4. Ensure that the Deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` deployment "redpanda-controller-operator" successfully rolled out 5. Install a [Redpanda custom resource](https://docs.redpanda.com/streaming/current/reference/k-crd/) to deploy a Redpanda cluster and Redpanda Console. `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda namespace: spec: clusterSpec: #enterprise: #licenseSecretRef: #name: #key: image: tag: v26.2.2 statefulset: extraVolumes: |- - name: redpanda-io-config configMap: name: redpanda-io-config extraVolumeMounts: |- - name: redpanda-io-config mountPath: /etc/redpanda-io-config additionalRedpandaCmdFlags: - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml" ``` - `metadata.name`: Name to assign the Redpanda cluster. - `metadata.namespace`: For cluster-scoped deployment, specify any namespace. For namespace-scoped deployment, must be the same namespace where the Redpanda Operator is deployed. - [`spec.clusterSpec`](https://docs.redpanda.com/streaming/current/reference/k-crd/#k8s-api-github-com-redpanda-data-redpanda-operator-api-redpanda-v1alpha2-redpandaclusterspec): This is where you can override default values in the Redpanda Helm chart. Here, you mount the [I/O configuration file](#prerequisites) to the Pods that run Redpanda. For other configuration details, see [Production considerations](#config). - `spec.clusterSpec.enterprise`: If you want to use enterprise features in Redpanda, uncomment this section and add the details of a Secret that stores your Enterprise Edition license key. For details, see [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). - `spec.clusterSpec.image.tag`: Deploys the latest version of Redpanda. - `spec.clusterSpec.statefulset`: Here, you mount the [I/O configuration file](#prerequisites) to the Pods that run Redpanda. For other configuration details, see [Production considerations](#config). 6. Apply the Redpanda resource: ```bash kubectl apply -f redpanda-cluster.yaml ``` 7. Wait for the Redpanda Operator to deploy Redpanda using the Helm chart: ```bash kubectl get redpanda --namespace --watch ``` NAME READY STATUS redpanda True Redpanda reconciliation succeeded This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing: ```bash kubectl get pod --namespace ``` If it’s taking too long, see [Troubleshooting](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). 8. Verify that each Redpanda broker is scheduled on only one Kubernetes node: ```bash kubectl get pod --namespace \ -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset ``` Expected output: example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2 ### Helm 1. Install cert-manager using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace ``` The Redpanda Helm chart enables TLS by default and uses cert-manager to manage TLS certificates. 2. Override the default values to mount your [I/O configuration file](#prerequisites) onto each Pod that runs Redpanda. `redpanda-values.yaml` ```yaml image: tag: {latest-redpanda-tag} statefulset: extraVolumes: |- - name: redpanda-io-config configMap: name: redpanda-io-config extraVolumeMounts: |- - name: redpanda-io-config mountPath: /etc/redpanda-io-config additionalRedpandaCmdFlags: - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml" ``` Redpanda reads from this file at startup to optimize itself for the given I/O parameters. If you want to use enterprise features in Redpanda, add the details of a Secret that stores your Enterprise Edition license key. `redpanda-values.yaml` ```yaml enterprise: licenseSecretRef: name: key: ``` For details, see [Add an Enterprise Edition License to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/kubernetes/). 3. Install the Redpanda Helm chart to deploy a Redpanda cluster and Redpanda Console. ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ (1) --namespace \ (2) --create-namespace \ --values redpanda-values.yaml ``` | 1 | This flag specifies the exact version of the Redpanda Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | Each deployment of the Redpanda Helm chart requires a separate namespace. Ensure you choose a unique namespace for each deployment. | 4. Wait for the Redpanda cluster to be ready: ```bash kubectl --namespace rollout status statefulset redpanda --watch ``` When the Redpanda cluster is ready, the output should look similar to the following: statefulset rolling update complete 3 pods at revision redpanda-8654f645b4... 5. Verify that each Redpanda broker is scheduled on only one Kubernetes node: ```bash kubectl get pod --namespace \ -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name -l \ app.kubernetes.io/component=redpanda-statefulset ``` Expected output: example-worker3 redpanda-0 example-worker2 redpanda-1 example-worker redpanda-2 ## [](#deploy-multiple-redpanda-clusters)Deploy multiple Redpanda clusters You can deploy multiple Redpanda clusters in the same Kubernetes cluster. This is useful for creating separate environments (such as production, staging, and development) or for organizing clusters by application or team. ### Operator When using the Redpanda Operator, you can deploy multiple Redpanda clusters by creating separate Redpanda custom resources. **Requirements:** - Use a cluster-scoped Redpanda Operator deployment (recommended) or separate namespace-scoped operators in different namespaces - Each cluster must be deployed in a unique namespace - Configure unique external port numbers for each cluster to avoid conflicts 1. Create a second Redpanda cluster in a different namespace: `redpanda-cluster-two.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda-staging namespace: redpanda-staging spec: clusterSpec: image: tag: v26.2.2 listeners: kafka: external: default: advertisedPorts: [31093] (1) admin: external: default: advertisedPorts: [31645] (1) http: external: default: advertisedPorts: [30083] (1) rpc: port: 33146 (1) schemaRegistry: external: default: advertisedPorts: [30084] (1) statefulset: extraVolumes: |- - name: redpanda-io-config configMap: name: redpanda-io-config extraVolumeMounts: |- - name: redpanda-io-config mountPath: /etc/redpanda-io-config additionalRedpandaCmdFlags: - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml" ``` | 1 | Configure unique port numbers for each cluster to avoid conflicts. Ensure these ports don’t conflict with your first cluster’s configuration. | | --- | --- | 2. Apply the second Redpanda resource: ```bash kubectl apply -f redpanda-cluster-two.yaml ``` 3. Wait for the second cluster to be ready: ```bash kubectl get redpanda --namespace redpanda-staging --watch ``` ### Helm When using Helm, deploy multiple Redpanda clusters by using separate namespaces and unique release names for each deployment. **Requirements:** - Each cluster must be deployed in a unique namespace - Use unique Helm release names for each deployment - Configure unique external port numbers for each cluster to avoid conflicts 1. Create configuration values for your second cluster: `redpanda-staging-values.yaml` ```yaml image: tag: v26.2.2 nameOverride: 'redpanda-staging' fullnameOverride: 'redpanda-staging' listeners: kafka: external: default: advertisedPorts: [31093] (1) admin: external: default: advertisedPorts: [31645] (1) http: external: default: advertisedPorts: [30083] (1) rpc: port: 33146 (1) schemaRegistry: external: default: advertisedPorts: [30084] (1) statefulset: extraVolumes: |- - name: redpanda-io-config configMap: name: redpanda-io-config extraVolumeMounts: |- - name: redpanda-io-config mountPath: /etc/redpanda-io-config additionalRedpandaCmdFlags: - "--io-properties-file=/etc/redpanda-io-config/io-config.yaml" ``` | 1 | Configure unique port numbers for each cluster to avoid conflicts. Ensure these ports don’t conflict with your first cluster’s configuration. | | --- | --- | 2. Install the second Redpanda cluster using a unique release name and namespace: ```bash helm install redpanda-staging redpanda/redpanda \ --version 26.2.2 \ --namespace redpanda-staging \ --create-namespace \ --values redpanda-staging-values.yaml ``` 3. Wait for the second cluster to be ready: ```bash kubectl --namespace redpanda-staging rollout status statefulset redpanda-staging --watch ``` > ❗ **IMPORTANT** > > When deploying multiple clusters, ensure that external listener ports are unique across all clusters to prevent conflicts. Also consider resource allocation and node capacity when planning multiple cluster deployments. ## [](#config)Production considerations This section provides advice for configuring the Redpanda in Kubernetes for production. If you’re using the Redpanda Operator, see: [cluster.redpanda.com/v1alpha2](https://docs.redpanda.com/streaming/current/reference/k-crd/) for all available settings. If you’re using the Redpanda Helm chart, see: [Redpanda Helm Chart Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) for all available settings. ### [](#version-pinning)Version pinning (Helm) If you use the Redpanda Helm chart to deploy Redpanda, it’s important to pin the version of the Helm chart to ensure that you have control over the version of Redpanda that you deploy. The Redpanda Helm chart version is independent of the Redpanda application version. The Redpanda application version can change even in patch releases of the Helm chart. This means that updates to the chart may roll out new versions of Redpanda. To avoid unexpected changes to your deployments, pin the version of the Helm chart. Pinning refers to the practice of specifying an exact version to use during deployment, rather than using the latest or unspecified version. When you pin the Helm chart version, you maintain consistent, predictable environments, especially in production. Using a specific version helps to: - **Ensure compatibility**: Guarantee that the deployed application behaves as tested, regardless of new chart versions being released. - **Avoid unexpected updates**: Prevent automatic updates that may introduce changes incompatible with the current deployment or operational practices. ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ --namespace \ --create-namespace ``` [Review the release notes](https://docs.redpanda.com/streaming/current/reference/releases/) to understand any significant changes, bug fixes, or potential disruptions that could affect your existing deployment. ### [](#name-overrides-helm)Name overrides (Helm) Deploying multiple instances of the same Helm chart in a Kubernetes cluster can lead to naming conflicts. Using `nameOverride` and `fullnameOverride` helps differentiate between them. If you have a production and staging environment for Redpanda, different names help to avoid confusion. - Use `nameOverride` to customize the labels `app.kubernetes.io/component=-statefulset` and `app.kubernetes.io/name=`. - Use `fullnameOverride` to customize the name of the StatefulSet and Services. ```yaml nameOverride: 'redpanda-production' fullnameOverride: 'redpanda-instance-prod' ``` ### [](#labels)Labels Kubernetes labels help you to organize, query, and manage your resources. Use labels to categorize Kubernetes resources in different deployments by environment, purpose, or team. ```yaml commonLabels: env: 'production' ``` ### [](#tolerations)Tolerations Tolerations and taints allow Pods to be scheduled onto nodes where they otherwise wouldn’t. If you have nodes dedicated to Redpanda with a taint `dedicated=redpanda:NoSchedule`, the following toleration allows the Redpanda brokers to be scheduled on them. ```yaml tolerations: - key: "dedicated" operator: "Equal" value: "redpanda" effect: "NoSchedule" ``` ### [](#docker-images)Docker images You can specify the image tag to deploy a known version of the Redpanda Docker image. By default, the image tag is set in `Chart.appVersion`. Avoid using the `latest` tag, which can lead to unexpected changes. A Redpanda deployment pulls more than the Redpanda broker image. To deploy from a private or mirrored registry, such as in an air-gapped environment, override the repository for each image that the Helm chart uses: ```yaml image: (1) repository: /redpandadata/redpanda tag: "v26.2.2" statefulset: sideCars: image: (2) repository: /redpandadata/redpanda-operator initContainerImage: (3) repository: /busybox tag: console: image: (4) registry: repository: redpandadata/console ``` | 1 | The Redpanda broker image. The chart also uses this image for init containers that configure the brokers. | | --- | --- | | 2 | The sidecar image, which runs the broker decommissioner, PVC unbinder, and rpk profile watcher. The chart also uses this image for the cluster configuration job. | | 3 | Used only by the optional set-datadir-ownership init container (statefulset.initContainers.setDataDirOwnership.enabled). If you don’t enable that init container, you don’t need to mirror this image. latest is this field’s chart default, but pin a specific tag for an air-gapped mirror, for the same reason you pin the Redpanda image tag above. | | 4 | The Redpanda Console image, configured through the Console subchart. | In the Redpanda resource, set the same fields under `spec.clusterSpec`: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda namespace: spec: clusterSpec: image: repository: /redpandadata/redpanda tag: "v26.2.2" statefulset: sideCars: image: repository: /redpandadata/redpanda-operator initContainerImage: repository: /busybox tag: console: image: registry: repository: redpandadata/console ``` > 📝 **NOTE** > > In Helm chart versions bundled with Redpanda Operator v25.1.x, the sidecar controllers use a separate image configured in `statefulset.sideCars.controllers.image`. In later versions, all sidecars use `statefulset.sideCars.image`. If you deploy with the Redpanda Operator, also override the Operator chart’s own [`image.repository`](https://docs.redpanda.com/streaming/current/reference/k-operator-helm-spec/#_image_repository). The section above covers every image the Redpanda chart pulls, but not the Operator Deployment’s own image, so an air-gapped Operator-based deployment still needs that field mirrored separately to avoid an `ImagePullBackOff` on the operator Pod itself. If you’re using a private registry, always ensure your nodes have the necessary credentials to pull the images. Pull secrets for the Redpanda Pods are set in `podTemplate.spec`. Redpanda Console is a subchart, so it takes its own pull secrets: ```yaml podTemplate: spec: imagePullSecrets: - name: console: imagePullSecrets: - name: ``` ### [](#number-of-redpanda-brokers)Number of Redpanda brokers The number of Redpanda brokers you deploy depends on your use case and the level of redundancy you require. For production, deploy at least three Redpanda brokers. Always deploy an odd number of brokers to avoid split-brain scenarios. ```yaml statefulset: replicas: 3 ``` > 📝 **NOTE** > > You must provision one dedicated worker node for each Redpanda broker that you plan to deploy in your Redpanda cluster. The default [`podAntiAffinity` rules](#affinity-rules) make sure that each Redpanda broker runs on its own worker node. See also: - [High Availability in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-high-availability/) - [Kubernetes Cluster Requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-worker-nodes) ### [](#tls)TLS By default, TLS (Transport Layer Security) is enabled for encrypted communication. Internal (`default`) and external (`external`) self-signed certificates are generated using cert-manager. See [TLS Certificates](#tls-certificates). ```yaml tls: enabled: true certs: # This key represents the name of the certificate. default: caEnabled: true # This key represents the name of the certificate. external: caEnabled: true ``` See also: [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) ### [](#authentication)Authentication If you want to authenticate clients connections to the Redpanda cluster, you can enable SASL authentication. ```yaml auth: sasl: enabled: true mechanism: "SCRAM-SHA-512" secretRef: "sasl-password-secret" users: [] ``` See also: [Configure Authentication for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) ### [](#bootstrap-user-with-custom-credentials)Bootstrap user with custom credentials When you enable authentication with `auth.sasl.enabled: true`, a default bootstrap user is created automatically. To deploy a cluster with your own bootstrap user credentials instead of the default, configure `auth.sasl.bootstrapUser`. This creates an initial superuser with your specified credentials at cluster formation time, before any client connections are accepted. This approach is recommended for production deployments where you want to control the initial superuser credentials for security compliance or automation requirements. ```yaml auth: sasl: enabled: true secretRef: redpanda-superusers users: [] bootstrapUser: name: "admin-bootstrap" # Optional, defaults to "kubernetes-controller" secretKeyRef: name: "bootstrap-user-secret" key: "password" mechanism: "SCRAM-SHA-512" # Options: SCRAM-SHA-256 or SCRAM-SHA-512 ``` The bootstrap user is automatically added to the `superusers` list and can be used to create additional users and configure ACLs after the cluster is deployed. See also: [Bootstrap superuser at cluster formation](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/#bootstrap) ### [](#resources)Resources By default, the resources allocated to Redpanda are for a development environment. In a production cluster, the resources you allocate should be proportionate to your machine type. You should determine and set these values before deploying the cluster. ```yaml resources: cpu: cores: 4 memory: enable_memory_locking: true container: max: 10Gi ``` See also: - [Manage Pod Resources in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/) - [Kubernetes Cluster Requirements and Recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) ### [](#storage)Storage In production, it’s best to use local PersistentVolumes (PVs) that are backed by NVMe devices to store the Redpanda data directory. NVMe devices outperform traditional SSDs or HDDs. Redpanda Data recommends creating StorageClasses that use the [local volume manager (LVM) CSI driver](https://github.com/metal-stack/csi-driver-lvm) to automatically provision PVs. The LVM allows you to group physical storage devices into a logical volume group. Allocating logical volumes from a logical volume group provides greater flexibility in terms of storage expansion and management. The LVM supports features such as resizing, snapshots, and striping, which are not available with the other drivers such as the local volume static provisioner. ```yaml storage: persistentVolume: enabled: true size: 100Gi storageClass: csi-driver-lvm-striped-xfs ``` For an example of configuring local PersistentVolumes backed by NVMe disks, see one of the following guides: - [Azure Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/aks-guide/#create-sc) (AKS) - [Elastic Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/eks-guide/#create-sc) (EKS) - [Google Kubernetes Engine](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/gke-guide/#create-sc) (GKE) See also: - [Supported Volume Types for Data in Redpanda](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-volume-types/) - [Kubernetes Cluster Requirements and Recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) - [Configure Storage for the Redpanda data directory in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/) ### [](#external-access)External access To make the Redpanda cluster accessible from outside the Kubernetes cluster, you can use NodePort or LoadBalancer Services. The default NodePort Service provides the lowest latency of all the Kubernetes Services because it does not include any unnecessary routing or middleware. Client connections go to the Redpanda brokers in the most direct way possible, through the worker nodes. By default, the fully qualified domain names (FQDNs) that brokers advertise are their internal addresses within the Kubernetes cluster, which are not reachable from outside the cluster. To make the cluster accessible from outside, each broker must advertise a domain that can be reached from outside the cluster. ```yaml external: enabled: true type: NodePort ``` See also: - [About Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/) - [Configure Listeners in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-configure-listeners/) ### [](#externaldns)ExternalDNS You should use ExternalDNS to manage DNS records for your Pods' domains. ExternalDNS synchronizes exposed Kubernetes Services with various DNS providers, rendering Kubernetes resources accessible through DNS servers. Benefits of ExternalDNS include: - **Automation**: ExternalDNS automatically configures public DNS records when you create, update, or delete Kubernetes Services or Ingresses. This eliminates the need for manual DNS configuration, which can be error-prone. - **Compatibility**: ExternalDNS is compatible with a wide range of DNS providers, including major cloud providers such as AWS, Google Cloud, and Azure, and DNS servers like CoreDNS and PowerDNS. - **Integration with other tools**: ExternalDNS can be used with other Kubernetes tools, such as ingress controllers or cert-manager for managing TLS certificates. ```yaml external: enabled: true type: LoadBalancer externalDns: enabled: true ``` See also: - [ExternalDNS with a NodePort Service](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-nodeport/#externaldns) - [ExternalDNS with LoadBalancer Services](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-loadbalancer/#externaldns) ### [](#logging)Logging By default, the log-level is set to `info`. In production, use the `info` logging level to avoid overwhelming the storage. For debugging purposes, temporarily change the logging level to `debug`. ```yaml logging: level: "info" ``` ### [](#monitoring)Monitoring By default, monitoring is disabled. If you have the [Prometheus Operator](https://prometheus-operator.dev/), enable monitoring to deploy a ServiceMonitor resource for Redpanda. Observability is essential in production environments. ```yaml monitoring: enabled: true ``` See also: [Monitor in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/) ### [](#statefulset-update-strategy)StatefulSet update strategy For smooth and uninterrupted updates, use the default `RollingUpdate` strategy. Additionally, set a PodDisruptionBudget to ensure that at least one Pod is available during updates. ```yaml statefulset: updateStrategy: type: "RollingUpdate" budget: maxUnavailable: 1 ``` See also: [Upgrade Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/) ### [](#affinity-rules)Affinity rules By default, `podAntiAffinity` rules stop the Kubernetes scheduler from placing multiple Redpanda brokers on the same node. These rules offer two benefits: - Minimize the risk of data loss by ensuring that a node’s failure results in the loss of only one Redpanda broker. - Prevent resource contention between brokers by ensuring they are never co-located on the same node. Affinities control Pod placement in the cluster based on various conditions. Set these according to your high availability and infrastructure needs. For example, this is a soft rule that tries to ensure the Kubernetes scheduler doesn’t place two Pods with the same `app: redpanda` label in the same zone. However, if it’s not possible, the scheduler can still place the Pods in the same zone. ```yaml statefulset: podAntiAffinity: topologyKey: kubernetes.io/hostname type: hard weight: 100 custom: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: "app" operator: "In" values: - "redpanda" topologyKey: "kubernetes.io/zone" ``` To completely disable `podAntiAffinity` rules (for example, in development environments where you want to run multiple brokers on the same node), you can override the default configuration: ```yaml statefulset: podTemplate: spec: affinity: podAntiAffinity: null ``` > ⚠️ **WARNING** > > Disabling `podAntiAffinity` rules is not recommended for production environments as it allows multiple brokers to be scheduled on the same node, increasing the risk of data loss if a node fails. See also: [High Availability in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-high-availability/) ### [](#graceful-shutdown)Graceful shutdown By default, Pods are given 90 seconds to shut down gracefully. If your brokers require additional time for a graceful shutdown, modify the `terminationGracePeriodSeconds`. ```yaml statefulset: terminationGracePeriodSeconds: 100 ``` See also: [Upgrade Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/) ### [](#service-account)Service account Restricting permissions is a best practice. Create a dedicated ServiceAccount for each Pod. To assign roles to this ServiceAccount, see [Role-based access control (RBAC)](#RBAC). ```yaml serviceAccount: create: true name: "redpanda-service-account" ``` ### [](#RBAC)Role-based access control (RBAC) RBAC is a method for providing permissions to ServiceAccounts based on roles. Some features such as rack awareness require both a ServiceAccount and RBAC to access resources using the Kubernetes API. ```yaml rbac: enabled: true annotations: {} ``` See also: [Enable Rack Awareness in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rack-awareness/) ## [](#perform-a-self-test)Perform a self test To understand the performance capabilities of your Redpanda cluster, Redpanda offers built-in self-test features that evaluate the performance of both disk and network operations. For more information, see [Disk and network self-test benchmarks](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/#self-test). ## [](#redpanda-console)Redpanda Console Redpanda Console is deployed by default when you deploy a Redpanda cluster using either the Redpanda Operator or the Redpanda Helm chart. This provides a web UI for managing and debugging your Redpanda cluster and applications. Both the Redpanda Operator and Helm chart automatically deploy Redpanda Console alongside your Redpanda cluster with sensible defaults. If you need to deploy Redpanda Console separately (for example, to connect to a Redpanda cluster running outside Kubernetes, or to have more granular control over the configuration), see [Deploy Redpanda Console on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy/). The standalone deployment option provides the option to connect to Redpanda clusters running outside Kubernetes. For resource recommendations and scaling guidance for Redpanda Console on Kubernetes, see [Redpanda Console Kubernetes Requirements and Recommendations](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/k-requirements/). ## [](#explore-the-default-kubernetes-components)Explore the default Kubernetes components By default, the Redpanda Helm chart deploys the following Kubernetes components: - [A StatefulSet](#statefulset) with three Pods. - [One PersistentVolumeClaim](#persistentvolumeclaim) for each Pod, each with a capacity of 20Gi. - [A headless ClusterIP Service and a NodePort Service](#service) for each Kubernetes node that runs a Redpanda broker. - [Self-Signed TLS Certificates](#tls-certificates). ### [](#statefulset)StatefulSet Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running. ```bash kubectl get statefulset --namespace ``` Example output: NAME READY AGE redpanda 3/3 3m11s StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as `redpanda-0`. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced. ```bash kubectl get pod --namespace ``` Expected output: ```none NAME READY STATUS RESTARTS AGE redpanda-0 1/1 Running 0 6m9s redpanda-1 1/1 Running 0 6m9s redpanda-2 1/1 Running 0 6m9s redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m redpanda-configuration-smqv7 0/1 Completed 0 6m9s ``` > 📝 **NOTE** > > The `redpanda-configuration` job updates the Redpanda runtime configuration. ### [](#persistentvolumeclaim)PersistentVolumeClaim Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using. ```bash kubectl get persistentvolumeclaims --namespace ``` Expected output: ```none NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s ``` ### [](#service)Service The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services: - Internal using the [Headless ClusterIP](#headless-clusterip-service) - External using the [NodePort](#nodeport-service) ```bash kubectl get service --namespace ``` Expected output: ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda ClusterIP None 5m37s redpanda-console ClusterIP 10.0.251.204 8080 5m redpanda-external NodePort 10.96.137.220 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s ``` #### [](#headless-clusterip-service)Headless ClusterIP Service The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other. An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other. ```bash kubectl --namespace exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml ``` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: redpanda-0.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-1.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-2.redpanda..svc.cluster.local. port: 33145 ``` #### [](#nodeport-service)NodePort Service External access is made available by a NodePort service that opens the following ports by default: | Listener | Node Port | Container Port | | --- | --- | --- | | Schema Registry | 30081 | 8081 | | HTTP Proxy | 30082 | 8083 | | Kafka API | 31092 | 9094 | | Admin API | 31644 | 9644 | To learn more, see [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/). ### [](#tls-certificates)TLS Certificates By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses [cert-manager](https://cert-manager.io/docs/) to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections. Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True - `redpanda-default-cert`: Self-signed certificate for internal communications. - `redpanda-default-root-certificate`: Root certificate authority for the internal certificate. - `redpanda-external-cert`: Self-signed certificate for external communications. - `redpanda-external-root-certificate`: Root certificate authority for the external certificate. By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/). > 📝 **NOTE** > > The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. ## [](#uninstall-redpanda)Uninstall Redpanda When you finish testing Redpanda, you can uninstall it from your Kubernetes cluster. The steps depend on how you installed Redpanda: using the Redpanda Operator or the Redpanda Helm chart. ### Operator Follow the steps in **exact order** to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection. 1. Delete all Redpanda-related custom resources: ```bash kubectl delete users --namespace --all kubectl delete topics --namespace --all kubectl delete schemas --namespace --all kubectl delete redpanda --namespace --all kubectl delete consoles --namespace --all ``` The Redpanda Operator creates a Console resource for each Redpanda resource, so you must delete Console resources too. The Redpanda Operator must still be running when you delete these resources. It removes their finalizers as part of its cleanup. If you uninstall the Redpanda Operator first, any remaining resource keeps its finalizer forever and the CRD deletion step below hangs. 2. Make sure requests for those resources return no results. Do not continue until every command returns `No resources found`: ```bash kubectl get users,topics,schemas,redpandas,consoles --namespace ``` 3. Uninstall the Redpanda Operator Helm release: ```bash helm uninstall redpanda-controller --namespace ``` Helm does not uninstall CRDs by default when using `helm uninstall` to avoid accidentally deleting existing custom resources. 4. Remove the CRDs. 1. List all Redpanda CRDs installed by the operator: ```bash kubectl api-resources --api-group='cluster.redpanda.com' ``` This command displays all CRDs defined by the Redpanda Operator. For example: ```bash NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User ``` 2. Delete the CRDs: ```bash kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl delete ``` This command lists all CRDs with the `cluster.redpanda.com` domain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually. 5. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ### Helm If you deployed Redpanda with the Redpanda Helm chart, follow these steps to uninstall it: 1. Uninstall the Helm release: ```bash helm uninstall redpanda --namespace ``` 2. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ## [](#troubleshoot)Troubleshoot Before troubleshooting your cluster, make sure that you have all the [prerequisites](#prerequisites). ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ### [](#statefulset-never-rolls-out)StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). ## [](#next-steps)Next steps After deploying Redpanda, validate your production readiness: - [Production readiness checklist](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness/) - Comprehensive validation of your deployment against production standards See the [Manage Kubernetes topics](https://docs.redpanda.com/streaming/current/manage/kubernetes/) to learn how to customize your deployment to meet your needs. ## [](#suggested-reading)Suggested reading - [High Availability in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-high-availability/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) ## Suggested labs - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 43: Production Readiness Checklist **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness.md --- # Production Readiness Checklist > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Production Readiness Checklist latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-production-readiness page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-production-readiness.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-production-readiness.adoc description: Comprehensive checklist for validating Redpanda deployments in Kubernetes against production readiness standards. learning-objective-1: Validate a Kubernetes-deployed Redpanda cluster against production readiness standards page-git-created-date: "2026-02-06" page-git-modified-date: "2026-08-20" support-status: supported --- Before running a production workload on Redpanda in Kubernetes, follow this readiness checklist. By completing this checklist, you will be able to: - Validate a Kubernetes-deployed Redpanda cluster against production readiness standards > 📝 **NOTE** > > For Linux deployments, see the [Production Readiness Checklist for Linux](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-readiness/). ## [](#critical-requirements)Critical requirements The Critical requirements checklist helps ensure that: - You have specified all required defaults and configuration items. - You have the optimal hardware setup. - You have enabled security. - You are set up to run in production. ### [](#redpanda-license)Redpanda license If using Enterprise features, validate that you are using a valid Enterprise license: Input ```bash kubectl exec -n -c redpanda -- rpk cluster license info -X user= -X pass= -X sasl.mechanism= ``` Output ```bash LICENSE INFORMATION =================== Organization: Your Company Name Type: enterprise Expires: Dec 31 2026 ``` Production deployments using Enterprise features (such as Tiered Storage, Schema Registry, or Continuous Data Balancing) must have a valid Enterprise license with a sufficient expiration date. See also: [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/) > 📝 **NOTE: Input** > > **SASL authentication flags** > > The `rpk` commands throughout this checklist include SASL authentication flags (`-X user`, `-X pass`, `-X sasl.mechanism`). If your cluster does not use SASL authentication, you can omit these flags from all commands. For example: > > Input > > ```bash > # With SASL authentication > kubectl exec -n -c redpanda -- rpk cluster health -X user= -X pass= -X sasl.mechanism= > > # Without SASL authentication > kubectl exec -n -c redpanda -- rpk cluster health > ``` > > Common SASL mechanisms are `SCRAM-SHA-256` or `SCRAM-SHA-512`. Update these values as needed for your deployment. ### [](#cluster-health)Cluster health Check that all brokers are connected and running. Run [`rpk cluster health`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-health/) to check the health of the cluster. No nodes should be down, and there should be zero leaderless or under-replicated partitions. Input ```bash kubectl exec -n -c redpanda -- rpk cluster health -X user= -X pass= -X sasl.mechanism= ``` Output ```bash CLUSTER HEALTH OVERVIEW ======================= Healthy: true Unhealthy reasons: [] Controller ID: 0 All nodes: [0 1 2] Nodes down: [] Leaderless partitions (0): [] Under-replicated partitions (0): [] ``` ### [](#minimum-broker-count)Minimum broker count You must have at least three brokers running to ensure production-level fault tolerance. Production clusters should have an odd number of brokers (3, 5, 7, etc.) for optimal consensus behavior. Verify the running broker count: Input ```bash kubectl get pods -n -l app.kubernetes.io/component=redpanda-statefulset ``` Output ```bash NAME READY STATUS RESTARTS AGE redpanda-0 2/2 Running 0 10d redpanda-1 2/2 Running 0 10d redpanda-2 2/2 Running 0 10d ``` Verify the configured replica count in your deployment: #### Helm Input ```bash helm get values redpanda -n | grep -A 1 "statefulset:" ``` Output ```bash statefulset: replicas: 3 ``` #### Operator Input ```bash kubectl get redpanda redpanda -n -o jsonpath='{.spec.clusterSpec.statefulset.replicas}' ``` Output ```bash 3 ``` See also: [Default Topic Replication Factor](#default-topic-replication-factor) ### [](#active-broker-membership)Active broker membership Verify that all brokers are in active state and not being decommissioned. Decommissioning is used to permanently remove a broker from the cluster, such as during node pool migrations or cluster downsizing. Brokers in a decommissioned state should not be present in production clusters unless actively performing a planned migration. Input ```bash kubectl exec -n -c redpanda -- rpk cluster info -b --detailed -X user= -X pass= -X sasl.mechanism= ``` Output ```bash NODE-ID NUM-CORES MEMBERSHIP-STATUS IS-ALIVE BROKER-VERSION 0 4 active true v24.2.4 1 4 active true v24.2.4 2 4 active true v24.2.4 ``` All brokers must show `active` status. If any broker shows the status `draining` or `decommissioned`, investigate immediately. See also: [Decommission Brokers](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-decommission-brokers/) ### [](#no-brokers-in-maintenance-mode)No brokers in maintenance mode Check that no brokers are in maintenance mode during normal operations. Maintenance mode is used when modifying brokers that will remain as members of the cluster, such as during rolling upgrades or hardware maintenance. While necessary during planned maintenance windows, brokers should not remain in maintenance mode during normal operations. Input ```bash kubectl exec -n -c redpanda -- rpk cluster maintenance status -X user= -X pass= -X sasl.mechanism= ``` Output ```bash NODE-ID ENABLED FINISHED ERRORS PARTITIONS ELIGIBLE TRANSFERRING FAILED 0 false - - - - - - 1 false - - - - - - 2 false - - - - - - ``` All brokers should show `ENABLED: false`. If any broker shows `ENABLED: true` outside of a planned maintenance window, investigate immediately. See also: [Maintenance Mode](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rolling-restart/) ### [](#consistent-redpanda-version)Consistent Redpanda version Check that Redpanda is running the [latest point release](https://github.com/redpanda-data/redpanda/releases) for the major version you’re on and that all brokers run the same version. **Verify Redpanda broker versions:** Input ```bash kubectl exec -n -c redpanda -- rpk cluster info -b --detailed -X user= -X pass= -X sasl.mechanism= ``` Output ```bash NODE-ID NUM-CORES MEMBERSHIP-STATUS IS-ALIVE BROKER-VERSION 0 4 active true v25.2.4 1 4 active true v25.2.4 2 4 active true v25.2.4 ``` All brokers must show the same `BROKER-VERSION`. Version mismatches between brokers can cause compatibility issues and must be resolved before advancing to production. **Verify Helm Chart or Operator version compatibility:** For Kubernetes deployments, you must also verify that your deployment tool (Helm Chart or Operator) version is compatible with your Redpanda version. The Helm Chart or Operator version must be within one minor version of the Redpanda version. For example, if running Redpanda v25.2.x, the Helm Chart or Operator version must be v25.1.x, v25.2.x, or v25.3.x. #### Helm Input ```bash helm list -n ``` Output ```bash NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION redpanda redpanda 1 2024-01-15 10:30:00.123456 -0800 PST deployed redpanda-5.2.4 v25.2.4 ``` The `CHART` column shows the Helm Chart version (for example, `redpanda-5.2.4`), which should be compatible with the `APP VERSION` (Redpanda version). #### Operator Input ```bash kubectl get deployment redpanda-controller-manager -n -o jsonpath='{.spec.template.spec.containers[0].image}' ``` Output ```bash docker.redpanda.com/redpandadata/redpanda-operator:v25.2.4 ``` The Operator version is shown in the image tag (for example, `v25.2.4`), which should be compatible with your Redpanda broker version. You can also check the Operator version using: Input ```bash kubectl get redpanda redpanda -n -o jsonpath='{.metadata.annotations.redpanda\.com/operator-version}' ``` **Version compatibility requirements:** - All Redpanda brokers must run the same version - The Helm Chart or Operator version must be within ±1 minor version of Redpanda version - Example: Redpanda v25.2.x requires Helm/Operator v25.1.x, v25.2.x, or v25.3.x - Running incompatible versions can lead to deployment failures or cluster instability. ### [](#version-pinning)Version pinning Verify that versions are explicitly pinned in your deployment configuration: #### Helm ```yaml image: tag: v24.2.4 # Pin specific Redpanda version console: enabled: true image: tag: v2.4.5 # Pin specific Console version connectors: enabled: true image: tag: v1.0.15 # Pin specific Connectors version ``` Verify pinned versions: Input ```bash helm get values redpanda -n ``` Output ```bash image: tag: v24.2.4 console: image: tag: v2.4.5 connectors: image: tag: v1.0.15 ``` #### Operator ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: image: tag: v24.2.4 # Pin specific Redpanda version console: enabled: true image: tag: v2.4.5 # Pin specific Console version connectors: enabled: true image: tag: v1.0.15 # Pin specific Connectors version ``` Verify pinned versions: Input ```bash kubectl get redpanda redpanda -n -o yaml | grep -A 1 "tag:" ``` Pin specific versions for Redpanda and all related components (Console, Connectors). This ensures all environments (dev/staging/prod) run the same tested versions, allows controlled upgrade testing before production rollout, and provides rollback capability to known-good versions. Avoid using the latest tag, version ranges (for example, v24.2.x), or unspecified tags, as these can result in unexpected upgrades that introduce breaking changes or cause downtime. ### [](#default-topic-replication-factor)Default topic replication factor Check that the default replication factor (≥3) is set appropriately for production. Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get default_topic_replications -X user= -X pass= -X sasl.mechanism= ``` Output ```bash 3 ``` Setting `default_topic_replications` to `3` or greater ensures new topics are created with adequate fault tolerance. See also: [Choose the Replication Factor](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/#choose-the-replication-factor) ### [](#existing-topics-replication-factor)Existing topics replication factor Check that all existing topics have adequate replication (default is `3`). Input ```bash kubectl exec -n -c redpanda -- rpk topic list -X user= -X pass= -X sasl.mechanism= ``` Output ```bash NAME PARTITIONS REPLICAS _schemas 1 3 orders 12 3 payments 8 3 user-events 16 3 ``` All production topics should have `REPLICAS` of three or greater. Topics with single-digit replication are at risk of data loss if a broker fails. See also: [Change Topic Replication Factor](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/topic-property-configuration/#change-topic-replication-factor) ### [](#persistent-storage-configuration)Persistent storage configuration Verify that you have configured persistent storage (not hostPath or emptyDir) for data persistence. Input ```bash kubectl get pvc -n ``` Output ```bash NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-a1b2c3d4-e5f6-7890-abcd-ef1234567890 100Gi RWO fast-ssd 10d datadir-redpanda-1 Bound pvc-b2c3d4e5-f6g7-8901-bcde-fg2345678901 100Gi RWO fast-ssd 10d datadir-redpanda-2 Bound pvc-c3d4e5f6-g7h8-9012-cdef-gh3456789012 100Gi RWO fast-ssd 10d ``` Verify the StatefulSet uses PersistentVolumeClaims: Input ```bash kubectl describe statefulset -n redpanda | grep -A 5 "Volume Claims" ``` Output ```bash Volume Claims: Name: datadir StorageClass: fast-ssd Labels: Annotations: Capacity: 100Gi ``` HostPath and emptyDir storage are not suitable for production as they lack durability guarantees. See also: [Persistent Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-persistent-storage/) ### [](#raidlvm-stripe-configuration-multiple-disks-only)RAID/LVM stripe configuration (multiple disks only) If using multiple physical disks, verify they are configured to stripe data across the disks as RAID-0 or LVM stripe (not linear/concat). Striping distributes data across multiple disks in parallel for improved I/O performance. Input ```bash # Check block device configuration on nodes kubectl debug node/ -it -- chroot /host /bin/bash lsblk -o NAME,TYPE,SIZE,MOUNTPOINT,FSTYPE lvs -o lv_name,stripes,stripe_size mdadm --detail /dev/md* # if using software RAID ``` Output ```bash # lsblk output NAME TYPE SIZE MOUNTPOINT FSTYPE nvme0n1 disk 1.8T nvme1n1 disk 1.8T vg0-data lvm 3.6T /var/lib/redpanda xfs # lvs output - note stripes > 1 indicates striping LV #Stripes StripeSize data 2 256.00k ``` Output ```bash # mdadm output /dev/md0: Raid Level : raid0 Array Size : 3515625472 (3.27 TiB) Raid Devices : 2 Number Major Minor RaidDevice State 0 259 0 0 active sync /dev/nvme0n1 1 259 1 1 active sync /dev/nvme1n1 ``` Using LVM linear/concat or JBOD instead of stripe/RAID-0 across multiple disks will severely degrade performance because data writes are serialized rather than parallelized. For optimal I/O throughput, configure multiple disks in a striped array that writes data across all disks simultaneously. Single disk configurations do not require striping. See also: [Storage](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#storage) ### [](#storage-performance-requirements)Storage performance requirements Ensure storage classes provide adequate IOPS and throughput for your workload by using the following specifications when selection a storage class: **Performance specifications:** - Use NVMe-based storage classes for production deployments - NFS (Network File System) is not supported - Test storage performance under load > ⚠️ **WARNING** > > Avoid cloud instance types that use multi-tenant or shared disks, as these can lead to unpredictable performance due to noisy neighbor effects. Examples of instances with shared/multi-tenant storage include AWS is4gen.xlarge and similar instance types across cloud providers. Instead, use instances with dedicated local NVMe storage or provisioned IOPS volumes that guarantee consistent performance. Multi-tenant disks can experience: - Unpredictable latency spikes from other tenants' workloads - Inconsistent throughput that varies based on neighbor activity - IOPS throttling that impacts Redpanda’s performance - Difficulty troubleshooting performance issues due to external factors See also: - [Storage requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#storage) - [Cloud Instance Types](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#cloud-instance-types) ### [](#cpu-and-memory-resource-limits)CPU and memory resource limits Verify Pods have resource requests and limits configured. Input ```bash kubectl get pod -n -o jsonpath='{.spec.containers[?(@.name=="redpanda")].resources}' | jq ``` Output ```bash { "limits": { "cpu": "4", "memory": "8Gi" }, "requests": { "cpu": "4", "memory": "8Gi" } } ``` All Redpanda Pods **must have**: - Identical CPU requests and limits (`requests.cpu == limits.cpu`) - Identical memory requests and limits (`requests.memory == limits.memory`) Setting requests equal to limits ensures the Pod receives the `Guaranteed` QoS class, which prevents CPU throttling and reduces the risk of Pod eviction. See also: [Manage Pod Resources](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/) ### [](#cpu-to-memory-ratio)CPU to memory ratio Ensure adequate memory allocation relative to CPU for optimal performance. Production deployments should provision at least 2 GiB of memory per CPU core. The ratio should be at least 1:2 (2 GiB per core). Verify the CPU to memory ratio in your configuration: #### Helm Input ```bash helm get values redpanda -n | grep -A 2 "resources:" ``` Output ```bash resources: cpu: cores: 4 memory: container: min: 8Gi max: 8Gi ``` #### Operator Input ```bash kubectl get redpanda redpanda -n -o jsonpath='{.spec.clusterSpec.resources}' | jq ``` Output ```bash { "cpu": { "cores": 4 }, "memory": { "container": { "min": "8Gi", "max": "8Gi" } } } ``` In the preceding examples, 4 CPU cores with 8 GiB memory provides a 1:2 ratio (2 GiB per core). See also: [Memory](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/#memory) ### [](#no-fractional-cpu-requests)No fractional CPU requests Ensure CPU requests use whole numbers for consistent performance. Fractional CPUs can lead to performance variability in production. Use whole integer values (`4`, `8`, or `16` are acceptable, while `3.5` or `7.5` are not). Verify CPU configuration: Input ```bash kubectl get pod -n -o jsonpath='{.spec.containers[?(@.name=="redpanda")].resources.requests.cpu}' ``` Output ```bash 4 ``` ### [](#authorization-enabled)Authorization enabled Verify Kafka authorization is enabled for access control. Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get kafka_enable_authorization -X user= -X pass= -X sasl.mechanism= ``` Output ```bash true ``` Without authorization enabled, any client can access Kafka APIs without authentication. See also: [Authorization](https://docs.redpanda.com/streaming/current/manage/security/authorization/) ### [](#production-mode-enabled)Production mode enabled Verify that developer mode and overprovisioned mode are disabled for production stability. Check developer mode: Input ```bash kubectl exec -n -c redpanda -- grep developer_mode /etc/redpanda/redpanda.yaml ``` Output ```bash developer_mode: false ``` Developer mode should never be enabled in production environments. Developer mode disables fsync and bypasses safety checks designed for production workloads. Check overprovisioned mode: Input ```bash kubectl exec -n -c redpanda -- grep overprovisioned /etc/redpanda/redpanda.yaml ``` Output ```bash overprovisioned: false ``` Overprovisioned mode bypasses critical resource checks and should never be enabled in production. This mode is intended only for development environments with constrained resources. Verify in Helm values that `resources.cpu.overprovisioned` is not explicitly set to `true` (it’s automatically calculated based on CPU allocation). ### [](#tls-enabled)TLS enabled Configure TLS encryption for all client and inter-broker communication. TLS prevents eavesdropping and man-in-the-middle attacks on network traffic. Verify TLS is enabled on all listeners: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config export -X user= -X pass= -X sasl.mechanism= | grep -A 10 "kafka_api:" ``` Output ```bash redpanda: kafka_api: - address: 0.0.0.0 port: 9093 name: internal authentication_method: sasl kafka_api_tls: - name: internal enabled: true cert_file: /etc/tls/certs/tls.crt key_file: /etc/tls/certs/tls.key ``` Required TLS listeners include: - **kafka\_api** - Client connections to Kafka API - **admin\_api** - Administrative REST API access - **rpc\_server** - Inter-broker communication - **schema\_registry** - Schema Registry API (if used) Verify certificates are properly mounted: Input ```bash kubectl exec -n -c redpanda -- ls -la /etc/tls/certs/ ``` Output ```bash total 16 -rw-r--r-- 1 redpanda redpanda 1234 Dec 15 10:00 ca.crt -rw-r--r-- 1 redpanda redpanda 1675 Dec 15 10:00 tls.crt -rw------- 1 redpanda redpanda 1704 Dec 15 10:00 tls.key ``` See also: [TLS Encryption](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) ### [](#authentication-enabled)Authentication enabled Configure appropriate authentication mechanisms to control access to Redpanda resources. Verify SASL users are configured: Input ```bash kubectl exec -n -c redpanda -- rpk acl user list -X user= -X pass= -X sasl.mechanism= ``` Output ```bash USERNAME admin app-producer app-consumer monitoring ``` Be sure to adhere to the following authentication requirements: - Set up SASL authentication for client connections - Configure TLS certificates for encryption (see preceding TLS configuration guidance) - Implement proper user management with principle of least privilege - Configure [ACLs (Access Control Lists)](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/) for resource authorization Verify ACLs are configured: Input ```bash kubectl exec -n -c redpanda -- rpk acl list -X user= -X pass= -X sasl.mechanism= ``` Output ```bash PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME OPERATION PERMISSION User:app-producer * TOPIC orders.* WRITE ALLOW User:app-consumer * TOPIC orders.* READ ALLOW User:app-consumer * GROUP consumer-group-1 READ ALLOW ``` See also: - [Authentication](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) - [Authorization](https://docs.redpanda.com/streaming/current/manage/security/authorization/) ### [](#network-security)Network security Secure network access to the cluster using Kubernetes-native controls. Verify NetworkPolicies are configured: Input ```bash kubectl get networkpolicy -n ``` Output ```bash NAME POD-SELECTOR AGE redpanda-allow-internal app.kubernetes.io/name=redpanda 10d redpanda-allow-clients app.kubernetes.io/name=redpanda 10d redpanda-deny-all-ingress app.kubernetes.io/name=redpanda 10d ``` Check NetworkPolicy rules: Input ```bash kubectl describe networkpolicy -n ``` Be sure to satisfy the following network security requirements: - Configure NetworkPolicies to restrict pod-to-pod communication - Use TLS for all client connections (see TLS configuration) - Secure admin API endpoints with [authentication](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) and [authorization](https://docs.redpanda.com/streaming/current/manage/security/authorization/) - Limit ingress traffic to only necessary ports and sources - Use Kubernetes Services to control external access - Set `publishNotReadyAddresses: true` on any [custom Services](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-custom-services/#publish-addresses-of-unready-brokers) that expose Redpanda brokers Verify services and exposed ports: Input ```bash kubectl get svc -n ``` Output ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) redpanda ClusterIP None 9093/TCP,9644/TCP,8082/TCP redpanda-external LoadBalancer 10.100.200.50 9093:30001/TCP ``` See also: [Listener Configuration](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-configure-listeners/) ### [](#pod-disruption-budget)Pod Disruption Budget Set up PDBs to control voluntary disruptions during maintenance. Input ```bash kubectl get pdb -n ``` Output ```bash NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE redpanda N/A 1 1 10d ``` Production deployments must have a PodDisruptionBudget with `maxUnavailable: 1` to prevent simultaneous broker disruptions during voluntary operations like node drains, upgrades, or autoscaler actions. See also: [Kubernetes Pod Disruption Budgets](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) ### [](#rack-awareness-and-topology-spread)Rack awareness and topology spread Configure topology spread constraints to distribute brokers across availability zones. For configuration instructions, see [Multi-AZ deployment](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-high-availability/#multi-az-deployment). Production deployments require each Redpanda broker to run in a different availability zone to ensure that a single zone failure does not cause loss of quorum. For a three-broker cluster, brokers must be distributed across three separate zones. To verify zone distribution, check your cluster configuration: - Verify `topologySpreadConstraints` are configured in your Helm values or Redpanda CR - Confirm nodes have zone labels (typically `topology.kubernetes.io/zone`) - Check that brokers are scheduled on nodes in different zones See also: [Rack Awareness](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rack-awareness/) ### [](#operator-crds-operator-deployments-only)Operator CRDs (Operator deployments only) > ⚠️ **WARNING** > > If your deployment uses the Redpanda Operator, all required Custom Resource Definitions (CRDs) must be installed with compatible versions. Without correct CRDs, the Operator cannot manage the cluster, leading to configuration drift, failed updates, and potential data loss. The required CRDs are below: - `clusters.cluster.redpanda.com` - Manages Redpanda cluster configuration - `topics.cluster.redpanda.com` - Manages topic lifecycle - `users.cluster.redpanda.com` - Manages SASL users - `schemas.cluster.redpanda.com` - Manages Schema Registry schemas If any CRDs are missing or incompatible with your Operator version, the Operator will fail to reconcile resources. Verify all required CRDs are installed: Input ```bash kubectl get crd | grep redpanda.com ``` Output ```bash clusters.cluster.redpanda.com topics.cluster.redpanda.com users.cluster.redpanda.com schemas.cluster.redpanda.com ``` ### [](#run-redpanda-tuners)Run Redpanda tuners Check that you have configured tuners for optimal performance. Tuners can significantly impact latency and throughput. In Kubernetes, tuners are configured through the Helm chart or may need to be run on worker nodes themselves. For details, see [Tune Kubernetes Worker Nodes for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/). ## [](#recommended-requirements)Recommended requirements The Recommended requirements checklist ensures that you can monitor and support your environment on a sustained basis. It includes the following checks: - You have adhered to day-2 operations best practices. - You can diagnose and recover from backup issues or failures. - You have configured monitoring, backup, and security scanning. ### [](#deployment-method)Deployment method Verify that the deployment method (Helm or Operator) is correctly identified for your cluster. Understanding your deployment method is important for troubleshooting, upgrades, and configuration management. #### Helm Input ```bash helm list -n ``` Output ```bash NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION redpanda redpanda 1 2024-01-15 10:30:00.123456 -0800 PST deployed redpanda-5.0.0 v24.1.1 ``` The presence of a Helm release (`CHART` displays `redpanda-5.0.0`) indicates a Helm-managed deployment. #### Operator Input ```bash kubectl get redpanda -n ``` Output ```bash NAME READY STATUS redpanda True Redpanda reconciliation succeeded ``` The presence of a Redpanda custom resource indicates an Operator-managed deployment. Knowing your deployment method helps determine which configuration approach to use (Helm values vs. Redpanda CR), how to perform upgrades and rollbacks, where to find deployment logs and troubleshooting information, and which documentation sections apply to your environment. See [Production Deployment Workflow](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-workflow/) for the complete deployment process. ### [](#xfs-filesystem)XFS filesystem Verify that data directories use XFS filesystem for optimal performance. Input ```bash kubectl exec -n -c redpanda -- df -khT /var/lib/redpanda/data ``` Output ```bash Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1 xfs 1.8T 14G 1.8T 1% /var/lib/redpanda/data ``` XFS provides better performance characteristics for Redpanda workloads compared to ext4. While ext4 is supported, XFS is strongly recommended for production deployments. See also: [Storage Requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/requirements/#storage) ### [](#pod-anti-affinity)Pod anti-affinity Configure Pod anti-affinity to spread brokers across nodes. Input ```bash kubectl get statefulset redpanda -n -o jsonpath='{.spec.template.spec.affinity}' | jq ``` Output ```bash { "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": [ { "labelSelector": { "matchLabels": { "app.kubernetes.io/name": "redpanda" } }, "topologyKey": "kubernetes.io/hostname" } ] } } ``` This prevents single node failures from affecting multiple brokers by ensuring each Redpanda Pod runs on a different node. See also: [Pod Anti-Affinity](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#affinity-rules) ### [](#node-isolation)Node isolation Configure taints/tolerations or nodeSelector for workload isolation. Input ```bash kubectl get statefulset redpanda -n -o jsonpath='{.spec.template.spec.nodeSelector}' | jq ``` Output ```bash { "workload-type": "redpanda" } ``` Isolating Redpanda workloads on dedicated nodes improves performance predictability by preventing resource contention with other applications. ### [](#partition-balancing)Partition balancing Configure automatic partition balancing across brokers and CPU cores. #### [](#continuous-data-balancing)Continuous Data Balancing [Continuous Data Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/) can help you manage production deployments by automatically rebalancing partition replicas across brokers based on disk usage and node changes. It also eliminates manual intervention and prevents performance degradation. > ❗ **IMPORTANT** > > You should enable Continuous Data Balancing for all licensed production clusters. Verify that Continuous Data Balancing is configured: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get partition_autobalancing_mode -X user= -X pass= -X sasl.mechanism= ``` Output ```bash continuous ``` The `continuous` setting enables automatic partition rebalancing based on: - Node additions or removals - High disk usage conditions - Broker availability changes Without Continuous Data Balancing, partition distribution becomes skewed over time, leading to hotspots and manual rebalancing operations. #### [](#core-balancing)Core Balancing [Intra-broker partition balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-balancing/#intra-broker-partition-balancing) distributes partition replicas across CPU cores within individual brokers. Check core balancing for CPU core partition distribution: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get core_balancing_on_core_count_change -X user= -X pass= -X sasl.mechanism= ``` Output ```bash true ``` When enabled, Redpanda continuously rebalances partitions between CPU cores on a broker for optimal resource utilization, which is especially beneficial after broker restarts or configuration changes. ### [](#system-requirements)System requirements Run system checks to get more details regarding your system configuration. Input ```bash kubectl exec -n -c redpanda -- rpk redpanda check -X user= -X pass= -X sasl.mechanism= ``` Output ```bash CONDITION REQUIRED CURRENT SEVERITY PASSED Data directory is writable true true Fatal true Free memory per CPU [MB] >= 2048 8192 Warning true NTP Synced true true Warning true Swappiness 1 1 Warning true ``` Review any failed checks and remediate before proceeding to production. See [rpk redpanda check](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-check/) for details on each validation. ### [](#debug-bundle)Debug bundle Verify that you can successfully generate and collect a debug bundle from your cluster. This proactive check ensures that if an issue occurs and you need to contact Redpanda support, you won’t face permission issues or silent collection failures that could delay troubleshooting. Generate a debug bundle: Input ```bash kubectl exec -n -c redpanda -- rpk debug bundle -o /tmp/bundle.zip ``` For additional options and arguments, see [rpk debug bundle](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-debug/rpk-debug-bundle/). Output ```bash Creating bundle file... Collecting cluster info... Collecting logs... Collecting configuration... Debug bundle saved to '/tmp/bundle.zip' ``` Debug bundles collect critical diagnostic information including cluster configuration and metadata, Redpanda logs from all brokers, system resource usage and performance metrics, and Kubernetes resource definitions. When testing bundle generation, watch for permission errors preventing log collection, insufficient disk space for bundle creation, network policies blocking bundle transfer, or RBAC restrictions on accessing Pod logs or exec. Testing bundle generation early ensures this critical troubleshooting tool works when you need it most. Debug bundles are often required by Redpanda support to diagnose production issues efficiently. See also: [Diagnostics Bundles in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/debug-bundle/generate/kubernetes/) ### [](#tiered-storage)Tiered Storage Configure [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) for extended data retention using object storage. Tiered Storage automatically offloads older data to cloud storage (S3, GCS, Azure Blob), enabling extended retention without expanding local disk capacity. Verify Tiered Storage configuration: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get cloud_storage_enabled -X user= -X pass= -X sasl.mechanism= ``` Output ```bash true ``` #### [](#benefits-of-tiered-storage)Benefits of Tiered Storage - Reduced local storage costs from offloading cold data to cheaper object storage - Longer data retention periods without provisioning additional disk - Required for advanced features like Remote Read Replicas and Iceberg integration - Disaster recovery capabilities through cloud-backed data To verify your Tiered Storage configuration: Input ```bash # Check bucket configuration kubectl exec -n -c redpanda -- rpk cluster config get cloud_storage_bucket -X user= -X pass= -X sasl.mechanism= # Check region/endpoint kubectl exec -n -c redpanda -- rpk cluster config get cloud_storage_region -X user= -X pass= -X sasl.mechanism= ``` See also: [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) ### [](#security-scanning)Security scanning Regularly scan container images and configurations for vulnerabilities to maintain security. #### [](#container-image-scanning)Container image scanning Verify that container images are scanned before deployment: Input ```bash # Check current image in use kubectl get statefulset redpanda -n -o jsonpath='{.spec.template.spec.containers[?(@.name=="redpanda")].image}' ``` Output ```bash docker.redpanda.com/redpandadata/redpanda:v24.2.4 ``` #### [](#security-scanning-best-practices)Security scanning best practices Security scanning best practices include: - Scan images using tools like Trivy, Snyk, or cloud-native scanners before deployment - Set up automated scanning in CI/CD pipelines - Monitor for CVE announcements and security advisories - Keep Redpanda and related components up-to-date with security patches (see [Rolling Upgrades](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/)) - Review Kubernetes RBAC policies and ServiceAccount permissions (see [Role Controller](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authorization/k-role-controller/)) #### [](#configuration-scanning)Configuration scanning Input ```bash # Scan Kubernetes manifests kubectl get redpanda,statefulset,deployment -n -o yaml > cluster-config.yaml # Use kubesec, kube-bench, or similar tools to scan cluster-config.yaml ``` Establish a regular cadence for security scanning (for example, weekly or with each deployment). ### [](#backup-and-recovery)Backup and recovery Implement and test backup and recovery processes to ensure business continuity. #### [](#backup-strategy-with-tiered-storage)Backup strategy with Tiered Storage Tiered Storage provides built-in backup capabilities by storing data in object storage. Verify Tiered Storage is configured: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get cloud_storage_enabled -X user= -X pass= -X sasl.mechanism= ``` #### [](#recovery-testing)Recovery testing Regularly test recovery procedures to validate RTO/RPO targets: Input ```bash # Test topic restoration from Tiered Storage kubectl exec -n -c redpanda -- rpk topic describe -X user= -X pass= -X sasl.mechanism= ``` For mission-critical workloads requiring active disaster recovery, consider implementing [Shadowing](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-shadow-linking/) to asynchronously replicate data to a standby cluster. Shadowing provides offset-preserving replication that maintains consumer positions, enabling faster recovery with lower RTO compared to restoration from backups. This Enterprise feature (available in Redpanda v25.3 or later) supports cross-region or cross-cloud disaster recovery with automatic failover capabilities. Configure and validate Tiered Storage for automatic data backup to object storage. Document and regularly test recovery procedures for different failure scenarios in non-production environments. Establish clear Recovery Time Objective (RTO) and Recovery Point Objective (RPO) targets, and maintain runbooks for disaster recovery scenarios. For Shadowing deployments, use the [Shadowing Failover Runbook](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-failover-runbook/) as a starting point. Verify that IAM roles and permissions for object storage access are correctly configured and tested. See also: - [Whole Cluster Restore](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-whole-cluster-restore/) - [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-shadow-linking/) - [Shadowing Failover Runbook](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-failover-runbook/) ### [](#audit-logging)Audit logging Enable and configure audit logging for compliance and security monitoring requirements. Verify your audit log configuration: Input ```bash kubectl exec -n -c redpanda -- rpk cluster config get audit_enabled -X user= -X pass= -X sasl.mechanism= ``` Output ```bash true ``` Check to ensure you know where audit logs are being written: Input ```bash # Check audit log topic kubectl exec -n -c redpanda -- rpk topic list -X user= -X pass= -X sasl.mechanism= | grep audit ``` Output ```bash _redpanda.audit_log 1 3 ``` The output values of `1` and `3` indicate the number of partitions and replicas, respectively, for the audit log topic. For production environments with compliance requirements (SOC 2, HIPAA, PCI DSS, GDPR), forward audit logs to your SIEM system and configure retention policies according to your regulatory obligations. Ensure the audit log topic has adequate replication and retention settings. See also: [Audit Logging](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/k-audit-logging/) ### [](#monitoring)Monitoring Check that [monitoring](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) is configured with [Prometheus](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/#configure-prometheus) and [Grafana](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/#generate-grafana-dashboard) to scrape metrics from all Redpanda brokers. Verify ServiceMonitor is configured: Input ```bash kubectl get servicemonitor -n ``` ### [](#system-log-retention)System log retention Check that Redpanda logs are being captured and stored for an appropriate period of time (minimally, seven days). Configure log forwarding using tools like Fluentd or your cloud provider’s logging solution to send logs to a central location for troubleshooting and compliance purposes. See also: [Diagnostics Bundles in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/debug-bundle/generate/kubernetes/) ### [](#environment-configuration)Environment configuration Check that you have a development or test environment configured to evaluate upgrades and configuration changes before applying them to production. ### [](#upgrade-policy)Upgrade policy Check that you have an upgrade policy defined and implemented. Redpanda supports [rolling upgrades](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/), so upgrades do not require downtime. However, make sure that upgrades are scheduled on a regular basis, ideally using automation with [Helm](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/) or GitOps workflows. ## [](#advanced-requirements)Advanced requirements The Advanced requirements checklist ensures full enterprise readiness, indicates that your system is operating at the highest level of availability, and can prevent or recover from the most serious incidents. The Advanced requirements checklist confirms the following: - You are proactively monitoring mission-critical workloads. - You have business continuity solutions in place. - You have integrated into enterprise security and operational systems. - Your enterprise is ready to run mission-critical workloads. ### [](#configure-alerts)Configure alerts A standard set of alerts for [Grafana](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/#generate-grafana-dashboard) or [Prometheus](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/#configure-prometheus) is provided in the [GitHub Redpanda observability repo](https://github.com/redpanda-data/observability). Customize these alerts for your specific needs. See also: [Monitoring Metrics](https://docs.redpanda.com/streaming/current/reference/monitor-metrics/) ### [](#deployment-automation)Deployment automation Review your deployment automation. Ensure that cluster configuration is managed using [Helm](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/) or GitOps workflows, and that all configuration is saved in source control. ### [](#monitor-security-settings)Monitor security settings Regularly review your cluster’s security settings using the `/v1/security/report` [Admin API](https://docs.redpanda.com/api/doc/admin/) endpoint. Investigate and address any issues identified in the alerts section. Input ```bash curl 'http://localhost:9644/v1/security/report' ``` View output ```bash { "interfaces": { "kafka": [ { "name": "test_kafka_listener", "host": "0.0.0.0", "port": 9092, "advertised_host": "0.0.0.0", "advertised_port": 9092, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_method": "None", "authorization_enabled": false } ], "rpc": { "host": "0.0.0.0", "port": 33145, "advertised_host": "127.0.0.1", "advertised_port": 33145, "tls_enabled": false, "mutual_tls_enabled": false }, "admin": [ { "name": "test_admin_listener", "host": "0.0.0.0", "port": 9644, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_methods": [], "authorization_enabled": false } ] }, "alerts": [ { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_TLS", "description": "\"kafka\" interface \"test_kafka_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHN", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authentication. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHZ", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "rpc", "issue": "NO_TLS", "description": "\"rpc\" interface is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_TLS", "description": "\"admin\" interface \"test_admin_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHZ", "description": "\"admin\" interface \"test_admin_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHN", "description": "\"admin\" interface \"test_admin_listener\" is not using authentication. This is insecure and not recommended." } ] } ``` ## [](#suggested-reading)Suggested reading - [Deploy for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/) - [Customize the Helm Chart](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 44: Production Deployment Workflow for Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-workflow.md --- # Production Deployment Workflow for Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Production Deployment Workflow for Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-production-workflow page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-production-workflow.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-production-workflow.adoc description: Learn how to deploy Redpanda in Kubernetes for production. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- The production deployment tasks involve Kubernetes administrators (admins) as well as Kubernetes users. > 💡 **TIP** > > To try Redpanda on a local Kubernetes cluster for development or testing, see the [guide for kind or minikube](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/local-guide/) instead. 1. All: [Review the requirements and recommendations](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/) to align on prerequisites. 2. Admin: [Tune the worker nodes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/) for best performance. 3. User: [Deploy Redpanda](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/) using either the Redpanda Operator or the Redpanda Helm chart. 4. All: [Validate production readiness](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness/) using the comprehensive checklist to ensure your deployment meets production standards. ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 45: Kubernetes Cluster Requirements and Recommendations **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements.md --- # Kubernetes Cluster Requirements and Recommendations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Kubernetes Cluster Requirements and Recommendations latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-requirements page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-requirements.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-requirements.adoc description: A list of requirements and recommendations for provisioning Kubernetes clusters and worker nodes for running Redpanda in production. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-02-06" support-status: supported --- This topic provides the requirements and recommendations for provisioning Kubernetes clusters and worker nodes for running Redpanda in production. ## [](#operating-system)Operating system - Minimum version required of RHEL/CentOS: 8. **Recommended**: 9+ - Minimum version required of Ubuntu: 20.04 LTS. **Recommended**: 22.04+ **Recommendation**: Linux kernel 4.19 or later for better performance. ## [](#kubernetes-version)Kubernetes version Minimum required Kubernetes version: 1.27.0-0 Make sure to do the following: 1. [Install kubectl](https://kubernetes.io/docs/tasks/tools/). 2. [Configure the `kubeconfig` file for your cluster](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/). ## [](#helm-version)Helm version Minimum required Helm version: 3.10.0 [Install Helm](https://helm.sh/docs/intro/install/). > 📝 **NOTE** > > Helm v3.18.0 is not supported due to a bug that causes errors such as: > > Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) > > To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ## [](#number-of-workers)Number of nodes Provision one physical node or virtual machine (VM) for each Redpanda broker that you plan to deploy in your Redpanda cluster. Each Redpanda broker requires its own dedicated node for the following reasons: - **Resource isolation**: Redpanda brokers are designed to make full use of available system resources, including CPU and memory. By dedicating a node to each broker, you ensure that these resources aren’t shared with other applications or processes, avoiding potential performance bottlenecks or contention. - **External networking**: External clients should connect directly to the broker that owns the partition they’re interested in. This means that each broker must be individually addressable. As clients must connect to the specific broker that is the leader of the partition, they need a mechanism to directly address each broker in the cluster. Assigning each broker to its own dedicated node makes this direct addressing feasible, since each node will have a unique address. See [External networking](#external-networking). - **Fault tolerance**: Ensuring each broker operates on a separate node enhances fault tolerance. If one node experiences issues, it won’t directly impact the other brokers. > 📝 **NOTE** > > The Redpanda Helm chart configures [`podAntiAffinity` rules](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/#statefulset-podantiaffinity) to make sure that each Redpanda broker runs on its own node. **Recommendations**: Deploy at least three Pod replicas. ## [](#node-updates)Prevent automatic node upgrades Ensure that node and operating system (OS) upgrades are manually managed when running Redpanda in production. Manual control avoids unplanned reboots or replacements that disrupt Redpanda brokers, causing service downtime, data loss, or quorum instability. Common issues with automatic node upgrades include: - Hard timeouts for graceful shutdowns that do not allow Redpanda brokers enough time to complete decommissioning or leadership transitions. - Replacements or reboots without ensuring data has been safely migrated or replicated, risking data loss. - Parallel upgrades across multiple nodes, which can disrupt quorum or reduce cluster availability. **Requirements**: - Disable automatic node maintenance or upgrades. To prevent managed Kubernetes services from automatically rebooting or upgrading nodes: - **Azure AKS**: [Set the OS upgrade channel to `None`](https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-os-image). - **Google GKE**: [Disable GKE auto-upgrades for node pools](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades). - **Amazon EKS**: [Disable EKS node auto-upgrades](https://docs.aws.amazon.com/eks/latest/userguide/automode.html). See also: [How to manually manage node upgrades](https://docs.redpanda.com/streaming/current/upgrade/k-migrate-node-pools-manually/). ## [](#cpu-and-memory)CPU and memory **Requirements**: - Each production node must have at least two physical CPU cores. - x86\_64 (Westmere or newer) and AWS Graviton processors are supported. - Each Redpanda Pod requires at least 2 GiB of memory per core. - Request a minimum of 2.22 GiB per core to meet Redpanda’s memory allocation strategy. See [Manage Pod Resources in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/) for detailed guidance and examples. - Each Redpanda broker must have at least 2 GB of memory per core. - Each Redpanda broker must have at least 2 MB of memory for each topic partition replica. The total memory available for partition replicas is determined as a percentage of the cluster’s total memory, which is controlled by the [`topic_partitions_memory_allocation_percent`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#topic_partitions_memory_allocation_percent) setting. Each partition replica consumes [`topic_memory_per_partition`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#topic_memory_per_partition) bytes from this pool. If insufficient memory is available, topic operations will fail. You can adjust the allocation ratio using `topic_partitions_memory_allocation_percent`, but doing so is not recommended, as lowering it may lead to instability or degraded performance. **Recommendations**: - Four physical cores for each node are strongly recommended. ## [](#pod-resource-configuration)Pod resource configuration To ensure stable performance and predictable scheduling in Kubernetes, configure Redpanda Pods with appropriate CPU and memory requests and limits: - Set `resources.requests.memory` and `resources.limits.memory` to the same value. - Request at least 2.22 GiB of memory per core to meet Redpanda’s heap and overhead requirements. - Set `resources.cpu.cores` to an even integer (for example, `4`, `6`, or `8`) to align with the Kubernetes static CPU manager policy. - Match CPU and memory resource settings for all containers in the Pod, including init containers and sidecars, to receive the `Guaranteed` QoS class. - Enable memory locking with the `--lock-memory` flag to prevent paging and improve performance. This configuration: - Grants Redpanda exclusive access to CPU cores and memory - Reduces the risk of throttling, eviction, and OOM kills - Provides predictable and isolated runtime performance See [Manage Pod Resources in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/) for configuration examples using both Helm and the Redpanda Operator. ## [](#storage)Storage **Requirements**: - NVMe (Non-Volatile Memory Express) drives are required for production deployments. NVMe drives provide the high throughput and low latency needed for optimal Redpanda performance. See also: [Disk and network self-test benchmarks](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/#self-test). - An XFS or ext4 file system. The Redpanda data directory (`/var/lib/redpanda/data`) and the Tiered Storage cache must be mounted on an XFS or ext4 file system. For information about supported volume types for different data in Redpanda, see [Supported Volume Types for Data in Redpanda](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-volume-types/). > ⚠️ **CAUTION** > > The Network File System (NFS) is unsupported for use as the storage mechanism for the Redpanda data directory or for the Tiered Storage cache. - A default StorageClass that can provision PersistentVolumes with at least 20Gi of storage. **Recommendations**: - Use an XFS file system for its enhanced performance with Redpanda workloads. - For setups with multiple disks, use a RAID-0 (striped) array. It boosts speed but lacks redundancy. A disk failure can lead to data loss. - Use local PersistentVolumes backed by NVMe disks. ## [](#security)Security **Recommendations**: - If you’re using a cloud platform, use [IAM roles](https://docs.redpanda.com/streaming/current/manage/security/iam-roles/) to restrict access to resources in your cluster. - Secure your Redpanda cluster with TLS encryption and SASL authentication. ## [](#external-networking)External networking - For external access, each node in your cluster must have a static, externally accessible IP address. - Minimum 10 GigE (10 Gigabit Ethernet) connection to ensure: - High data throughput - Reduced data transfer latency - Scalability for increased network traffic **Recommendations**: [Use a NodePort Service for external access](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#external-access). ## [](#tuning)Tuning Before deploying Redpanda to production, each node that runs Redpanda must be tuned to optimize the Linux kernel for Redpanda processes. See [Tune Kubernetes Worker Nodes for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/). ## [](#object-storage-providers-for-tiered-storage)Object storage providers for Tiered Storage Redpanda supports the following storage providers for Tiered Storage: - Amazon Simple Storage Service (S3) - Google Cloud Storage (GCS), using the Google Cloud Platform S3 API - Azure Blob Storage (ABS) ## [](#cloud-instance-types)Cloud instance types **Recommendations**: - Use a cloud instance type that supports locally attached NVMe devices with an XFS file system. NVMe devices offer high I/O operations per second (IOPS) and minimal latency, while XFS offers enhanced performance with Redpanda workloads. ### [](#amazon)Amazon EKS defaults to the ext4 file system. Use XFS instead where possible. - General purpose: General-purpose instances provide a balance of compute, memory, and networking resources, and they can be used for a variety of diverse workloads. - [M5d](https://aws.amazon.com/ec2/instance-types/m5/) - [M5ad](https://aws.amazon.com/ec2/instance-types/m5/) - [M5dn](https://aws.amazon.com/ec2/instance-types/m5/) - [M6gd](https://aws.amazon.com/ec2/instance-types/m6g/) - [M7gd](https://aws.amazon.com/ec2/instance-types/m7g/) - Memory optimized: Memory-optimized instances are designed to deliver fast performance for workloads that process large data sets in memory. - [R5ad](https://aws.amazon.com/ec2/instance-types/r5/) - [R5d](https://aws.amazon.com/ec2/instance-types/r5/) - [R5dn](https://aws.amazon.com/ec2/instance-types/r5/) - [R6gd](https://aws.amazon.com/ec2/instance-types/r6g/) - [R6id](https://aws.amazon.com/ec2/instance-types/r6i/) - [R6idn](https://aws.amazon.com/ec2/instance-types/r6i/) - [R7gd](https://aws.amazon.com/ec2/instance-types/r7g/) - [X2gd](https://aws.amazon.com/ec2/instance-types/x2/) - [X2idn](https://aws.amazon.com/ec2/instance-types/x2i/) - [X2iedn](https://aws.amazon.com/ec2/instance-types/x2i/) - [z1d](https://aws.amazon.com/ec2/instance-types/z1d/) - Storage optimized: Storage-optimized instances are designed for workloads that require high, sequential read and write access to very large data sets on local storage. They are optimized to deliver tens of thousands of low-latency, random IOPS to applications. - [I4g, Is4gen, Im4gn](https://aws.amazon.com/ec2/instance-types/i4g/) - [I4i](https://aws.amazon.com/ec2/instance-types/i4i/) - [I3](https://aws.amazon.com/ec2/instance-types/i3/) - [I3en](https://aws.amazon.com/ec2/instance-types/i3en/) - Compute optimized: Compute-optimized instances deliver cost-effective high performance at a low price per compute ratio for running advanced compute-intensive workloads. - [C5d](https://aws.amazon.com/ec2/instance-types/c5/) - [C5ad](https://aws.amazon.com/ec2/instance-types/c5/) ### [](#azure)Azure AKS often defaults to the ext4 file system. Use XFS instead where possible. - General purpose: General purpose VM sizes provide balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers. - [Standard\_D2d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) - [Standard\_D4d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) - [Standard\_D32d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) ### [](#google)Google GKE often defaults to the ext4 file system. Use XFS instead where possible. - General purpose: The general-purpose machine family has the best price-performance with the most flexible vCPU to memory ratios, and provides features that target most standard and cloud-native workloads. - [C3 machine series with local SSD](https://cloud.google.com/compute/docs/general-purpose-machines#c3-with-local-ssd) - [N2 machine series](https://cloud.google.com/compute/docs/general-purpose-machines#n2_series) - [N2D machine series](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) - Memory optimized: The memory-optimized machine family provides the most compute and memory resources of any Compute Engine machine family offering. They are ideal for workloads that require higher memory-to-vCPU ratios than the high-memory machine types in the general-purpose N1 machine series. - [M3 machine series](https://cloud.google.com/compute/docs/memory-optimized-machines#m3_series) - Compute optimized: Compute-optimized VM instances are ideal for compute-intensive and high-performance computing (HPC) workloads. - [C2D machine series](https://cloud.google.com/compute/docs/compute-optimized-machines#c2d_series) - [C2 machine series](https://cloud.google.com/compute/docs/compute-optimized-machines#c2_machine_types) ## [](#next-steps)Next steps After meeting these requirements, proceed to: - [Deploy Redpanda for production](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/) - [Validate production readiness](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness/) with the comprehensive checklist ## [](#suggested-reading)Suggested reading - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) - [Sizing Guidelines](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing/) - Resources for Kubernetes managed services: ### GKE - [Local SSDs](https://cloud.google.com/compute/docs/disks/local-ssd) - [About local SSDs for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/local-ssd) - [Provision ephemeral storage with local SSDs](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd) ### EKS - [Instance store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-volumes.html) - [Choosing an Amazon EC2 instance type](https://docs.aws.amazon.com/eks/latest/userguide/choosing-instance-type.html) - [EKS Persistent Volumes for Instance Store](https://aws.amazon.com/blogs/containers/eks-persistent-volumes-for-instance-store/) ### AKS - [AKS storage](https://docs.microsoft.com/en-us/azure/aks/concepts-storage) - [Using NVMe instances in Azure Kubernetes Service](https://medium.com/cooking-with-azure/using-nvm-e-instances-in-azure-kubernetes-service-40c587dbd67b) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 46: Deploy a Stretch Cluster on Kubernetes **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters.md --- # Deploy a Stretch Cluster on Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy a Stretch Cluster on Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-stretch-clusters page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-stretch-clusters.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc description: Deploy a Kafka-compatible stretch cluster with Redpanda across multiple Kubernetes clusters for multi-region high availability and automatic failover. page-topic-type: how-to personas: platform_operator, platform_engineer learning-objective-1: Install multicluster operators with bootstrapped TLS certificates across Kubernetes clusters learning-objective-2: Deploy StretchCluster and RedpandaBrokerPool resources to form a single Redpanda cluster learning-objective-3: Diagnose and resolve common Stretch Cluster deployment issues page-git-created-date: "2026-07-22" page-git-modified-date: "2026-08-24" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Get high availability across regions or data centers by running a single logical Redpanda cluster, called a Stretch Cluster, across multiple Kubernetes clusters. A Stretch Cluster gives Apache Kafka® workloads synchronous multi-region replication: it fails over automatically through Raft consensus, you manage it as a single unit, and existing Kafka clients and applications work with it unchanged through Redpanda’s Kafka-compatible API. > 💡 **TIP** > > To compare this topology with the other deployment options, see [Choose Your Deployment](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-choose-deployment/). For asynchronous disaster recovery between two independent clusters, see [Shadowing Overview](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/). After reading this page, you will be able to: - Install multicluster operators with bootstrapped TLS certificates across Kubernetes clusters - Deploy StretchCluster and RedpandaBrokerPool resources to form a single Redpanda cluster - Diagnose and resolve common Stretch Cluster deployment issues A Stretch Cluster deployment includes: - **StretchCluster resource**: Cluster-wide configuration, including the Redpanda version, listeners, and networking mode. - **RedpandaBrokerPool resources**: Groups of brokers in each Kubernetes cluster, with per-pool replica counts, resources, and storage. - **Multicluster operators**: One Redpanda Operator in each Kubernetes cluster. The operators coordinate through Raft consensus to manage the Stretch Cluster as a single unit. The rpk commands for this operator plane live under [`rpk k8s multicluster`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster/). - **Cross-cluster networking**: Direct broker-to-broker communication across Kubernetes clusters for Raft replication. ![Three Kubernetes clusters, each running a Redpanda Operator and a RedpandaBrokerPool. The operators peer over TLS to form a Raft group; the brokers replicate partitions to each other through Raft.](https://docs.redpanda.com/streaming/current/shared/_images/stretch-cluster-architecture.svg) Figure 1. Stretch Cluster architecture The operators form the control plane: they coordinate through their own Raft group over TLS and manage the StretchCluster as a single unit. The brokers form the data plane: they replicate partition data directly with each other across clusters, without going through the operators. This design protects the control plane against split brain. You install every operator with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes. Operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. Leader election happens at two levels. Within each Kubernetes cluster, operator replicas elect a local leader through a standard Kubernetes Lease, so you can run multiple operator replicas per cluster for high availability. Across clusters, only each cluster’s local leader participates in the Raft group, representing its cluster with a single identity. > 📝 **NOTE** > > The peer list is fixed at installation time. To add or remove a participating cluster, rerun the bootstrap command with the new set of contexts, then upgrade every operator installation with the updated `multicluster.peers` list and restart the operator Deployments. ## [](#limitations)Limitations - **RedpandaBrokerPool scope**: RedpandaBrokerPool works only with StretchCluster resources. The CRD is installed only in multicluster mode, and standalone Redpanda resources cannot use it. To manage broker pools in a standalone cluster, use the beta [NodePool resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-node-pools/) instead. - **Tiered Storage**: Not supported on Stretch Clusters in this release. This also means that features that depend on Tiered Storage are unavailable, including [Iceberg topics](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/) and [Cloud Topics](https://docs.redpanda.com/streaming/current/manage/kubernetes/cloud-topics/k-cloud-topics/). - **Distance**: Keep all locations on the same continent. Transoceanic deployments are technically possible, but produce latency degrades with distance, and pairwise latencies over the documented budgets cause broker liveness failures. See [Prerequisites](#prerequisites). ## [](#prerequisites)Prerequisites Before deploying a Stretch Cluster, ensure you have: - **Redpanda Operator 26.2 or later** on each Kubernetes cluster, installed in multicluster mode. Multicluster mode installs the StretchCluster and RedpandaBrokerPool CRDs. See [Install multicluster operators](#install-multicluster-operators). - **Redpanda Enterprise Edition license**: The multicluster operator requires a valid license to start. Contact your Redpanda account team or [request a trial license](https://redpanda.com/try-redpanda). - **rpk with the `k8s` plugin**: The [rpk CLI](https://docs.redpanda.com/streaming/current/reference/rpk/) version 26.2 or later, with the [`k8s` plugin](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s/) installed: ```bash rpk k8s install ``` The `k8s` plugin is a managed rpk plugin: it also downloads automatically, with checksum verification, the first time you run any `rpk k8s` command. - **At least three Kubernetes clusters, running a [supported Kubernetes version](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/#compatibility-matrix)**: The operators form a Raft group that needs a quorum of `⌊n/2⌋ + 1` members to elect a leader. With three clusters, the deployment tolerates the loss of one cluster. With only two clusters, the loss of either cluster stops the operator control plane, so two-cluster deployments are not recommended. Stretch Clusters work on managed Kubernetes services, including Amazon EKS, Google GKE, and Azure AKS, and on self-managed clusters. - **Cross-cluster networking**: One of the networking modes described in [Choose a networking mode](#choose-a-networking-mode). - **Network requirements**: - Pod-to-pod connectivity between all clusters. - Inter-cluster network latency under 50 ms for optimal performance. Broker liveness checks begin to fail above roughly 100 ms, even with WAN tuning. In practice, this means choosing regions on the same continent, for example AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget. For replication across greater distances, use [Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/) instead. - Sufficient bandwidth for cross-cluster replication traffic. - **Kubernetes API access**: Each operator must be able to reach the Kubernetes API servers of all peer clusters, and you need a kubeconfig with contexts for all participating clusters to run the bootstrap command. - **cert-manager**: [cert-manager](https://cert-manager.io/docs/installation/) installed on all clusters. The operator uses cert-manager to issue TLS certificates for the Redpanda listeners. > 📝 **NOTE** > > You do not need to create TLS certificates for inter-operator communication yourself. The [`rpk k8s multicluster bootstrap`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-bootstrap/) command generates an ECDSA P-256 certificate authority (CA) and per-operator certificates automatically. ## [](#choose-a-networking-mode)Choose a networking mode Brokers in a Stretch Cluster replicate data and maintain Raft consensus across Kubernetes clusters, so every broker must be reachable from every other cluster. Set the mode in the StretchCluster’s `spec.networking.crossClusterMode` field: | Mode | How it works | Requirements | | --- | --- | --- | | mesh (default) | Per-pod Services use selectors, and a service mesh such as Cilium Cluster Mesh or Istio mirrors the Service endpoints across clusters. | Service mesh installed on all clusters with multi-cluster federation configured. | | flat | The operator renders per-pod Services headless, without selectors, and manages their endpoints with pod IPs directly. Brokers communicate over routable pod IPs. | CNI configuration that routes pod IPs across clusters, non-overlapping pod CIDR ranges, and firewall rules that allow traffic on the Redpanda ports (by default, 9093 for the Kafka API, 9644 for the Admin API, and 33145 for internal RPC). | | mcs | The operator exports per-pod Services through the Kubernetes Multi-Cluster Services (MCS) API, and brokers advertise addresses in the clusterset.local DNS domain. | An MCS API implementation installed on all clusters, such as GKE multi-cluster Services or Submariner. | Choose `flat` for the best performance when you control the network infrastructure, `mesh` when you already run a service mesh with multi-cluster federation, or `mcs` for a Kubernetes-native approach on platforms that support the MCS API. ### [](#cross-cloud-deployments)Cross-cloud deployments You can stretch a cluster across Kubernetes clusters in different cloud providers, but you must first make the clouds' networks mutually routable. In addition to the general network requirements, cross-cloud deployments need: - **Site-to-site VPN connectivity between all clouds**: A full mesh of IPsec VPN tunnels between the provider networks, for example AWS VPN Gateway, Google Cloud HA VPN, and Azure VPN Gateway, with BGP or static routes advertising each cloud’s CIDR ranges. Node IPs in every cloud must be routable from the others. - **Non-overlapping CIDR ranges** for the VPC or VNet networks and for the pod networks in every cloud. - **Pairwise latency under 100 ms** between all regions, which means choosing regions on the same continent. Broker liveness checks fail at higher latencies, even with the WAN tuning shown in this page’s StretchCluster example. - **A CNI that supports cross-cluster networking**, such as Cilium with Cluster Mesh (`mesh` mode), installed in place of each cloud’s default CNI. Avoid double encryption: if the VPN layer already encrypts all cross-cloud traffic, don’t also enable CNI-level node encryption, such as WireGuard, on the same path. > 📝 **NOTE** > > Cross-cloud egress is billed at each provider’s internet egress rate, which is typically several times the cost of same-cloud, cross-region traffic. Factor replication traffic into your cost planning. ## [](#install-multicluster-operators)Install multicluster operators Install the Redpanda Operator in multicluster mode on each Kubernetes cluster. 1. Install cert-manager on each cluster, if it’s not already installed: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update for ctx in ; do helm install cert-manager jetstack/cert-manager \ --kube-context "$ctx" \ --namespace cert-manager \ --create-namespace \ --set crds.enabled=true done ``` Replace ``, ``, and `` with the kubeconfig context names of your clusters. To list your contexts, run `kubectl config get-contexts`. 2. Create a Secret containing your Enterprise Edition license on each cluster: ```bash for ctx in ; do kubectl create namespace redpanda --context "$ctx" --dry-run=client -o yaml \ | kubectl apply --context "$ctx" -f - kubectl create secret generic redpanda-license \ --namespace redpanda \ --context "$ctx" \ --from-file=license.key= done ``` 3. Bootstrap TLS certificates and peer configuration: ```bash rpk k8s multicluster bootstrap \ --context \ --context \ --context \ --namespace redpanda \ --loadbalancer ``` This command: - Generates an ECDSA P-256 CA, valid for 10 years, and an individual certificate for each operator, valid for one year. - Provisions a LoadBalancer Service named `-multicluster-peer` in each cluster for operator peer communication on port 9443, and includes the resolved LoadBalancer addresses in the certificate Subject Alternative Names (SANs). - Creates the TLS and kubeconfig Secrets that the operators read at startup, in the given namespace on each cluster. Every operator caches kubeconfigs for all peer clusters, so a newly elected leader can manage resources in the other clusters immediately. When the command completes, it prints a `multicluster.peers` block with each cluster’s name and peer address. Use these values in the Helm installation. > 📝 **NOTE** > > The bootstrap command uses your kubeconfig context names as the cluster names, which also become the prefixes of the generated Secrets and Services. To use different names, such as `us-east`, pass `--name-override =` for each cluster. Bootstrap command flags | Flag | Description | | --- | --- | | --context | Kubeconfig context of a participating cluster. Repeat for each cluster. | | --kubeconfig | Path to a kubeconfig file. If you omit --context, all contexts in the file are used. | | --namespace | Namespace for the generated Secrets and Services. Default: redpanda. | | --loadbalancer | Provision a LoadBalancer Service per cluster for peer communication and include its external address in the certificate SANs. Without this flag, use --dns-override to provide the peer addresses. | | --loadbalancer-timeout | How long to wait per cluster for a LoadBalancer address. Default: 10m. | | --dns-override | Override or add a peer address for a cluster, in context=address format. Repeat for each override. | | --name-override | Override the cluster name for a context, in context=name format. Defaults to the context name. Repeat for each override. | | --create-namespace | Create the namespace if it doesn’t exist. Default: true. | | --organization | Organization field for the generated certificates. Default: Redpanda. | | --output=yaml | Emit the generated resources as YAML instead of applying them, for GitOps workflows. Use with --output-dir to write one file per cluster, which is required when emitting TLS Secrets because each cluster’s private key must not be applied to any other cluster. | 4. Get the addresses required for the Helm installation. The bootstrap command already printed the peer addresses. To retrieve them again later, read them from the LoadBalancer Services: ```bash for ctx in ; do kubectl get service "-multicluster-peer" \ --namespace redpanda \ --context "$ctx" \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}{"\n"}' done ``` You also need the Kubernetes API server address of each cluster: ```bash kubectl cluster-info --context ``` 5. Install the Redpanda Operator with multicluster mode enabled on the first cluster: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda/operator \ --kube-context \ --namespace redpanda \ --version v26.2.2 \ --set fullnameOverride= \ --set crds.enabled=true \ --set multicluster.enabled=true \ --set multicluster.name= \ --set multicluster.apiServerExternalAddress= \ --set-json 'multicluster.peers=[{"name":"","address":""},{"name":"","address":""},{"name":"","address":""}]' \ --set enterprise.licenseSecretRef.name=redpanda-license \ --set enterprise.licenseSecretRef.key=license.key ``` Replace the placeholders: - ``: The cluster’s name from the bootstrap output. This is the kubeconfig context name unless you passed `--name-override`. - ``: The API server address from `kubectl cluster-info`. It must be reachable from the other clusters. - ``: The peer addresses from the bootstrap output. > ❗ **IMPORTANT** > > The `fullnameOverride` and `multicluster.name` values must match the cluster’s name from the bootstrap output. The operator reads the Secrets that the bootstrap command created, and their names are derived from the cluster name. The `multicluster.peers` list must contain all clusters, including the one you’re installing on, and must be identical in every installation. The peer addresses are the LoadBalancer addresses from the bootstrap command, not the API server addresses. 6. Repeat the installation for the other clusters, changing only the release name, `--kube-context`, `fullnameOverride`, `multicluster.name`, and `multicluster.apiServerExternalAddress`. 7. Verify that the operators are running: ```bash for ctx in ; do kubectl get pods --namespace redpanda --context "$ctx" done ``` Each operator Pod shows `1/1` in the `READY` column after the operators form a Raft group. If a Pod does not become ready, see [Troubleshooting](#troubleshooting). ## [](#create-the-stretchcluster-and-broker-pools)Create the StretchCluster and broker pools After the operators are running, deploy a StretchCluster resource and one RedpandaBrokerPool for each Kubernetes cluster. 1. (Recommended) Create a Secret containing SASL superuser credentials on each cluster: ```bash ADMIN_PASSWORD=$(openssl rand -base64 24) echo "Save this password: ${ADMIN_PASSWORD}" for ctx in ; do kubectl create secret generic redpanda-users \ --namespace redpanda \ --context "$ctx" \ --from-literal=users.txt="admin:${ADMIN_PASSWORD}:SCRAM-SHA-256" \ --from-literal=password="${ADMIN_PASSWORD}" done ``` The `users.txt` key defines the cluster’s superusers in `username:password:mechanism` format, with one user per line. The `password` key holds the same password on its own, so that the User resource in a later step can reference it. 2. Create a StretchCluster resource. Save the following to `stretchcluster.yaml`. You apply this same manifest to every participating cluster, and the spec must be identical everywhere: the operators compare the spec across clusters before reconciling, and if any copy differs, they block changes until the specs are aligned. See [StretchCluster specs differ across clusters](#stretchcluster-specs-differ-across-clusters). ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: StretchCluster metadata: name: cluster namespace: redpanda spec: networking: crossClusterMode: mesh (1) auth: (2) sasl: enabled: true mechanism: SCRAM-SHA-256 secretRef: redpanda-users storage: persistentVolume: enabled: true size: 100Gi resources: cpu: cores: "4" memory: container: max: 8Gi enterprise: licenseSecretRef: name: redpanda-license key: license.key config: cluster: # Five replicas spread over three locations keep a Raft majority # through the loss of any one location. default_topic_replications: 5 # Wait 10 min before relocating replicas off an unavailable broker # to avoid cross-region rebalancing on brief network blips. partition_autobalancing_node_availability_timeout_sec: 600 # Increase the liveness probe interval to accommodate WAN latency. node_status_interval: 1000 # Increase Raft timeouts for WAN latency. raft_heartbeat_timeout_ms: 5000 replicate_append_timeout_ms: 5000 # Cache health metadata longer to reduce cross-cluster API calls. health_monitor_max_metadata_age: 60000 ``` | 1 | mesh is the default. Set flat for direct pod routing or mcs for the Multi-Cluster Services API. See Choose a networking mode. | | --- | --- | | 2 | Omit the auth block to deploy without SASL authentication. secretRef references the Secret created in the previous step.For all StretchCluster fields, see the StretchCluster reference.Apply the manifest to every cluster:for ctx in ; do kubectl apply -f stretchcluster.yaml --context "$ctx" done | 3. Create one RedpandaBrokerPool for each Kubernetes cluster. Each RedpandaBrokerPool defines a group of brokers in one Kubernetes cluster and binds to the StretchCluster through `spec.clusterRef`. Redpanda Data recommends five brokers distributed 2/2/1 across the three Kubernetes clusters, with `default_topic_replications: 5`. The odd broker count means the cluster always keeps a clean three-of-five majority through the loss of any one location, with the single-broker location acting as a tiebreaker. With five replicas on five brokers, every broker also holds a replica of every partition, so any surviving location holds a complete copy of the data even if the cluster loses quorum. Save the following to `brokerpool-cluster1.yaml`: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: RedpandaBrokerPool metadata: name: pool- namespace: redpanda spec: clusterRef: (1) group: cluster.redpanda.com kind: StretchCluster name: cluster replicas: 2 # 2/2/1 across the three clusters image: (2) repository: docker.redpanda.com/redpandadata/redpanda tag: v26.2.2 tls: (3) enabled: true certs: default: caEnabled: true services: perPod: (4) local: annotations: service.cilium.io/global: "true" remote: annotations: service.cilium.io/global: "true" ``` > 💡 **TIP** > > To spread partition replicas across failure domains within each Kubernetes cluster, set `spec.rackAwareness.enabled: true` on the pool. Rack awareness requires the operator Helm chart to be installed with `rbac.createAdditionalControllerCRs=true`. See [Enable Rack Awareness in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rack-awareness/). 4. Verify that the StretchCluster is ready and the pools are bound: ```bash kubectl get stretchcluster --namespace redpanda --context kubectl get redpandabrokerpool --namespace redpanda --context ``` The StretchCluster’s `READY` column shows `True` when the cluster is fully operational. Each cluster creates a StatefulSet and Pods for its local pool. 5. If you enabled SASL, create the superuser’s credentials with a User resource. The `users.txt` Secret defines which users are superusers, but does not create their login credentials. Save the following to `user-admin.yaml` and apply it to one cluster: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: User metadata: name: admin namespace: redpanda spec: cluster: clusterRef: group: cluster.redpanda.com kind: StretchCluster name: cluster authentication: type: scram-sha-256 password: valueFrom: secretKeyRef: name: redpanda-users key: password ``` ```bash kubectl apply -f user-admin.yaml --context kubectl get user --namespace redpanda --context ``` The `SYNCED` column shows `True` when the user is created. For more details about the User resource, see [Manage Users and ACLs with the Redpanda Operator](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-user-controller/). 6. After all brokers are running, verify that they form a single cluster: ```bash kubectl exec --namespace redpanda --context -it -- \ rpk cluster info -X user=admin -X pass= -X sasl.mechanism=SCRAM-SHA-256 ``` The output lists the brokers from all pools across all Kubernetes clusters. If you deployed without SASL, omit the `-X` flags. ## [](#understand-failure-scenarios)Understand failure scenarios With the recommended 2/2/1 layout and `default_topic_replications: 5`, at most two of any partition’s five replicas share a location, so the deployment behaves as follows: - **One location down**: Every partition keeps a three-of-five majority, leaders re-elect automatically (along your leader preference, if set), and clients keep working with zero data loss. After the configured timeouts, continuous partition balancing decommissions the unreachable brokers (see [Automatically eject ghost brokers](#automatically-eject-ghost-brokers)). When the location returns, its brokers rejoin as fresh brokers, replication is fully restored, and pinned leaders migrate back. - **Two or more locations down**: Partitions lose their Raft majority and the cluster stops serving writes, and the operator control plane loses quorum and stops reconciling. This is an availability outage, not data loss: acknowledged writes persist on the surviving disks, and the cluster resumes when a location is restored. If a lost location cannot be restored, contact Redpanda Support before attempting recovery. - **Beyond quorum loss**: A Stretch Cluster cannot survive the simultaneous loss of a majority of its locations. For protection against failures of that scale, or for an isolated recovery copy, use [Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/) between independent clusters. Failing over to a shadow cluster costs a replication-lag window (a recovery point objective, or RPO, greater than zero) and requires repointing clients. For how the two features combine, see [Stretch Clusters and Shadowing](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-choose-deployment/#stretch-clusters-and-shadowing). ## [](#optimize-for-region-failures)Optimize for region failures To keep a Stretch Cluster performing predictably through a region outage, enable rack awareness on every broker pool, pin partition leaders near your clients, and let Redpanda eject brokers that are permanently gone. ### [](#enable-rack-awareness-on-broker-pools)Enable rack awareness on broker pools Rack awareness is the foundation for both leader pinning and safe replica placement. Set it on each RedpandaBrokerPool: ```yaml spec: rbac: enabled: true (1) rackAwareness: enabled: true nodeAnnotation: topology.kubernetes.io/zone (2) ``` | 1 | Rack awareness requires per-pool RBAC, and the operator Helm chart must be installed with rbac.createAdditionalControllerCRs=true. | | --- | --- | | 2 | The Kubernetes Node label whose value becomes each broker’s rack ID. The default is topology.kubernetes.io/zone, which gives each availability zone its own rack. To treat each Kubernetes cluster or region as one rack, use a label whose value is shared by all nodes in that cluster, such as topology.kubernetes.io/region. | ### [](#pin-partition-leaders-near-your-clients)Pin partition leaders near your clients Without a leader preference, partition leaders can move to any region after a failure, including regions far from your producers. For example, when a broker fails and its replacement starts in a distant region, leaders can end up there. Pin leaders to the racks closest to your clients with the `[storage.tiered.config.default_leaders_preference](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#default_leaders_preference)` cluster property in the StretchCluster spec: ```yaml spec: config: cluster: default_leaders_preference: "ordered_racks:," (1) ``` | 1 | Rack IDs in priority order, which are the values of the node label configured in rackAwareness.nodeAnnotation. Redpanda places leaders in the first listed rack when it’s available and fails over down the list in order. ordered_racks requires Redpanda 26.1 or later. To balance leaders across racks without a priority order, use racks:, instead. | | --- | --- | Leaders fail over along the preference list when the preferred racks are unavailable and move back automatically when they recover. To let consumers read from replicas in their own region instead of fetching from a leader in another region, pair leader pinning with [follower fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/). For details, see [Configure Leader Pinning](https://docs.redpanda.com/streaming/current/develop/produce-data/leader-pinning/). ### [](#automatically-eject-ghost-brokers)Automatically eject ghost brokers When an entire region is lost, its brokers remain in the cluster as unreachable "ghost" brokers until they are decommissioned. With continuous partition balancing, Redpanda decommissions them automatically after the configured timeouts, so you don’t have to clean up stale brokers manually after a regional outage: ```yaml spec: config: cluster: partition_autobalancing_mode: continuous (1) # Wait before moving partitions off an unavailable broker. partition_autobalancing_node_availability_timeout_sec: 600 # After this additional timeout, decommission the broker entirely. partition_autobalancing_node_autodecommission_timeout_sec: 3600 (2) ``` | 1 | Automatic ejection applies only in continuous mode, which is an Enterprise feature. | | --- | --- | | 2 | The total time before an unreachable broker is ejected is approximately the availability timeout plus the auto-decommission timeout. | > ⚠️ **CAUTION** > > Keep these timeouts comfortably longer than any transient cross-region network disruption you expect. If they’re too short, a brief WAN outage can trigger cross-region partition rebalancing or even decommission healthy brokers. ## [](#upgrade-the-operators)Upgrade the operators Upgrade the multicluster operators one Kubernetes cluster at a time. The operator Raft group needs a quorum of `⌊n/2⌋ + 1` members, so upgrading two operators at once in a three-cluster deployment drops quorum and stalls reconciliation until the operators rejoin. 1. Before you start, verify that the Raft group is healthy on every cluster with [`rpk k8s multicluster status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-status/): ```bash rpk k8s multicluster status \ --context \ --context \ --context \ --namespace redpanda ``` 2. Run `helm upgrade` on the first cluster, then wait for that cluster’s operator Deployment to roll and for the Raft group to report healthy again. 3. Repeat for each remaining cluster, one at a time. The operators are not on the data path. While an operator restarts, the brokers continue to serve traffic; only reconciliation of new changes pauses. ## [](#troubleshooting)Troubleshooting Run the diagnostic command to check the health of a multicluster deployment: ```bash rpk k8s multicluster status \ --context \ --context \ --context \ --namespace redpanda ``` This command validates: - Operator Pod health and Deployment configuration - TLS certificate validity and SANs - Raft consensus state and leader agreement - Unique cluster names across all peers - Peer configuration consistency - CA certificate consistency across clusters To collect a diagnostics bundle for Redpanda Support, run [`rpk k8s multicluster bundle`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-bundle/) with the same connection flags. ### [](#operators-not-forming-a-raft-group)Operators not forming a Raft group If operator Pods remain in the `0/1 READY` state and logs show `unreachable` peer messages: 1. Check the operator logs for TLS or peer connection errors: ```bash kubectl logs --namespace redpanda --context -l app.kubernetes.io/name=operator --tail=100 ``` 2. Verify that the bootstrap-created Secrets exist on each cluster: ```bash kubectl get secrets --namespace redpanda --context | grep multicluster ``` 3. Check the certificate contents and SANs: ```bash kubectl get secret --namespace redpanda --context \ -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout ``` The SANs must include the peer addresses configured in `multicluster.peers`. If the LoadBalancer addresses changed after bootstrap, rerun the bootstrap command to regenerate the certificates, then restart the operator Deployments on all clusters. Each bootstrap run generates a new CA, and the operators load their certificates only at startup. 4. Verify network connectivity to the peer Services on port 9443 from inside each cluster. ### [](#redpandabrokerpools-not-binding-to-the-stretchcluster)RedpandaBrokerPools not binding to the StretchCluster If RedpandaBrokerPool resources don’t report `Bound`: 1. Check the StretchCluster’s conditions for error messages: ```bash kubectl get stretchcluster --namespace redpanda -o yaml ``` 2. Verify that each pool’s `clusterRef` matches the StretchCluster’s name: ```bash kubectl get redpandabrokerpool --namespace redpanda \ -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.clusterRef.name}{"\n"}{end}' ``` 3. Check the operator logs for role-based access control (RBAC) errors: ```bash kubectl logs --namespace redpanda -l app.kubernetes.io/name=operator | grep -i "forbidden\|unauthorized" ``` ### [](#stretchcluster-specs-differ-across-clusters)StretchCluster specs differ across clusters If you change the StretchCluster on one cluster but not the others, the operators detect the drift and block all changes until the specs are aligned. While blocked, the operators still remediate local failures, such as restarting downed pods, but do not apply any spec changes. The drift is reported in the StretchCluster’s `SpecSynced` condition and in the operator logs: ```bash kubectl get stretchcluster --namespace redpanda --context \ -o jsonpath='{.status.conditions[?(@.type=="SpecSynced")].message}' ``` The message names the clusters and the top-level fields that differ, for example `StretchCluster .spec differs on clusters: (fields: config)`. To recover, apply the same manifest to every cluster. ### [](#bootstrap-user-password-mismatch)Bootstrap user password mismatch If the StretchCluster doesn’t deploy brokers and the operator logs show `bootstrap user password mismatch`, the operators created the internal `-bootstrap-user` Secret independently with different passwords. This can happen when the StretchCluster manifest is applied to all clusters at nearly the same time. Find the Raft leader with `rpk k8s multicluster status`, compare the Secret across clusters, then delete the copies that differ from the leader’s cluster and let the operator recreate them: ```bash for ctx in ; do printf '%s: ' "$ctx" kubectl get secret cluster-bootstrap-user --namespace redpanda --context "$ctx" \ -o jsonpath='{.data.password}' | sha256sum done kubectl delete secret cluster-bootstrap-user --namespace redpanda --context ``` If the StretchCluster still reports no brokers after the Secrets converge, restart the operator Deployments to clear the stalled reconciliation: ```bash for ctx in ; do kubectl rollout restart deployment --namespace redpanda --context "$ctx" done ``` ### [](#broker-crash-loops-after-its-region-recovers)Broker crash-loops after its region recovers If a region comes back online after its broker was automatically decommissioned (see [Automatically eject ghost brokers](#automatically-eject-ghost-brokers)), the returning broker cannot rejoin with its old identity: the pod starts but never becomes ready, and its logs show that the broker was removed from the cluster. The operator recovers from this automatically. When the pod stays not-ready past a threshold (five minutes by default) and its on-disk identity conflicts with the cluster’s authoritative broker list, the operator wipes the stale disk by deleting the pod’s PersistentVolumeClaim and Pod. The broker reschedules onto a fresh disk, joins with a new broker ID, and continuous balancing moves partition replicas back onto it. To change the threshold, set the operator’s `--wipe-stale-disk-after` flag. A negative duration disables the wipe. To recover immediately, or if you disabled the automatic wipe, delete the broker’s PersistentVolumeClaim and Pod yourself: ```bash kubectl delete pvc --namespace redpanda --context --wait=false kubectl delete pod --namespace redpanda --context ``` ### [](#brokers-cannot-communicate-across-clusters)Brokers cannot communicate across clusters If brokers in different Kubernetes clusters cannot reach each other: 1. Get a broker Pod IP from another cluster and test connectivity from the local cluster: ```bash kubectl get pods --namespace redpanda --context -o wide kubectl run -it --rm debug --image=busybox --restart=Never -- \ nc -zv 33145 ``` 2. Check for NetworkPolicies that block cross-cluster traffic: ```bash kubectl get networkpolicy --namespace redpanda ``` 3. For `flat` mode, verify that routes to the remote pod CIDRs exist. For `mesh` mode, verify that the per-pod Services are mirrored to the remote clusters. For `mcs` mode, verify that `ServiceExport` and `ServiceImport` resources exist for the per-pod Services. 4. Review the broker logs for connection errors: ```bash kubectl logs --namespace redpanda | grep -i "connection\|error" ``` ## [](#next-steps)Next steps - [Enable rack awareness](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rack-awareness/) to optimize replica placement across failure domains within each Kubernetes cluster. - [Monitor your Stretch Cluster](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-stretch-clusters/) to scrape the operator metrics and deploy the prebuilt Grafana dashboard. - [Configure leader pinning](https://docs.redpanda.com/streaming/current/develop/produce-data/leader-pinning/) to keep partition leaders in specific regions for latency optimization. - [Configure cluster balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-balancing/) for automatic partition rebalancing across brokers. ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 47: Tune Kubernetes Worker Nodes for Production **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers.md --- # Tune Kubernetes Worker Nodes for Production > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Tune Kubernetes Worker Nodes for Production latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/k-tune-workers page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/k-tune-workers.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/k-tune-workers.adoc description: To get the best performance from your hardware, set Redpanda to production mode and run the autotuner tool. The autotuner identifies your hardware configuration and tunes itself to give you the best performance. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-07-17" support-status: supported --- To get the best performance from your hardware, set Redpanda to production mode on each worker node and run the [autotuner tool](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/). The autotuner identifies the hardware configuration of your worker node and optimizes the Linux kernel to give you the best performance. ## [](#prerequisites)Prerequisites Make sure that your current Linux user has root privileges. The autotuner requires privileged access to the Linux kernel settings. ## [](#install-redpanda)Install Redpanda To run the autotuner, you need to install the Redpanda binary on each worker node. ### Fedora/RedHat ```bash # Run the setup script to download and install the repo curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | sudo -E bash && \ # Use yum to install redpanda sudo yum install redpanda -y ``` ### Debian/Ubuntu ```bash # Run the setup script to download and install the repo curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | sudo -E bash && \ # Use apt to install redpanda sudo apt install redpanda -y ``` ## [](#run-the-autotuner)Run the autotuner To tune the Linux kernel of your worker nodes, run the autotuner on each worker node that will host a Redpanda broker. 1. Set Redpanda to production mode: ```bash sudo rpk redpanda mode production ``` 2. Run the autotuner: ```bash sudo rpk redpanda tune all ``` Expected output: ```none TUNER APPLIED ENABLED SUPPORTED ERROR aio_events true true true ballast_file true true true clocksource true true true coredump false false true cpu true true true disk_irq true true true disk_nomerges true true true disk_scheduler true true true disk_write_cache false true false Disk write cache tuner is only supported in GCP fstrim false false true net true true true swappiness true true true transparent_hugepages false false true ``` Changes to the Linux kernel are not persisted. If a worker node restarts, make sure to run `sudo rpk redpanda tune all` on it again. > 💡 **TIP** > > You can use a privileged DaemonSet to schedule the autotuner on each worker node that runs a Redpanda broker. Apply taints to Nodes that successfully complete the autotuner command. Use tolerations on your Pods so that they are scheduled only on tuned worker nodes. For details about the autotuner, including how to enable or disable an individual tuner, see the [rpk redpanda tune](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/) command reference. ## [](#io)Generate optimal I/O configuration settings After tuning the Linux kernel, you can optimize Redpanda for the I/O capabilities of your worker node by using `rpk` to run benchmarks that capture its read/write IOPS and bandwidth capabilities. After running the benchmarks `rpk` saves the results to an I/O configuration file (`io-config.yaml`) that Redpanda can read upon startup to optimize itself for the worker node. > 📝 **NOTE** > > Unlike the autotuner, it isn’t necessary to run `rpk iotune` each time Redpanda is started, as its I/O output configuration file can be reused for each worker node that runs on the same type of hardware. 1. Run the I/O benchmark on your worker node: ```bash rpk iotune ``` Example output: `/etc/redpanda/io-config.yaml` ```yaml disks: - mountpoint: / read_iops: 40952 read_bandwidth: 5638210048 write_iops: 6685 write_bandwidth: 1491679488 ``` When this command is successful, the I/O configuration file is saved to `/etc/redpanda/io-config.yaml` by default. 2. Copy the file to your local machine. 3. Create a ConfigMap in the same namespace in which you will deploy Redpanda to store the I/O configuration file: ```bash kubectl create configmap redpanda-io-config --namespace --from-file= ``` You will mount this file onto the Pods that run Redpanda so that Redpanda can read it at startup. See [Deploy Redpanda for Production in Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/). For more details about this procedure, see [Optimize I/O](https://docs.redpanda.com/streaming/current/manage/io-optimization/). ## [](#next-steps)Next steps [Deploy the Redpanda cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/). ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 48: Deploy a Local Development Cluster with kind or minikube **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/local-guide.md --- # Deploy a Local Development Cluster with kind or minikube > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy a Local Development Cluster with kind or minikube latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/kubernetes/local-guide page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/kubernetes/local-guide.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/kubernetes/local-guide.adoc description: Deploy a local Redpanda cluster with Redpanda Console using the Helm chart. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-07-30" support-status: supported --- Deploy a local Redpanda cluster with Redpanda Console using the Helm chart. Explore the essentials of how Redpanda works in Kubernetes and what components are deployed by default. Then, use `rpk` both as an internal client and an external client to interact with your Redpanda cluster from the command line. > ⚠️ **CAUTION: Only for development and testing** > > Only for development and testing > > Do not use kind or minikube for production workloads. Instead, try one of the following environments: > > - [Azure Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/aks-guide/) (AKS) > > - [Elastic Kubernetes Service](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/eks-guide/) (EKS) > > - [Google Kubernetes Engine](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/gke-guide/) (GKE) ## [](#prerequisites)Prerequisites Before you begin, make sure that you have the correct software for your Kubernetes platform: ### kind - [Install `kubectl`](https://kubernetes.io/docs/tasks/tools/). Minimum required Kubernetes version: 1.27.0-0 ```bash kubectl version --client ``` - [Install Helm](https://helm.sh/docs/intro/install/). Minimum required Helm version: 3.10.0 ```bash helm version ``` - [Install kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) - [Install Docker](https://docs.docker.com/get-docker/) ### minikube - [Install `kubectl`](https://kubernetes.io/docs/tasks/tools/). Minimum required Kubernetes version: 1.27.0-0 ```bash kubectl version --client ``` - [Install Helm](https://helm.sh/docs/intro/install/). Minimum required Helm version: 3.10.0 ```bash helm version ``` - [Install minikube](https://minikube.sigs.k8s.io/docs/start/) ## [](#create-a-kubernetes-cluster)Create a Kubernetes cluster In this step, you create one master and three worker nodes (one worker node for each Redpanda broker). ### kind 1. Define a cluster in the `kind.yaml` configuration file: ```bash cat <kind.yaml --- apiVersion: kind.x-k8s.io/v1alpha4 kind: Cluster nodes: - role: control-plane - role: worker - role: worker - role: worker EOF ``` 2. Create the Kubernetes cluster from the configuration file: ```bash kind create cluster --config kind.yaml ``` ### minikube 1. Create the Kubernetes cluster: ```bash minikube start --nodes 4 ``` 2. Prevent applications from being scheduled on the Kubernetes control plane node: ```bash kubectl taint node \ -l node-role.kubernetes.io/control-plane="" \ node-role.kubernetes.io/control-plane=:NoSchedule ``` > 📝 **NOTE** > > The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). ## [](#deploy-redpanda-and-redpanda-console)Deploy Redpanda and Redpanda Console In this step, you deploy Redpanda with self-signed TLS certificates. Redpanda Console is included as a subchart in the Redpanda Helm chart. ### Operator 1. Make sure that you have permission to install custom resource definitions (CRDs): ```bash kubectl auth can-i create CustomResourceDefinition --all-namespaces ``` You should see `yes` in the output. You need these cluster-level permissions to install [cert-manager](https://cert-manager.io/docs/) and Redpanda Operator CRDs in the next steps. 2. Install [cert-manager](https://cert-manager.io/docs/installation/helm/) using Helm: ```bash helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --set crds.enabled=true --namespace cert-manager --create-namespace ``` TLS is enabled by default. The Redpanda Helm chart uses cert-manager to manage TLS certificates by default. 3. Deploy the Redpanda Operator: 1. To deploy in cluster scope, use: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ (1) --set crds.enabled=true (2) ``` | 1 | This flag specifies the exact version of the Redpanda Operator Helm chart to use for deployment. By setting this value, you pin the chart to a specific version, which prevents automatic updates that might introduce breaking changes or new features that have not been tested in your environment. | | --- | --- | | 2 | This flag ensures that the CRDs are installed as part of the Redpanda Operator deployment.This command deploys the Redpanda Operator in cluster scope (default in v25.2+), allowing it to manage Redpanda clusters across multiple namespaces. | 2. To deploy in namespace scope (managing only resources within its deployment namespace), use: ```bash helm upgrade --install redpanda-controller redpanda/operator \ --namespace \ --create-namespace \ --version v26.2.2 \ --set crds.enabled=true \ --set 'additionalCmdFlags=["--namespace="]' (1) ``` | 1 | This flag restricts the Redpanda Operator to manage resources only within the specified namespace. | | --- | --- | 4. Ensure that the Deployment is successfully rolled out: ```bash kubectl --namespace rollout status --watch deployment/redpanda-controller-operator ``` deployment "redpanda-controller-operator" successfully rolled out 5. Install a [Redpanda custom resource](https://docs.redpanda.com/streaming/current/reference/k-crd/) in the same namespace as the Redpanda Operator: `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: clusterSpec: external: domain: customredpandadomain.local statefulset: initContainers: setDataDirOwnership: enabled: true ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` 6. Wait for the Redpanda Operator to deploy Redpanda using the Helm chart: ```bash kubectl get redpanda --namespace --watch ``` NAME READY STATUS redpanda True Redpanda reconciliation succeeded This step may take a few minutes. You can watch for new Pods to make sure that the deployment is progressing: ```bash kubectl get pod --namespace ``` If it’s taking too long, see [Troubleshoot](#troubleshoot). ### Helm 1. Add the Redpanda Helm chart repository and install cert-manager using Helm: ```bash helm repo add redpanda https://charts.redpanda.com helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager --set crds.enabled=true --namespace cert-manager --create-namespace ``` The Redpanda Helm chart uses cert-manager to manage TLS certificates. 2. Install Redpanda using Helm: ```bash helm repo add redpanda https://charts.redpanda.com/ helm repo update helm install redpanda redpanda/redpanda \ --version 26.2.2 \ --namespace \ --create-namespace \ --set external.domain=customredpandadomain.local \ --set statefulset.initContainers.setDataDirOwnership.enabled=true ``` The installation displays some tips for getting started. 3. Wait for the Redpanda cluster to be ready: ```bash kubectl --namespace rollout status statefulset redpanda --watch ``` When the Redpanda cluster is ready, the output should look similar to the following: ```plain statefulset rolling update complete 3 pods at revision redpanda-8654f645b4... ``` If your cluster remains in a pending state, see [Troubleshoot](#troubleshoot). ## [](#start-streaming)Start streaming Each Redpanda broker comes with `rpk`, which is a CLI tool for connecting to and interacting with Redpanda brokers. You can use `rpk` inside one of the Redpanda broker’s Docker containers to create a topic, produce messages to it, and consume messages from it. 1. Create an alias to simplify the `rpk` commands: ```bash alias internal-rpk="kubectl --namespace exec -i -t redpanda-0 -c redpanda -- rpk" ``` 2. Create a topic called `twitch-chat`: ### Operator 1. Create a [Topic resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/): `topic.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Topic metadata: name: twitch-chat spec: kafkaApiSpec: brokers: - "redpanda-0.redpanda..svc.cluster.local:9093" - "redpanda-1.redpanda..svc.cluster.local:9093" - "redpanda-2.redpanda..svc.cluster.local:9093" tls: caCertSecretRef: name: "redpanda-default-cert" key: "ca.crt" ``` 2. Apply the Topic resource in the same namespace as your Redpanda cluster: ```bash kubectl apply -f topic.yaml --namespace ``` 3. Check the logs of the Redpanda Operator to confirm that the topic was created: ```bash kubectl logs -l app.kubernetes.io/name=operator -c manager --namespace ``` You should see that the Redpanda Operator reconciled the Topic resource. Example output ```json { "level":"info", "ts":"2023-09-25T16:20:09.538Z", "logger":"TopicReconciler.Reconcile", "msg":"Starting reconcile loop", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432" } { "level":"info", "ts":"2023-09-25T16:20:09.581Z", "logger":"TopicReconciler.Reconcile", "msg":"reconciliation finished in 43.436125ms, next run in 3s", "controller":"topic", "controllerGroup":"cluster.redpanda.com", "controllerKind":"Topic", "Topic": { "name":"twitch-chat", "namespace":"" }, "namespace":"", "name":"twitch-chat", "reconcileID":"c0cf9abc-a553-48b7-9b6e-2de3cdfb4432", "result": { "Requeue":false, "RequeueAfter":3000000000 } } ``` ### Helm ```bash internal-rpk topic create twitch-chat ``` Example output: TOPIC STATUS twitch-chat OK 3. Describe the topic: ```bash internal-rpk topic describe twitch-chat ``` Expected output: ```none SUMMARY ======= NAME twitch-chat PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG partition_count 1 DYNAMIC_TOPIC_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG replication_factor 1 DYNAMIC_TOPIC_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG ``` 4. Produce a message to the topic: ```bash internal-rpk topic produce twitch-chat ``` 5. Type a message, then press Enter: ```text Pandas are fabulous! ``` Example output: ```text Produced to partition 0 at offset 0 with timestamp 1663282629789. ``` 6. Press Ctrl+C to finish producing messages to the topic. 7. Consume one message from the topic: ```bash internal-rpk topic consume twitch-chat --num 1 ``` Expected output: Your message is displayed along with its metadata: ```json { "topic": "twitch-chat", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ## [](#explore-your-topic-in-redpanda-console)Explore your topic in Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. In this step, you use port-forwarding to access Redpanda Console on your local network. > 💡 **TIP** > > Because you’re using the Community Edition of Redpanda Console, you should not expose Redpanda Console outside your local network. The Community Edition of Redpanda Console does not provide authentication, and it connects to the Redpanda cluster as superuser. To use the Enterprise Edition, you need a license key. See [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). 1. Expose Redpanda Console to your localhost: ```bash kubectl --namespace port-forward svc/redpanda-console 8080:8080 ``` The `kubectl port-forward` command actively runs in the command-line window. To execute other commands while the command is running, open another command-line window. 2. Open Redpanda Console on [http://localhost:8080](http://localhost:8080). All your Redpanda brokers are listed along with their IP addresses and IDs. 3. Go to **Topics** > **twitch-chat**. The message that you produced to the topic is displayed along with some other details about the topic. 4. Press Ctrl+C in the command-line to stop the port-forwarding process. ## [](#configure-external-access-to-redpanda)Configure external access to Redpanda If you want to connect to the Redpanda cluster with external clients, Redpanda brokers must advertise an externally accessible address that external clients can connect to. External clients are common in Internet of Things (IoT) environments, or if you use external services that do not implement VPC peering in your network. When you created the cluster, you set the `external.domain` configuration to `customredpandadomain.local`, which means that your Redpanda brokers are advertising the following addresses: - `redpanda-0.customredpandadomain.local` - `redpanda-1.customredpandadomain.local` - `redpanda-2.customredpandadomain.local` To access your Redpanda brokers externally, you can map your worker nodes' IP addresses to these domains. > ⚠️ **CAUTION** > > IP addresses can change. If the IP addresses of your worker nodes change, you must update your `/etc/hosts` file with the new mappings. > > In a production environment, it’s a best practice to use ExternalDNS to manage DNS records for your brokers. See [Use ExternalDNS for external access](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-deployment/#externaldns). > 📝 **NOTE** > > These steps work only on Linux operating systems. 1. Add mappings in your `/etc/hosts` file between your worker nodes' IP addresses and their custom domain names: ```bash sudo true && kubectl --namespace get endpoints,node -A -o go-template='{{ range $_ := .items }}{{ if and (eq .kind "Endpoints") (eq .metadata.name "redpanda-external") }}{{ range $_ := (index .subsets 0).addresses }}{{ $nodeName := .nodeName }}{{ $podName := .targetRef.name }}{{ range $node := $.items }}{{ if and (eq .kind "Node") (eq .metadata.name $nodeName) }}{{ range $_ := .status.addresses }}{{ if eq .type "InternalIP" }}{{ .address }} {{ $podName }}.customredpandadomain.local{{ "\n" }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}' | envsubst | sudo tee -a /etc/hosts ``` `/etc/hosts` 203.0.113.3 redpanda-0.customredpandadomain.local 203.0.113.5 redpanda-1.customredpandadomain.local 203.0.113.7 redpanda-2.customredpandadomain.local 2. Save the root certificate authority (CA) to your local file system outside Kubernetes: ```bash kubectl --namespace get secret redpanda-external-root-certificate -o go-template='{{ index .data "ca.crt" | base64decode }}' > ca.crt ``` 3. Install `rpk` on your local Linux machine, not on a Pod: ### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. 4. Configure `rpk` to connect to your cluster using the [pre-configured profile](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` Replace `` with the name that you want to give this `rpk` profile. 5. Test the connection: ```bash rpk cluster info ``` ## [](#explore-the-default-kubernetes-components)Explore the default Kubernetes components By default, the Redpanda Helm chart deploys the following Kubernetes components: - [A StatefulSet](#statefulset) with three Pods. - [One PersistentVolumeClaim](#persistentvolumeclaim) for each Pod, each with a capacity of 20Gi. - [A headless ClusterIP Service and a NodePort Service](#service) for each Kubernetes node that runs a Redpanda broker. - [Self-Signed TLS Certificates](#tls-certificates). ### [](#statefulset)StatefulSet Redpanda is a stateful application. Each Redpanda broker needs to store its own state (topic partitions) in its own storage volume. As a result, the Helm chart deploys a StatefulSet to manage the Pods in which the Redpanda brokers are running. ```bash kubectl get statefulset --namespace ``` Example output: NAME READY AGE redpanda 3/3 3m11s StatefulSets ensure that the state associated with a particular Pod replica is always the same, no matter how often the Pod is recreated. Each Pod is also given a unique ordinal number in its name such as `redpanda-0`. A Pod with a particular ordinal number is always associated with a PersistentVolumeClaim with the same number. When a Pod in the StatefulSet is deleted and recreated, it is given the same ordinal number and so it mounts the same storage volume as the deleted Pod that it replaced. ```bash kubectl get pod --namespace ``` Expected output: ```none NAME READY STATUS RESTARTS AGE redpanda-0 1/1 Running 0 6m9s redpanda-1 1/1 Running 0 6m9s redpanda-2 1/1 Running 0 6m9s redpanda-console-5ff45cdb9b-6z2vs 1/1 Running 0 5m redpanda-configuration-smqv7 0/1 Completed 0 6m9s ``` > 📝 **NOTE** > > The `redpanda-configuration` job updates the Redpanda runtime configuration. ### [](#persistentvolumeclaim)PersistentVolumeClaim Redpanda brokers must be able to store their data on disk. By default, the Helm chart uses the default StorageClass in the Kubernetes cluster to create a PersistentVolumeClaim for each Pod. The default StorageClass in your Kubernetes cluster depends on the Kubernetes platform that you are using. ```bash kubectl get persistentvolumeclaims --namespace ``` Expected output: ```none NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-redpanda-0 Bound pvc-3311ade3-de84-4027-80c6-3d8347302962 20Gi RWO standard 75s datadir-redpanda-1 Bound pvc-4ea8bc03-89a6-41e4-b985-99f074995f08 20Gi RWO standard 75s datadir-redpanda-2 Bound pvc-45c3555f-43bc-48c2-b209-c284c8091c45 20Gi RWO standard 75s ``` ### [](#service)Service The clients writing to or reading from a given partition have to connect directly to the leader broker that hosts the partition. As a result, clients need to be able to connect directly to each Pod. To allow internal and external clients to connect to each Pod that hosts a Redpanda broker, the Helm chart configures two Services: - Internal using the [Headless ClusterIP](#headless-clusterip-service) - External using the [NodePort](#nodeport-service) ```bash kubectl get service --namespace ``` Expected output: ```none NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE redpanda ClusterIP None 5m37s redpanda-console ClusterIP 10.0.251.204 8080 5m redpanda-external NodePort 10.96.137.220 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8080:30081/TCP 5m37s ``` #### [](#headless-clusterip-service)Headless ClusterIP Service The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other. An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster. When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other. ```bash kubectl --namespace exec redpanda-0 -c redpanda -- cat etc/redpanda/redpanda.yaml ``` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: redpanda-0.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-1.redpanda..svc.cluster.local. port: 33145 - host: address: redpanda-2.redpanda..svc.cluster.local. port: 33145 ``` #### [](#nodeport-service)NodePort Service External access is made available by a NodePort service that opens the following ports by default: | Listener | Node Port | Container Port | | --- | --- | --- | | Schema Registry | 30081 | 8081 | | HTTP Proxy | 30082 | 8083 | | Kafka API | 31092 | 9094 | | Admin API | 31644 | 9644 | To learn more, see [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/). ### [](#tls-certificates)TLS Certificates By default, TLS is enabled in the Redpanda Helm chart. The Helm chart uses [cert-manager](https://cert-manager.io/docs/) to generate four Certificate resources that provide Redpanda with self-signed certificates for internal and external connections. Having separate certificates for internal and external connections provides security isolation. If an external certificate or its corresponding private key is compromised, it doesn’t affect the security of internal communications. ```bash kubectl get certificate --namespace ``` NAME READY redpanda-default-cert True redpanda-default-root-certificate True redpanda-external-cert True redpanda-external-root-certificate True - `redpanda-default-cert`: Self-signed certificate for internal communications. - `redpanda-default-root-certificate`: Root certificate authority for the internal certificate. - `redpanda-external-cert`: Self-signed certificate for external communications. - `redpanda-external-root-certificate`: Root certificate authority for the external certificate. By default, all listeners are configured with the same certificate. To configure separate TLS certificates for different listeners, see [TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/). > 📝 **NOTE** > > The Redpanda Helm chart provides self-signed certificates for convenience. In a production environment, it’s best to use certificates from a trusted Certificate Authority (CA) or integrate with your existing CA infrastructure. ## [](#uninstall-redpanda)Uninstall Redpanda When you finish testing Redpanda, you can uninstall it from your Kubernetes cluster. The steps depend on how you installed Redpanda: using the Redpanda Operator or the Redpanda Helm chart. ### Operator Follow the steps in **exact order** to avoid race conditions between the Redpanda Operator’s reconciliation loop and Kubernetes garbage collection. 1. Delete all Redpanda-related custom resources: ```bash kubectl delete users --namespace --all kubectl delete topics --namespace --all kubectl delete schemas --namespace --all kubectl delete redpanda --namespace --all kubectl delete consoles --namespace --all ``` The Redpanda Operator creates a Console resource for each Redpanda resource, so you must delete Console resources too. The Redpanda Operator must still be running when you delete these resources. It removes their finalizers as part of its cleanup. If you uninstall the Redpanda Operator first, any remaining resource keeps its finalizer forever and the CRD deletion step below hangs. 2. Make sure requests for those resources return no results. Do not continue until every command returns `No resources found`: ```bash kubectl get users,topics,schemas,redpandas,consoles --namespace ``` 3. Uninstall the Redpanda Operator Helm release: ```bash helm uninstall redpanda-controller --namespace ``` Helm does not uninstall CRDs by default when using `helm uninstall` to avoid accidentally deleting existing custom resources. 4. Remove the CRDs. 1. List all Redpanda CRDs installed by the operator: ```bash kubectl api-resources --api-group='cluster.redpanda.com' ``` This command displays all CRDs defined by the Redpanda Operator. For example: ```bash NAME SHORTNAMES APIVERSION NAMESPACED KIND redpandas rp cluster.redpanda.com/v1alpha2 true Redpanda schemas sc cluster.redpanda.com/v1alpha2 true Schema topics cluster.redpanda.com/v1alpha2 true Topic users rpu cluster.redpanda.com/v1alpha2 true User ``` 2. Delete the CRDs: ```bash kubectl get crds -o name | grep cluster.redpanda.com | xargs kubectl delete ``` This command lists all CRDs with the `cluster.redpanda.com` domain suffix and deletes them, ensuring only Redpanda CRDs are removed. Helm does not delete CRDs automatically to prevent data loss, so you must run this step manually. 5. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ### Helm If you deployed Redpanda with the Redpanda Helm chart, follow these steps to uninstall it: 1. Uninstall the Helm release: ```bash helm uninstall redpanda --namespace ``` 2. (Optional) Delete any leftover PVCs or Secrets in the namespace: > ⚠️ **CAUTION** > > The following command deletes all PVCs and Secrets in the namespace, which may remove unrelated resources if the namespace is shared with other applications. ```bash kubectl delete pvc,secret --all --namespace ``` ## [](#delete-the-cluster)Delete the cluster To delete your Kubernetes cluster: ### kind ```bash kind delete cluster ``` ### minikube ```bash minikube delete ``` To remove the convenience alias created during the quickstart: ```bash unalias internal-rpk ``` ## [](#troubleshoot)Troubleshoot Before troubleshooting your cluster, make sure that you have all the [prerequisites](#prerequisites). ### [](#helm-v3-18-0-is-not-supported-json-number-error)Helm v3.18.0 is not supported (json.Number error) If you are using Helm v3.18.0, you may encounter errors such as: Error: INSTALLATION FAILED: execution error at (redpanda/templates/entry-point.yaml:17:4): invalid Quantity expected string or float64 got: json.Number (1) This is due to a bug in Helm v3.18.0. To avoid similar errors, upgrade to a later version. For more details, see the [Helm GitHub issue](https://github.com/helm/helm/issues/30880). ### [](#statefulset-never-rolls-out)StatefulSet never rolls out If the StatefulSet Pods remain in a pending state, they are waiting for resources to become available. To identify the Pods that are pending, use the following command: ```bash kubectl get pod --namespace ``` The response includes a list of Pods in the StatefulSet and their status. To view logs for a specific Pod, use the following command. ```bash kubectl logs -f --namespace ``` You can use the output to debug your deployment. ### [](#didnt-match-pod-anti-affinity-rules)Didn’t match pod anti-affinity rules If you see this error, your cluster does not have enough nodes to satisfy the anti-affinity rules: Warning FailedScheduling 18m default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. The Helm chart configures default `podAntiAffinity` rules to make sure that only one Pod running a Redpanda broker is scheduled on each worker node. To learn why, see [Number of workers](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-requirements/#number-of-workers). To resolve this issue, do one of the following: - Create additional worker nodes. - Modify the anti-affinity rules (for development purposes only). If adding nodes is not an option, you can modify the `podAntiAffinity` rules in your StatefulSet to be less strict. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: statefulset: podAntiAffinity: type: soft ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml statefulset: podAntiAffinity: type: soft ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set statefulset.podAntiAffinity.type=soft ``` ### [](#unable-to-mount-volume)Unable to mount volume If you see volume mounting errors in the Pod events or in the Redpanda logs, ensure that each of your Pods has a volume available in which to store data. - If you’re using StorageClasses with dynamic provisioners (default), ensure they exist: ```bash kubectl get storageclass ``` - If you’re using PersistentVolumes, ensure that you have one PersistentVolume available for each Redpanda broker, and that each one has the storage capacity that’s set in `storage.persistentVolume.size`: ```bash kubectl get persistentvolume --namespace ``` To learn how to configure different storage volumes, see [Configure Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/storage/k-configure-storage/). ### [](#failed-to-pull-image)Failed to pull image When deploying the Redpanda Helm chart, you may encounter Docker rate limit issues because the default registry URL is not recognized as a Docker Hub URL. The domain `docker.redpanda.com` is used for statistical purposes, such as tracking the number of downloads. It mirrors Docker Hub’s content while providing specific analytics for Redpanda. Failed to pull image "docker.redpanda.com/redpandadata/redpanda:v": rpc error: code = Unknown desc = failed to pull and unpack image "docker.redpanda.com/redpandadata/redpanda:v": failed to copy: httpReadSeeker: failed open: unexpected status code 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit To fix this error, do one of the following: - Replace the `image.repository` value in the Helm chart with `docker.io/redpandadata/redpanda`. Switching to Docker Hub avoids the rate limit issues associated with `docker.redpanda.com`. #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: image: repository: docker.io/redpandadata/redpanda ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `docker-repo.yaml` ```yaml image: repository: docker.io/redpandadata/redpanda ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values docker-repo.yaml ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set image.repository=docker.io/redpandadata/redpanda ``` - Authenticate to Docker Hub by logging in with your Docker Hub credentials. The `docker.redpanda.com` site acts as a reflector for Docker Hub. As a result, when you log in with your Docker Hub credentials, you will bypass the rate limit issues. ### [](#dig-not-defined)Dig not defined This error means that you are using an unsupported version of [Helm](https://helm.sh/docs/intro/install/): Error: parse error at (redpanda/templates/statefulset.yaml:203): function "dig" not defined To fix this error, ensure that you are using the minimum required version: 3.10.0. ```bash helm version ``` ### [](#repository-name-already-exists)Repository name already exists If you see this error, remove the `redpanda` chart repository, then try installing it again. ```bash helm repo remove redpanda helm repo add redpanda https://charts.redpanda.com helm repo update ``` ### [](#fatal-error-during-checker-data-directory-is-writable-execution)Fatal error during checker "Data directory is writable" execution This error appears when Redpanda does not have write access to your configured storage volume under `storage` in the Helm chart. Error: fatal error during checker "Data directory is writable" execution: open /var/lib/redpanda/data/test\_file: permission denied To fix this error, set `statefulset.initContainers.setDataDirOwnership.enabled` to `true` so that the initContainer can set the correct permissions on the data directories. ### [](#cannot-patch-redpanda-with-kind-statefulset)Cannot patch "redpanda" with kind StatefulSet This error appears when you run `helm upgrade` with the `--values` flag but do not include all your previous overrides. Error: UPGRADE FAILED: cannot patch "redpanda" with kind StatefulSet: StatefulSet.apps "redpanda" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden To fix this error, include all the value overrides from the previous installation using either the `--set` or the `--values` flags. > ⚠️ **WARNING** > > Do not use the `--reuse-values` flag to upgrade from one version of the Helm chart to another. This flag stops Helm from using any new values in the upgraded chart. ### [](#cannot-patch-redpanda-console-with-kind-deployment)Cannot patch "redpanda-console" with kind Deployment This error appears if you try to upgrade your deployment and you already have `console.enabled` set to `true`. Error: UPGRADE FAILED: cannot patch "redpanda-console" with kind Deployment: Deployment.apps "redpanda-console" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map\[string\]string{"app.kubernetes.io/instance":"redpanda", "app.kubernetes.io/name":"console"}, MatchExpressions:\[\]v1.LabelSelectorRequirement(nil)}: field is immutable To fix this error, set `console.enabled` to `false` so that Helm doesn’t try to deploy Redpanda Console again. ### [](#helm-is-in-a-pending-rollback-state)Helm is in a pending-rollback state An interrupted Helm upgrade process can leave your Helm release in a `pending-rollback` state. This state prevents further actions like upgrades, rollbacks, or deletions through standard Helm commands. To fix this: 1. Identify the Helm release that’s in a `pending-rollback` state: ```bash helm list --namespace --all ``` Look for releases with a status of `pending-rollback`. These are the ones that need intervention. 2. Verify the Secret’s status to avoid affecting the wrong resource: ```bash kubectl --namespace get secret --show-labels ``` Identify the Secret associated with your Helm release by its `pending-rollback` status in the labels. > ⚠️ **WARNING** > > Ensure you have correctly identified the Secret to avoid unintended consequences. Deleting the wrong Secret could impact other deployments or services. 3. Delete the Secret to clear the `pending-rollback` state: ```bash kubectl --namespace delete secret -l status=pending-rollback ``` After clearing the `pending-rollback` state: - **Retry the upgrade**: Restart the upgrade process. You should investigate the initial failure to avoid getting into the `pending-rollback` state again. - **Perform a rollback**: If you need to roll back to a previous release, use `helm rollback ` to revert to a specific, stable release version. ### [](#crash-loop-backoffs)Crash loop backoffs If a broker crashes after startup, or gets stuck in a crash loop, it can accumulate an increasing amount of stored state. This accumulated state not only consumes additional disk space but also prolongs the time required for each subsequent restart to process it. To prevent infinite crash loops, the Redpanda Helm chart sets the [`crash_loop_limit`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_limit) broker configuration property to `5`. The crash loop limit is the number of consecutive crashes that can happen within one hour of each other. By default, the broker terminates immediately after hitting the `crash_loop_limit`. The Pod running Redpanda remains in a `CrashLoopBackoff` state until its internal consecutive crash counter is reset to zero. To facilitate debugging in environments where a broker is stuck in a crash loop, you can also set the [`crash_loop_sleep_sec`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#crash_loop_sleep_sec) broker configuration property. This setting determines how long the broker sleeps before terminating the process after reaching the crash loop limit. By providing a window during which the Pod remains available, you can SSH into it and troubleshoot the issue. Example configuration: ```yaml config: node: crash_loop_limit: 5 crash_loop_sleep_sec: 60 ``` In this example, when the broker hits the `crash_loop_limit` of 5, it will sleep for 60 seconds before terminating the process. This delay allows administrators to access the Pod and troubleshoot. To troubleshoot a crash loop backoff: 1. Check the Redpanda logs from the most recent crashes: ```bash kubectl logs --namespace ``` > 📝 **NOTE** > > Kubernetes retains logs only for the current and the previous instance of a container. This limitation makes it difficult to access logs from earlier crashes, which may contain vital clues about the root cause of the issue. Given these log retention limitations, setting up a centralized logging system is crucial. Systems such as [Loki](https://grafana.com/docs/loki/latest/) or [Datadog](https://www.datadoghq.com/product/log-management/) can capture and store logs from all containers, ensuring you have access to historical data. 2. Resolve the issue that led to the crash loop backoff. 3. Reset the crash counter to zero to allow Redpanda to restart. You can do any of the following to reset the counter: - Make changes to any of the following sections in the Redpanda Helm chart to trigger an update: - `config.node` - `config.tunable` For example: ```yaml config: node: crash_loop_limit: ``` - Delete the `startup_log` file in the broker’s data directory. ```bash kubectl exec --namespace -- rm /var/lib/redpanda/data/startup_log ``` > 📝 **NOTE** > > It might be challenging to execute this command within a Pod that is in a `CrashLoopBackoff` state due to the limited time during which the Pod is available before it restarts. Wrapping the command in a loop might work. - Wait one hour since the last crash. The crash counter resets after one hour. To avoid future crash loop backoffs and manage the accumulation of small segments effectively: - [Monitor](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/) the size and number of segments regularly. - Optimize your Redpanda configuration for segment management. - Consider implementing [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/kubernetes/tiered-storage/k-tiered-storage/) to manage data more efficiently. ### [](#a-redpanda-enterprise-edition-license-is-required)A Redpanda Enterprise Edition license is required During a Redpanda upgrade, if enterprise features are enabled and a valid Enterprise Edition license is missing, Redpanda logs a warning and aborts the upgrade process on the first broker. This issue prevents a successful upgrade. A Redpanda Enterprise Edition license is required to use the currently enabled features. To apply your license, downgrade this broker to the pre-upgrade version and provide a valid license key via rpk using 'rpk cluster license set ', or via Redpanda Console. To request an enterprise license, please visit . To try Redpanda Enterprise for 30 days, visit . For more information, see . If you encounter this message, follow these steps to recover: 1. [Roll back the affected broker to the original version](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#roll-back). 2. Do one of the following: - [Apply a valid Redpanda Enterprise Edition license](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) to the cluster. - Disable enterprise features. If you do not have a valid license and want to proceed without using enterprise features, you can disable the enterprise features in your Redpanda configuration. 3. Retry the upgrade. For more troubleshooting steps, see [Troubleshoot Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/errors-solutions/k-resolve-errors/). ## [](#next-steps)Next steps - [Try an example in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about Redpanda Console](https://docs.redpanda.com/streaming/current/manage/console/) - [Learn more about rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) > 💡 **TIP** > > When you’re ready to use a registered domain, make sure to remove your entries from the `/etc/hosts` file, and see [Configure External Access through a NodePort Service](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/k-nodeport/#use-the-default-redpanda-subdomains). ## [](#suggested-reading)Suggested reading - [Networking and Connectivity in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-networking-and-connectivity/) - [Configure TLS for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/tls/) - [Configure SASL for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) - [Redpanda Helm Specification](https://docs.redpanda.com/streaming/current/reference/k-redpanda-helm-spec/) - [Redpanda CRD Reference](https://docs.redpanda.com/streaming/current/reference/k-crd/) - [Redpanda Console README](https://github.com/redpanda-data/console) on GitHub ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 49: Deploy on Linux **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual.md --- # Deploy on Linux > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy on Linux latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/index.adoc description: Learn about deployment options on Linux, as well as considerations for high availability and sizing. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-11-19" support-status: supported --- - [Requirements and Recommendations](production/requirements/) A list of requirements and recommendations for provisioning servers to run Redpanda in production. - [Linux Deployment Options](production/) Deploy Redpanda on Linux for development or for production. - [Sizing Use Cases](sizing-use-cases/) How to size Redpanda clusters for low, medium, and high throughput use cases in your data center or in object storage. - [Sizing Guidelines](sizing/) Learn about considerations to size your Redpanda cluster to handle the volume of data being produced, replicated, and consumed. - [Linux System Tuning](linux-system-tuning/) Learn how Redpanda applies automatic tuning to your Linux system. --- # Page 50: Linux System Tuning **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/linux-system-tuning.md --- # Linux System Tuning > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Linux System Tuning latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/linux-system-tuning page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/linux-system-tuning.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/linux-system-tuning.adoc description: Learn how Redpanda applies automatic tuning to your Linux system. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-08-13" support-status: supported --- Redpanda includes several features to tune your Linux systems for optimal performance. You can trigger hardware-based optimizations through the `rpk redpanda tune` command, also called the autotuner. Software-based configurations are managed automatically through the `systemd` suite of tools. ## [](#user-triggered-hardware-tuning)User-triggered hardware tuning You can invoke the Redpanda autotuner using the `rpk redpanda tune` command to optimize native Linux nodes. It analyzes the hardware configuration of the worker node and sets appropriate kernel options. You can choose to set certain hardware parameters using `ulimit` or other system tools while using the autotuner to manage others. ### [](#using-the-autotuner)Using the autotuner The Redpanda autotuner has a large array of hardware tuners to choose from. You may choose to either execute all available tuners, or to apply only a subset of them. This is particularly important when you wish to maintain specific configurations and prevent Redpanda from altering these settings. Redpanda recommends running the autotuner as part of your production deployment workflows. This ensures any hardware configurations or updates to the autotuner itself are incorporated quickly and efficiently into your system. For the full list of available tuners, refer to [`rpk redpanda tune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/). When using Kubernetes, you should tune your system at the host node level. For more information on Kubernetes configuration and execution, refer to [Kubernetes worker node tuning](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/). ## [](#automated-software-configurations)Automated software configurations Redpanda makes use of systemd to manage operating system and software-based limits, such as memlocks, file handle limits, core limits, and certain distribution-specific configurations. These help ensure the system executes stably and efficiently. Some key examples of automated configurations include: - Detection of hung services and infinite looping scenarios. This helps ensure you don’t have blocking services or processes that may interfere with Redpanda’s stream processing. - Configuration of asynchronous input/output interfaces and scheduler affinity. This is particularly critical when you rely on real-time processing of data streams for your applications. - Use of systemd slices (see [this blog post](https://www.scylladb.com/2019/09/25/isolating-workloads-with-systemd-slices/) for more details), which allows Redpanda to efficiently use system resources while isolating tasks for scalability and performance reasons. ## [](#monitoring-configurations)Monitoring configurations While Redpanda provides numerous metrics, the configurations applied using the autotuner and systemd are not exposed internally. You should continue monitoring your system health through standard means. For example, the systemd configuration sets the file handle limit to 800,000 references. If you are concerned about approaching this limit, you should monitor it through standard system monitoring, such as the use of the `lsof` utility. --- # Page 51: Linux Deployment Options **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production.md --- # Linux Deployment Options > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Linux Deployment Options latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/index.adoc description: Deploy Redpanda on Linux for development or for production. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- - [Deploy for Development](dev-deployment/) Steps to deploy a Redpanda sandbox cluster. - [Deploy for Production: Automated](production-deployment-automation/) Deploy Redpanda using automation tools like Terraform and Ansible. - [Deploy for Production: Manual](production-deployment/) Steps to deploy a Redpanda production cluster. - [Production Readiness Checklist](production-readiness/) --- # Page 52: Deploy for Development **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/dev-deployment.md --- # Deploy for Development > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy for Development latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/dev-deployment page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/dev-deployment.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/dev-deployment.adoc description: Steps to deploy a Redpanda sandbox cluster. page-git-created-date: "2025-08-15" page-git-modified-date: "2026-03-31" support-status: supported --- You can deploy Redpanda using well-known configuration properties optimized for a development or test environment. This configuration uses less system resources and simplifies setup, but it’s not suitable for a production deployment. For example, in [development mode](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-mode/#development-mode), the default `group_topic_partitions` is 3, and the default `storage_min_free_bytes` is 1 GiB. In production mode, the default `group_topic_partitions` is 16, and the default `storage_min_free_bytes` is 5 GiB. > 📝 **NOTE** > > - Development mode enables write caching by default. This is a relaxed mode of [`acks=all`](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/#acksall) that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. Write caching provides lower latency while still ensuring that a majority of brokers acknowledge the write. For more information, or to disable this, see [write caching](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/#configure-write-caching). > > - Development mode also bypasses `fsync`, acknowledging messages before they’re stored to disk. This reduces the durability of messages, could cause potential data loss, and could give unrealistic performance characteristics for a production environment. To deploy for a production environment, see [Deploy for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/). Or to try out Redpanda in Docker, see [Redpanda Quickstart](https://docs.redpanda.com/streaming/current/get-started/quick-start/). ## [](#prerequisites)Prerequisites Make sure you meet the [hardware and software requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/requirements/). ### [](#tcpip-ports)TCP/IP ports Redpanda uses the following default ports: | Port | Purpose | | --- | --- | | 9092 | Kafka API | | 8082 | HTTP Proxy | | 8081 | Schema Registry | | 9644 | Admin API and Prometheus | | 33145 | internal RPC | ## [](#install-redpanda)Install Redpanda Install Redpanda on each system you want to be part of your cluster. There are binaries available for Fedora/RedHat or Debian systems. Unless you intend to run Redpanda in FIPS-compliance mode, the following packages should accommodate your needs (for both Debian and Redhat based systems): `redpanda` - Contains the Redpanda application and all supporting libraries - Depends on `redpanda-tuner` and either `redpanda-rpk` or `redpanda-rpk-fips` `redpanda-rpk` - Contains the pure GoLang compiled `rpk` application - If you wish to use `rpk` only, then this is the only required install package `redpanda-tuner` - Contains the files used to run Redpanda tuners - Depends on `redpanda-rpk` or `redpanda-rpk-fips` > ❗ **IMPORTANT** > > Redpanda’s Linux package repositories are moving from `dl.redpanda.com` to `linux.pkg.redpanda.com`, with a new GPG signing key. The legacy `dl.redpanda.com` repository is scheduled to shut down on **September 28, 2026**. If you currently install or upgrade Redpanda from `dl.redpanda.com`, re-point your package manager before then. For steps, including a manual configuration option for environments that cannot pipe a script to `bash`, see [Migrate the Linux Package Repository](https://docs.redpanda.com/streaming/current/migrate/linux-package-repository/). ### Fedora/RedHat ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash && sudo yum install redpanda -y ``` > ❗ **IMPORTANT** > > To install a version of Redpanda that is older than the latest available version, you must also specify each dependency. For example, if installing `24.3.1~rc1-1` and you fail to specify each package and its version, you may encounter the following error: > > ```bash > yum install redpanda=24.3.1~rc1-1 > Reading package lists... Done > Building dependency tree... Done > Reading state information... Done > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > redpanda : Depends: redpanda-rpk (= 24.3.1~rc1-1) but it is not going to be installed or > redpanda-rpk-fips (= 24.3.1~rc1-1) but it is not going to be installed > Depends: redpanda-tuner (= 24.3.1~rc1-1) but it is not going to be installed > E: Unable to correct problems, you have held broken packages. > ``` > > To troubleshoot this error, specify the full list of versions for each package. In this case: > > ```bash > yum install \ > redpanda-tuner=24.3.1~rc1-1 \ > redpanda-rpk=24.3.1~rc1-1 \ > redpanda=24.3.1~rc1-1 > ``` ### Debian/Ubuntu ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | \ sudo -E bash && sudo apt install redpanda -y ``` > ❗ **IMPORTANT** > > To install a version of Redpanda that is older than the latest available version, you must also specify each dependency. For example, if installing `24.3.1~rc1-1` and you fail to specify each package and its version, you may encounter the following error: > > ```bash > apt install redpanda=24.3.1~rc1-1 > Reading package lists... Done > Building dependency tree... Done > Reading state information... Done > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > redpanda : Depends: redpanda-rpk (= 24.3.1~rc1-1) but it is not going to be installed or > redpanda-rpk-fips (= 24.3.1~rc1-1) but it is not going to be installed > Depends: redpanda-tuner (= 24.3.1~rc1-1) but it is not going to be installed > E: Unable to correct problems, you have held broken packages. > ``` > > To troubleshoot this error, specify the full list of versions for each package. In this case: > > ```bash > apt install \ > redpanda-tuner=24.3.1~rc1-1 \ > redpanda-rpk=24.3.1~rc1-1 \ > redpanda=24.3.1~rc1-1 > ``` ## [](#install-redpanda-for-fips-compliance)Install Redpanda for FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. To install Redpanda for FIPS compliance, install the packages `redpanda-fips` and `redpanda-rpk-fips`, which automatically pull in all required dependencies. `redpanda-fips` - Contains the OpenSSL FIPS-approved module and scripts required to set up and run Redpanda in FIPS-compliance mode. - Depends upon the successful installation of the `redpanda` package. - Includes the `fips.so` cryptographic provider (built from OpenSSL v3.0.9, which is the latest FIPS 140-2 approved module) and a copy of the OpenSSL application. - Executes `openssl fipsinstall` against the `fips.so` module, which generates a `fipsmodule.cnf` file that is used during the module’s POST (power-on-self-test) to validate the integrity of the module. `redpanda-rpk-fips` - Contains a version of `rpk` built with the [Microsoft GoLang compiler](https://github.com/microsoft/go) and [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl) to which `rpk` is linked, and uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > Redpanda’s Linux package repositories are moving from `dl.redpanda.com` to `linux.pkg.redpanda.com`, with a new GPG signing key. The legacy `dl.redpanda.com` repository is scheduled to shut down on **September 28, 2026**. If you currently install or upgrade Redpanda from `dl.redpanda.com`, re-point your package manager before then. For steps, including a manual configuration option for environments that cannot pipe a script to `bash`, see [Migrate the Linux Package Repository](https://docs.redpanda.com/streaming/current/migrate/linux-package-repository/). ### RHEL To install Redpanda for FIPS compliance, run: ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash && sudo yum install redpanda -y ``` > 📝 **NOTE** > > Alternatively, you could run `sudo yum install -y redpanda-fips`, which also picks up and includes the `redpanda` install package. If you wish to only use `rpk` on a FIPS host, run: ```bash sudo yum install -y redpanda-fips redpanda-rpk-fips ``` ### Debian/Ubuntu To install Redpanda for FIPS compliance, run: ```bash sudo apt install redpanda-fips redpanda-rpk-fips ``` > 📝 **NOTE** > > Alternatively, you could run `sudo apt install redpanda-fips`, which also picks up and includes the `redpanda` install package. If you wish to only use `rpk` on a FIPS host, run: ```bash sudo apt install -y redpanda-rpk-fips ``` See also: [Configure Redpanda for FIPS](https://docs.redpanda.com/streaming/current/manage/security/fips-compliance/) ## [](#install-redpanda-console)Install Redpanda Console For detailed instructions on installing and configuring Redpanda Console, see [Deploy Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy/). ## [](#bootstrapping)Bootstrap broker configurations Each broker requires a set of broker configurations that determine how all brokers communicate with each other and with clients. Bootstrapping a cluster configures the [listeners](https://docs.redpanda.com/streaming/current/reference/glossary/#listener), [seed servers](https://docs.redpanda.com/streaming/current/reference/glossary/#seed-server), and [advertised listeners](https://docs.redpanda.com/streaming/current/reference/glossary/#advertised-listener), which ensure proper network connectivity and accessibility. Starting in version 23.3.8, `rpk` enhances the bootstrapping process with additional flags for configuring advertised listener addresses directly. Use the [`rpk redpanda config bootstrap`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-config-bootstrap/) command to bootstrap Redpanda: ```bash sudo rpk redpanda config bootstrap --self --advertised-kafka --ips ,, && \ sudo rpk redpanda config set redpanda.empty_seed_starts_cluster false ``` Replace the following placeholders: - ``: The `--self` flag tells Redpanda the interfaces to bind to for the Kafka API, the RPC API, and the Admin API. These addresses determine on which network interface and port Redpanda listens for incoming connections. - Set the listener address to `0.0.0.0` to listen on all network interfaces available on the machine. - Set the listener address to a specific IP address to bind the listener to that address, restricting connections to that interface. - ``: The `--advertised-kafka` flag sets a different advertised Kafka address, which is useful for scenarios where the accessible address differs from the bind address. > ❗ **IMPORTANT** > > Redpanda does not allow advertised addresses set to `0.0.0.0`. If you set any advertised addresses to `0.0.0.0`, Redpanda will output startup validation errors. - ``: The `--ips` flag lists all the seed servers in the cluster, including the one being started. > 📝 **NOTE** > > The `--ips` flag must be set _identically_ (with nodes listed in identical order) on each node. Bootstrapping Redpanda updates your `/etc/redpanda/redpanda.yaml` configuration file: `/etc/redpanda/redpanda.yaml` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: port: 33145 - host: address: port: 33145 - host: address: port: 33145 rpc_server: address: port: 33145 kafka_api: - address: port: 9092 admin: - address: port: 9644 advertised_rpc_api: address: port: 33145 advertised_kafka_api: - address: port: 9092 ``` ### [](#recommendations)Recommendations - Redpanda Data strongly recommends at least three seed servers when forming a cluster. A larger number of seed servers increases the robustness of consensus and minimizes any chance that new clusters get spuriously formed after brokers are lost or restarted without any data. - It’s important to have one or more seed servers in each fault domain (for example, in each rack or cloud AZ). A higher number provides a stronger guarantee that clusters don’t fracture unintentionally. - It’s possible to change the seed servers for a short period of time after a cluster has been created. For example, you may want to designate one additional broker as a seed server to increase availability. To do this without cluster downtime, add the new broker to the [`seed_servers`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) property and restart Redpanda to apply the change on a broker-by-broker basis. ### [](#listeners-for-mixed-environments)Listeners for mixed environments For clusters serving both internal and external clients, configure multiple listeners for the Kafka API to separate internal from external traffic. For more details, see [Configure Listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/). ## [](#start-redpanda)Start Redpanda To start Redpanda: ```bash sudo systemctl start redpanda-tuner redpanda ``` When a Redpanda cluster starts, it instantiates a controller Raft group with all the seed servers specified in the `--ips` flag. After all seed servers complete their startup procedure and become accessible, the cluster is then available. After that, non-seed servers start up and are added to the cluster. ## [](#start-redpanda-console)Start Redpanda Console For instructions on starting and managing Redpanda Console, see [Deploy Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy/). ## [](#verify-the-installation)Verify the installation To verify that the Redpanda cluster is up and running, use `rpk` to get information about the cluster: ```bash rpk cluster info ``` You should see a list of advertised addresses. To create a topic: ```bash rpk topic create ``` If topics were initially created in a test environment with a replication factor of `1`, use `rpk topic alter-config` to change the topic replication factor: ```bash rpk topic alter-config --set replication.factor=3 ``` ## [](#perform-a-self-test)Perform a self test To understand the performance capabilities of your Redpanda cluster, Redpanda offers built-in self-test features that evaluate the performance of both disk and network operations. For more information, see [Disk and network self-test benchmarks](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/#self-test). ## [](#next-steps)Next steps - If clients connect from a different subnet, see [Configure Listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/). - Observability is essential. See [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/). ## [](#suggested-reading)Suggested reading - [Configure Cluster Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/) - [Redpanda Console Configuration](https://docs.redpanda.com/streaming/current/console/config/configure-console/) - [Work with Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/) - [Work with HTTP Proxy](https://docs.redpanda.com/streaming/current/develop/http-proxy/) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 53: Deploy for Production: Automated **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation.md --- # Deploy for Production: Automated > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: "Deploy for Production: Automated" latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/production-deployment-automation page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/production-deployment-automation.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/production-deployment-automation.adoc description: Deploy Redpanda using automation tools like Terraform and Ansible. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- If you use automation tools like Terraform and Ansible in your environment, you can use them to quickly provision a Redpanda cluster. Terraform can set up the infrastructure and output a properly-formatted `hosts.ini` file, and Ansible can use that `hosts.ini` file as input to install Redpanda. If you already have an infrastructure provisioning framework, you can supply your own hosts file (without using Terraform), and you can use Ansible to install Redpanda. This recommended automated deployment provides a production-usable way to deploy and maintain a cluster. For unique configurations, you can work directly with the Ansible and Terraform modules to integrate them into your environment. ## [](#prerequisites)Prerequisites 1. Install Terraform following the [Terraform documentation](https://learn.hashicorp.com/tutorials/terraform/install-cli). 2. Install Ansible following the [Ansible documentation](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html). Different operating systems may have specific Ansible dependencies. 3. Clone the [`deployment-automation` GitHub repository](https://github.com/redpanda-data/deployment-automation/): ```bash git clone https://github.com/redpanda-data/deployment-automation.git ``` 4. Change into the directory: ```bash cd deployment-automation ``` ## [](#use-terraform-to-set-up-infrastructure)Use Terraform to set up infrastructure ### AWS The recommended [Terraform module for Redpanda](https://registry.terraform.io/modules/redpanda-data/redpanda-cluster/aws/latest) deploys virtual machines on AWS EC2. To create an AWS Redpanda cluster, review the [default variables](https://github.com/redpanda-data/deployment-automation/blob/main/aws/main.tf) and make any edits necessary for your environment. 1. In the `deployment-automation` folder, change into the `aws` directory: ```bash cd aws ``` 2. Set AWS credentials. Terraform provides multiple ways to set the AWS secret and key. See the [Terraform documentation](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables). 3. Initialize Terraform: ```bash terraform init ``` 4. Create the cluster with `terraform apply`: ```bash terraform apply -var='public_key_path=~/.ssh/id_rsa.pub' -var='subnet_id=' -var='vpc_id=' ``` - Terraform configures `public_key_path` on the brokers to remotely connect with SSH. If the public key path isn’t the default `~/.ssh/id_rsa.pub`, then you need to set it. - If you don’t have a default VPC defined, then you need to set `subnet_id` and `vpc_id`. For a complete and up-to-date list of configuration options, see the [Terraform module](https://registry.terraform.io/modules/redpanda-data/redpanda-cluster/aws/latest): > 📝 **NOTE** > > For acceptable `distro` names: > > ```bash > data "aws_ami" "ami" { > most_recent = true > > filter { > name = "name" > values = [ > "ubuntu/images/hvm-ssd/ubuntu-*-amd64-server-*", > "ubuntu/images/hvm-ssd/ubuntu-*-arm64-server-*", > "Fedora-Cloud-Base-*.x86_64-hvm-us-west-2-gp2-0", > "debian-*-amd64-*", > "debian-*-hvm-x86_64-gp2-*", > "amzn2-ami-hvm-2.0.*-x86_64-gp2", > "RHEL*HVM-*-x86_64*Hourly2-GP2", > "al2023-ami-2023.*-kernel-*-x86_64" > ] > } > > filter { > name = "architecture" > values = [var.machine_architecture] > } > > filter { > name = "name" > values = ["*${var.distro}*"] > } > > filter { > name = "virtualization-type" > values = ["hvm"] > } > > owners = ["099720109477", "125523088429", "136693071363", "137112412989", "309956199498"] > # Canonical, Fedora, Debian (new), Amazon, RedHat > } > ``` ### GCP 1. In the `deployment-automation` folder, change into the `gcp` directory: ```bash cd gcp ``` 2. Set GCP credentials. Terraform provides multiple ways to set the GCP secret and key. See the [Terraform documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/getting_started). 3. This module generates a properly-configured GCP network within your GCP project by default. You can disable this functionality by commenting out that section of the main.tf and providing your own subnet to var.subnet. 4. You will need to either create a GCP project or provide a Project ID for an existing project. 5. Initialize Terraform: ```bash terraform init ``` 6. Create the cluster: ```bash terraform apply ``` The following example shows how to create a three-broker cluster: ```bash terraform apply --var="public_key_path=~/.ssh/id_rsa.pub" --var "ssh_user=ubuntu" --var="project_name=$GCP_PROJECT_ID" ``` For a full, up-to-date list of configuration options, see the [Terraform module](https://registry.terraform.io/modules/redpanda-data/redpanda-cluster/gcp/latest): ## [](#use-ansible-to-install-redpanda)Use Ansible to install Redpanda 1. From the [`deployment-automation`](https://github.com/redpanda-data/deployment-automation/) folder, set the required Ansible variables: ```bash export CLOUD_PROVIDER= export DEPLOYMENT_PREFIX= export ANSIBLE_COLLECTIONS_PATH=${PWD}/artifacts/collections export ANSIBLE_ROLES_PATH=${PWD}/artifacts/roles export ANSIBLE_INVENTORY=${PWD}/artifacts/hosts_${CLOUD_PROVIDER}_${DEPLOYMENT_PREFIX}.ini ``` 2. Install the roles required by Ansible: ```bash ansible-galaxy install -r requirements.yml ``` ### [](#configure-a-hosts-file)Configure a hosts file > 💡 **TIP** > > Redpanda Data recommends incorporating variables into your $ANSIBLE\_INVENTORY file for every host. Edits made to properties outside of the playbook may be overwritten. If you used Terraform to deploy the instances, the `hosts.ini` is configured automatically in the [`artifacts`](https://github.com/redpanda-data/deployment-automation/tree/main/artifacts) directory. If you didn’t use Terraform, then you must manually update the `[redpanda]` section. When you open the file, you see something like the following: ```ini [redpanda] ip ansible_user=ssh_user ansible_become=True private_ip=pip ip ansible_user=ssh_user ansible_become=True private_ip=pip ip ansible_user=ssh_user ansible_become=True private_ip=pip [monitor] ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1 ``` Under the `[redpanda]` section, replace the following: | Property | Description | | --- | --- | | ip | The public IP address of the machine. | | ansible_user | The username for Ansible to use to SSH to the machine. | | private_ip | The private IP address of the machine. This could be the same as the public IP address. | You can add additional properties to configure features like rack awareness and Tiered Storage. The `[monitor]` section is only required if you want the playbook to install and configure a basic Prometheus and Grafana setup for observability. If you have a centralized monitoring setup or if you don’t require monitoring, then remove this section. ### [](#run-a-playbook)Run a playbook Use the [Ansible Collection for Redpanda](https://galaxy.ansible.com/redpanda/cluster) to build a Redpanda cluster. The recommended Redpanda playbook enables TLS encryption and Tiered Storage. If you prefer, you can download the modules and required roles and create your own playbook. For example, if you want to handle your own data directory, you can toggle that part off, and Redpanda ensures that the permissions are correct. If you want to generate your own security certificates, you can. To install and start a Redpanda cluster in one command with the Redpanda playbook, run: ```bash ansible-playbook --private-key -v ansible/provision-cluster.yml ``` > 📝 **NOTE** > > - The private key corresponds to the public key in the `distro_user` SSH configuration. > > - To use your own playbook, replace `provision-cluster.yml` with your playbook name. > > - When you use a playbook to create a cluster, you should also use the playbook for subsequent operations, like upgrades. The Ansible modules safely handle rolling upgrades, but you must comply with [Redpanda version path requirements](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/). #### [](#custom-configuration)Custom configuration You can specify any available Redpanda configuration value, or set of values, by passing a JSON dictionary as an Ansible `extra-var`. These values are spliced with the calculated configuration and only override the values that you specify. Values must be unset manually with `rpk`. There are two sub-dictionaries you can specify: `redpanda.cluster` and `redpanda.node`. For more information, see [Cluster Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/) and [Broker Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/). ```bash export JSONDATA='{"cluster":{"auto_create_topics_enabled":"true"},"node":{"developer_mode":"false"}}' ansible-playbook ansible/.yml --private-key artifacts/testkey -e redpanda="${JSONDATA}" ``` > 📝 **NOTE** > > Adding whitespace to the JSON breaks configuration merging. Use `rpk` and standard Kafka tools to produce and consume from the Redpanda cluster. #### [](#configure-prometheus-and-grafana)Configure Prometheus and Grafana Include a `[monitor]` section in your hosts file if you want the playbook to install and configure a basic Prometheus and Grafana setup for observability. Redpanda emits Prometheus metrics that can be scraped with a central collector. If you already have a centralized monitoring setup or if you don’t require monitoring, then this is unnecessary. To run the `deploy-monitor.yml` playbook: ```bash ansible-playbook ansible/deploy-monitor.yml \ --private-key '' ``` #### [](#configure-redpanda-console)Configure Redpanda Console To install Redpanda Console, add the `redpanda_broker` role to a group with `install_console: true`. The standard playbooks automatically install Redpanda Console on hosts in the `[client]` group. #### [](#build-the-cluster-with-tls-enabled)Build the cluster with TLS enabled Configure TLS with externally-provided and signed certificates. Then run the `provision-cluster-tls.yml` playbook, specifying the certificate locations on new hosts. You can either pass the variables in the command line or edit the file and pass them there. Consider whether you want public access to the Kafka API and Admin API endpoints. For example: ```bash ansible-playbook ansible/provision-cluster-tls.yml \ --private-key '' \ --extra-vars create_demo_certs=false \ --extra-vars advertise_public_ips=false \ --extra-vars handle_certs=false \ --extra-vars redpanda_truststore_file='' ``` It is important to use a signed certificate from a valid CA for production environments. The playbook uses locally-signed certificates that are not recommended for production use. Provide a valid certificate using these variables: ```bash redpanda_certs_dir: /etc/redpanda/certs redpanda_csr_file: "{{ redpanda_certs_dir }}/node.csr" redpanda_key_file: "{{ redpanda_certs_dir }}/node.key" redpanda_cert_file: "{{ redpanda_certs_dir }}/node.crt" redpanda_truststore_file: "{{ redpanda_certs_dir }}/truststore.pem" ``` For testing, you could deploy a local CA to generate private keys and signed certificates: ```bash ansible-playbook ansible/provision-cluster-tiered-storage.yml \ --private-key '' ``` #### [](#add-brokers-to-an-existing-cluster)Add brokers to an existing cluster To add brokers to a cluster, you must add them to the hosts file and run the relevant playbook again. You can add `skip_node=true` to the existing hosts to avoid the playbooks being rerun on them. #### [](#upgrade-a-cluster)Upgrade a cluster The playbook is designed to be idempotent, so it should be suitable for running as part of a CI/CD pipeline or through Ansible Tower. The playbook upgrades the packages and then performs a rolling upgrade, where one broker at a time is upgraded and safely restarted. For all upgrade requirements and recommendations, see [Upgrade Redpanda](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/). It is important to test that your upgrade path is safe before using it in production. To upgrade a cluster, run the playbook with a specific target version: ```bash ansible-playbook --private-key ~/.ssh/id_rsa ansible/.yml -e redpanda_version=22.3.10-1 ``` By default, the playbook selects the latest version of the Redpanda packages, but an upgrade is only performed if the `redpanda_install_status` variable is set to `latest`: ```bash ansible-playbook --private-key ~/.ssh/id_rsa ansible/.yml -e redpanda_install_status=latest ``` To upgrade clusters with SASL authentication: ```bash export JSONDATA='{"cluster":{"auto_create_topics_enabled":"true"},"node":{"developer_mode":"false"}}' ansible-playbook ansible/.yml --private-key artifacts/testkey -e redpanda="${JSONDATA}" ``` Similarly, you can put the `redpanda_rpk_opts` into a YAML file protected with Ansible vault. ```bash ansible-playbook --private-key ~/.ssh/id_rsa ansible/.yml --extra-vars=redpanda_install_status=latest --extra-vars @vault-file.yml --ask-vault-pass ``` #### [](#redpanda-ansible-collection-values)Redpanda Ansible Collection values You can pass the following variables as `-e var=value` when running Ansible: | Property | Default value | Description | | --- | --- | --- | | redpanda_organization | redpanda-test | Set this to identify your organization in the asset management system. | | redpanda_cluster_id | redpanda | This helps identify the cluster. | | advertise_public_ips | false | Configure Redpanda to advertise the broker’s public IPs for client communication instead of private IPs. This enables using the cluster from outside its subnet.Note: This is not recommended for production deployments, because your brokers will be public. | | grafana_admin_pass | | Grafana admin user’s password. | | ephemeral_disk | false | Enable file system check for attached disk.This is useful when using attached disks in instances with ephemeral operating system disks like Azure L Series. This allows a file system repair at boot time and ensures that the drive is remounted automatically after a reboot. | | redpanda_mode | production | Enables hardware optimization. | | redpanda_admin_api_port | 9644 | | | redpanda_kafka_port | 9092 | | | redpanda_rpc_port | 33145 | | | redpanda_schema_registry_port | 8081 | | | is_using_unstable | false | Enables access to unstable builds. | | redpanda_version | latest | Version; for example, 22.2.2-1 or 22.3.1~rc1-1. If this value is set, then the package is upgraded if the installed version is lower than what has been specified. | | redpanda_rpk_opts | | Command line options to be passed to instances where rpk is used on the playbook. For example, superuser credentials can be specified as --user myuser --password mypassword. | | redpanda_install_status | present | If redpanda_version is set to latest, then changing redpanda_install_status to latest causes an upgrade; otherwise, the currently-installed version remains. | | redpanda_data_directory | /var/lib/redpanda/data | Path where Redpanda keeps its data. | | redpanda_key_file | /etc/redpanda/certs/node.key | TLS: Path to private key. | | redpanda_cert_file | /etc/redpanda/certs/node.crt | TLS: Path to signed certificate. | | redpanda_truststore_file | /etc/redpanda/certs/truststore.pem | TLS: Path to truststore. | | tls | false | Set to true to configure Redpanda to use TLS. This can be set on each broker, although this may lead to errors configuring rpk. | | skip_node | false | Broker configuration to prevent the redpanda_broker role being applied to this specific broker. Use carefully when adding new brokers to avoid existing brokers from being reconfigured. | | restart_node | false | Broker configuration to prevent Redpanda brokers from being restarted after updating. Use with care: This can cause rpk to be reconfigured but the broker is not restarted and therefore is in an inconsistent state. | | rack | undefined | Broker configuration to enable rack awareness. Rack awareness is enabled cluster-wide if at least one broker has this set. | | tiered_storage_bucket_name | | Set bucket name to enable Tiered Storage. | | schema_registry_replication_factor | 1 | The replication factor of Schema Registry’s internal storage topic. | | aws_region | | The region to be used if Tiered Storage is enabled. | ### [](#troubleshooting)Troubleshooting On macOS, Python may be [unable to fork workers](https://stackoverflow.com/questions/50168647/multiprocessing-causes-python-to-crash-and-gives-an-error-may-have-been-in-progr). You may see something like the following: ```bash ok: [34.209.26.177] => {“changed”: false, “stat”: {“exists”: false}} objc[57889]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. objc[57889]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. ERROR! A worker was found in a dead state ``` Try setting an environment variable to resolve the error: ```bash export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ``` ## [](#next-steps)Next steps - If clients connect from a different subnet, see [Configure Listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/). - Observability is essential in production environments. See [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/). ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 54: Deploy for Production: Manual **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment.md --- # Deploy for Production: Manual > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: "Deploy for Production: Manual" latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/production-deployment page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/production-deployment.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/production-deployment.adoc description: Steps to deploy a Redpanda production cluster. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- You can deploy Redpanda for production with a default deployment, which uses recommended deployment tools, or with a custom deployment, which uses unsupported deployment tools. > 📝 **NOTE** > > - See [Deploy for Production: Automated](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/) to use Terraform and Ansible to deploy Redpanda. > > - See [Redpanda Quickstart](https://docs.redpanda.com/streaming/current/get-started/quick-start/) to try out Redpanda in Docker or [Deploy for Development](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/dev-deployment/). ## [](#prerequisites)Prerequisites Make sure you meet the [hardware and software requirements](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/requirements/). ### [](#tcpip-ports)TCP/IP ports Redpanda uses the following default ports: | Port | Purpose | | --- | --- | | 9092 | Kafka API | | 8082 | HTTP Proxy | | 8081 | Schema Registry | | 9644 | Admin API and Prometheus | | 33145 | internal RPC | ## [](#select-deployment-type)Select deployment type To start deploying Redpanda for production, choose your deployment type: - [Default deployment](#default-deployment): Use recommended deployment tools. - [Custom deployment](#custom-deployment): Use unsupported deployment tools. ## [](#default-deployment)Default deployment This section describes how to set up a production cluster of Redpanda. ### [](#install-redpanda)Install Redpanda Install Redpanda on each system you want to be part of your cluster. There are binaries available for Fedora/RedHat or Debian systems. Unless you intend to run Redpanda in FIPS-compliance mode, the following packages should accommodate your needs (for both Debian and Redhat based systems): `redpanda` - Contains the Redpanda application and all supporting libraries - Depends on `redpanda-tuner` and either `redpanda-rpk` or `redpanda-rpk-fips` `redpanda-rpk` - Contains the pure GoLang compiled `rpk` application - If you wish to use `rpk` only, then this is the only required install package `redpanda-tuner` - Contains the files used to run Redpanda tuners - Depends on `redpanda-rpk` or `redpanda-rpk-fips` > ❗ **IMPORTANT** > > Redpanda’s Linux package repositories are moving from `dl.redpanda.com` to `linux.pkg.redpanda.com`, with a new GPG signing key. The legacy `dl.redpanda.com` repository is scheduled to shut down on **September 28, 2026**. If you currently install or upgrade Redpanda from `dl.redpanda.com`, re-point your package manager before then. For steps, including a manual configuration option for environments that cannot pipe a script to `bash`, see [Migrate the Linux Package Repository](https://docs.redpanda.com/streaming/current/migrate/linux-package-repository/). #### Fedora/RedHat ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash && sudo yum install redpanda -y ``` > ❗ **IMPORTANT** > > To install a version of Redpanda that is older than the latest available version, you must also specify each dependency. For example, if installing `24.3.1~rc1-1` and you fail to specify each package and its version, you may encounter the following error: > > ```bash > yum install redpanda=24.3.1~rc1-1 > Reading package lists... Done > Building dependency tree... Done > Reading state information... Done > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > redpanda : Depends: redpanda-rpk (= 24.3.1~rc1-1) but it is not going to be installed or > redpanda-rpk-fips (= 24.3.1~rc1-1) but it is not going to be installed > Depends: redpanda-tuner (= 24.3.1~rc1-1) but it is not going to be installed > E: Unable to correct problems, you have held broken packages. > ``` > > To troubleshoot this error, specify the full list of versions for each package. In this case: > > ```bash > yum install \ > redpanda-tuner=24.3.1~rc1-1 \ > redpanda-rpk=24.3.1~rc1-1 \ > redpanda=24.3.1~rc1-1 > ``` #### Debian/Ubuntu ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.deb.sh' | \ sudo -E bash && sudo apt install redpanda -y ``` > ❗ **IMPORTANT** > > To install a version of Redpanda that is older than the latest available version, you must also specify each dependency. For example, if installing `24.3.1~rc1-1` and you fail to specify each package and its version, you may encounter the following error: > > ```bash > apt install redpanda=24.3.1~rc1-1 > Reading package lists... Done > Building dependency tree... Done > Reading state information... Done > Some packages could not be installed. This may mean that you have > requested an impossible situation or if you are using the unstable > distribution that some required packages have not yet been created > or been moved out of Incoming. > The following information may help to resolve the situation: > > The following packages have unmet dependencies: > redpanda : Depends: redpanda-rpk (= 24.3.1~rc1-1) but it is not going to be installed or > redpanda-rpk-fips (= 24.3.1~rc1-1) but it is not going to be installed > Depends: redpanda-tuner (= 24.3.1~rc1-1) but it is not going to be installed > E: Unable to correct problems, you have held broken packages. > ``` > > To troubleshoot this error, specify the full list of versions for each package. In this case: > > ```bash > apt install \ > redpanda-tuner=24.3.1~rc1-1 \ > redpanda-rpk=24.3.1~rc1-1 \ > redpanda=24.3.1~rc1-1 > ``` ### [](#install-redpanda-for-fips-compliance)Install Redpanda for FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. To install Redpanda for FIPS compliance, install the packages `redpanda-fips` and `redpanda-rpk-fips`, which automatically pull in all required dependencies. `redpanda-fips` - Contains the OpenSSL FIPS-approved module and scripts required to set up and run Redpanda in FIPS-compliance mode. - Depends upon the successful installation of the `redpanda` package. - Includes the `fips.so` cryptographic provider (built from OpenSSL v3.0.9, which is the latest FIPS 140-2 approved module) and a copy of the OpenSSL application. - Executes `openssl fipsinstall` against the `fips.so` module, which generates a `fipsmodule.cnf` file that is used during the module’s POST (power-on-self-test) to validate the integrity of the module. `redpanda-rpk-fips` - Contains a version of `rpk` built with the [Microsoft GoLang compiler](https://github.com/microsoft/go) and [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl) to which `rpk` is linked, and uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > Redpanda’s Linux package repositories are moving from `dl.redpanda.com` to `linux.pkg.redpanda.com`, with a new GPG signing key. The legacy `dl.redpanda.com` repository is scheduled to shut down on **September 28, 2026**. If you currently install or upgrade Redpanda from `dl.redpanda.com`, re-point your package manager before then. For steps, including a manual configuration option for environments that cannot pipe a script to `bash`, see [Migrate the Linux Package Repository](https://docs.redpanda.com/streaming/current/migrate/linux-package-repository/). #### RHEL To install Redpanda for FIPS compliance, run: ```bash curl -1sLf 'https://linux.pkg.redpanda.com/setup-redpanda.rpm.sh' | \ sudo -E bash && sudo yum install redpanda -y ``` > 📝 **NOTE** > > Alternatively, you could run `sudo yum install -y redpanda-fips`, which also picks up and includes the `redpanda` install package. If you wish to only use `rpk` on a FIPS host, run: ```bash sudo yum install -y redpanda-fips redpanda-rpk-fips ``` #### Debian/Ubuntu To install Redpanda for FIPS compliance, run: ```bash sudo apt install redpanda-fips redpanda-rpk-fips ``` > 📝 **NOTE** > > Alternatively, you could run `sudo apt install redpanda-fips`, which also picks up and includes the `redpanda` install package. If you wish to only use `rpk` on a FIPS host, run: ```bash sudo apt install -y redpanda-rpk-fips ``` See also: [Configure Redpanda for FIPS](https://docs.redpanda.com/streaming/current/manage/security/fips-compliance/) ### [](#install-redpanda-console)Install Redpanda Console For comprehensive installation and configuration instructions, see [Deploy Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy/). ### [](#tune-the-linux-kernel-for-production)Tune the Linux kernel for production To get the best performance from your hardware, set Redpanda to production mode on each node and run the autotuner tool. The autotuner identifies the hardware configuration of your node and optimizes the Linux kernel to give you the best performance. By default, Redpanda is installed in development mode, which turns off hardware optimization. 1. Make sure that your current Linux user has root privileges. The autotuner requires privileged access to the Linux kernel settings. 2. Set Redpanda to run in [production mode](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-mode/#production-mode): ```bash sudo rpk redpanda mode production ``` 3. Tune the Linux kernel: ```bash sudo rpk redpanda tune all ``` Changes to the Linux kernel are not persisted. If a node restarts, make sure to run the autotuner again. To automatically tune the Linux kernel on a Redpanda broker after the node restarts, enable the `redpanda-tuner` service, which runs `rpk redpanda tune all`: - For RHEL, after installing the rpm package, run `systemctl` to both start and enable the `redpanda-tuner` service: ```bash sudo systemctl start redpanda-tuner sudo systemctl enable redpanda-tuner ``` - For Ubuntu, after installing the apt package, run `systemctl` to start the `redpanda-tuner` service (which is already enabled): ```bash sudo systemctl start redpanda-tuner ``` For more details, see the [autotuner reference](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/). ### [](#generate-optimal-io-configuration-settings)Generate optimal I/O configuration settings After tuning the Linux kernel, you can optimize Redpanda for the I/O capabilities of your worker node by using `rpk` to run benchmarks that capture its read/write IOPS and bandwidth capabilities. After running the benchmarks `rpk` saves the results to an I/O configuration file (`io-config.yaml`) that Redpanda reads upon startup to optimize itself for the node. > 📝 **NOTE** > > Unlike the autotuner, it isn’t necessary to run `rpk iotune` each time Redpanda is started, as its I/O output configuration file can be reused for each node that runs on the same type of hardware. Run [rpk iotune](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/): ```bash sudo rpk iotune # takes 10mins ``` For reference, a local NVMe SSD should yield around 1 GB/s sustained writes. `rpk iotune` captures SSD wear and tear and gives accurate measurements of what your hardware is capable of delivering. Run this before benchmarking. If you’re on AWS, GCP, or Azure, creating a new instance and upgrading to an image with a recent Linux kernel version is often the easiest way to work around bad devices. ### [](#bootstrapping)Bootstrap broker configurations Each broker requires a set of broker configurations that determine how all brokers communicate with each other and with clients. Bootstrapping a cluster configures the [listeners](https://docs.redpanda.com/streaming/current/reference/glossary/#listener), [seed servers](https://docs.redpanda.com/streaming/current/reference/glossary/#seed-server), and [advertised listeners](https://docs.redpanda.com/streaming/current/reference/glossary/#advertised-listener), which ensure proper network connectivity and accessibility. Starting in version 23.3.8, `rpk` enhances the bootstrapping process with additional flags for configuring advertised listener addresses directly. Use the [`rpk redpanda config bootstrap`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-config-bootstrap/) command to bootstrap Redpanda: ```bash sudo rpk redpanda config bootstrap --self --advertised-kafka --ips ,, && \ sudo rpk redpanda config set redpanda.empty_seed_starts_cluster false ``` Replace the following placeholders: - ``: The `--self` flag tells Redpanda the interfaces to bind to for the Kafka API, the RPC API, and the Admin API. These addresses determine on which network interface and port Redpanda listens for incoming connections. - Set the listener address to `0.0.0.0` to listen on all network interfaces available on the machine. - Set the listener address to a specific IP address to bind the listener to that address, restricting connections to that interface. - ``: The `--advertised-kafka` flag sets a different advertised Kafka address, which is useful for scenarios where the accessible address differs from the bind address. > ❗ **IMPORTANT** > > Redpanda does not allow advertised addresses set to `0.0.0.0`. If you set any advertised addresses to `0.0.0.0`, Redpanda will output startup validation errors. - ``: The `--ips` flag lists all the seed servers in the cluster, including the one being started. > 📝 **NOTE** > > The `--ips` flag must be set _identically_ (with nodes listed in identical order) on each node. Bootstrapping Redpanda updates your `/etc/redpanda/redpanda.yaml` configuration file: `/etc/redpanda/redpanda.yaml` ```yaml redpanda: data_directory: /var/lib/redpanda/data empty_seed_starts_cluster: false seed_servers: - host: address: port: 33145 - host: address: port: 33145 - host: address: port: 33145 rpc_server: address: port: 33145 kafka_api: - address: port: 9092 admin: - address: port: 9644 advertised_rpc_api: address: port: 33145 advertised_kafka_api: - address: port: 9092 ``` #### [](#recommendations)Recommendations - Redpanda Data strongly recommends at least three seed servers when forming a cluster. A larger number of seed servers increases the robustness of consensus and minimizes any chance that new clusters get spuriously formed after brokers are lost or restarted without any data. - It’s important to have one or more seed servers in each fault domain (for example, in each rack or cloud AZ). A higher number provides a stronger guarantee that clusters don’t fracture unintentionally. - It’s possible to change the seed servers for a short period of time after a cluster has been created. For example, you may want to designate one additional broker as a seed server to increase availability. To do this without cluster downtime, add the new broker to the [`seed_servers`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) property and restart Redpanda to apply the change on a broker-by-broker basis. #### [](#listeners-for-mixed-environments)Listeners for mixed environments For clusters serving both internal and external clients, configure multiple listeners for the Kafka API to separate internal from external traffic. For more details, see [Configure Listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/). ### [](#start-redpanda)Start Redpanda To start Redpanda: ```bash sudo systemctl start redpanda-tuner redpanda ``` When a Redpanda cluster starts, it instantiates a controller Raft group with all the seed servers specified in the `--ips` flag. After all seed servers complete their startup procedure and become accessible, the cluster is then available. After that, non-seed servers start up and are added to the cluster. ### [](#start-redpanda-console)Start Redpanda Console For instructions on starting and managing Redpanda Console, see [Deploy Redpanda Console on Linux](https://docs.redpanda.com/streaming/current/deploy/console/linux/deploy/). ### [](#verify-the-installation)Verify the installation To verify that the Redpanda cluster is up and running, use `rpk` to get information about the cluster: ```bash rpk cluster info ``` You should see a list of advertised addresses. To create a topic: ```bash rpk topic create ``` If topics were initially created in a test environment with a replication factor of `1`, use `rpk topic alter-config` to change the topic replication factor: ```bash rpk topic alter-config --set replication.factor=3 ``` ### [](#enable-monitoring)Enable monitoring [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/). Observability is essential in production environments. ## [](#custom-deployment)Custom deployment This section provides information for creating your own automation for deploying Redpanda clusters without using any of the tools that Redpanda supports for setting up a cluster, such as Ansible Playbook, Helm Chart, or Kubernetes Operator. > 💡 **TIP** > > Redpanda strongly recommends using one of these supported deployment tools. See [Automate Deploying for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/). ### [](#configure-a-bootstrap-file)Configure a bootstrap file Redpanda cluster configuration is written with the Admin API and the `rpk cluster config` CLIs. In the special case where you want to provide configuration to Redpanda before it starts for the first time, you can write a `.bootstrap.yaml` file in the same directory as `redpanda.yaml`. This file is only read on the first startup of the cluster. Any subsequent changes to `.bootstrap.yaml` are ignored, so changes to cluster configuration must be done with the Admin API. The content format is a YAML dictionary of cluster configuration properties. For example, to initialize a cluster with Admin API authentication enabled and a single superuser, the `.bootstrap.yaml` file would contain the following: ```yaml admin_api_require_auth: true superusers: - alice ``` With this configuration, the Admin API is not accessible until you bootstrap a user account. ### [](#bootstrap-a-user-account)Bootstrap a user account When using username/password authentication, it’s helpful to be able to create one user before the cluster starts for the first time. Do this by setting the `RP_BOOTSTRAP_USER` environment variable when starting Redpanda for the first time. The value has the format `RP_BOOTSTRAP_USER=username:password[:mechanism]`. The only supported values for `mechanism` are `SCRAM-SHA-512` or `SCRAM-SHA-256`; if it is omitted, Redpanda defaults to `SCRAM-SHA-256`. For example: `RP_BOOTSTRAP_USER=alice:letmein:SCRAM-SHA-512`. > 📝 **NOTE** > > `RP_BOOTSTRAP_USER` only creates a user account. You must still set up authentication using cluster configuration. ### [](#secure-the-admin-api)Secure the Admin API The Admin API is used to create SASL user accounts and ACLs, so it’s important to think about how you secure it when creating a cluster. - No authentication, but listening only on 127.0.0.1: This may be appropriate if your Redpanda processes are running in an environment where only administrators can access the host. - mTLS authentication: You can generate client and server x509 certificates before starting Redpanda for the first time, refer to them in `redpanda.yaml`, and use the client certificate when accessing the Admin API. - Username/password authentication: Use the combination of `admin_api_require_auth`, `superusers`, and `RP_BOOTSTRAP_USER` to access the Admin API username/password authentication. You probably still want to enable TLS on the Admin API endpoint to protect credentials in flight. ### [](#configure-the-seed-servers)Configure the seed servers Seed servers help new brokers join a cluster by directing requests from newly-started brokers to an existing cluster. The [`seed_servers`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#seed_servers) broker property controls how Redpanda finds its peers when initially forming a cluster. It is dependent on the [`empty_seed_starts_cluster`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#empty_seed_starts_cluster) broker property. Starting with Redpanda version 22.3, you should explicitly set `empty_seed_starts_cluster` to `false` on every broker, and every broker in the cluster should have the same value set for `seed_servers`. With this set of configurations, Redpanda clusters form with these guidelines: - When a broker starts and it is a seed server (its address is in the `seed_servers` list), it waits for all other seed servers to start up, and it forms a cluster with all seed servers as members. - When a broker starts and it is not a seed server, it sends requests to the seed servers to join the cluster. It is essential that all seed servers have identical values for the `seed_servers` list. Redpanda strongly recommends at least three seed servers when forming a cluster. Each seed server decreases the likelihood of unintentionally forming a split brain cluster. To ensure brokers can always discover the cluster, at least one seed server should be available at all times. By default, for backward compatibility, `empty_seed_starts_cluster` is set to `true`, and Redpanda clusters form with the guidelines used prior to version 22.3: - When a broker starts with an empty `seed_servers` list, it creates a single broker cluster with itself as the only member. - When a broker starts with a non-empty `seed_servers` list, it sends requests to the brokers in that list to join the cluster. You should never have more than one broker with an empty `seed_servers` list, which would result in the creation of multiple clusters. > ❗ **IMPORTANT** > > Redpanda expects its storage to be persistent, and it’s an error to erase a broker’s drive and restart it. However, in some environments (like when migrating to a different Node pool on Kubernetes), truly persistent storage is unavailable, and brokers may find their data volumes erased. For such environments, Redpanda recommends setting `empty_seed_starts_cluster` to false and designating a set of seed servers such that they couldn’t lose their storage simultaneously. ### [](#do-not-configure-broker-ids)Do not configure broker IDs Redpanda automatically generates unique broker IDs for each new broker and assigns it to the [`node_id`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) field in the broker configuration. This ensures safe and consistent cluster operations without requiring manual configuration. > ⚠️ **WARNING: Do not set node_id manually.** > > Do not set `node_id` manually. > > Redpanda assigns unique IDs automatically to prevent issues such as: > > - Brokers with empty disks rejoining the cluster. > > - Conflicts during recovery or scaling. > > > Manually setting or reusing `node_id` values, even for decommissioned brokers, can cause cluster inconsistencies and operational failures. ## [](#perform-a-self-test)Perform a self test To understand the performance capabilities of your Redpanda cluster, Redpanda offers built-in self-test features that evaluate the performance of both disk and network operations. For more information, see [Disk and network self-test benchmarks](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/#self-test). ### [](#upgrade-considerations)Upgrade considerations Deployment automation should place each broker into maintenance mode and wait for it to drain leadership before restarting it with a newer version of Redpanda. For more information, see [Upgrade](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/). If upgrading multiple [feature release versions of Redpanda](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/#find-a-new-version) in succession, make sure to verify that each version upgrades to completion before proceeding to the next version. You can verify by reading the `/v1/features` Admin API endpoint and checking that `cluster_version` has increased. Starting with Redpanda version 23.1, the `/v1/features` endpoint also includes a `node_latest_version` attribute, and installers can verify that the cluster has activated any new functionality from a previous upgrade by checking for `cluster_version` == `node_latest_version`. ## [](#next-steps)Next steps - If clients connect from a different subnet, see [Configure Listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/). - Observability is essential in production environments. See [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/). ## [](#suggested-reading)Suggested reading - [Configure Cluster Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/) - [Redpanda Console Configuration](https://docs.redpanda.com/streaming/current/console/config/configure-console/) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 55: Production Readiness Checklist **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-readiness.md --- # Production Readiness Checklist > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Production Readiness Checklist latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/production-readiness page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/production-readiness.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/production-readiness.adoc page-git-created-date: "2025-08-15" page-git-modified-date: "2026-07-28" support-status: supported --- Before running a production workload on Redpanda, follow this readiness checklist. Redpanda Data recommends using the [automated deployment instructions](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/) with Ansible. If you cannot deploy with Ansible, use the [manual deployment instructions](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/). > 📝 **NOTE** > > For Kubernetes deployments, see the [Production Readiness Checklist for Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-production-readiness/). ## [](#critical-requirements)Critical requirements The Critical requirements checklist helps you to confirm that: - All required defaults and configuration items are specified. - You have the optimal hardware setup. - Security is enabled. - You are set up to run in production. ### [](#redpanda-license)Redpanda license Check that the [Redpanda License](https://docs.redpanda.com/streaming/current/get-started/licensing/) has been loaded into the cluster configuration. This is required to enable [Enterprise](https://docs.redpanda.com/streaming/current/get-started/licensing/#redpanda-enterprise-edition) features. Input ```bash rpk cluster license info ``` Output ```bash LICENSE INFORMATION =================== Organization: Redpanda Owlshop LLC Type: enterprise Expires: Mar 25 2025 ``` ### [](#cluster-health)Cluster health Check that all brokers are connected and running. Run [`rpk cluster info`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-info/) to check the health of the cluster. No nodes should be down, and there should be zero leaderless or under-replicated partitions. Then run [`rpk cluster health`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-health/). The cluster should be listed as healthy. Input ```bash rpk cluster info ``` Output ```bash CLUSTER ======= redpanda.be267958-279d-49cd-ae86-98fc7ed2de48 BROKERS ======= ID HOST PORT RACK 0* 54.70.51.189 9092 us-west-2a 1 35.93.178.18 9092 us-west-2b 2 35.91.121.126 9092 us-west-2c ``` Input ```bash rpk cluster health ``` Output ```bash CLUSTER HEALTH OVERVIEW ======================= Healthy: true Unhealthy reasons: [] Controller ID: 0 All nodes: [0 1 2] Nodes down: [] Leaderless partitions (0): [] Under-replicated partitions (0): [] ``` ### [](#production-mode)Production mode enabled Check that Redpanda is running in production mode. To check the status of a Redpanda broker, check its broker configuration in `/etc/redpanda/redpanda.yaml`. Both [`developer_mode`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#developer_mode) and [`overprovisioned`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-start/) should be `false` or should not be present in the file. If either configuration is set to `true` on any broker, then the cluster is not in full production mode and must be corrected. Input ```bash grep -E 'developer_mode|overprovisioned' /etc/redpanda/redpanda.yaml ``` Output ```bash developer_mode: false overprovisioned: false ``` ### [](#redpanda-reqs)System meets Redpanda requirements Run [`sudo rpk redpanda check`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-check/) to ensure that your system meets Redpanda’s requirements. > 📝 **NOTE** > > This command requires sudo because it’s looking in `/proc` or `/sys`, which may be read restricted. Input ```bash sudo rpk redpanda check ``` Output ```bash System check results CONDITION REQUIRED CURRENT SEVERITY PASSED Ballast file present true true Warning true Clock Source tsc tsc Warning true Config file valid true true Fatal true Connections listen backlog size >= 4096 4096 Warning true Data directory filesystem type xfs xfs Warning true Data directory is writable true true Fatal true Data partition free space [GB] >= 10 1755.29 Warning true Dir '/var/lib/redpanda/data' IRQs affinity set true true Warning true Dir '/var/lib/redpanda/data' IRQs affinity static true true Warning true Dir '/var/lib/redpanda/data' nomerges tuned true true Warning true Dir '/var/lib/redpanda/data' scheduler tuned true true Warning true Free memory per CPU [MB] 2048 per CPU 7659 Warning true Fstrim systemd service and timer active true true Warning true I/O config file present true true Warning true Kernel Version 3.19 5.15.0-1056-aws Warning true Max AIO Events >= 1048576 1048576 Warning true Max syn backlog size >= 4096 4096 Warning true NIC IRQs affinity static true true Warning true NTP Synced true true Warning true RFS Table entries >= 32768 32768 Warning true Swap enabled true true Warning true Swappiness 1 1 Warning true Transparent huge pages active true true Warning true ``` ### [](#redpanda-lmr)Latest Redpanda version Check that Redpanda is running the [latest point release](https://github.com/redpanda-data/redpanda/releases) on every node for the major version you’re on. Input ```bash /usr/bin/redpanda --version ``` Output ```bash 26.2.2 - fd30e45 ``` ### [](#cpu-memory)Correct CPUs and memory configured Check that you have the [correct number of CPUs and sufficient memory](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/requirements/#cpu-and-memory) to run Redpanda. Input ```bash journalctl -u redpanda | grep "System resources" ``` Output ```bash Mar 25 12:16:18 ip-172-31-10-199 rpk[3957]: INFO 2024-03-25 12:16:18,105 [shard 0:main] main - application.cc:350 - System resources: { cpus: 8, available memory: 55.578GiB, reserved memory: 3.890GiB} ``` ### [](#mounted-disks)Disks correctly mounted Check that the correct disks are mounted, and if multiple devices are used, they are configured as RAID-0. Other RAID configurations can have significantly worse latencies. The file system should be type XFS. If XFS is unavailable, ext4 is an appropriate alternative. Input ```bash grep data_directory /etc/redpanda/redpanda.yaml data_directory: /var/lib/redpanda/data df -khT /var/lib/redpanda/data ``` Output for NVMe with XFS ```bash Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1 xfs 1.8T 14G 1.8T 1% /mnt/vectorized ``` Output for mdadm RAID mount point ```bash Filesystem Type Size Used Avail Use% Mounted on /dev/md0 xfs 14T 99G 14T 1% /mnt/vectorized ``` Example for how to get more details about the RAID array: Input ```bash mdadm --detail /dev/md0 ``` Output ```bash /dev/md0: Version : 1.2 Creation Time : Thu Apr 18 11:03:46 2024 Raid Level : raid0 Array Size : 14648172544 (13969.59 GiB 14999.73 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Apr 18 11:03:46 2024 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Layout : -unknown- Chunk Size : 512K Consistency Policy : none Name : ip-172-31-24-82:0 (local to host ip-172-31-24-82) UUID : e9574118:10d562bf:ed3ca2d9:68ccc3a6 Events : 0 Number Major Minor RaidDevice State 0 259 2 0 active sync /dev/nvme2n1 1 259 0 1 active sync /dev/nvme1n1 ``` Use these results to verify that the expected disks are present and the expected RAID level is set. (Typically, this would be `raid0` in a production system, as data resilience is provided by Raft across Redpanda brokers, rather than by RAID.) ### [](#auth-enable)Authentication enabled Check that [authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/) is set up (or other mitigations are in place). Without SASL authentication enabled, anybody can potentially connect to the Redpanda brokers. Input ```bash rpk cluster config get kafka_enable_authorization ``` Output ```bash true ``` ### [](#super-users)Superusers configured Check that the [Admin API is secured](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/#secure-the-admin-api), and any users defined in the superusers configuration are appropriately protected with strong credentials. See also: xref:manage:security/authentication.adoc#create-superusers ### [](#tls-enabled)TLS enabled Check that all public interfaces have [TLS enabled](https://docs.redpanda.com/streaming/current/manage/security/encryption/). Input ```bash journalctl -u redpanda.service | grep tls ``` Output ```bash Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,513 [shard 0:main] main - application.cc:772 - redpanda.cloud_storage_disable_tls:0 - Disable TLS for all S3 connections Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,514 [shard 0:main] main - application.cc:772 - redpanda.kafka_mtls_principal_mapping_rules:{nullopt} - Principal Mapping Rules for mTLS Authentication on the Kafka API Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,514 [shard 0:main] main - application.cc:772 - **redpanda.admin_api_tls:{{name: , tls_config: { enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 }}} - TLS configuration for admin HTTP server Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - **redpanda.kafka_api_tls:{{name: , tls_config: { enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 }}} - TLS configuration for Kafka API endpoint Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - **redpanda.rpc_server_tls:{ enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 } - TLS configuration for RPC server Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - pandaproxy.pandaproxy_api_tls:{} - TLS configuration for Pandaproxy api Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - **pandaproxy_client.broker_tls:{ enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 } - TLS configuration for the brokers Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - **schema_registry.schema_registry_api_tls:{{name: , tls_config: { enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 }}} - TLS configuration for Schema Registry API Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - **schema_registry_client.broker_tls:{ enabled: 1** key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 } - TLS configuration for the brokers Jun 06 12:41:35 ip-172-31-31-199 rpk[9673]: INFO 2024-06-06 12:41:35,515 [shard 0:main] main - application.cc:772 - audit_log_client.broker_tls:{ enabled: 1 key/cert files: {{ key_file: /etc/redpanda/certs/node.key cert_file: /etc/redpanda/certs/node.crt }} ca file: {/etc/redpanda/certs/truststore.pem} client_auth_required: 0 } - TLS configuration for the brokers ``` Using the logs on each broker, check to verify that the following interfaces have TLS enabled: - Kafka API - Admin REST API - Internal RPC Server - Schema Registry - HTTP Proxy (Pandaproxy) In the logs, verify `enabled: 1`. See also: [Multiple listeners](https://docs.redpanda.com/streaming/current/manage/security/listener-configuration/#multiple-listeners) > 📝 **NOTE** > > You can also use the [`/v1/security/report`](https://docs.redpanda.com/api/doc/admin/operation/operation-get_security_report) Admin API endpoint to generate a security report for your cluster and verify TLS, authentication, and authorization settings: > > ```bash > curl 'http://localhost:9644/v1/security/report' > ``` ### [](#redpanda-tuners)Run Redpanda tuners Check that you have run tuners on all cluster hosts. This can have a significant impact on latency and throughput. [Redpanda tuners](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/) ensure that the operating system is configured for optimal performance. In Kubernetes, you may need to run the tuners on the hosts themselves, rather than in containers. Input ```bash systemctl status redpanda-tuner ``` Output ```bash redpanda-tuner.service - Redpanda Tuner Loaded: loaded (/lib/systemd/system/redpanda-tuner.service; enabled; vendor preset: enabled) Active: active (exited) since Mon 2024-03-25 12:03:51 UTC; 48min ago Process: 3795 ExecStart=/usr/bin/rpk redpanda tune all $CPUSET (code=exited, status=0/SUCCESS) Main PID: 3795 (code=exited, status=0/SUCCESS) Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: cpu true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: disk_irq true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: disk_nomerges true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: disk_scheduler true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: disk_write_cache false true false Disk write cache tuner is only supported in GCP Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: fstrim false false true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: net true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: swappiness true true true Mar 25 12:03:51 ip-172-31-10-199 rpk[3795]: transparent_hugepages false false true Mar 25 12:03:51 ip-172-31-10-199 systemd[1]: Finished Redpanda Tuner. ``` Check that [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/) has been run on all hosts. Ensure that the mountpoint listed in this configuration file matches the mountpoint for Redpanda’s data directory, usually `/var/lib/redpanda`. See [Generate optimal I/O configuration settings](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/#generate-optimal-io-configuration-settings). See also: - [Tune the Linux kernel for production](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/#tune-the-linux-kernel-for-production) - [Tune Kubernetes Worker Nodes for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-tune-workers/) Input ```bash cat /etc/redpanda/io-config.yaml disks: - mountpoint: /mnt/vectorized read_iops: 413115 read_bandwidth: 1882494592 write_iops: 182408 write_bandwidth: 788050688 ``` ### [](#disk-perf)Check disk performance Run [`rpk cluster self-test status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-self-test-status/) to ensure that disk performance is within an acceptable range. See also: [Cluster Diagnostics](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/) Input ```bash rpk cluster self-test status ``` Output ```bash NODE ID: 1 | STATUS: IDLE ========================= NAME 512KB sequential r/w throughput disk test INFO write run TYPE disk TEST ID e13b2c93-2417-458b-87be-fac409089513 TIMEOUTS 0 DURATION 30000ms IOPS 984 req/sec THROUGHPUT 492.1MiB/sec LATENCY P50 P90 P99 P999 MAX 4095us 4095us 4351us 4607us 5119us ``` ### [](#hostnames-interfaces)Advertised hostnames use correct interfaces Check that the advertised hostnames are operating on the correct network interfaces. For clusters with multiple interfaces (for example, a public and private IP address), set [`advertised_kafka_api`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#advertised_kafka_api) to the public interface and set [`advertised_rpc_api`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#advertised_rpc_api) to the private interface. These should be hostnames, not IP addresses. Example ```bash grep -A2 advertised /etc/redpanda/redpanda.yaml advertised_kafka_api: - address: myhostname.customdomain.com port: '9092' advertised_rpc_api: address: myinternalhostname.customdomain.com port: '33145' ``` ### [](#continuous-db)Confirm Continuous Data Balancing configuration Run [`rpk cluster config get partition_autobalancing_mode`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-config-get/) to ensure that [Continuous Data Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/) is configured and enabled. Input ```bash rpk cluster config get partition_autobalancing_mode ``` Output ```bash continuous ``` ### [](#debug-bundle)Generate debug bundle Check that you can generate a debug bundle from each host and upload it to [Redpanda support](https://support.redpanda.com/hc/en-us/requests/new). This is how you can collect data and export it to Redpanda support. Input ```bash sudo rpk debug bundle ``` Output ```bash Creating bundle file... Debug bundle saved to '1711372017-bundle.zip' ``` See also: - [rpk debug bundle](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-debug/rpk-debug-bundle/) - [Diagnostics Bundles in Kubernetes](https://docs.redpanda.com/streaming/current/troubleshoot/debug-bundle/generate/kubernetes/) ### [](#topic-rf)Topic replication factor Check that all topics have a replication factor greater than one. Input ```bash rpk topic list ``` Output ```bash NAME PARTITIONS REPLICAS bad 1 1 good 1 3 ``` Redpanda Data recommends that you set `minimum_topic_replications` and `default_topic_replications` to at least 3. ```bash rpk cluster config set minimum_topic_replications=3 rpk cluster config set default_topic_replications=3 ``` See also: [Change topic replication factor](https://docs.redpanda.com/streaming/current/migrate/data-migration/#change-topic-replication-factor) ### [](#maintenance-mode)No brokers in maintenance mode Check that no brokers are in maintenance mode. Input ```bash rpk cluster maintenance status ``` Output ```bash NODE-ID ENABLED FINISHED ERRORS PARTITIONS ELIGIBLE TRANSFERRING FAILED 1 false - - - - - - 2 false - - - - - - 3 false - - - - - - ``` See also: [Remove a broker from maintenance mode](https://docs.redpanda.com/streaming/current/manage/node-management/#place-a-broker-in-maintenance-mode) ### [](#decom-state)No brokers in decommissioned state Check that no brokers are in a decommissioned state. Input ```bash rpk cluster info -b --detailed ``` Output ```bash NODE-ID NUM-CORES MEMBERSHIP-STATUS IS-ALIVE BROKER-VERSION 0 1 active true v24.1.6 - 5e880f6fd1a610d0991b00e32c012a03b14888ca 1 1 active true v24.1.6 - 5e880f6fd1a610d0991b00e32c012a03b14888ca 2 1 active true v24.1.6 - 5e880f6fd1a610d0991b00e32c012a03b14888ca ``` See also: [Decommission Brokers](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/decommission-brokers/) ## [](#recommended-requirements)Recommended requirements The Recommended requirements checklist confirms that you can monitor and support your environment on a sustained basis. It includes the following checks: - You have adhered to day-2 operations best practices. - You can diagnose and recover from issues or failures. ### [](#environment-configuration)Environment configuration Check that you have a [development environment](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/dev-deployment/) or test environment configured to evaluate upgrades and new versions before rolling them straight to production. ### [](#monitoring)Monitoring Check that [monitoring](https://docs.redpanda.com/streaming/current/manage/monitoring/) is configured with [Prometheus](https://docs.redpanda.com/streaming/current/manage/monitoring/#configure-prometheus), [Grafana](https://docs.redpanda.com/streaming/current/manage/monitoring/#generate-grafana-dashboard), or [Datadog](https://www.datadoghq.com/product/log-management/) to scrape metrics from all Redpanda brokers at a regular interval. ### [](#system-log-retention)System log retention Check that system logs are being captured and stored for an appropriate period of time (minimally, 7 days). On bare metal, this may be journald. On Kubernetes you may need to have fluentd or an equivalent configured, with logs sent to a central location. See also: [rpk debug bundle](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-debug/rpk-debug-bundle/) ### [](#upgrade-policy)Upgrade policy Check that you have an upgrade policy defined and implemented. Redpanda Enterprise Edition supports [rolling upgrades](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/#perform-a-rolling-upgrade), so upgrades do not require downtime. However, make sure that upgrades are scheduled on a regular basis, ideally using automation such as [Ansible](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/#use-ansible-to-install-redpanda) or [Helm](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/). ### [](#high-availability)High availability If you have [high availability](https://docs.redpanda.com/streaming/current/manage/high-availability/) requirements, check that the cluster is configured across multiple availability zones or fault domains. Input ```bash rpk cluster info ``` Output ```bash CLUSTER ======= redpanda.be267958-279d-49cd-ae86-98fc7ed2de48 BROKERS ======= ID HOST PORT RACK 0* 54.70.51.189 9092 us-west-2a 1 35.93.178.18 9092 us-west-2b 2 35.91.121.126 9092 us-west-2c ``` Check that [rack awareness](https://docs.redpanda.com/streaming/current/manage/rack-awareness/#configure-rack-awareness) is configured correctly. Input ```bash rpk cluster config get enable_rack_awareness ``` Output ```bash true ``` See also: - [Multi-AZ deployments](https://docs.redpanda.com/streaming/current/manage/high-availability/#multi-az-deployments) - [Configure rack awareness in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rack-awareness/#configure-rack-awareness) ## [](#advanced-requirements)Advanced requirements The Advanced requirements checklist ensures full enterprise readiness, indicates that your system is operating at the highest level of availability, and can prevent or recover from the most serious incidents. The advanced requirements confirm the following: - You are proactively monitoring mission-critical workloads, business continuity solutions, and integration into enterprise security systems. - Your enterprise is ready to run mission-critical workloads. ### [](#configure-alerts)Configure alerts A standard set of alerts for [Grafana](https://docs.redpanda.com/streaming/current/manage/monitoring/#generate-grafana-dashboard) or [Prometheus](https://docs.redpanda.com/streaming/current/manage/monitoring/#configure-prometheus) is provided in the [GitHub Redpanda observability repo](https://github.com/redpanda-data/observability). However, you should customize these alerts for your specific needs. See also: [Monitoring Metrics](https://docs.redpanda.com/streaming/current/reference/monitor-metrics/) ### [](#backup-and-disaster-recovery-dr-solution)Backup and disaster recovery (DR) solution Check that you have a backup and disaster recovery (DR) solution in place. You can configure backup and restore using [Tiered Storage Whole Cluster Recovery](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/whole-cluster-restore/). Be sure to confirm that the backup and DR solution has been tested. For disaster recovery, confirm that a standby cluster is configured and running with replication (such as [MirrorMaker2](https://docs.redpanda.com/streaming/current/migrate/data-migration/)). Also verify that your monitoring ensures that MirrorMaker2 is running and checks replication traffic. See [High-availability deployment of Redpanda: Patterns and considerations](https://redpanda.com/blog/high-availability-software-deployment-patterns-part-1) for more details about HA and DR options. ### [](#deployment-automation)Deployment automation Review your deployment automation. Specifically, if you need to reprovision a cluster, ensure that cluster installation is managed using automation such as [Terraform](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/#use-terraform-to-set-up-infrastructure), [Ansible](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/#use-ansible-to-install-redpanda), or [Helm](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/), and that the configuration is saved in source control. ### [](#audit-logs)Audit logs Check that your [audit logs](https://docs.redpanda.com/streaming/current/manage/audit-logging/#audit-log-flow) are forwarded to an enterprise security information and event management (SIEM) system. ### [](#monitor-security-settings)Monitor security settings Regularly review your cluster’s security settings using the [`/v1/security/report`](https://docs.redpanda.com/api/doc/admin/operation/operation-get_security_report) Admin API endpoint. Investigate and address any issues identified in the alerts section. Input ```bash curl 'http://localhost:9644/v1/security/report' ``` View output ```bash { "interfaces": { "kafka": [ { "name": "test_kafka_listener", "host": "0.0.0.0", "port": 9092, "advertised_host": "0.0.0.0", "advertised_port": 9092, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_method": "None", "authorization_enabled": false } ], "rpc": { "host": "0.0.0.0", "port": 33145, "advertised_host": "127.0.0.1", "advertised_port": 33145, "tls_enabled": false, "mutual_tls_enabled": false }, "admin": [ { "name": "test_admin_listener", "host": "0.0.0.0", "port": 9644, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_methods": [], "authorization_enabled": false } ] }, "alerts": [ { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_TLS", "description": "\"kafka\" interface \"test_kafka_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHN", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authentication. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHZ", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "rpc", "issue": "NO_TLS", "description": "\"rpc\" interface is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_TLS", "description": "\"admin\" interface \"test_admin_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHZ", "description": "\"admin\" interface \"test_admin_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHN", "description": "\"admin\" interface \"test_admin_listener\" is not using authentication. This is insecure and not recommended." } ] } ``` ## [](#suggested-reading)Suggested reading - [Deploy for Production: Manual](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/) - [Deploy for Production: Automated](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/) --- # Page 56: Requirements and Recommendations **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/requirements.md --- # Requirements and Recommendations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Requirements and Recommendations latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/production/requirements page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/production/requirements.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/production/requirements.adoc description: A list of requirements and recommendations for provisioning servers to run Redpanda in production. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- This topic provides the requirements and recommendations for provisioning servers to run Redpanda in production. ## [](#operating-system)Operating system - Minimum version required of RHEL/CentOS: 8. **Recommended**: 9+ - Minimum version required of Ubuntu: 20.04 LTS. **Recommended**: 22.04+ **Recommendation**: Linux kernel 4.19 or later for better performance. ## [](#number-of-workers)Number of nodes Provision one physical node or virtual machine (VM) for each Redpanda broker that you plan to deploy in your Redpanda cluster. Each Redpanda broker requires its own dedicated node for the following reasons: - **Resource isolation**: Redpanda brokers are designed to make full use of available system resources, including CPU and memory. By dedicating a node to each broker, you ensure that these resources aren’t shared with other applications or processes, avoiding potential performance bottlenecks or contention. - **External networking**: External clients should connect directly to the broker that owns the partition they’re interested in. This means that each broker must be individually addressable. As clients must connect to the specific broker that is the leader of the partition, they need a mechanism to directly address each broker in the cluster. Assigning each broker to its own dedicated node makes this direct addressing feasible, since each node will have a unique address. See [External networking](#external-networking). - **Fault tolerance**: Ensuring each broker operates on a separate node enhances fault tolerance. If one node experiences issues, it won’t directly impact the other brokers. **Recommendations**: Deploy at least three Redpanda brokers. ## [](#node-updates)Prevent automatic node upgrades Ensure that node and operating system (OS) upgrades are manually managed when running Redpanda in production. Manual control avoids unplanned reboots or replacements that disrupt Redpanda brokers, causing service downtime, data loss, or quorum instability. Common issues with automatic node upgrades include: - Hard timeouts for graceful shutdowns that do not allow Redpanda brokers enough time to complete decommissioning or leadership transitions. - Replacements or reboots without ensuring data has been safely migrated or replicated, risking data loss. - Parallel upgrades across multiple nodes, which can disrupt quorum or reduce cluster availability. **Requirements**: - Disable automatic node maintenance or upgrades. ## [](#cpu-and-memory)CPU and memory **Requirements**: - Each production node must have at least two physical CPU cores. - x86\_64 (Westmere or newer) and AWS Graviton processors are supported. - Each Redpanda broker must have at least 2 GB of memory per core. - Each Redpanda broker must have at least 2 MB of memory for each topic partition replica. The total memory available for partition replicas is determined as a percentage of the cluster’s total memory, which is controlled by the [`topic_partitions_memory_allocation_percent`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#topic_partitions_memory_allocation_percent) setting. Each partition replica consumes [`topic_memory_per_partition`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#topic_memory_per_partition) bytes from this pool. If insufficient memory is available, topic operations will fail. You can adjust the allocation ratio using `topic_partitions_memory_allocation_percent`, but doing so is not recommended, as lowering it may lead to instability or degraded performance. **Recommendations**: - Four physical cores for each node are strongly recommended. ## [](#storage)Storage **Requirements**: - NVMe (Non-Volatile Memory Express) drives are required for production deployments. NVMe drives provide the high throughput and low latency needed for optimal Redpanda performance. See also: [Disk and network self-test benchmarks](https://docs.redpanda.com/streaming/current/troubleshoot/cluster-diagnostics/diagnose-issues/#self-test). - An XFS or ext4 file system. The Redpanda data directory (`/var/lib/redpanda/data`) and the Tiered Storage cache must be mounted on an XFS or ext4 file system. > ⚠️ **CAUTION** > > The Network File System (NFS) is unsupported for use as the storage mechanism for the Redpanda data directory or for the Tiered Storage cache. **Recommendations**: - Use an XFS file system for its enhanced performance with Redpanda workloads. - For setups with multiple disks, use a RAID-0 (striped) array. It boosts speed but lacks redundancy. A disk failure can lead to data loss. ## [](#security)Security **Recommendations**: - If you’re using a cloud platform, use [IAM roles](https://docs.redpanda.com/streaming/current/manage/security/iam-roles/) to restrict access to resources in your cluster. - Secure your Redpanda cluster with TLS encryption and SASL authentication. ## [](#external-networking)External networking - For external access, each node in your cluster must have a static, externally accessible IP address. - Minimum 10 GigE (10 Gigabit Ethernet) connection to ensure: - High data throughput - Reduced data transfer latency - Scalability for increased network traffic ## [](#tuning)Tuning Before deploying Redpanda to production, each node that runs Redpanda must be tuned to optimize the Linux kernel for Redpanda processes. See [Deploy for Production: Manual](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/). ## [](#object-storage-providers-for-tiered-storage)Object storage providers for Tiered Storage Redpanda supports the following storage providers for Tiered Storage: - Amazon Simple Storage Service (S3) - Google Cloud Storage (GCS), using the Google Cloud Platform S3 API - Azure Blob Storage (ABS) ## [](#cloud-instance-types)Cloud instance types **Recommendations**: - Use a cloud instance type that supports locally attached NVMe devices with an XFS file system. NVMe devices offer high I/O operations per second (IOPS) and minimal latency, while XFS offers enhanced performance with Redpanda workloads. ### [](#amazon)Amazon - General purpose: General-purpose instances provide a balance of compute, memory, and networking resources, and they can be used for a variety of diverse workloads. - [M5d](https://aws.amazon.com/ec2/instance-types/m5/) - [M5ad](https://aws.amazon.com/ec2/instance-types/m5/) - [M5dn](https://aws.amazon.com/ec2/instance-types/m5/) - [M6gd](https://aws.amazon.com/ec2/instance-types/m6g/) - [M7gd](https://aws.amazon.com/ec2/instance-types/m7g/) - Memory optimized: Memory-optimized instances are designed to deliver fast performance for workloads that process large data sets in memory. - [R5ad](https://aws.amazon.com/ec2/instance-types/r5/) - [R5d](https://aws.amazon.com/ec2/instance-types/r5/) - [R5dn](https://aws.amazon.com/ec2/instance-types/r5/) - [R6gd](https://aws.amazon.com/ec2/instance-types/r6g/) - [R6id](https://aws.amazon.com/ec2/instance-types/r6i/) - [R6idn](https://aws.amazon.com/ec2/instance-types/r6i/) - [R7gd](https://aws.amazon.com/ec2/instance-types/r7g/) - [X2gd](https://aws.amazon.com/ec2/instance-types/x2/) - [X2idn](https://aws.amazon.com/ec2/instance-types/x2i/) - [X2iedn](https://aws.amazon.com/ec2/instance-types/x2i/) - [z1d](https://aws.amazon.com/ec2/instance-types/z1d/) - Storage optimized: Storage-optimized instances are designed for workloads that require high, sequential read and write access to very large data sets on local storage. They are optimized to deliver tens of thousands of low-latency, random IOPS to applications. - [I4g, Is4gen, Im4gn](https://aws.amazon.com/ec2/instance-types/i4g/) - [I4i](https://aws.amazon.com/ec2/instance-types/i4i/) - [I3](https://aws.amazon.com/ec2/instance-types/i3/) - [I3en](https://aws.amazon.com/ec2/instance-types/i3en/) - Compute optimized: Compute-optimized instances deliver cost-effective high performance at a low price per compute ratio for running advanced compute-intensive workloads. - [C5d](https://aws.amazon.com/ec2/instance-types/c5/) - [C5ad](https://aws.amazon.com/ec2/instance-types/c5/) ### [](#azure)Azure - General purpose: General purpose VM sizes provide balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers. - [Standard\_D2d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) - [Standard\_D4d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) - [Standard\_D32d\_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/ddv5-series?tabs=sizebasic) ### [](#google)Google - General purpose: The general-purpose machine family has the best price-performance with the most flexible vCPU to memory ratios, and provides features that target most standard and cloud-native workloads. - [C3 machine series with local SSD](https://cloud.google.com/compute/docs/general-purpose-machines#c3-with-local-ssd) - [N2 machine series](https://cloud.google.com/compute/docs/general-purpose-machines#n2_series) - [N2D machine series](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines) - Memory optimized: The memory-optimized machine family provides the most compute and memory resources of any Compute Engine machine family offering. They are ideal for workloads that require higher memory-to-vCPU ratios than the high-memory machine types in the general-purpose N1 machine series. - [M3 machine series](https://cloud.google.com/compute/docs/memory-optimized-machines#m3_series) - Compute optimized: Compute-optimized VM instances are ideal for compute-intensive and high-performance computing (HPC) workloads. - [C2D machine series](https://cloud.google.com/compute/docs/compute-optimized-machines#c2d_series) - [C2 machine series](https://cloud.google.com/compute/docs/compute-optimized-machines#c2_machine_types) ## [](#next-steps)Next steps - [Linux Deployment Options](../) ## [](#suggested-reading)Suggested reading - [Sizing Guidelines](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing/) - [Manage Disk Space](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 57: Sizing Use Cases **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing-use-cases.md --- # Sizing Use Cases > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Sizing Use Cases latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/sizing-use-cases page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/sizing-use-cases.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/sizing-use-cases.adoc description: How to size Redpanda clusters for low, medium, and high throughput use cases in your data center or in object storage. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- The following scenarios provide estimates and advice for sizing Redpanda clusters for different throughput and retention use cases in your data center and in object storage. For details about sizing considerations, see [Sizing Guidelines](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing/). > 📝 **NOTE** > > These use cases assume a happy path with known metrics and expected outputs, but many other factors can influence performance, such as batch size and other sources of network traffic. ## [](#low-throughput)Low throughput | Metric | Value | | --- | --- | | Producer throughput | 75 MB/sec (600 Mbps) | | Producer rate | 300 messages per second | | Consumer throughput | 75 MB/sec (600 Mbps) | | Consumer rate | 300 messages per second | | Data retention | 3 days | | Average message size | 250 KB | | Failure tolerance | 1 node | In this use case, despite the relatively low throughput of 150 MB/sec (producer plus consumer), it’s important to calculate the expected bandwidth utilization and to use a network testing tool like iPerf to verify that the bandwidth is available and sustainable. With a single topic with a replication factor of three, producing 75 MB/sec generates an additional 150 MB/sec of data transmitted over the network for replication, and it generates a further 75 MB/sec for the consumers. The 150 MB/sec of bandwidth for replication is full duplex (where each byte sent by a broker is received by some other broker). The 75 MB/sec producer and consumer flows, however, are half-duplex, because the client endpoint in each case is outside of the cluster. Therefore, the intra-cluster bandwidth is 225 MB for incoming and outgoing flows: - 150 MB/sec of intra-cluster full duplex bandwidth - 75 MB/sec of ingress from producers - 75 MB/sec of egress to consumers Three nodes satisfy Redpanda’s minimum deployment requirement (so Raft can form quorums) and also the single node failure tolerance. Divide the bandwidth total by the node count (3) to get the per-node bandwidth requirements. The throughput is not high enough to warrant any more than two cores and a single NVMe SSD disk. Be mindful of predicted growth of CPU and disk usage, and estimate when the cluster might need to scale up or scale out. With an average producer throughput of 75 MB/sec and a replication factor of three, each node writes 254 GB of data each hour and 6.4 TB of data each day. For three days of data retention, each node needs at least 20 TB of storage. > 📝 **NOTE** > > This assumes that each node could be a leader or a follower, and there are a sufficient number of partitions for good distribution. A typical node is the leader for 1/Nth of the partitions in a cluster with N nodes and a follower for 2/Nths of the partitions. However, the per node bandwidth could vary if distribution is uneven. You may have an inexact distribution of load during Redpanda partition balancing or when the client library doesn’t write to each partition evenly. The following machine specifications provide a minimum for a bare metal cluster or its cloud-based equivalent. | | Bare Metal | AWS | GCP | Azure | | --- | --- | --- | --- | --- | | Instance Type | - | m5.large | n2-standard-2 | F2s_v2 | | Nodes | 3 | 3 | 3 | 3 | | Cores | 2 | 2 | 2 | 2 | | Memory | 4 GB | 8 GB | 8 GB | 4 GB | | Instance Storage | 20 TB (NVMe) | - | - | 16 GB (SSD) | | Persistent Storage | - | 20 TB (gb3) | 20 TB (Zonal SSD PD) | 20 TB (Standard SSD) | | Network | 4 Gbps | Up to 10 Gbps | 10 Gbps | 5 Gbps | | Tiered Storage | False | False | False | False | ## [](#medium-throughput)Medium throughput | Metric | Value | | --- | --- | | Producer throughput | ~500 MB/sec (~4,000 Mbps) | | Producer rate | 2,000 messages per second | | Consumer throughput | ~1,000 MB/sec (~8,000 Mbps) | | Consumer rate | 4,000 messages per second | | Data retention | 24 hours | | Average message size | 250 KB | | Failure tolerance | 1 node | Producing an average of 500 MB/sec and consuming an average of 1,000 MB/sec equates to 2,500 MB/sec (20 Gbps) of network bandwidth for replication traffic. This is attainable but expensive with cloud providers, and these speeds are not as prevalent within a typical data center. With at least one partition for each core, the 500 MB/sec of data from producers is evenly distributed between the nodes. For example, with three nodes, each node receives approximately 167 MB/sec. However, that bandwidth value increases with data replication. | Producer MB/sec | Consumer MB/sec | Avg. Replication Factor | Nodes | Writes per node MB/sec | Reads per node MB/sec | | --- | --- | --- | --- | --- | --- | | 500 | 1,500 | 3 | 3 | 500/3 * 3 = 500 | 1500/3 = 500 | | 500 | 1,500 | 3 | 5 | 500/5 * 3 = 300 | 1500/5 = 300 | | 500 | 1,500 | 3 | 7 | 500/7 * 3 = 215 | 1500/7 = 215 | | 500 | 1,500 | 5 | 7 | 500/7 * 5 = 358 | 1500/7 = 215 | The additional 500 MB/sec for consumer throughput is for Tiered Storage and the bandwidth required to archive log segments to object storage. When Tiered Storage is enabled on a topic, it essentially adds another consumer’s worth of bandwidth on the network. To balance the available local disk, consider exactly how many reads can be serviced from local storage. Different instance types or locally attached NVMe SSD disks provide different amounts of local storage, and therefore different amounts of available data without going back to object storage. A topic with Tiered Storage enabled can write data to faster local storage managed by local retention settings, and at the same time, it can write data to object storage managed by different retention settings, or left to grow for a longer period. Consumers that generally keep up with producers stream from local storage, but at this velocity that window of opportunity is narrower. The object store enables a consumer to read from an older offset when necessary. | | Bare Metal | AWS | GCP | Azure | | --- | --- | --- | --- | --- | | Instance Type | - | i3en.6xlarge | n2-standard-32 | F48s_v2 | | Nodes | 3 | 3 | 3 | 3 | | Cores | 24 | 24 | 32 | 48 | | Memory | 192 GB | 192 GB | 128 GB | 96 GB | | Instance Storage | 30 TB (NVMe) | 15 TB (NVM3) | 9 TB (SSD) | 384 GiB (SSD) | | Persistent Storage | - | - | - | 20 TB (Standard SSD) | | Available Local Retention | 17 hrs | 8 hrs | 5 hrs | 9 days | | Network | 25 Gbps | 25 Gbps | 32 Gbps | 21 Gbps | | Tiered Storage | True | True | True | True | ## [](#high-throughput)High throughput | Metric | Value | | --- | --- | | Producer throughput | 1,000 MB/sec (8,000 Mbps) | | Producer rate | 4,000 messages per second | | Consumer throughput | 2,000 MB/sec (16,000 Mbps) | | Consumer rate | 8,000 messages per second | | Data retention | 24 hours | | Average message size | 250 KB | | Failure tolerance | 2 nodes | This use case has many topics, hundreds of partitions, and a high throughput. The combined producer and replication data equates to 8 Gbps of network traffic, plus 16 Gbps for the consumers and 8 Gbps for Tiered Storage. In total, that’s at least 32 Gbps of network bandwidth required to sustain this level of throughput. Writing at 1,000 MB/sec is near the upper limit of what a single NVMe disk can sustain. At this scale, you get significant performance gains by distributing the writes over many cores and disks to better leverage Redpanda’s thread-per-core model. For example, given five nodes with 24 cores each, start with at least one partition for each core (120 partitions in total) and scale up. Redpanda generates over 3 TB of writes each hour and over 80 TB each day. Local storage is going to fill up quickly, and the window of opportunity for consumers to read from local storage is going to be shorter than in the other scenarios. In this use case, [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) is essential. | | Bare Metal | AWS | GCP | Azure | | --- | --- | --- | --- | --- | | Instance Type | - | i3en.12xlarge | n2-standard-48 | F48s_v2 | | Nodes | 5 | 5 | 5 | 5 | | Cores | 24 | 48 | 48 | 48 | | Memory | 192 GB | 384 GB | 192 GB | 96 GB | | Instance Storage | 30 TB (NVMe) | 30 TB (NVM3) | 9 TB (SSD) | 384 TB (SSD) | | Persistent Storage | - | - | - | 30 TB (Ultra SSD) | | Available Local Retention | 14 hrs | 7 hrs | 4 hrs | 7 days | | Network | 25 Gbps | 25 Gbps | 32 Gbps | 21 Gbps | | Tiered Storage | True | True | True | True | ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 58: Sizing Guidelines **URL**: https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing.md --- # Sizing Guidelines > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Sizing Guidelines latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: redpanda/manual/sizing page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: redpanda/manual/sizing.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/deploy/pages/redpanda/manual/sizing.adoc description: Learn about considerations to size your Redpanda cluster to handle the volume of data being produced, replicated, and consumed. page-git-created-date: "2025-08-15" page-git-modified-date: "2025-08-15" support-status: supported --- For best performance, size your Redpanda cluster to handle the volume of data being produced, replicated, and consumed. The following variables affect cluster sizing: - Throughput of data (after compression, if applied) - Topic replication factor - Number of producers and consumers Throughput and retention requirements can cause bottlenecks in the system. On an undersized cluster, clients could saturate the available network bandwidth, a disk could run out of IOPS and be unable to keep up with writes, or you could simply run out of disk space. On an oversized Redpanda cluster, you could overpay for unnecessary infrastructure. For sizing estimates and advice for various throughput and retention scenarios, see [Sizing Use Cases](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing-use-cases/). In general, choose the number of nodes based on the following criteria, and add nodes for fault tolerance. This ensures that the system can operate with full throughput in a degraded state. - **Network bandwidth**: Total bandwidth must account for maximum simultaneous writes and reads, multiplied by the replication factor. - **Memory per core**: Allocate a minimum of 2 GB memory for each CPU core. Additional memory could improve performance. - **Memory per partition**: Allocate a minimum of 2 MB of memory for each topic partition replica. For example: If you have 10,000 total partitions with a replication factor of 3, you need at least 60 GB of memory across all brokers: 10,000 partitions × 3 replicas × 2 MB = 60,000 MB (60 GB) If these partitions are evenly distributed across three brokers, each broker needs at least 20 GB of memory. - **Storage strategy**: Use Tiered Storage to unify historical and real-time data cost-effectively. - **Performance testing**: Run hardware and Redpanda benchmark tests to establish a performance baseline. Choose instance types that prioritize storage and network performance: - **AWS**: Test with i3en (NVMe SSD), i4i (NVMe), or is4gen (Intel-based NVMe) instances, or other NVMe-backed types - **Azure**: Test with Lsv2-series instances (high I/O performance) or other high-performance storage types - **GCP**: Test with n2-standard instances with local SSD or other high-performance local storage types ## [](#sizing-considerations)Sizing considerations ### [](#network)Network To understand the network usage of a basic Redpanda cluster, consider a cluster with three nodes, a topic with a single partition and a replication factor of three, and a single producer and consumer. For every 75 MB written to the partition’s leader, 150 MB is transmitted across the network to other nodes for replication, and 75 MB is transmitted to the consumer. ![3 node Redpanda cluster](https://docs.redpanda.com/streaming/current/shared/_images/3-node-rp-cluster-alt.png) The 150 MB/sec of bandwidth for replication is full duplex (where each byte sent by a broker is received by some other broker). The 75 MB/sec producer and consumer flows, however, are half-duplex, because the client endpoint in each case is outside of the cluster. In a well-balanced scenario, the intra-cluster bandwidth is 225 MB for incoming and outgoing flows: - 150 MB/sec of intra-cluster full duplex bandwidth - 75 MB/sec of ingress from producers half-duplex bandwidth - 75 MB/sec of egress to consumers half-duplex bandwidth Even with the same amount of data produced, increasing the replication factor or the number of consumers increases bandwidth utilization. It’s important to measure the network bandwidth between nodes, and between clients and nodes, to make sure that you’re getting the expected performance from the network. This is especially important for cloud deployments where network bandwidth is not always guaranteed. Short tests may give unrealistically good results because of burst bandwidth, where instances can use a network I/O credit mechanism to burst for a limited time beyond their baseline bandwidth. To get realistic results, soak test the network to understand how it behaves over longer periods of time. The following example uses [iPerf3](https://iperf.fr/) to test the network bandwidth between two Debian-based servers. To do a full network stress test, run iPerf3 for every combination of network routes. ```bash redpanda1:~ sudo apt -y update; sudo apt -y install iperf3 redpanda1:~ iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- redpanda2:~ sudo apt update; sudo apt install iperf3 redpanda2:~ iperf3 -c redpanda1 -p 5201 -t 300 Connecting to host redpanda1, port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.11 GBytes 9.57 Gbits/sec 0 1.64 MBytes [ 5] 1.00-2.00 sec 1.11 GBytes 9.53 Gbits/sec 0 1.64 MBytes [ 5] 2.00-3.00 sec 1.11 GBytes 9.53 Gbits/sec 0 1.64 MBytes [ 5] 3.00-4.00 sec 1.11 GBytes 9.53 Gbits/sec 0 1.72 MBytes [ 5] 4.00-5.00 sec 1.11 GBytes 9.53 Gbits/sec 0 1.72 MBytes ... ``` ### [](#cpu-and-memory)CPU and memory Redpanda is designed to scale up to utilize all available hardware and scale out to distribute performance across multiple nodes. Topic partitions are the unit of parallelization in Redpanda. Adding partitions is how you scale to meet workload demands. Redpanda implements a thread-per-core programming model through its use of the [Seastar](https://seastar.io/) library. This allows Redpanda to pin each of its application threads to a CPU core to avoid context switching and blocking, significantly improving processing performance and efficiency. Redpanda can handle approximately one GB/sec of writes for each core, depending on the workload. Since NVMe disks can have a sustained write speed of over one GB/sec, it takes two cores to saturate a single NVMe disk. A general recommendation is to have 100 MB/sec for each core. Redpanda is basically a distributed transaction log with well-understood access patterns. It appends data to the end of log files and sequentially reads data from log files. Because Redpanda understands its own access patterns better than the operating system does, Redpanda chooses to bypass the Linux page cache and manages its own memory and disk I/O. This gives Redpanda complete control over the underlying hardware to optimize I/O performance, deliver predictable tail latencies, and minimize its memory footprint. A minimum of 2 GB of memory for each core is recommended, but more is better. ### [](#storage)Storage Your best storage solution for your workload depends on your performance and data retention requirements. If high throughput and low latency is most important, then use locally attached NVMe SSD disks. This is also a good option in the cloud. Just remember that in the cloud, local disks are ephemeral, so data is wiped when an instance is restarted. An alternative in the cloud is to use SSD-backed network-attached storage to persist data between instance restarts. Most cloud providers have options for guaranteeing throughput and provisioned IOPS performance of network-attached storage, although network-attached storage almost always exhibits slightly higher tail latencies than direct-attached storage. For example, AWS io2 volumes offer up to 64,000 IOPS and 1,000 MB/sec throughput with single-digit millisecond latency. This is an expensive option, so if you can trade performance for cost, then AWS gp3 volumes are a good alternative. GCP has comparable options with high-end Extreme persistent disks and the lesser SSD persistent disks. Likewise, Azure has Ultra, Premium, and Standard persistent disk options for choosing the right balance of performance versus cost. Whichever option you choose, benchmark Redpanda’s underlying storage for read and write performance, at least from an I/O perspective. Fio is a good tool for replicating Redpanda’s sequential write pattern and load. The following example shows how to run fio on a Debian-based server: > 📝 **NOTE** > > Ensure that the fio job runs against the chosen block device. By default, fio operates in the local directory. ```bash sudo apt -y update; sudo apt -y install fio cd /var/lib/redpanda/data sudo tee fio-seq-write.job >/dev/null << EOF [global] name=fio-seq-write filename=fio-seq-write rw=write bs=16K direct=1 numjobs=4 group_reporting time_based runtime=300 # 5 minute runtime [file1] size=10G ioengine=libaio iodepth=16 EOF sudo fio fio-seq-write.job ``` Key performance metrics: - IOPS = Input and output operations per second. IOPS represents how many sequential write operations per second the volume can handle. - BW = Average bandwidth measured in MB per second. Bandwidth divided by the write block size (for example, bs=16K) is the IOPS. - slat = Submission latency. The time in microseconds to submit the I/O to the kernel. - clat = Completion latency. The time in microseconds after slat until the device has completed the I/O. - lat = Overall latency in microseconds. - clat percentiles = Completion tail latency. Pay particular attention to p90 and above. This is a good indication of whether the volume can deliver predictable, consistent performance. ### [](#data-retention)Data retention Retention properties control how long messages are kept on disk before they’re deleted or compacted. You can configure data retention until message age or aggregate message size in the topic is exceeded. Setting retention properties (at the topic level or the cluster level) is the best way to prevent old messages from accumulating on disk to the point that the disk becomes full. See also: [Configure message retention](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/#configure-message-retention) and [Set retention limits](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#set-retention-limits) ### [](#tiered-storage)Tiered Storage Redpanda Tiered Storage enables multi-tiered object storage. It archives log segments to object storage in near real time while maintaining the ability for brokers to fetch and serve these archived segments to slow consumers transparently and without any client configuration. With only local storage, data retention is limited to the provisioned capacity: you must provision more nodes to increase capacity. Adding nodes is expensive, because you’re forced to overprovision infrastructure regardless of whether you need the additional compute power. In most cases, overprovisioning leads to underutilization and higher operational costs. Tiered Storage can be combined with local storage to provide long-term data retention and disaster recovery on a per-topic basis. Retention properties work the same for Tiered Storage topics and local storage topics. Data is retained in the cloud until it reaches the configured time or size limit. Ideally, a cluster should be sized such that the cluster’s local storage can service the majority of its consumers within a normal amount of lag, with Tiered Storage used to service any slow readers (for example, in the event of some downstream failure). When Tiered Storage is enabled on a topic, it copies closed log segments to the configured storage bucket or container. Log segments are closed when the value of [`log_segment_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_segment_size) has been reached, so a topic’s object store lags behind the local copy. You can set an idle timeout to force Redpanda to periodically archive the contents of open log segments to object storage. This is useful if a topic’s write rate is low and log segments are kept open for long periods of time. Adjusting how much data the object store lags behind the local copy allows Redpanda to meet stricter recovery point-in-time objectives. This is encapsulated in the Kafka API, so clients can continue to produce and consume data from Redpanda in the same way. Consumers that keep up with producers continue to read from local storage and are subject to the local data retention policy. Consumers that want to read from older offsets do so with the same consumer API, and Redpanda handles fetching the necessary log segments from object storage. See also: [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) ### [](#production-settings)Production settings Before running performance benchmark testing, set Redpanda into production mode and run the autotuner tool ([rpk redpanda tune all](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/)) on every node. This enables the necessary hardware optimizations and ensures that kernel parameters are set correctly. See also: [Set Redpanda production mode](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/#set-redpanda-production-mode) and [autotuner reference](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-tune/) ### [](#open-messaging-benchmark)Open Messaging Benchmark Performance benchmarking a distributed system like Redpanda requires careful orchestration, instrumentation, and measurement. Every cluster destined for production should be subject to performance benchmarking for validation and confidence in the setup. The [Open Messaging Benchmark](https://github.com/redpanda-data/openmessaging-benchmark) (OMB) framework simplifies the process. OMB contains extensible tests that replicate realworld stress on a streaming platform to measure throughput and latency over given time periods. OMB can verify that a Redpanda cluster, deployed in your own data center or in the cloud, is sized appropriately for your use case. See also: [Redpanda Benchmarks](https://github.com/redpanda-data/openmessaging-benchmark/blob/main/driver-redpanda/README.md) ## [](#assess-throughput)Assess throughput This section describes how to use the [`rpk topic analyze`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-analyze/) command to check how much work your Redpanda cluster is handling. It shows the number of messages the cluster is processing and the size of the data groups (batches). This information helps you decide if you need to add more servers or make changes to your setup. This command shows you the throughput of your Redpanda cluster: ```bash rpk topic analyze --regex '*' --print-all --time-range -1m:end ``` The arguments are: - `--regex '*'`: Analyzes all topics. - `--print-all`: Prints all the metrics. - `--time-range -1m:end`: Analyzes the last minute of data. Example output: ```bash SUMMARY ======= TOPICS 6 PARTITIONS 17 TOTAL THROUGHPUT (BYTES/S) 1361.9166666666667 TOTAL BATCH RATE (BATCHES/S) 2.9833333333333334 AVERAGE BATCH SIZE (BYTES) 456.50837988826817 TOPIC SUMMARY ============= TOPIC PARTITIONS BYTES-PER-SECOND BATCHES-PER-SECOND AVERAGE-BYTES-PER-BATCH _redpanda.audit_log 12 61 0.1 610 _redpanda.transform_logs 1 890.2666666666667 0.7833333333333333 1136.5106382978724 _schemas 1 0 0 0 edu-filtered-domains 1 14.283333333333333 0.1 142.83333333333334 logins 1 144.61666666666667 1 144.61666666666667 transactions 1 251.75 1 251.75 PARTITION BATCH RATE (BATCHES/S) ================================ TOPIC P25 P50 P75 P99 _redpanda.audit_log 0.016666666666666666 0.016666666666666666 0.03333333333333333 0.03333333333333333 _redpanda.transform_logs 0.7833333333333333 0.7833333333333333 0.7833333333333333 0.7833333333333333 _schemas 0 0 0 0 edu-filtered-domains 0.1 0.1 0.1 0.1 logins 1 1 1 1 transactions 1 1 1 1 PARTITION BATCH SIZE (BYTES) ============================ TOPIC P25 P50 P75 P99 _redpanda.audit_log 608 610 610 611 _redpanda.transform_logs 895 895 895 895 _schemas 0 0 0 0 edu-filtered-domains 141 141 141 141 logins 144 144 144 144 transactions 255 255 255 255 ``` - **Total throughput:** Indicates the total amount of data processed by the cluster every second. - **Total batch rate:** Shows the number of message batches processed per second. A higher rate suggests increased activity, which may require more CPU or I/O resources. - **Average batch size:** Reflects the average size of each message batch. Large or inconsistent batch sizes may indicate the need to adjust producer settings or verify storage capacity. - **Topic and partition summaries:** Provides details on resource usage by individual topics. For example, if a single topic (such as `_redpanda.transform_logs` in the example output) is responsible for most throughput, it may need optimization or additional resources. - **Percentiles (P25, P50, P75, P99):** Offers insights into workload distribution across partitions. Consistent values suggest balanced workloads, while significant variations may highlight areas that need rebalancing or capacity adjustments. ### [](#plan-for-capacity)Plan for capacity Compare the current throughput and batch rate with your cluster’s hardware limits, such as network bandwidth, disk IOPS, or CPU capacity. If usage is nearing these limits, consider scaling up (upgrading hardware) or scaling out (adding brokers). Monitor trends over time to anticipate when expansion is necessary. ### [](#address-bottlenecks)Address bottlenecks If specific topics or partitions consistently show higher loads, it may indicate uneven workload distribution. Redistribute partitions or adjust replication factors to balance the load more effectively. ## [](#suggested-reading)Suggested reading - [Four sizing principles for Redpanda production clusters](https://redpanda.com/blog/sizing-redpanda-cluster-best-practices) - [Free guide - Architecture and sizing guidelines for your Redpanda clusters](https://go.redpanda.com/redpanda-sizing-guidelines) - [Thread-per-core buffer management for a modern Kafka-API storage system](https://redpanda.com/blog/tpc-buffers?utm_medium=content&utm_assetname=sizing_guide&utm_assettype=report&utm_source=gated_content&utm_campaign=tpc_architecture_blog) - [A guide to benchmarking the performance of Redpanda](https://redpanda.com/blog/self-hosted-redpanda-benchmarking) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 59: Develop **URL**: https://docs.redpanda.com/streaming/current/develop.md --- # Develop > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Develop latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/index.adoc description: Develop doc topics. page-git-created-date: "2023-05-30" page-git-modified-date: "2023-08-21" support-status: supported --- - [Kafka Compatibility](kafka-clients/) Kafka clients, version 0.11 or later, are compatible with Redpanda. Validations and exceptions are listed. - [Benchmark Redpanda](benchmark/) Learn how to measure the performance of a Redpanda cluster deployed on AWS EC2 instances with the OpenMessaging Benchmark. - [Use Redpanda with the HTTP Proxy API](http-proxy/) HTTP Proxy exposes a REST API to list topics, produce events, and subscribe to events from topics using consumer groups. - [Topics](manage-topics/) Learn how to manage topics in Redpanda, including creation, configuration, and advanced features. - [Edit Topic Configuration in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/edit-topic-configuration/) Learn how to use Redpanda Console to edit the configuration of existing topics in a cluster. - [Produce Data](produce-data/) Learn how to configure producers and idempotent producers. - [Consume Data](consume-data/) Learn about consumer offsets and follower fetching. - [Data Transforms](data-transforms/) Learn about WebAssembly data transforms within Redpanda. - [Transactions](transactions/) Learn how to use transactions; for example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. --- # Page 60: Benchmark Redpanda **URL**: https://docs.redpanda.com/streaming/current/develop/benchmark.md --- # Benchmark Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Benchmark Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: benchmark page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: benchmark.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/benchmark.adoc description: Learn how to measure the performance of a Redpanda cluster deployed on AWS EC2 instances with the OpenMessaging Benchmark. page-git-created-date: "2023-07-24" page-git-modified-date: "2024-07-25" support-status: supported --- Learn how to measure the performance of a Redpanda cluster deployed on AWS EC2 instances with the Linux Foundation’s OpenMessaging Benchmark. Run the same tests and workloads that Redpanda uses to demonstrate significantly better performance than Apache Kafka. ## [](#about-openmessaging-benchmark)About OpenMessaging Benchmark The [Linux Foundation’s OpenMessaging Benchmark](https://openmessaging.cloud/docs/benchmarks/) (OMB) Framework is an open-source, cloud-based benchmark framework that supports several messaging systems, including Kafka, and is configurable for workloads representing real-world use cases. Redpanda Data provides a [fork of OMB on Github](https://github.com/redpanda-data/openmessaging-benchmark) with some updates: - Fixed coalescing of asynchronous consumer offset requests in the OMB Kafka driver. - Support for Kafka 3.2.0 clients. ### [](#omb-workloads)OMB workloads An OMB workload is a benchmark configuration that sets the producers, consumers, topics, and messages used by a test, as well as the production rate and duration of each test. An OMB workload is specified in a YAML configuration file. Example workload configuration file The content of an OMB workload configuration file, copied from Redpanda Data’s [fork of OMB](https://github.com/redpanda-data/openmessaging-benchmark/blob/main/workloads/1-topic-1-partition-1kb.yaml): ```none name: 1 topic / 1 partition / 1Kb topics: 1 partitionsPerTopic: 1 keyDistributor: "NO_KEY" messageSize: 1024 payloadFile: "payload/payload-1Kb.data" subscriptionsPerTopic: 1 consumerPerSubscription: 1 producersPerTopic: 1 producerRate: 50000 consumerBacklogSizeGB: 0 testDurationMinutes: 15 ``` The `keyDistributor` property configures how keys are distributed and assigned to messages. - `NO_KEY` sets `null` for all keys. - `KEY_ROUND_ROBIN` cycles through a finite set of keys in round-robin fashion. - `RANDOM_NANO` returns random keys based on `System.nanoTime()`. ## [](#set-up-benchmark)Set up benchmark Running OMB with Redpanda requires setting up your local environment to provision and start a Redpanda cluster in AWS. 1. Install CLI tools. - [Maven](https://maven.apache.org/install.html) - [Terraform](https://developer.hashicorp.com/terraform/downloads) with [terraform-inventory plugin](https://github.com/adammck/terraform-inventory) - [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) (v2.11 or higher) - Python 3 and pip - A window manager like [tmux](https://github.com/tmux/tmux/wiki) or [screen](https://linux.die.net/man/1/screen) that supports detachable screen sessions. > 💡 **TIP** > > Redpanda Data recommends running the benchmark executable with a window manager that supports detachable screen sessions, like tmux or screen, so the benchmark can continue to run in the background even after you disconnect. 2. Clone the Redpanda Data fork of OMB. ```bash git clone https://github.com/redpanda-data/openmessaging-benchmark ``` The repository contains a directory for the Redpanda driver, `openmessaging-benchmark/driver-redpanda`. Subsequent steps read and configure files in that directory. 3. Customize the `openmessaging-benchmark/driver-redpanda/pom.xml` file with your Kafka client version if necessary (currently 3.3.1): `pom.xml` ```xml ... org.apache.kafka kafka-clients 3.3.1 ... ``` 4. From the repository root directory, build the benchmark client. ```bash cd openmessaging-benchmark mvn clean install -Dlicense.skip=true ``` 5. From the Redpanda driver directory, install the Ansible roles required for deploying Redpanda. ```bash cd driver-redpanda/deploy ansible-galaxy install -r requirements.yaml ``` 6. Configure AWS credentials and SSH keys. 1. [Install](https://aws.amazon.com/cli/) and [configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) AWS CLI. 2. Generate SSH keys: ```bash ssh-keygen -f ~/.ssh/redpanda_aws ``` When prompted for a passphrase, set a blank passphrase by pressing Enter twice. 3. Verify the SSH key files were created. ```bash ls ~/.ssh/redpanda_aws* ``` 7. Provision a Redpanda cluster to deploy on AWS with Terraform. 1. Customize the `openmessaging-benchmark/deploy/terraform.tfvars` Terraform configuration file for your environment. Default Terraform configuration for Redpanda on AWS The default contents of `openmessaging-benchmark/driver-redpanda/deploy/terraform.tfvars`: ```none public_key_path = "~/.ssh/redpanda_aws.pub" region = "us-west-2" az = "us-west-2a" ami = "ami-0d31d7c9fc9503726" profile = "default" instance_types = { "redpanda" = "i3en.6xlarge" "client" = "m5n.8xlarge" "prometheus" = "c5.2xlarge" } num_instances = { "client" = 4 "redpanda" = 3 "prometheus" = 1 } ``` 2. From the Redpanda driver deployment directory, initialize the Terraform deployment of Redpanda on AWS. ```bash cd driver-redpanda/deploy terraform init terraform apply -auto-approve ``` > 📝 **NOTE** > > The `terraform apply` command prompts you for an owner name (`var.owner`) that is used to tag all the cloud resources that will be created. Once the installation is complete, you will see a confirmation message listing the resources that have been installed. 8. Run the Ansible playbook to install and start the Redpanda cluster. Redpanda can run with or without TLS and SASL enabled. - To run Redpanda **without TLS and SASL**: ```bash ansible-playbook deploy.yaml ``` - To run Redpanda **with TLS and SASL**: ```bash ansible-playbook deploy.yaml -e "tls_enabled=true sasl_enabled=true" ``` If the path to your SSH private key isn’t `~/.ssh/redpanda_aws`, add the `--private-key` flag to your Ansible command. ```bash ansible-playbook deploy.yaml --private-key= ``` > 📝 **NOTE** > > Beginning with Ansible 2.14, references to `args: warn` within Ansible tasks cause a fatal error and halt the execution of the playbook. You may find instances of this in the components installed by `ansible-galaxy`, particularly in the `cloudalchemy.grafana` task in `dashboards.yml`. To resolve this issue, removing the `warn` line in from the yml file. ## [](#run-benchmark)Run benchmark Connect to the benchmark’s client and run the benchmark with a custom workload. 1. Connect with SSH to the benchmark client, with its IP address retrieved from the `client_ssh_host` output of Terraform. ```bash ssh -i ~/.ssh/redpanda_aws ubuntu@$(terraform output --raw client_ssh_host) ``` 2. On the client, navigate to the `/opt/benchmark` directory. ```bash cd /opt/benchmark ``` 3. Create a workload configuration file. For example, create a `.yaml` file with one topic, 144 partitions, 500 MBps producer rate, four producers, and four consumers: ```bash cat > workloads/1-topic-144-partitions-500mb-4p-4c.yaml << EOF name: 500mb/sec rate; 4 producers 4 consumers; 1 topic with 144 partitions topics: 1 partitionsPerTopic: 144 messageSize: 1024 useRandomizedPayloads: true randomBytesRatio: 0.5 randomizedPayloadPoolSize: 1000 subscriptionsPerTopic: 1 consumerPerSubscription: 4 producersPerTopic: 4 producerRate: 500000 consumerBacklogSizeGB: 0 testDurationMinutes: 30 EOF ``` Alternatively, you can use an existing workload file from the Redpanda repo, in `openmessaging-benchmark/driver-redpanda/deploy/workloads/`. Workloads from Redpanda vs. Kafka comparison The workloads from the [Redpanda vs. Kafka benchmark comparison](https://redpanda.com/blog/redpanda-vs-kafka-performance-benchmark) can be gotten from the chart in the comparison: ![kafka vs redpanda performance 8](https://images.ctfassets.net/paqvtpyf8rwu/2lpkGM01nrl0s87xSBISno/6c25504b1f6e7c8015ef193433bd077e/kafka_vs_redpanda_performance_8.png) 4. Create or reuse a client configuration file. This file configures the Redpanda producer and consumer clients, as well as topics. The rest of the guide uses the `openmessaging-benchmark/driver-redpanda/redpanda-ack-all-group-linger-1ms.yaml` configuration file. Client configuration from Redpanda vs. Kafka comparison The client configuration from the [Redpanda vs. Kafka benchmark comparison](https://redpanda.com/blog/redpanda-vs-kafka-performance-benchmark) can be gotten from the code listing in the comparison: ```yaml topicConfig: | min.insync.replicas=2 flush.messages=1 flush.ms=0 producerConfig: | acks=all linger.ms=1 batch.size=131072 consumerConfig: | auto.offset.reset=earliest enable.auto.commit=false auto.commit.interval.ms=0 max.partition.fetch.bytes=131072 ``` > 💡 **TIP** > > Configure `reset=false` and manually delete the generated topic after the benchmark completes. Otherwise, when `reset=true`, the benchmark can fail due to it erroneously trying to delete the `_schemas` topic. 5. Run the benchmark with your workload and client configuration. ```bash sudo bin/benchmark -d \ driver-redpanda/redpanda-ack-all-group-linger-1ms.yaml \ workloads/1-topic-144-partitions-500mb-4p-4c.yaml ``` ## [](#view-benchmark-results)View benchmark results After a run completes, the benchmark generates results as `*.json` files in `/opt/benchmark`. Redpanda provides a Python script, `generate_charts.py`, to generate charts of benchmark results. To run the script: 1. Copy the results from the client to your local machine. ```bash exit; # back to your local machine mkdir ~/results scp -i ~/.ssh/redpanda_aws ubuntu@$(terraform output --raw client_ssh_host):/opt/benchmark/*.json ~/results/ ``` 2. From the root directory of the repository, install the prerequisite packages for the Python script. ```bash cd ../../bin # openmessaging-benchmark/bin python3 -m pip -r install requirements.txt ``` 3. To list all options, run the script with the `-h` flag. ```bash ./generate_charts.py -h ``` 4. To generate charts from your `~/results/` directory, first create an `~/output` directory, then run the script with `--results` and `--output` options set accordingly. ```bash mkdir ~/output ./generate_charts.py --results ~/results --output ~/output ``` 5. In `~/output`, verify the generated charts are in an HTML page with charts for throughput, publish latency, end-to-end latency, publish rate, and consume rate. ## [](#tear-down-benchmark)Tear down benchmark When done running the benchmark, tear down the Redpanda cluster. ```bash terraform destroy -auto-approve ``` ## [](#suggested-reading)Suggested reading - [Redpanda vs. Apache Kafka: A performance comparison (2022 update)](https://redpanda.com/blog/redpanda-vs-kafka-performance-benchmark) - [Performance update: Redpanda vs. Kafka with KRaft](https://redpanda.com/blog/kafka-kraft-vs-redpanda-performance-2023) - [Why `fsync()`: Losing unsynced data on a single node leads to global data loss](https://redpanda.com/blog/why-fsync-is-needed-for-data-safety-in-kafka-or-non-byzantine-protocols) --- # Page 61: Consume Data **URL**: https://docs.redpanda.com/streaming/current/develop/consume-data.md --- # Consume Data > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Consume Data latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: consume-data/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: consume-data/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/consume-data/index.adoc description: Learn about consumer offsets and follower fetching. page-git-created-date: "2023-05-30" page-git-modified-date: "2024-02-26" support-status: supported --- - [Consumer Offsets](consumer-offsets/) Redpanda uses an internal topic, `__consumer_offsets`, to store committed offsets from each Kafka consumer that is attached to Redpanda. - [Follower Fetching](follower-fetching/) Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. - [Filter Messages with JavaScript in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters/) Learn how to filter Kafka records using custom JavaScript code within Redpanda Console. - [View Deserialized Messages in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/record-deserialization/) Learn how Redpanda Console deserializes messages. - [Paginate Messages in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/paginate-messages-events/) Enable Continuous Pagination on a topic's Messages tab to scroll through all records instead of being capped by Max results. --- # Page 62: Consumer Offsets **URL**: https://docs.redpanda.com/streaming/current/develop/consume-data/consumer-offsets.md --- # Consumer Offsets > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Consumer Offsets latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: consume-data/consumer-offsets page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: consume-data/consumer-offsets.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/consume-data/consumer-offsets.adoc description: Redpanda uses an internal topic, __consumer_offsets, to store committed offsets from each Kafka consumer that is attached to Redpanda. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-07-30" support-status: supported --- In Redpanda, all messages are organized by [topic](https://docs.redpanda.com/streaming/current/reference/glossary/#topic) and distributed across multiple partitions, based on a [partition strategy](https://www.redpanda.com/guides/kafka-tutorial-kafka-partition-strategy). For example, when using the round robin strategy, a producer writing to a topic with five partitions would distribute approximately 20% of the messages to each [partition](https://docs.redpanda.com/streaming/current/reference/glossary/#partition). Within a partition, each message (once accepted and acknowledged by the partition leader) is permanently assigned a unique sequence number called an [offset](https://docs.redpanda.com/streaming/current/reference/glossary/#offset). Offsets enable consumers to resume processing from a specific point, such as after an application outage. If an outage prevents your application from receiving events, you can use the consumer offset to retrieve only the events that occurred during the downtime. By default, the first message in a partition is assigned offset 0, the next is offset 1, and so on. You can manually specify a specific start value for offsets if needed. Once assigned, offsets are immutable, ensuring that the order of messages within a partition is preserved. ## [](#how-consumers-use-offsets)How consumers use offsets As a consumer reads messages from Redpanda, it can save its progress by “committing the offset” (known as an [offset commit](https://docs.redpanda.com/streaming/current/reference/glossary/#offset-commit)), an action initiated by the consumer, not Redpanda. Kafka client libraries provide an API for committing offsets, which communicates with Redpanda using the [consumer group](https://docs.redpanda.com/streaming/current/reference/glossary/#consumer-group) API. Each committed offset is stored as a message in the `__consumer_offsets` topic, which is a private Redpanda topic that stores committed offsets from each Kafka consumer attached to Redpanda, allowing the consumer to resume processing from the last committed point. Redpanda exposes the `__consumer_offsets` key to enable the many tools in the Kafka ecosystem that rely on this value for their operation, providing greater ecosystem interoperability with environments and applications. When a consumer group works together to consume data from topics, the partitions are divided among the consumers in the group. For example, if a topic has 12 partitions, and there are two consumers, each consumer would be assigned six partitions to consume. If a new consumer starts later and joins this consumer group, a rebalance occurs, such that each consumer ends up with four partitions to consume. You specify a consumer group by setting the `group.id` property to a unique name for the group. Kafka tracks the maximum offset it has consumed in each partition and can commit offsets to ensure it can resume processing from the same point in the event of a restart. Kafka allows offsets for a consumer group to be stored on a designated broker, known as the group coordinator. All consumers in the group send their offset commits and fetch requests to this group coordinator. > 📝 **NOTE** > > More advanced consumers can read data from Redpanda without using a consumer group by requesting to read a specific topic, partition, and offset range. This pattern is often used by stream processing systems such as Apache Spark and Apache Flink, which have their own mechanisms for assigning work to consumers. Redpanda Console derives its consumer group lists and lag information from committed consumer group offsets, so these consumers appear in Console only if they also commit offsets to a consumer group. For example, Spark Structured Streaming does not commit offsets, so its consumers never appear. Flink commits offsets only when it is configured to do so on checkpoint completion. When the group coordinator receives an OffsetCommitRequest, it appends the request to the [compacted](https://kafka.apache.org/documentation/#compaction) Kafka topic `__consumer_offsets`. The broker sends a successful offset commit response to the consumer only after all the replicas of the offsets topic receive the offsets. If the offsets fail to replicate within a configurable timeout, the offset commit fails and the consumer may retry the commit after backing off. The brokers periodically compact the `__consumer_offsets` topic, because it only needs to maintain the most recent offset commit for each partition. The coordinator also caches the offsets in an in-memory table to serve offset fetches quickly. ## [](#commit-strategies)Commit strategies There are several strategies for managing offset commits: ### [](#automatic-offset-commit)Automatic offset commit Auto commit is the default commit strategy, where the client automatically commits offsets at regular intervals. This is set with the `enable.auto.commit` property. The client then commits offsets every `auto.commit.interval.ms` milliseconds. The primary advantage of the auto commit approach is its simplicity. After it is configured, the consumer requires no additional effort. Commits are managed in the background. However, the consumer is unaware of what was committed or when. As a result, after an application restart, some messages may be reprocessed (since consumption resumes from the last committed offset, which may include already-processed messages). The strategy guarantees at-least-once delivery. > 📝 **NOTE** > > If your consume configuration is set up to consume and write to another data store, and the write to that datastore fails, the consumer might not recover when it is auto-committed. It may not only duplicate messages, but could also drop messages intended to be in another datastore. Make sure you understand the trade-off possibilities associated with this default behavior. ### [](#manual-offset-commit)Manual offset commit The manual offset commit strategy gives consumers greater control over when commits occur. This approach is typically used when a consumer needs to align commits with an external system, such as database transactions in an RDBMS. The main advantage of manual commits is that they allow you to decide exactly when a record is considered consumed. You can use two API calls for this: `commitSync` and `commitAsync`, which differ in their blocking behavior. #### [](#synchronous-commit)Synchronous commit The advantage of synchronous commits is that consumers can take appropriate action before continuing to consume messages, albeit at the expense of increased latency (while waiting for the commit to return). The commit (`commitSync`) will also retry automatically, until it either succeeds or receives an unrecoverable error. The following example shows a synchronous commit: ```java consumer.subscribe(Arrays.asList("foo", "bar")); while (true) { ConsumerRecords records = consumer.poll(100); for (ConsumerRecord record : records) { // process records here ... // ... and at the appropriate point, call commit (not after every message) consumer.commitSync(); } } ``` #### [](#asynchronous-commit)Asynchronous commit The advantage of asynchronous commits is lower latency, because the consumer does not pause to wait for the commit response. However, there is no automatic retry of the commit (`commitAsync`) if it fails. There is also increased coding complexity (due to the asynchronous callbacks). The following example shows an asynchronous commit in which the consumer will not block. Instead, the commit call registers a callback, which is executed once the commit returns: ```java void callback() { // executed when the commit returns } consumer.subscribe(Arrays.asList("foo", "bar")); while (true) { ConsumerRecords records = consumer.poll(100); for (ConsumerRecord record : records) { // process records here ... // ... and at the appropriate point, call commit consumer.commitAsync(callback); } } ``` ### [](#external-offset-management)External offset management The external offset management strategy allows consumers to manage offsets independently of Redpanda. In this approach: - Consumers bypass the consumer group API and directly assign partitions instead of subscribing to a topic. - Offsets are not committed to Redpanda, but are instead stored in an external storage system. Because consumers that use this strategy do not commit offsets to Redpanda, they do not appear in the consumer group lists in Redpanda Console. This is expected behavior for systems such as Apache Spark Structured Streaming, which manages offsets entirely in its own checkpoint storage. Systems such as Apache Flink can also commit offsets to Kafka when a checkpoint completes. In that configuration, they follow the hybrid strategy described in the next section and do appear in Console. To implement an external offset management strategy: 1. Set `enable.auto.commit` to `false`. 2. Use `assign(Collection)` to assign partitions. 3. Use the offset provided with each ConsumerRecord to save your position. 4. Upon restart, use `seek(TopicPartition, long)` to restore the position of the consumer. ### [](#hybrid-offset-management)Hybrid offset management The hybrid offset management strategy allows consumers to handle their own consumer rebalancing while still leveraging Redpanda’s offset commit functionality. In this approach: - Consumers bypass the consumer group API and directly assign partitions instead of subscribing to a topic. - Offsets are committed to Redpanda. ## [](#offset-commit-best-practices)Offset commit best practices Follow these best practices to optimize offset commits. ### [](#avoid-over-committing)Avoid over-committing The purpose of a commit is to save consumer progress. More frequent commits reduce the amount of data to re-read after an application restart, as the commit interval directly affects the recovery point objective (RPO). Because a lower RPO is desirable, application designers may believe that committing frequently is a good design choice. However, committing too frequently can result in adverse consequences. While individually small, each commit still results in a message being written to the `__consumer_offsets` topic, because the position of the consumer against every partition must be recorded. At high commit rates, this workload can become a bottleneck for both the client and the server. Additionally, many Kafka client implementations do not coalesce offset commits, meaning redundant commits in a backlog still need to be processed. In many Kafka client implementations, offset commits aren’t coalesced at the client; so if a backlog of commits forms (when using the asynchronous commit API), the earlier commits still need to be processed, even though they are effectively redundant. **Best practice**: Monitor commit latency to ensure commits are timely. If you notice performance issues, commit less frequently. ### [](#use-unique-consumer-groups)Use unique consumer groups Like many topics, the consumer group topic has multiple partitions to help with performance. When writing commit messages, Redpanda groups all of the commits for a consumer group into a specific partition to maintain ordering. Reusing a consumer group across multiple applications, even for different topics, forces all commits to use a single partition, negating the benefits of partitioning. **Best practice**: Assign a unique consumer group to each application to distribute the commit load across all partitions. ### [](#tune-the-consumer-group)Tune the consumer group In highly parallel applications, frequent consumer group heartbeats can create unnecessary overhead. For example, 3,200 consumers checking every 500 milliseconds generate 6,400 heartbeats per second. You can optimize this behavior by increasing the `heartbeat.interval.ms` (along with `session.timeout.ms`). **Best practice**: Adjust heartbeat and session timeout settings to reduce unnecessary overhead in large-scale applications. ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 63: Follower Fetching **URL**: https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching.md --- # Follower Fetching > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Follower Fetching latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: consume-data/follower-fetching page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: consume-data/follower-fetching.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/consume-data/follower-fetching.adoc description: Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. page-git-created-date: "2023-08-03" page-git-modified-date: "2026-07-28" support-status: supported --- Learn about follower fetching and how to configure a Redpanda consumer to fetch records from the closest replica. ## [](#about-follower-fetching)About follower fetching **Follower fetching** enables a consumer to fetch records from the closest replica of a topic partition, regardless of whether it’s a leader or a follower. For a Redpanda cluster deployed across different data centers and availability zones (AZs), restricting a consumer to fetch only from the leader of a partition can incur greater costs and have higher latency than fetching from a follower that is geographically closer to the consumer. With follower fetching (proposed in [KIP-392](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica)), the fetch protocol is extended to support a consumer fetching from any replica. This includes [Remote Read Replicas](https://docs.redpanda.com/streaming/current/manage/remote-read-replicas/). The first fetch from a consumer is processed by a Redpanda leader broker. The leader checks for a replica (itself or a follower) that has a rack ID that matches the consumer’s rack ID. If a replica with a matching rack ID is found, the fetch request returns records from that replica. Otherwise, the fetch is handled by the leader. ## [](#configure-follower-fetching)Configure follower fetching Redpanda decides which replica a consumer fetches from. If the consumer configures its `client.rack` property, Redpanda by default selects a replica from the same rack as the consumer, if available. To enable follower fetching in Redpanda, configure properties for the consumer and the Redpanda cluster and broker: - For a Redpanda cluster, set the [`enable_rack_awareness`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_rack_awareness) property to `true`. - For each Redpanda broker, set the [`rack`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#rack) property to a rack ID. - For each consumer, set the `client.rack` property to a rack ID. > 💡 **TIP** > > Follower fetching spreads consumers across replicas to distribute the read load. If many consumers also fetch the same data concurrently, [fetch read coalescing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/fetch-read-coalescing/) complements it by removing the redundant work within each shard. ## [](#suggested-videos)Suggested videos - [YouTube - Redpanda Office Hour: Follower Fetching (52 mins)](https://www.youtube.com/watch?v=wV6gH5_yVaw&ab_channel=RedpandaData) ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 64: Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms.md --- # Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/index.adoc description: Learn about WebAssembly data transforms within Redpanda. page-git-created-date: "2023-12-22" page-git-modified-date: "2024-04-30" support-status: supported --- - [How Data Transforms Work](how-transforms-work/) Learn how Redpanda data transforms work. - [Data Transforms Quickstarts](run-transforms-index/) Choose your deployment environment to get started with building and deploying your first transform function in Redpanda. - [Develop Data Transforms](build/) Learn how to initialize a data transforms project and write transform functions in your chosen language. - [Configure Data Transforms](configure/) Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment. - [Deploy Data Transforms](deploy/) Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. - [Write Integration Tests for Transform Functions](test/) Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. - [Monitor Data Transforms](monitor/) This topic provides guidelines on how to monitor the health of your data transforms and view logs. - [Manage Data Transforms in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/data-transforms/) Use Redpanda Console to monitor the status and performance metrics of your transform functions. You can also view detailed logs and delete transform functions when they are no longer needed. - [Upgrade the Data Transforms SDK](upgrade/) Upgrading the SDK version in your data transforms project ensures compatibility with the latest features and fixes. This guide provides step-by-step instructions to upgrade the SDK version for all supported SDK languages. - [Versioning and Compatibility for Data Transforms](versioning-compatibility/) The data transforms SDKs use semantic versioning to ensure compatibility and stability. Use this guide to learn the SDKs that are compatible with different versions of Redpanda, and what guarantees are provided regarding SDK and Redpanda compatibility. - [Prebuilt Data Transforms](labs/) Explore labs that include examples of transform functions and instructions on how to deploy and run them. --- # Page 65: Develop Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/build.md --- # Develop Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Develop Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/build page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/build.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/build.adoc description: Learn how to initialize a data transforms project and write transform functions in your chosen language. page-topic-type: how-to personas: streaming_developer, application_developer learning-objective-1: Initialize a data transforms project using the rpk CLI learning-objective-2: Build transform functions that process records and write to output topics learning-objective-3: Implement multi-topic routing patterns with Schema Registry integration page-git-created-date: "2024-07-31" page-git-modified-date: "2026-08-13" support-status: supported --- Learn how to initialize a data transforms project and write transform functions in your chosen language. After reading this page, you will be able to: - Initialize a data transforms project using the rpk CLI - Build transform functions that process records and write to output topics - Implement multi-topic routing patterns with Schema Registry integration ## [](#prerequisites)Prerequisites You must have the following development tools installed on your host machine: - The [`rpk` command-line client](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed on your host machine and configured to connect to your Redpanda cluster. - For Golang projects, you must have at least version 1.20 of [Go](https://go.dev/doc/install). - For Rust projects, you must have the latest stable version of [Rust](https://rustup.rs/). - For JavaScript and TypeScript projects, you must have the [latest long-term-support release of Node.js](https://nodejs.org/en/download/package-manager). ## [](#init)Initialize a data transforms project To initialize a data transforms project, use the following command to set up the project files in your current directory. This command adds the latest version of the [SDK](https://docs.redpanda.com/streaming/current/reference/data-transforms/sdks/) as a project dependency: ```bash rpk transform init --language= --name= ``` If you do not include the `--language` flag, the command prompts you for the language. Supported languages include: - `tinygo-no-goroutines` (does not include [Goroutines](https://golangdocs.com/goroutines-in-golang)) - `tinygo-with-goroutines` - `rust` - `javascript` - `typescript` For example, if you choose `tinygo-no-goroutines`, `rpk` creates the following project files: . ├── go.mod ├── go.sum ├── README.md ├── transform.go └── transform.yaml The `transform.go` file contains a boilerplate transform function. The `transform.yaml` file specifies the configuration settings for the transform function. See also: [Configure Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/) ## [](#build-transform-functions)Build transform functions You can develop your transform logic with one of the available SDKs that allow your transform code to interact with a Redpanda cluster. #### Go All transform functions must register a callback with the `OnRecordWritten()` method. You should run any initialization steps in the `main()` function because it’s only run once when the transform function is first deployed. You can also use the standard predefined [`init()` function](https://go.dev/doc/effective_go#init). ```go package main import ( "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Register your transform function. // This is a good place to perform other setup too. transform.OnRecordWritten(myTransform) } // myTransform is where you read the record that was written, and then you can // output new records that will be written to the destination topic func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust All transform functions must register a callback with the `on_record_written()` method. You should run any initialization steps in the `main()` function because it’s only run once when the transform function is first deployed. ```rust use redpanda_transform_sdk::*; fn main() { // Register your transform function. // This is a good place to perform other setup too. on_record_written(my_transform); } // my_transform is where you read the record that was written, and then you can // return new records that will be written to the output topic fn my_transform(event: WriteEvent, writer: &mut RecordWriter) -> Result<(), Box> { writer.write(event.record)?; Ok(()) } ``` #### JavaScript All transform functions must register a callback with the `onRecordWritten()` method. You should run any initialization steps outside of the callback so that they are only run once when the transform function is first deployed. ```js // src/index.js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // This is a good place to perform setup steps. // Register your transform function. onRecordWritten((event, writer) => { // This is where you read the record that was written, and then you can // output new records that will be written to the destination topic writer.write(event.record); }); ``` If you need to use Node.js standard modules in your transform function, you must configure the [`polyfillNode` plugin](https://github.com/cyco130/esbuild-plugin-polyfill-node) for [esbuild](https://esbuild.github.io/). This plugin allows you to polyfill Node.js APIs that are not natively available in the Redpanda JavaScript runtime environment. `esbuild.js` ```js import * as esbuild from 'esbuild'; import { polyfillNode } from 'esbuild-plugin-polyfill-node'; await esbuild.build({ plugins: [ polyfillNode({ globals: { buffer: true, // Allow a global Buffer variable if referenced. process: false, // Don't inject the process global, the Redpanda JavaScript runtime does that. }, polyfills: { crypto: true, // Enable crypto polyfill // Add other polyfills as needed }, }), ], }); ``` ### [](#errors)Error handling By distinguishing between recoverable and critical errors, you can ensure that your transform functions are both resilient and robust. Handling recoverable errors internally helps maintain continuous operation, while allowing critical errors to escape ensures that the system can address severe issues effectively. Redpanda tracks the offsets of records that transform functions have processed. If an error escapes the Wasm virtual machine (VM), the VM will fail. When the Wasm engine detects this failure and starts a new VM, the transform function retries processing the input topics from the last processed offset, potentially leading to repeated failures if the underlying issue is not resolved. Handling errors internally by logging them and continuing to process subsequent records can help maintain continuous operation. However, this approach can result in silently discarding problematic records, which may lead to unnoticed data loss if the logs are not monitored closely. #### Go ```go package main import ( "log" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(myTransform) } func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { record := event.Record() if record.Key == nil { // Handle the error internally by logging it log.Println("Error: Record key is nil") // Skip this record and continue to process other records return nil } // Allow errors with writes to escape return writer.Write(record) } ``` #### Rust ```rust use redpanda_transform_sdk::*; use log::error; fn main() { // Set up logging env_logger::init(); on_record_written(my_transform); } fn my_transform(event: WriteEvent, writer: &mut RecordWriter) -> anyhow::Result<()> { let record = event.record; if record.key().is_none() { // Handle the error internally by logging it error!("Error: Record key is nil"); // Skip this record and continue to process other records return Ok(()); } // Allow errors with writes to escape return writer.write(record) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Register your transform function. onRecordWritten((event, writer) => { const record = event.record; if (!record.key) { // Handle the error internally by logging it console.error("Error: Record key is nil"); // Skip this record and continue to process other records return; } // Allow errors with writes to escape writer.write(record); }); ``` When you deploy this transform function, and produce a message without a key, you’ll get the following in the logs: ```js { "body": { "stringValue": "2024/06/20 08:17:33 Error: Record key is nil\n" }, "timeUnixNano": 1718871455235337000, "severityNumber": 13, "attributes": [ { "key": "transform_name", "value": { "stringValue": "test" } }, { "key": "node", "value": { "intValue": 0 } } ] } ``` You can view logs for transform functions using the `rpk transform logs ` command. To ensure that you are notified of any errors or issues in your data transforms, Redpanda provides metrics that you can use to monitor the state of your data transforms. See also: - [View logs for transform functions](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/#logs) - [Monitor data transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/) - [Configure transform logging](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#log) - [`rpk transform logs` reference](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-logs/) ### [](#avoid-state-management)Avoid state management Relying on in-memory state across transform invocations can lead to inconsistencies and unpredictable behavior. Data transforms operate with at-least-once semantics, meaning a transform function might be executed more than once for a given record. Redpanda may also restart a transform function at any point, which causes its state to be lost. ### [](#env-vars)Access environment variables You can access both [built-in and custom environment variables](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#environment-variables) in your transform function. In this example, environment variables are checked once during initialization: #### Go ```go package main import ( "fmt" "os" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Check environment variables before registering the transform function. outputTopic1, ok := os.LookupEnv("REDPANDA_OUTPUT_TOPIC_1") if ok { fmt.Printf("Output topic 1: %s\n", outputTopic1) } else { fmt.Println("Only one output topic is set") } // Register your transform function. transform.OnRecordWritten(myTransform) } func myTransform(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust ```rust use redpanda_transform_sdk::*; use std::env; use log::error; fn main() { // Set up logging env_logger::init(); // Check environment variables before registering the transform function. match env::var("REDPANDA_OUTPUT_TOPIC_1") { Ok(output_topic_1) => println!("Output topic 1: {}", output_topic_1), Err(_) => println!("Only one output topic is set"), } // Register your transform function. on_record_written(my_transform); } fn my_transform(_event: WriteEvent, _writer: &mut RecordWriter) -> anyhow::Result<()> { Ok(()) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Check environment variables before registering the transform function. const outputTopic1 = process.env.REDPANDA_OUTPUT_TOPIC_1; if (outputTopic1) { console.log(`Output topic 1: ${outputTopic1}`); } else { console.log("Only one output topic is set"); } // Register your transform function. onRecordWritten((event, writer) => { return writer.write(event.record); }); ``` ### [](#write-to-specific-output-topics)Write to specific output topics You can configure your transform function to write records to specific output topics based on message content, enabling powerful routing and fan-out patterns. This capability is useful for: - Filtering messages by criteria and routing to different topics - Fan-out patterns that distribute data from one input topic to multiple output topics - Event routing based on message type or schema - Data distribution for downstream consumers Wasm transforms provide a simpler alternative to external connectors like Kafka Connect for in-broker data routing, with lower latency and no additional infrastructure to manage. #### [](#basic-json-validation-example)Basic JSON validation example The following example shows a filter that outputs only valid JSON from the input topic into the output topic. The transform writes invalid JSON to a different output topic. ##### Go ```go import ( "encoding/json" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(filterValidJson) } func filterValidJson(event transform.WriteEvent, writer transform.RecordWriter) error { if json.Valid(event.Record().Value) { return writer.Write(event.Record()) } // Send invalid records to separate topic return writer.Write(event.Record(), transform.ToTopic("invalid-json")) } ``` ##### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { on_record_written(filter_valid_json); } fn filter_valid_json(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { let value = event.record.value().unwrap_or_default(); if serde_json::from_slice::(value).is_ok() { writer.write(event.record)?; } else { // Send invalid records to separate topic writer.write_with_options(event.record, WriteOptions::to_topic("invalid-json"))?; } Ok(()) } ``` ##### JavaScript The JavaScript SDK does not support writing records to a specific output topic. #### [](#multi-topic-fanout)Multi-topic fan-out with Schema Registry This example shows how to route batched updates from a single input topic to multiple output topics based on a routing field in each message. Messages are encoded with the [Schema Registry wire format](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/#wire-format) for validation against the output topic schema. Consider using this pattern with Iceberg-enabled topics to fan out data directly into lakehouse tables. Input message example ```json { "updates": [ {"table": "orders", "data": {"order_id": "123", "amount": 99.99}}, {"table": "inventory", "data": {"product_id": "P456", "quantity": 50}}, {"table": "customers", "data": {"customer_id": "C789", "name": "Jane"}} ] } ``` [Configure the transform](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/) with multiple output topics: ```yaml name: event-router input_topic: events output_topics: - orders - inventory - customers ``` The transform extracts each update and routes it to the appropriate topic based on the `table` field. Schemas are registered dynamically in the `main()` function using the Schema Registry client, which returns the schema IDs needed for encoding messages in the wire format. > 📝 **NOTE** > > In this example, it is assumed that you have created the output topics and have the schema definitions ready. The transform registers the schemas dynamically on startup using the `{topic-name}-value` naming convention for schema subjects (for example, `orders-value`, `inventory-value`). ##### Go `go.mod` ```go module fanout-example go 1.20 require github.com/redpanda-data/redpanda/src/transform-sdk/go/transform v1.1.0 // v1.1.0+ required ``` `transform.go`: ```go package main import ( "encoding/binary" "encoding/json" "log" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform/sr" ) // Input message structure with array of updates type BatchMessage struct { Updates []TableUpdate `json:"updates"` } // Individual table update with routing field type TableUpdate struct { Table string `json:"table"` // Routing field - determines output topic Data json.RawMessage `json:"data"` // The actual data to write } // Schema IDs for each output topic, registered dynamically at startup var schemaIDs = make(map[string]int) func main() { // Create Schema Registry client client := sr.NewClient() // Define schemas for each output topic schemas := map[string]string{ "orders": `{"type":"record","name":"Order","fields":[{"name":"order_id","type":"string"},{"name":"amount","type":"double"}]}`, "inventory": `{"type":"record","name":"Inventory","fields":[{"name":"product_id","type":"string"},{"name":"quantity","type":"int"}]}`, "customers": `{"type":"record","name":"Customer","fields":[{"name":"customer_id","type":"string"},{"name":"name","type":"string"}]}`, } // Register schemas and store their IDs for topic, schemaStr := range schemas { subject := topic + "-value" schema := sr.Schema{ Schema: schemaStr, Type: sr.TypeAvro, } result, err := client.CreateSchema(subject, schema) if err != nil { log.Fatalf("Failed to register schema for %s: %v", topic, err) } schemaIDs[topic] = result.ID log.Printf("Registered schema for %s with ID %d", topic, result.ID) } log.Printf("Starting fanout transform with schema IDs: %v", schemaIDs) transform.OnRecordWritten(routeUpdates) } func routeUpdates(event transform.WriteEvent, writer transform.RecordWriter) error { var batch BatchMessage if err := json.Unmarshal(event.Record().Value, &batch); err != nil { log.Printf("Failed to parse batch message: %v", err) return nil // Skip invalid records } // Process each update in the batch for i, update := range batch.Updates { schemaID, exists := schemaIDs[update.Table] if !exists { log.Printf("Unknown table in update %d: %s", i, update.Table) continue } if err := writeUpdate(update, schemaID, writer, event); err != nil { log.Printf("Failed to write update %d to %s: %v", i, update.Table, err) } } return nil } func writeUpdate(update TableUpdate, schemaID int, writer transform.RecordWriter, event transform.WriteEvent) error { // Create Schema Registry wire format: [magic_byte, schema_id (4 bytes BE), data...] value := make([]byte, 5) value[0] = 0 // magic byte binary.BigEndian.PutUint32(value[1:5], uint32(schemaID)) value = append(value, update.Data...) record := transform.Record{ Key: event.Record().Key, Value: value, } return writer.Write(record, transform.ToTopic(update.Table)) } ``` ##### Rust `Cargo.toml` ```toml [package] name = "fanout-rust-example" version = "0.1.0" edition = "2021" [dependencies] redpanda-transform-sdk = "1.1.0" # v1.1.0+ required for WriteOptions API redpanda-transform-sdk-sr = "1.1.0" serde = { version = "1", features = ["derive"] } serde_json = "1" log = "0.4" env_logger = "0.11" [profile.release] opt-level = "z" lto = true strip = true ``` `src/main.rs`: ```rust use redpanda_transform_sdk::*; use redpanda_transform_sdk_sr::{SchemaRegistryClient, Schema, SchemaFormat}; use serde::Deserialize; use std::collections::HashMap; use std::error::Error; use std::sync::OnceLock; use log::{info, error}; #[derive(Deserialize)] struct BatchMessage { updates: Vec, } #[derive(Deserialize)] struct TableUpdate { table: String, data: serde_json::Value, } // Schema IDs for each output topic, registered dynamically at startup static SCHEMA_IDS: OnceLock> = OnceLock::new(); fn main() { // Initialize logging env_logger::init(); // Create Schema Registry client let mut client = SchemaRegistryClient::new(); // Define schemas for each output topic let schemas = [ ("orders", r#"{"type":"record","name":"Order","fields":[{"name":"order_id","type":"string"},{"name":"amount","type":"double"}]}"#), ("inventory", r#"{"type":"record","name":"Inventory","fields":[{"name":"product_id","type":"string"},{"name":"quantity","type":"int"}]}"#), ("customers", r#"{"type":"record","name":"Customer","fields":[{"name":"customer_id","type":"string"},{"name":"name","type":"string"}]}"#), ]; let mut schema_ids = HashMap::new(); // Register schemas and store their IDs for (topic, schema_str) in schemas { let subject = format!("{}-value", topic); let schema = Schema::new(schema_str.to_string(), SchemaFormat::Avro, vec![]); match client.create_schema(&subject, schema) { Ok(result) => { let id = result.id(); // SchemaId type schema_ids.insert(topic.to_string(), id.0); // Extract i32 from SchemaId wrapper info!("Registered schema for {} with ID {}", topic, id.0); } Err(e) => { error!("Failed to register schema for {}: {}", topic, e); panic!("Schema registration failed"); } } } let _ = SCHEMA_IDS.set(schema_ids); info!("Starting fanout transform with schema IDs"); on_record_written(route_updates); } fn write_update( update: &TableUpdate, schema_id: i32, writer: &mut RecordWriter, event: &WriteEvent, ) -> Result<(), Box> { // Create Schema Registry wire format: [magic_byte, schema_id (4 bytes BE), data...] let mut value = vec![0u8; 5]; value[0] = 0; // magic byte value[1..5].copy_from_slice(&schema_id.to_be_bytes()); let data_bytes = serde_json::to_vec(&update.data)?; value.extend_from_slice(&data_bytes); let key = event.record.key().map(|k| k.to_vec()); let record = BorrowedRecord::new(key.as_deref(), Some(&value)); writer.write_with_options(record, WriteOptions::to_topic(&update.table))?; Ok(()) } fn route_updates(event: WriteEvent, writer: &mut RecordWriter) -> Result<(), Box> { let batch: BatchMessage = serde_json::from_slice(event.record.value().unwrap_or_default())?; let schema_ids = SCHEMA_IDS.get().unwrap(); for update in batch.updates.iter() { if let Some(&schema_id) = schema_ids.get(&update.table) { write_update(update, schema_id, writer, &event)?; } } Ok(()) } ``` ##### JavaScript The JavaScript SDK does not support writing records to specific output topics. For multi-topic fan-out, use the Go or Rust SDK. ### [](#connect-to-the-schema-registry)Connect to the Schema Registry You can use the Schema Registry client library to read and write schemas as well as serialize and deserialize records. This client library is useful when working with schema-based topics in your data transforms. See also: - [Redpanda Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/) - [Go Schema Registry client reference](https://docs.redpanda.com/streaming/current/reference/data-transforms/golang-sdk/) - [Rust Schema Registry client reference](https://docs.redpanda.com/streaming/current/reference/data-transforms/rust-sdk/) - [JavaScript Schema Registry client reference](https://docs.redpanda.com/streaming/current/reference/data-transforms/js/js-sdk-sr/) ## [](#next-steps)Next steps [Configure Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/) ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) - [Data Transforms SDKs](https://docs.redpanda.com/streaming/current/reference/data-transforms/sdks/) - [`rpk transform` commands](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 66: Configure Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/configure.md --- # Configure Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/configure page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/configure.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/configure.adoc description: Learn how to configure data transforms in Redpanda, including editing the transform.yaml file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine's environment. page-git-created-date: "2024-07-31" page-git-modified-date: "2025-04-08" support-status: supported --- Learn how to configure data transforms in Redpanda, including editing the `transform.yaml` file, environment variables, and memory settings. This topic covers both the configuration of transform functions and the WebAssembly (Wasm) engine’s environment. ## [](#configure-transform-functions)Configure transform functions This section covers how to configure transform functions using the `transform.yaml` configuration file, command-line overrides, and environment variables. ### [](#config-file)Transform configuration file When you [initialize](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/#init) a data transforms project, a `transform.yaml` file is generated in the provided directory. You can use this configuration file to configure the transform function with settings, including input and output topics, the language used for the data transform, and any environment variables. - `name`: The name of the transform function. - `description`: A description of what the transform function does. - `input-topic`: The topic from which data is read. - `output-topics`: A list of up to eight topics to which the transformed data is written. - `language`: The language used for the transform function. The language is set to the one you defined during [initialization](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/#init). - `env`: A dictionary of custom environment variables that are passed to the transform function. Do not prefix keys with `REDPANDA_`. Check the list of all [limitations](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/#limitations). Here is an example of a transform.yaml file: ```yaml name: redpanda-example description: | This transform function is an example to demonstrate how to configure data transforms in Redpanda. input-topic: example-input-topic output-topics: - example-output-topic-1 - example-output-topic-2 language: tinygo-no-goroutines env: DATA_TRANSFORMS_ARE_AWESOME: 'true' ``` ### [](#cl)Override configurations with command-line options You can set the name of the transform function, environment variables, and input and output topics on the command-line when you deploy the transform. These command-line settings take precedence over those specified in the `transform.yaml` file. See [Deploy Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/deploy/) ### [](#built-in)Built-In environment variables As well as custom environment variables set in either the [command-line](#cl) or the [configuration file](#config-file), Redpanda makes some built-in environment variables available to your transform functions. These variables include: - `REDPANDA_INPUT_TOPIC`: The input topic specified. - `REDPANDA_OUTPUT_TOPIC_0..REDPANDA_OUTPUT_TOPIC_N`: The output topics in the order specified on the command line or in the configuration file. For example, `REDPANDA_OUTPUT_TOPIC_0` is the first variable, `REDPANDA_OUTPUT_TOPIC_1` is the second variable, and so on. Transform functions are isolated from the broker’s internal environment variables to maintain security and encapsulation. Each transform function only uses the environment variables explicitly provided to it. ## [](#configure-the-wasm-engine)Configure the Wasm engine This section covers how to configure the Wasm engine environment using Redpanda cluster configuration properties. ### [](#enable-transforms)Enable data transforms To use data transforms, you must enable it for a Redpanda cluster using the [`data_transforms_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_enabled) property. ### [](#configure-memory-resources-for-data-transforms)Configure memory resources for data transforms Redpanda reserves memory for each transform function within the broker. You need enough memory for your input record and output record to be in memory at the same time. Set the following based on the number of functions you have and the amount of memory you anticipate needing. - [`data_transforms_per_core_memory_reservation`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_per_core_memory_reservation): Increase this setting if you plan to deploy a large number of data transforms or if your transforms are memory-intensive. Reducing it may limit the number of concurrent transforms. - [`data_transforms_per_function_memory_limit`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_per_function_memory_limit): Adjust this setting if individual transform functions require more memory to process records efficiently. Reducing it may cause memory errors in complex transforms. The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. When that limit is hit, Redpanda cannot allocate memory for the VM and the transforms stay in `errored` states. ### [](#binary-size)Configure maximum binary size You can set the maximum size for a deployable Wasm binary that the broker can store using the [`data_transforms_binary_max_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_binary_max_size) property. Increase this setting if your Wasm binaries are larger than the default limit. Setting it too low may prevent deployment of valid transform functions. ### [](#commit-interval)Configure commit interval You can set the interval at which data transforms commit their progress using the [`data_transforms_commit_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_commit_interval_ms) property. Adjust this setting to control how frequently the transform function’s progress is committed. Shorter intervals may provide more frequent progress updates but can increase load. Longer intervals reduce load but may delay progress updates. ### [](#log)Configure transform logging The following properties configure logging for data transforms: - [`data_transforms_logging_buffer_capacity_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_logging_buffer_capacity_bytes): Increase this value if your transform logs are large or if you need to buffer more log data before flushing. Reducing this value may cause more frequent log flushing. - [`data_transforms_logging_flush_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_logging_flush_interval_ms): Adjust this value to control how frequently logs are flushed to the `transform_logs` topic. Shorter intervals provide more frequent log updates but can increase load. Longer intervals reduce load but may delay log updates. - [`data_transforms_logging_line_max_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_logging_line_max_bytes): Increase this value if your log messages are frequently truncated. Setting this value too low may truncate important log information. ### [](#runtime-limit)Configure runtime limits You can set the maximum runtime for starting up a data transform and the time it takes for a single record to be transformed using the [`data_transforms_runtime_limit_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#data_transforms_runtime_limit_ms) property. Adjust this value only if your transform functions need more time to process each record or to start up. ## [](#next-steps)Next steps [Deploy Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/deploy/) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 67: Deploy Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/deploy.md --- # Deploy Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Deploy Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/deploy page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/deploy.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/deploy.adoc description: Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. page-git-created-date: "2024-07-31" page-git-modified-date: "2025-04-08" support-status: supported --- Learn how to build, deploy, share, and troubleshoot data transforms in Redpanda. ## [](#prerequisites)Prerequisites Before you begin, ensure that you have the following: - [Data transforms enabled](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#enable-transforms) in your Redpanda cluster. - The [`rpk` command-line client](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed on your host machine and configured to connect to your Redpanda cluster. - A [data transform](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/) project. ## [](#build)Build the Wasm binary To build a Wasm binary: 1. Ensure your project directory contains a `transform.yaml` file. 2. Build the Wasm binary using the [`rpk transform build`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-build/) command. ```bash rpk transform build ``` You should now have a Wasm binary named `.wasm`, where `` is the name specified in your `transform.yaml` file. This binary is your data transform function, ready to be deployed to a Redpanda cluster or hosted on a network for others to use. ## [](#deploy)Deploy the Wasm binary You can deploy your transform function using the [`rpk transform deploy`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-deploy/) command. 1. Validate your setup against the pre-deployment checklist: - Do you meet the [Prerequisites](#prerequisites)? - Does your transform function access any environment variables? If so, make sure to set them in the `transform.yaml` file or in the command-line when you deploy the binary. - Do your configured input and output topics already exist? Input and output topics must exist in your Redpanda cluster before you deploy the Wasm binary. 2. Deploy the Wasm binary: ```bash rpk transform deploy ``` When the transform function reaches Redpanda, it starts processing new records that are written to the input topic. ### [](#reprocess)Reprocess records In some cases, you may need to reprocess records from an input topic that already contains data. Processing existing records can be useful, for example, to process historical data into a different format for a new consumer, to re-create lost data from a deleted topic, or to resolve issues with a previous version of a transform that processed data incorrectly. To reprocess records, you can specify the starting point from which the transform function should process records in each partition of the input topic. The starting point can be either a partition offset or a timestamp. > 📝 **NOTE** > > The `--from-offset` flag is only effective the first time you deploy a transform function. On subsequent deployments of the same function, Redpanda resumes processing from the last committed offset. To reprocess existing records using an existing function, [delete the function](#delete) and redeploy it with the `--from-offset` flag. To deploy a transform function and start processing records from a specific partition offset, use the following syntax: ```bash rpk transform deploy --from-offset +/- ``` In this example, the transform function will start processing records from the beginning of each partition of the input topic: ```bash rpk transform deploy --from-offset +0 ``` To deploy a transform function and start processing records from a specific timestamp, use the following syntax: ```bash rpk transform deploy --from-timestamp @ ``` In this example, the transform function will start processing from the first record in each partition of the input topic that was committed after the given timestamp: ```bash rpk transform deploy --from-timestamp @1617181723 ``` ### [](#share-wasm-binaries)Share Wasm binaries You can also deploy data transforms on a Redpanda cluster by providing an addressable path to the Wasm binary. This is useful for sharing transform functions across multiple clusters or teams within your organization. For example, if the Wasm binary is hosted at `https://my-site/my-transform.wasm`, use the following command to deploy it: ```bash rpk transform deploy --file=https://my-site/my-transform.wasm ``` ## [](#edit-existing-transform-functions)Edit existing transform functions To make changes to an existing transform function: 1. [Make your changes to the code](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/). 2. [Rebuild](#build) the Wasm binary. 3. [Redeploy](#deploy) the Wasm binary to the same Redpanda cluster. When you redeploy a Wasm binary with the same name, it will resume processing from the last offset it had previously processed. If you need to [reprocess existing records](#reprocess), you must delete the transform function, and redeploy it with the `--from-offset` flag. Deploy-time configuration overrides must be provided each time you redeploy a Wasm binary. Otherwise, they will be overwritten by default values or the configuration file’s contents. ## [](#delete)Delete a transform function To delete a transform function, use the following command: ```bash rpk transform delete ``` For more details about this command, see [rpk transform delete](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-delete/). > 💡 **TIP** > > You can also [delete transform functions in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/data-transforms/#delete). ## [](#troubleshoot)Troubleshoot This section provides guidance on how to diagnose and troubleshoot issues with building or deploying data transforms. ### [](#invalid-transform-environment)Invalid transform environment This error means that one or more of your configured custom environment variables are invalid. Check your custom environment variables against the list of [limitations](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/#limitations). ### [](#invalid-webassembly)Invalid WebAssembly This error indicates that the binary is missing a required callback function: Invalid WebAssembly - the binary is missing required transform functions. Check the broker support for the version of the data transforms SDK being used. All transform functions must register a callback with the `OnRecordWritten()` method. For more details, see [Develop Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/). ## [](#next-steps)Next steps [Set up monitoring](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/) for data transforms. ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 68: How Data Transforms Work **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work.md --- # How Data Transforms Work > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: How Data Transforms Work latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/how-transforms-work page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/how-transforms-work.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/how-transforms-work.adoc description: Learn how Redpanda data transforms work. page-git-created-date: "2023-12-22" page-git-modified-date: "2026-05-26" support-status: supported --- Redpanda provides the framework to build and deploy inline transformations (data transforms) on data written to Redpanda topics, delivering processed and validated data to consumers in the format they expect. Redpanda does this directly inside the broker, eliminating the need to manage a separate stream processing environment or use third-party tools. ![Data transforms in a broker](https://docs.redpanda.com/streaming/current/shared/_images/wasm1.png) Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types. To learn how to build and deploy data transforms, see [Data Transforms in Linux Quickstart](https://docs.redpanda.com/streaming/current/develop/data-transforms/run-transforms/). ## [](#data-transforms-with-webassembly)Data transforms with WebAssembly Data transforms use [WebAssembly](https://webassembly.org/) (Wasm) engines inside a Redpanda broker, allowing Redpanda to control the entire transform lifecycle. For example, Redpanda can stop and start transforms when partitions are moved or to free up system resources for other tasks. Data transforms take data from an input topic and map it to one or more output topics. For each topic partition, a leader is responsible for handling the data. Redpanda runs a Wasm virtual machine (VM) on the same CPU core (shard) as these partition leaders to execute the transform function. Transform functions are the specific implementations of code that carry out the transformations. They read data from input topics, apply the necessary processing logic, and write the transformed data to output topics. To execute a transform function, Redpanda uses just-in-time (JIT) compilation to compile the bytecode in memory, write it to an executable space, then run the directly translated machine code. This JIT compilation ensures efficient execution of the machine code, as it is tailored to the specific hardware it runs on. When you deploy a data transform to a Redpanda broker, it stores the Wasm bytecode and associated metadata, such as input and output topics and environment variables. The broker then replicates this data across the cluster using internal Kafka topics. When the data is distributed, each shard runs its own instance of the transform function. This process includes several resource management features: - Each shard can run only one instance of the transform function at a time to ensure efficient resource utilization and prevent overload. - Memory for each function is reserved within the broker with the `data_transforms_per_core_memory_reservation` and `data_transforms_per_function_memory_limit` properties. See [Configure memory for data transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#resources). - CPU time is dynamically allocated to the Wasm runtime to ensure that the code does not run forever and cannot block the broker from handling traffic or doing other work, such as Tiered Storage uploads. ## [](#flow-of-data-transforms)Flow of data transforms When a shard becomes the leader of a given partition on the input topic of one or more active transforms, Redpanda does the following: 1. Spins up a Wasm VM using the JIT-compiled Wasm module. 2. Pushes records from the input partition into the Wasm VM. 3. Writes the output. The output partition may exist on the same broker or on another broker in the cluster. Within Redpanda, a single Raft controller manages cluster information, including data transforms. On every shard, Redpanda knows what data transforms exist in the cluster, as well as metadata about the transform function, such as input and output topics and environment variables. ![Wasm architecture in Redpanda](https://docs.redpanda.com/streaming/current/shared/_images/wasm_architecture.png) Each transform function reads from a specified input topic and writes to a specified output topic. The transform function processes every record produced to an input topic and returns zero or more records that are then produced to the specified output topic. Data transforms are applied to all partitions on an input topic. A record is processed after it has been successfully written to disk on the input topic. Because the transform happens in the background after the write finishes, the transform doesn’t affect the original produced record, doesn’t block writes to the input topic, and doesn’t block produce and consume requests. A new transform function reads the input topic from the latest offset. That is, it only reads new data produced to the input topic: it does not read records produced to the input topic before the transform was deployed. If a partition leader moves from one broker to another, then the instance of the transform function assigned to that partition moves with it. When a partition replica [loses leadership](https://docs.redpanda.com/streaming/current/get-started/architecture/#partition-leadership-elections), the broker hosting that partition replica stops the instance of the transform function running on the same shard. The broker that is now hosting the partition’s new leader starts the transform function on the same shard as that leader, and the transform function resumes from the last committed offset. If the previous instance of the transform function failed to commit its latest offsets before moving with the partition leader (for example, if the broker crashed), then it’s likely that the new instance will reprocess some events. For broker failures, transform functions have at-least-once semantics, because records are retried from the committed last offset, and offsets are committed periodically. For more information, see [Data Transforms in Linux Quickstart](https://docs.redpanda.com/streaming/current/develop/data-transforms/run-transforms/). ## [](#limitations)Limitations This section outlines the limitations of data transforms. These constraints are categorized into general limitations affecting the overall functionality and specific limitations related to giving data transforms access to custom environment variables. ### [](#general)General - **No external access**: Transform functions have no external access to disk or network resources. - **Single message transforms**: Only single record transforms are supported, but multiple output records from a single input record are supported. For aggregations, joins, or complex transformations, consider using [Redpanda Connect](https://docs.redpanda.com/connect/get-started/about/) or [Apache Flink](https://flink.apache.org/). - **Output topic limit**: Up to eight output topics are supported. - **Delivery semantics**: Transform functions have at-least-once delivery. - **Transactions API**: When clients use the Kafka Transactions API on partitions of an input topic, transform functions process only committed records. ### [](#javascript)JavaScript - **No native extensions**: Native Node.js extensions are not supported. Packages that require compiling native code or interacting with low-level system features cannot be used. - **Limited Node.js standard modules**: Only modules that can be polyfilled by the [esbuild plugin](https://www.npmjs.com/package/esbuild-plugin-polyfill-node#implemented-polyfills) can be used. Even if a module can be polyfilled, certain functionalities, such as network connections, will not work because the necessary browser APIs are not exposed in the Redpanda JavaScript runtime environment. For example, while the plugin can provide stubs for some Node.js modules such as `http` and `process`, these stubs will not work in the Redpanda JavaScript runtime environment. - **No write options**: The JavaScript SDK does not support write options, such as specifying which output topic to write to. ### [](#environment-variables)Environment variables - **Maximum number of variables**: You can set up to 128 custom environment variables. - **Reserved prefix**: Variable keys must not start with `REDPANDA_`. This prefix is reserved for [built-in environment variables](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#built-in). - **Key length**: Each key must be less than 128 bytes in length. - **Total value length**: The combined length of all values for the environment variables must be less than 2000 bytes. - **Encoding**: All keys and values must be encoded in UTF-8. - **Control characters**: Keys and values must not contain any control characters, such as null bytes. ## [](#suggested-reading)Suggested reading - [Golang SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/golang-sdk/) - [Rust SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/rust-sdk/) - [`rpk transform` commands](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 69: Data Transforms in Kubernetes Quickstart **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/k-run-transforms.md --- # Data Transforms in Kubernetes Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Data Transforms in Kubernetes Quickstart latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/k-run-transforms page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/k-run-transforms.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/k-run-transforms.adoc description: Learn how to build and deploy your first transform function in Kubernetes deployments. page-git-created-date: "2023-12-22" page-git-modified-date: "2025-07-31" support-status: supported --- Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types. Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with [`rpk transform`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) commands. > 📝 **NOTE** > > You should build and deploy transforms from a separate, non-production machine (host machine). Using a separate host machine avoids potential resource conflicts and stability issues on the nodes that run your brokers. See also: [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) ## [](#prerequisites)Prerequisites You must have the following: - [A Redpanda cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/) running at least version 26.2. - External access to the Kafka API and the Admin API. Ensure that your Redpanda cluster has [external access](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/external/) enabled and is accessible from your host machine using the advertised addresses. > 💡 **TIP** > > For a tutorial on setting up a Redpanda cluster with external access, see [Get Started with Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/get-started-dev/). - Development tools installed on your host machine: - For Golang, you must have at least version 1.20 of [Go](https://go.dev/doc/install). - For Rust, you must have the latest stable version of [Rust](https://rustup.rs/). - The [`rpk` command-line client](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed on your host machine and configured to connect to your Redpanda cluster. - For JavaScript and TypeScript projects, you must have the [latest long-term-support release of Node.js](https://nodejs.org/en/download/package-manager). You can use a [pre-configured `rpk` profile](https://docs.redpanda.com/streaming/current/manage/kubernetes/networking/k-connect-to-redpanda/#rpk-profile): ```bash rpk profile create --from-profile <(kubectl get configmap --namespace redpanda-rpk -o go-template='{{ .data.profile }}') ``` Replace `` with the name that you want to give this `rpk` profile. ## [](#enable-data-transforms)Enable data transforms Data transforms is disabled on all clusters by default. Before you can deploy data transforms to a cluster, you must first enable the feature. 1. To enable data transforms, set the `data_transforms_enabled` cluster property to `true`: ### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: config: cluster: data_transforms_enabled: true ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` ### Helm #### --values `write-caching.yaml` ```yaml config: cluster: data_transforms_enabled: true ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values write-caching.yaml --reuse-values ``` #### --set ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --set config.cluster.data_transforms_enabled=true ``` 2. Restart all brokers: ```bash kubectl rollout restart statefulset redpanda --namespace= ``` 3. Wait for all Pods to restart: ```bash kubectl rollout status statefulset redpanda --namespace= --watch ``` ## [](#create-a-data-transforms-project)Create a data transforms project The easiest way to create a new data transforms project is to use the [`rpk transform init` command](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-init/). This command generates template files and installs any dependencies for your chosen language. Create and initialize a data transforms project: ### Go ```bash rpk transform init --language=tinygo --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── go.mod ├── go.sum ├── README.md ├── transform.go └── transform.yaml The `transform.go` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. ### Rust ```bash rpk transform init --language=rust --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── Cargo.lock ├── Cargo.toml ├── README.md ├── src │ └── main.rs └── transform.yaml The `src/main.rs` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. ### JavaScript ```bash rpk transform init --language=javascript --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── README.md ├── esbuild.js ├── node\_modules ├── package-lock.json ├── package.json ├── src │ └── index.js └── transform.yaml The `src/index.js` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. The `esbuild.js` file is the build script for your project. This file configures the build process using esbuild, a fast JavaScript bundler. It ensures that your code is bundled correctly and includes any necessary polyfills for Node.js standard modules that are not natively available in the Redpanda JavaScript runtime environment. Now that you have a project set up, you can run some examples to learn how to work with data transforms. Make sure to copy the provided transform functions and paste them into your source file. For example, the `transform.go` file for Go projects, or the `src/main.rs` file for Rust. ## [](#run-examples)Run examples This section provides some examples of transform functions to teach you the basics of writing and deploying data transforms. It’s best to try each example in order, one after the other. ### [](#copy-records-from-one-topic-to-another)Copy records from one topic to another This transform function copies the same data from an input topic to an output topic. 1. Paste this transform function into your source file: #### Go ```go package main import ( "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Make sure to register your callback and perform other setup in main transform.OnRecordWritten(copyRecordsToOutput) } // This will be called for each record in the input topic. // The records returned will be written to the output topic. func copyRecordsToOutput(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { // Make sure to register your callback and perform other setup in main on_record_written(copy_records_to_output); } // This will be called for each record in the input topic. // The records returned will be written to the output topic. fn copy_records_to_output(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { writer.write(event.record)?; Ok(()) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Register your callback function in the entry point of your script. onRecordWritten(copyRecordsToOutput); // This function will be called for each record in the input topic. // The records returned will be written to the output topic. function copyRecordsToOutput(event, writer) { writer.write(event.record); } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic ``` 4. Deploy the Wasm binary to your cluster: ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic ``` 5. Produce two new records to the input topic. ```bash echo "hello\nworld" | rpk topic produce input-topic ``` 6. [Open Redpanda Console](http://localhost:8080/topics) and check the records in both the input topic and the output topic. They should be the same. You can also verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` ### [](#convert-csv-input-to-json-output)Convert CSV input to JSON output This example is a transform function that converts CSV inputs into JSON outputs. 1. Prepare the project files: #### Go Paste this transform function into your source file: ```go package main import ( "bytes" "encoding/csv" "encoding/json" "errors" "io" "strconv" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(csvToJsonTransform) } type ItemQuantity struct { Item string `json:"item"` Quantity int `json:"quantity"` } func csvToJsonTransform(event transform.WriteEvent, writer transform.RecordWriter) error { // The input data is a CSV (without a header row) that is structured as: // key, item, quantity reader := csv.NewReader(bytes.NewReader(event.Record().Value)) // Improve performance by reusing the result slice. reader.ReuseRecord = true for { row, err := reader.Read() if err == io.EOF { break } else if err != nil { return err } if len(row) != 3 { return errors.New("unexpected number of rows") } // Convert the last column into an int quantity, err := strconv.Atoi(row[2]) if err != nil { return err } // Marshall the JSON value iq := ItemQuantity{ Item: row[1], Quantity: quantity, } v, err := json.Marshal(&iq) if err != nil { return err } // Add the output record using the first column as the key. r := transform.Record{ Key: []byte(row[0]), Value: v, } if err := writer.Write(r); err != nil { return err } } return nil } ``` #### Rust 1. Add the following dependencies to the `Cargo.toml` file: ```toml csv = "1.3.0" serde_json = "1.0.111" serde = { version = "1.0.195", features = ["derive"] } ``` 2. Run the following command to update your dependencies: ```bash cargo build ``` 3. Paste this transform function into your source file: ```rust use anyhow::Result; use redpanda_transform_sdk::*; use serde::{Deserialize, Serialize}; use csv::ReaderBuilder; use serde_json; #[derive(Serialize, Deserialize)] struct MarketStock { item: String, quantity: i32, } fn main() { on_record_written(csv_to_json_transform); } fn csv_to_json_transform(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { // The input data is a CSV (without a header row) that is defined as the MarketStock structure. let mut reader = ReaderBuilder::new().has_headers(false).from_reader(event.record.value().unwrap_or_default()); // For each record in our CSV for result in reader.deserialize() { let stock: MarketStock = match result { Ok(record) => record, Err(err) => { eprintln!("CSV deserialize error: {}", err); continue; // Skip the invalid record and continue processing } }; // Convert it to JSON let value = serde_json::to_vec(&stock)?; // Then output it with the same key. writer.write(BorrowedRecord::new(event.record.key(), Some(&value)))?; } Ok(()) } ``` #### JavaScript Paste this transform function into your source file: ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; onRecordWritten(csvToJsonTransform); function csvToJsonTransform(event, writer) { // The input data is a CSV (without a header row) that is structured as: // key, item, quantity const input = event.record.value.text(); const rows = input.split('\n'); for (const row of rows) { const columns = row.split(','); if (columns.length !== 2) { throw new Error('unexpected number of columns'); } const quantity = parseInt(columns[1], 10); if (isNaN(quantity)) { throw new Error('invalid quantity'); } const itemQuantity = { item: columns[0], quantity: quantity, }; event.record.value = JSON.stringify(itemQuantity); writer.write(event.record); } } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic ``` 4. Deploy the Wasm binary to your cluster. If you have already deployed another example, this new transform function will replace it. ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic ``` 5. Produce CSV records to the input topic. ```bash echo "apples,10\npears,11\noranges,5" | rpk topic produce input-topic -k market-stock ``` 6. [Open Redpanda Console](http://localhost:8080/topics) and check the records in both the input topic and the output topic. You should see the following values: ```json { "item": "oranges", "quantity": 5 } { "item": "apples", "quantity": 10 } { "item": "pears", "quantity": 11 } ``` You can also verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` ### [](#validate-json)Validate JSON This example is a filter that outputs only valid JSON from the input topic into the output topic. Invalid JSON is written to a different output topic. 1. Paste this transform function into your source file: #### Go ```go import ( "encoding/json" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(filterValidJson) } func filterValidJson(event transform.WriteEvent, writer transform.RecordWriter) error { if json.Valid(event.Record().Value) { return w.Write(e.Record()) } // Send invalid records to separate topic return writer.Write(e.Record(), transform.ToTopic("invalid-json")) } ``` #### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { on_record_written(filter_valid_json); } fn filter_valid_json(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { let value = event.record.value().unwrap_or_default(); if serde_json::from_slice::(value).is_ok() { writer.write(event.record)?; } else { // Send invalid records to separate topic writer.write_with_options(event.record, WriteOptions::to_topic("invalid-json"))?; } Ok(()) } ``` #### JavaScript The JavaScript SDK does not support writing records to a specific output topic. As a result, this transform function writes only valid JSON messages to the output topic. Invalid messages are logged, instead of written to a separate output topic. ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; onRecordWritten(filterValidJson); function filterValidJson(event, writer) { const recordValue = event.record.value.text(); if (isValidJson(recordValue)) { writer.write(event.record); } else { console.log('Invalid JSON detected') } } function isValidJson(str) { try { JSON.parse(str); return true; } catch (e) { return false; } } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic invalid-json ``` 4. Deploy the Wasm binary to your cluster. If you have already deployed another example, this new transform function will replace it. ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic --output-topic=invalid-json ``` 5. Produce an invalid JSON record a one valid one to the input topic. ```bash echo '{"valid":"json"}' | rpk topic produce input-topic -k json echo 'invalid json' | rpk topic produce input-topic -k json ``` 6. Verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` You should see only the valid JSON from the input topic. Invalid JSON messages are written to the `invalid-json` topic. ## [](#clean-up)Clean up Your transform function will continue processing new records in the input topic until you delete it. To delete the transform function: ```bash rpk transform delete data-transforms-tutorial --no-confirm ``` ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) - [Golang SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/golang-sdk/) - [Rust SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/rust-sdk/) - [`rpk transform` commands](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 70: Prebuilt Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/labs.md --- # Prebuilt Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Prebuilt Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/labs page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/labs.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/labs.adoc description: Explore labs that include examples of transform functions and instructions on how to deploy and run them. page-git-created-date: "2024-07-31" page-git-modified-date: "2024-08-01" support-status: supported --- Explore labs that include examples of transform functions and instructions on how to deploy and run them. - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/): Flatten JSON messages in topics using data transforms. - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/): Query live tracking data from the International Space Station and convert it from JSON to Avro using data transforms. - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/): Filter messages from one topic into another using regular expressions (regex) and data transforms. - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/): Convert timestamps from various forms, such as epochs to strings. - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/): Redact personally identifiable information (PII) in topics using data transforms. --- # Page 71: Monitor Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor.md --- # Monitor Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Monitor Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/monitor page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/monitor.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/monitor.adoc description: This topic provides guidelines on how to monitor the health of your data transforms and view logs. page-git-created-date: "2024-07-31" page-git-modified-date: "2025-04-08" support-status: supported --- This topic provides guidelines on how to monitor the health of your data transforms and view logs. ## [](#prerequisites)Prerequisites [Set up monitoring](https://docs.redpanda.com/streaming/current/manage/monitoring/) for your cluster. ## [](#performance)Performance You can identify performance bottlenecks by monitoring latency and CPU usage: - [`redpanda_transform_execution_latency_sec`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_execution_latency_sec) - [`redpanda_wasm_engine_cpu_seconds_total`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_wasm_engine_cpu_seconds_total) If latency is high, investigate the transform logic for inefficiencies or consider scaling the resources. High CPU usage might indicate the need for optimization in the code or an increase in [allocated CPU resources](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/). ## [](#reliability)Reliability Tracking execution errors and error states helps in maintaining the reliability of your data transforms: - [`redpanda_transform_execution_errors`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_execution_errors) - [`redpanda_transform_failures`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_failures) - [`redpanda_transform_state`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_state) Make sure to [implement robust error handling and logging](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/#errors) within your transform functions to help with troubleshooting. ## [](#resource-usage)Resource usage Monitoring memory usage metrics and total execution time ensures that the Wasm engine does not exceed allocated resources, helping in efficient resource management: - [`redpanda_wasm_engine_memory_usage`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_wasm_engine_memory_usage) - [`redpanda_wasm_engine_max_memory`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_wasm_engine_max_memory) - [`redpanda_wasm_binary_executable_memory_usage`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_wasm_binary_executable_memory_usage) If memory usage is consistently high or exceeds the maximum allocated memory: - Review and optimize your transform functions to reduce memory consumption. This step can involve optimizing data structures, reducing memory allocations, and ensuring efficient handling of records. - Consider increasing the allocated memory for the Wasm engine. Adjust the [`data_transforms_per_core_memory_reservation`](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#resources) and [`data_transforms_per_function_memory_limit settings`](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#resources) to provide more memory to each function and the overall Wasm engine. ## [](#throughput)Throughput Keeping track of read and write bytes and processor lag helps in understanding the data flow through your transforms, enabling better capacity planning and scaling: - [`redpanda_transform_read_bytes`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_read_bytes) - [`redpanda_transform_write_bytes`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_write_bytes) - [`redpanda_transform_processor_lag`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_transform_processor_lag) If there is a significant lag or low throughput, investigate potential bottlenecks in the data flow or consider scaling your infrastructure to handle higher throughput. ## [](#logs)View logs for data transforms Runtime logs for transform functions are written to an internal topic called `_redpanda.transform_logs`. You can read these logs by using the [`rpk transform logs`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-logs/) command. ```bash rpk transform logs ``` Replace `` with the [configured name](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/) of the transform function. > 💡 **TIP** > > You can also [view logs in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/data-transforms/#logs). By default, Redpanda provides several settings to manage logging for data transforms, such as buffer capacity, flush interval, and maximum log line length. These settings ensure that logging operates efficiently without overwhelming the system. However, you may need to adjust these settings based on your specific requirements and workloads. For information on how to configure logging, see the [Configure transform logging](https://docs.redpanda.com/streaming/current/develop/data-transforms/configure/#log) section of the configuration guide. ## [](#suggested-reading)Suggested reading - [Data transforms metrics](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#data_transform_metrics) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 72: Data Transforms Quickstarts **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/run-transforms-index.md --- # Data Transforms Quickstarts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Data Transforms Quickstarts latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/run-transforms-index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/run-transforms-index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/run-transforms-index.adoc description: Choose your deployment environment to get started with building and deploying your first transform function in Redpanda. page-git-created-date: "2023-12-22" page-git-modified-date: "2025-04-16" support-status: supported --- - [Data Transforms in Linux Quickstart](https://docs.redpanda.com/streaming/current/develop/data-transforms/run-transforms/) Learn how to build and deploy your first transform function in Linux deployments. - [Data Transforms in Kubernetes Quickstart](https://docs.redpanda.com/streaming/current/develop/data-transforms/k-run-transforms/) Learn how to build and deploy your first transform function in Kubernetes deployments. --- # Page 73: Data Transforms in Linux Quickstart **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/run-transforms.md --- # Data Transforms in Linux Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Data Transforms in Linux Quickstart latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/run-transforms page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/run-transforms.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/run-transforms.adoc description: Learn how to build and deploy your first transform function in Linux deployments. page-git-created-date: "2023-12-22" page-git-modified-date: "2025-07-31" support-status: supported --- Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types. Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with [`rpk transform`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) commands. > 📝 **NOTE** > > You should build and deploy transforms from a separate, non-production machine (host machine). Using a separate host machine avoids potential resource conflicts and stability issues on the nodes that run your brokers. See also: [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) ## [](#prerequisites)Prerequisites You must have the following: - [A Redpanda cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/) running at least version 26.2. - External access to the Kafka API and the Admin API. - Development tools installed on your host machine: - For Golang, you must have at least version 1.20 of [Go](https://go.dev/doc/install). - For Rust, you must have the latest stable version of [Rust](https://rustup.rs/). - The [`rpk` command-line client](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed on your host machine and configured to connect to your Redpanda cluster. - For JavaScript and TypeScript projects, you must have the [latest long-term-support release of Node.js](https://nodejs.org/en/download/package-manager). ## [](#enable-data-transforms)Enable data transforms Data transforms is disabled on all clusters by default. Before you can deploy data transforms to a cluster, you must first enable the feature. 1. To enable data transforms, set the `data_transforms_enabled` cluster property to `true`: ```bash rpk cluster config set data_transforms_enabled true ``` 2. Restart all brokers: ```bash rpk redpanda stop rpk redpanda start ``` ## [](#create-a-data-transforms-project)Create a data transforms project The easiest way to create a new data transforms project is to use the [`rpk transform init` command](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform-init/). This command generates template files and installs any dependencies for your chosen language. Create and initialize a data transforms project: ### Go ```bash rpk transform init --language=tinygo --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── go.mod ├── go.sum ├── README.md ├── transform.go └── transform.yaml The `transform.go` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. ### Rust ```bash rpk transform init --language=rust --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── Cargo.lock ├── Cargo.toml ├── README.md ├── src │ └── main.rs └── transform.yaml The `src/main.rs` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. ### JavaScript ```bash rpk transform init --language=javascript --name=data-transforms-tutorial ``` A successful command generates project files in your current directory: . ├── README.md ├── esbuild.js ├── node\_modules ├── package-lock.json ├── package.json ├── src │ └── index.js └── transform.yaml The `src/index.js` file is the source file for your transform function. The `transform.yaml` file is the configuration for your transform function. The `transform.yaml` file already contains the name of your transform function and the language that you specified in the `rpk transform init` command. The `esbuild.js` file is the build script for your project. This file configures the build process using esbuild, a fast JavaScript bundler. It ensures that your code is bundled correctly and includes any necessary polyfills for Node.js standard modules that are not natively available in the Redpanda JavaScript runtime environment. Now that you have a project set up, you can run some examples to learn how to work with data transforms. Make sure to copy the provided transform functions and paste them into your source file. For example, the `transform.go` file for Go projects, or the `src/main.rs` file for Rust. ## [](#run-examples)Run examples This section provides some examples of transform functions to teach you the basics of writing and deploying data transforms. It’s best to try each example in order, one after the other. ### [](#copy-records-from-one-topic-to-another)Copy records from one topic to another This transform function copies the same data from an input topic to an output topic. 1. Paste this transform function into your source file: #### Go ```go package main import ( "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { // Make sure to register your callback and perform other setup in main transform.OnRecordWritten(copyRecordsToOutput) } // This will be called for each record in the input topic. // The records returned will be written to the output topic. func copyRecordsToOutput(event transform.WriteEvent, writer transform.RecordWriter) error { return writer.Write(event.Record()) } ``` #### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { // Make sure to register your callback and perform other setup in main on_record_written(copy_records_to_output); } // This will be called for each record in the input topic. // The records returned will be written to the output topic. fn copy_records_to_output(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { writer.write(event.record)?; Ok(()) } ``` #### JavaScript ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; // Register your callback function in the entry point of your script. onRecordWritten(copyRecordsToOutput); // This function will be called for each record in the input topic. // The records returned will be written to the output topic. function copyRecordsToOutput(event, writer) { writer.write(event.record); } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic ``` 4. Deploy the Wasm binary to your cluster: ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic ``` 5. Produce two new records to the input topic. ```bash echo "hello\nworld" | rpk topic produce input-topic ``` 6. [Open Redpanda Console](http://localhost:8080/topics) and check the records in both the input topic and the output topic. They should be the same. You can also verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` ### [](#convert-csv-input-to-json-output)Convert CSV input to JSON output This example is a transform function that converts CSV inputs into JSON outputs. 1. Prepare the project files: #### Go Paste this transform function into your source file: ```go package main import ( "bytes" "encoding/csv" "encoding/json" "errors" "io" "strconv" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(csvToJsonTransform) } type ItemQuantity struct { Item string `json:"item"` Quantity int `json:"quantity"` } func csvToJsonTransform(event transform.WriteEvent, writer transform.RecordWriter) error { // The input data is a CSV (without a header row) that is structured as: // key, item, quantity reader := csv.NewReader(bytes.NewReader(event.Record().Value)) // Improve performance by reusing the result slice. reader.ReuseRecord = true for { row, err := reader.Read() if err == io.EOF { break } else if err != nil { return err } if len(row) != 3 { return errors.New("unexpected number of rows") } // Convert the last column into an int quantity, err := strconv.Atoi(row[2]) if err != nil { return err } // Marshall the JSON value iq := ItemQuantity{ Item: row[1], Quantity: quantity, } v, err := json.Marshal(&iq) if err != nil { return err } // Add the output record using the first column as the key. r := transform.Record{ Key: []byte(row[0]), Value: v, } if err := writer.Write(r); err != nil { return err } } return nil } ``` #### Rust 1. Add the following dependencies to the `Cargo.toml` file: ```toml csv = "1.3.0" serde_json = "1.0.111" serde = { version = "1.0.195", features = ["derive"] } ``` 2. Run the following command to update your dependencies: ```bash cargo build ``` 3. Paste this transform function into your source file: ```rust use anyhow::Result; use redpanda_transform_sdk::*; use serde::{Deserialize, Serialize}; use csv::ReaderBuilder; use serde_json; #[derive(Serialize, Deserialize)] struct MarketStock { item: String, quantity: i32, } fn main() { on_record_written(csv_to_json_transform); } fn csv_to_json_transform(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { // The input data is a CSV (without a header row) that is defined as the MarketStock structure. let mut reader = ReaderBuilder::new().has_headers(false).from_reader(event.record.value().unwrap_or_default()); // For each record in our CSV for result in reader.deserialize() { let stock: MarketStock = match result { Ok(record) => record, Err(err) => { eprintln!("CSV deserialize error: {}", err); continue; // Skip the invalid record and continue processing } }; // Convert it to JSON let value = serde_json::to_vec(&stock)?; // Then output it with the same key. writer.write(BorrowedRecord::new(event.record.key(), Some(&value)))?; } Ok(()) } ``` #### JavaScript Paste this transform function into your source file: ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; onRecordWritten(csvToJsonTransform); function csvToJsonTransform(event, writer) { // The input data is a CSV (without a header row) that is structured as: // key, item, quantity const input = event.record.value.text(); const rows = input.split('\n'); for (const row of rows) { const columns = row.split(','); if (columns.length !== 2) { throw new Error('unexpected number of columns'); } const quantity = parseInt(columns[1], 10); if (isNaN(quantity)) { throw new Error('invalid quantity'); } const itemQuantity = { item: columns[0], quantity: quantity, }; event.record.value = JSON.stringify(itemQuantity); writer.write(event.record); } } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic ``` 4. Deploy the Wasm binary to your cluster. If you have already deployed another example, this new transform function will replace it. ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic ``` 5. Produce CSV records to the input topic. ```bash echo "apples,10\npears,11\noranges,5" | rpk topic produce input-topic -k market-stock ``` 6. [Open Redpanda Console](http://localhost:8080/topics) and check the records in both the input topic and the output topic. You should see the following values: ```json { "item": "oranges", "quantity": 5 } { "item": "apples", "quantity": 10 } { "item": "pears", "quantity": 11 } ``` You can also verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` ### [](#validate-json)Validate JSON This example is a filter that outputs only valid JSON from the input topic into the output topic. Invalid JSON is written to a different output topic. 1. Paste this transform function into your source file: #### Go ```go import ( "encoding/json" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) func main() { transform.OnRecordWritten(filterValidJson) } func filterValidJson(event transform.WriteEvent, writer transform.RecordWriter) error { if json.Valid(event.Record().Value) { return w.Write(e.Record()) } // Send invalid records to separate topic return writer.Write(e.Record(), transform.ToTopic("invalid-json")) } ``` #### Rust ```rust use anyhow::Result; use redpanda_transform_sdk::*; fn main() { on_record_written(filter_valid_json); } fn filter_valid_json(event: WriteEvent, writer: &mut RecordWriter) -> Result<()> { let value = event.record.value().unwrap_or_default(); if serde_json::from_slice::(value).is_ok() { writer.write(event.record)?; } else { // Send invalid records to separate topic writer.write_with_options(event.record, WriteOptions::to_topic("invalid-json"))?; } Ok(()) } ``` #### JavaScript The JavaScript SDK does not support writing records to a specific output topic. As a result, this transform function writes only valid JSON messages to the output topic. Invalid messages are logged, instead of written to a separate output topic. ```js import { onRecordWritten } from "@redpanda-data/transform-sdk"; onRecordWritten(filterValidJson); function filterValidJson(event, writer) { const recordValue = event.record.value.text(); if (isValidJson(recordValue)) { writer.write(event.record); } else { console.log('Invalid JSON detected') } } function isValidJson(str) { try { JSON.parse(str); return true; } catch (e) { return false; } } ``` 2. Build the transform into a Wasm binary: ```bash rpk transform build ``` 3. Create topics to apply the transform function to: ```bash rpk topic create input-topic output-topic invalid-json ``` 4. Deploy the Wasm binary to your cluster. If you have already deployed another example, this new transform function will replace it. ```bash rpk transform deploy --input-topic=input-topic --output-topic=output-topic --output-topic=invalid-json ``` 5. Produce an invalid JSON record a one valid one to the input topic. ```bash echo '{"valid":"json"}' | rpk topic produce input-topic -k json echo 'invalid json' | rpk topic produce input-topic -k json ``` 6. Verify the content of the output topic in the command-line: ```bash rpk topic consume output-topic ``` You should see only the valid JSON from the input topic. Invalid JSON messages are written to the `invalid-json` topic. ## [](#clean-up)Clean up Your transform function will continue processing new records in the input topic until you delete it. To delete the transform function: ```bash rpk transform delete data-transforms-tutorial --no-confirm ``` ## [](#suggested-reading)Suggested reading - [How Data Transforms Work](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) - [Golang SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/golang-sdk/) - [Rust SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/rust-sdk/) - [`rpk transform` commands](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-transform/rpk-transform/) ## Suggested labs - [Flatten JSON Messages](https://docs.redpanda.com/labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](https://docs.redpanda.com/labs/data-transforms/issdemo-go/) - [Filter Messages into a New Topic using a Regex](https://docs.redpanda.com/labs/data-transforms/regex-go/) - [Convert Timestamps using Rust](https://docs.redpanda.com/labs/data-transforms/ts-converter-rust/) - [Redact Information in JSON Messages](https://docs.redpanda.com/labs/data-transforms/redaction-go/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 74: Write Integration Tests for Transform Functions **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/test.md --- # Write Integration Tests for Transform Functions > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Write Integration Tests for Transform Functions latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/test page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/test.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/test.adoc description: Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. page-git-created-date: "2024-07-31" page-git-modified-date: "2026-08-13" support-status: supported --- Learn how to write integration tests for data transform functions in Redpanda, including setting up unit tests and using testcontainers for integration tests. This guide covers how to write both unit tests and integration tests for your transform functions. While unit tests focus on testing individual components in isolation, integration tests verify that the components work together as expected in a real environment. ## [](#unit-tests)Unit tests You can create unit tests for transform functions by mocking the interfaces injected into the transform function and asserting that the input and output work correctly. This typically includes mocking the `WriteEvent` and `RecordWriter` interfaces. ```go package main import ( "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" "github.com/redpanda-data/redpanda/src/transform-sdk/go/transform" ) // MockWriteEvent is a mock implementation of the WriteEvent interface. type MockWriteEvent struct { mock.Mock } func (m *MockWriteEvent) Record() transform.Record { args := m.Called() return args.Get(0).(transform.Record) } // MockRecordWriter is a mock implementation of the RecordWriter interface. type MockRecordWriter struct { mock.Mock } func (m *MockRecordWriter) Write(record transform.Record) error { args := m.Called(record) return args.Error(0) } // copyRecord copies the record to the output topic. func copyRecord(event transform.WriteEvent, writer transform.RecordWriter) error { record := event.Record() return writer.Write(record) } // TestCopyRecord tests the copyRecord function. func TestCopyRecord(t *testing.T) { // Create mocks for the WriteEvent and RecordWriter event := new(MockWriteEvent) writer := new(MockRecordWriter) // Set up the expected behavior record := transform.Record{Value: []byte("test")} event.On("Record").Return(record) writer.On("Write", record).Return(nil) // Call the function under test err := copyRecord(event, writer) // Assert that no error occurred and that the expectations were met assert.NoError(t, err) event.AssertExpectations(t) writer.AssertExpectations(t) } ``` To run your unit tests, use the following command: ```bash go test ``` This will execute all tests in the current directory. ## [](#integration-tests)Integration tests Integration tests verify that your transform functions work correctly in a real Redpanda environment. You can use [testcontainers](https://github.com/testcontainers/testcontainers-go/tree/main) to set up and manage a Redpanda instance for testing. For more detailed examples and helper code for setting up integration tests, refer to the SDK integration tests on [GitHub](https://github.com/redpanda-data/redpanda/tree/dev/src/transform-sdk/tests). --- # Page 75: Upgrade the Data Transforms SDK **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/upgrade.md --- # Upgrade the Data Transforms SDK > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Upgrade the Data Transforms SDK latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/upgrade page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/upgrade.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/upgrade.adoc description: Upgrading the SDK version in your data transforms project ensures compatibility with the latest features and fixes. This guide provides step-by-step instructions to upgrade the SDK version for all supported SDK languages. page-git-created-date: "2024-07-31" page-git-modified-date: "2024-08-01" support-status: supported --- Upgrading the SDK version in your data transforms project ensures compatibility with the latest features and fixes. This guide provides step-by-step instructions to upgrade the SDK version for all supported SDK languages. ## [](#prerequisites)Prerequisites Before upgrading, check the [compatibility matrix](https://docs.redpanda.com/streaming/current/develop/data-transforms/versioning-compatibility/) to ensure the new SDK version is compatible with your Redpanda version. ## [](#upgrade-your-local-sdk-version)Upgrade your local SDK version ### Go 1. Open your project’s root directory. 2. Run the following command to update the SDK: ```bash go get github.com/redpanda-data/redpanda/src/transform-sdk/go/transform@v ``` 3. Clean up the `go.mod` and `go.sum` files: ```bash go mod tidy ``` ### Rust 1. Open the `Cargo.toml` file in your project’s root directory. This file specifies the dependencies for your Rust project. 2. Locate the line that specifies the Redpanda SDK version. It will look something like this: ```toml [dependencies] redpanda-transform-sdk = "" ``` 3. Change the version to the one you want to upgrade to. 4. Run the following command to update the dependencies: ```bash cargo update -p redpanda-transform-sdk ``` ### JavaScript 1. Open the `package.json` file in your project’s root directory. 2. Locate the line that specifies the data transforms SDK version. It will look something like this: ```json { "dependencies": { "@redpanda-data/transform-sdk": "" }, } ``` 3. Run the following command to update the SDK: ```bash npm install --save @redpanda-data/transform-sdk@ ``` 4. Verify the update by checking the `package.json` file, the `package-lock.json` file, and the `node_modules/` directory. ## [](#next-steps)Next steps Run your [tests](https://docs.redpanda.com/streaming/current/develop/data-transforms/test/) to ensure everything works correctly with the new SDK version. --- # Page 76: Versioning and Compatibility for Data Transforms **URL**: https://docs.redpanda.com/streaming/current/develop/data-transforms/versioning-compatibility.md --- # Versioning and Compatibility for Data Transforms > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Versioning and Compatibility for Data Transforms latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: data-transforms/versioning-compatibility page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: data-transforms/versioning-compatibility.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/data-transforms/versioning-compatibility.adoc description: The data transforms SDKs use semantic versioning to ensure compatibility and stability. Use this guide to learn the SDKs that are compatible with different versions of Redpanda, and what guarantees are provided regarding SDK and Redpanda compatibility. page-git-created-date: "2024-07-31" page-git-modified-date: "2026-08-13" support-status: supported --- The data transforms SDKs use semantic versioning to ensure compatibility and stability. Use this guide to learn the SDKs that are compatible with different versions of Redpanda, and what guarantees are provided regarding SDK and Redpanda compatibility. ## [](#semantic-versioning)Semantic versioning The data transforms SDKs use semantic versioning, which is a versioning scheme with a three-part number: `major.minor.patch`. When a new version is released, one part of the version number is changed. The changed part of the version number signifies the level of change: - **Major**: Indicate breaking changes. - **Minor**: Indicate backward-compatible functionality. - **Patch**: Indicate backward-compatible bug fixes. For example, in the version number `1.2.3`, `1` is the major version, `2` is the minor version, and `3` is the patch version. ## [](#compatibility-matrix)Compatibility matrix This compatibility matrix provides detailed information on which versions of the SDK are supported with which versions of Redpanda. Always consult the compatibility matrix to ensure that you are using compatible versions. SDK versions are backwards-compatible and will continue to be supported in newer versions of Redpanda. However, newer SDK versions may not work with older versions of Redpanda. Always check this compatibility matrix when upgrading the SDK or Redpanda. ### [](#golang-sdk)Golang SDK | SDK version | Redpanda version | | --- | --- | | 1.1.x | 24.2.x | | 1.0.x | 24.2.x, 24.1.x, 23.3.x | ### [](#rust-sdk)Rust SDK | SDK version | Redpanda version | | --- | --- | | 1.1.x | 24.2.x | | 1.0.x | 24.2.x, 24.1.x, 23.3.x | ### [](#javascript-sdk)JavaScript SDK | SDK version | Redpanda version | | --- | --- | | 1.1.x | 24.2.x | | 1.0.x | 24.2.x, 24.1.x, 23.3.x | ## [](#best-practices)Best practices - **Stay updated**: Regularly check the [release notes](#rn) for updates to both the data transform SDK and Redpanda. Ensure that you are using compatible versions as specified in the [compatibility matrix](#matrix). - **Test upgrades**: Before upgrading either the SDK or Redpanda, test the upgrade in a staging environment. This helps ensure that your data transforms continue to work as expected. - **Monitor logs and metrics**: Watch the logs and metrics to catch any issues early. Ensure that you [set up monitoring and alerting](https://docs.redpanda.com/streaming/current/develop/data-transforms/monitor/) to notify you of any problems with your data transforms. ## [](#rn)Release notes Release notes for SDK versions are included in the Redpanda release notes. This ensures that all changes related to data transforms and SDKs are documented in one place. To view the release notes: 1. Visit the [Redpanda GitHub releases page](https://github.com/redpanda-data/redpanda/releases). 2. Search for the term `wasm` to filter the release notes for relevant updates to the data transform SDKs. ## [](#suggested-reading)Suggested reading - [Upgrade the Data Transforms SDK](https://docs.redpanda.com/streaming/current/develop/data-transforms/upgrade/) - [Golang SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/golang-sdk/) - [Rust SDK for Data Transforms](https://docs.redpanda.com/streaming/current/reference/data-transforms/rust-sdk/) --- # Page 77: Use Redpanda with the HTTP Proxy API **URL**: https://docs.redpanda.com/streaming/current/develop/http-proxy.md --- # Use Redpanda with the HTTP Proxy API > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use Redpanda with the HTTP Proxy API latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: http-proxy page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: http-proxy.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/http-proxy.adoc description: HTTP Proxy exposes a REST API to list topics, produce events, and subscribe to events from topics using consumer groups. page-git-created-date: "2023-05-30" page-git-modified-date: "2025-05-07" support-status: supported --- Redpanda HTTP Proxy (`pandaproxy`) allows access to your data through a REST API. For example, you can list topics or brokers, get events, produce events, subscribe to events from topics using consumer groups, and commit offsets for a consumer. See the [HTTP Proxy API reference](https://docs.redpanda.com/api/doc/http-proxy/) for a full list of available endpoints. ## [](#prerequisites)Prerequisites ### [](#start-redpanda)Start Redpanda The first step is to start up Redpanda. HTTP Proxy is enabled by default on port 8082. To change the proxy port, edit `redpanda.yaml`: #### redpanda.yaml ```yaml ... pandaproxy: pandaproxy_api: - address: 0.0.0.0 port: 8082 ... ``` #### Kubernetes Cluster Resource ```yaml apiVersion: redpanda.vectorized.io/v1alpha1 kind: Cluster ... spec: ... resources: pandaproxyApi: - port: 8082 ... ``` > 📝 **NOTE** > > The remainder of this guide is based on the assumption that the HTTP Proxy port is 8082. ## [](#authenticate-with-http-proxy)Authenticate with HTTP Proxy HTTP Proxy supports authentication using SCRAM credentials or OIDC tokens. The authentication method depends on the [`authentication_method`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#http_proxy_auth_method) broker property and the cluster’s [`http_authentication`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#http_authentication) settings. ### [](#scram-authentication)SCRAM Authentication If HTTP Proxy is configured to support SASL, you can provide the SCRAM username and password as part of the Basic Authentication header in your request. For example, to list topics as an authenticated user: #### curl ```bash curl -s -u ":" "http://:8082/topics" ``` #### NodeJS ```javascript let options = { auth: { username: "", password: "" }, }; axios .get("http://:8082/topics", options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python auth = ("", "") res = requests.get("http://:8082/topics", auth=auth).json() pretty(res) ``` ### [](#oidc-authentication)OIDC Authentication If HTTP Proxy is configured to support OIDC, you can provide an OIDC token in the Authorization header. For example: #### curl ```bash curl -s -H "Authorization: Bearer " "http://:8082/topics" ``` #### NodeJS ```javascript let options = { headers: { Authorization: `Bearer ` }, }; axios .get("http://:8082/topics", options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python headers = {"Authorization": "Bearer "} res = requests.get("http://:8082/topics", headers=headers).json() pretty(res) ``` For details about configuring OIDC authentication, see [OIDC Authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/#oidc-http). ## [](#set-up-libraries)Set up libraries You need an app that calls the HTTP Proxy endpoint. This app can be curl (or a similar CLI), or it could be your own custom app written in any language. Below are curl, JavaScript and Python examples. > 📝 **NOTE** > > In the examples, `` refers to your Redpanda cluster’s hostname or IP address. All following examples use a `base_uri` variable that combines the protocol, host, and port for consistency across curl, JavaScript, and Python examples. ### curl Curl is likely already installed on your system. If not, see [curl download instructions](https://curl.se/download.html). Set the base URI for your HTTP Proxy: ```bash base_uri="http://:8082" ``` ### NodeJS > 📝 **NOTE** > > This is based on the assumption that you’re in the root directory of an existing NodeJS project. See [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) for an example of a NodeJS project. In a terminal window, run: ```bash npm install axios ``` Import the library into your code: ```javascript const axios = require('axios'); const base_uri = 'http://:8082'; ``` ### Python In a terminal window, run: ```bash pip install requests ``` Import the library into your code: ```python import requests import json def pretty(text): print(json.dumps(text, indent=2)) base_uri = "http://:8082" ``` ## [](#create-a-topic)Create a topic To create a test topic for this guide, use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/). You can configure `rpk` for your Redpanda deployment, using [profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/), flags, or [environment variables](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-x-options/#environment-variables). To create a topic named `test_topic` with three partitions, run: ```bash rpk topic create test_topic -p 3 ``` For more information, see the [rpk topic create](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-create/) reference. ## [](#access-your-data)Access your data Here are some sample commands to produce and consume streams: ### [](#get-list-of-topics)Get list of topics #### curl ```bash curl -s "$base_uri/topics" ``` #### NodeJS ```javascript axios .get(`${base_uri}/topics`) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application. If your file name is `index.js` for example, you would run the following command: ```bash node index.js ``` #### Python ```python res = requests.get(f"{base_uri}/topics").json() pretty(res) ``` Expected output: ```bash ["test_topic"] ``` ### [](#send-events-to-a-topic)Send events to a topic Use POST to send events in the REST endpoint query. The header must include the following line: Content-Type:application/vnd.kafka.json.v2+json The following commands show how to send events to `test_topic`: #### curl ```bash curl -s \ -X POST \ "$base_uri/topics/test_topic" \ -H "Content-Type: application/vnd.kafka.json.v2+json" \ -d '{ "records":[ { "value":"Redpanda", "partition":0 }, { "value":"HTTP proxy", "partition":1 }, { "value":"Test event", "partition":2 } ] }' ``` #### NodeJS ```javascript let payload = { records: [ { "value":"Redpanda", "partition": 0 }, { "value":"HTTP proxy", "partition": 1 }, { "value":"Test event", "partition": 2 } ]}; let options = { headers: { "Content-Type" : "application/vnd.kafka.json.v2+json" }}; axios .post(`${base_uri}/topics/test_topic`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/topics/test_topic", data=json.dumps( dict(records=[ dict(value="Redpanda", partition=0), dict(value="HTTP Proxy", partition=1), dict(value="Test Event", partition=2) ])), headers={"Content-Type": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output (may be formatted differently depending on the chosen application): ```bash {"offsets":[{"partition":0,"offset":0},{"partition":2,"offset":0},{"partition":1,"offset":0}]} ``` ### [](#get-events-from-a-topic)Get events from a topic After events have been sent to the topic, you can retrieve these same events. #### curl ```bash curl -s \ "$base_uri/topics/test_topic/partitions/0/records?offset=0&timeout=1000&max_bytes=100000"\ -H "Accept: application/vnd.kafka.json.v2+json" ``` #### NodeJS ```javascript let options = { headers: { accept: "application/vnd.kafka.json.v2+json" }, params: { offset: 0, timeout: "1000", max_bytes: "100000", }, }; axios .get(`${base_uri}/topics/test_topic/partitions/0/records`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.get( url=f"{base_uri}/topics/test_topic/partitions/0/records", params={"offset": 0, "timeout":1000,"max_bytes":100000}, headers={"Accept": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output: ```bash [{"topic":"test_topic","key":null,"value":"Redpanda","partition":0,"offset":0}] ``` ### [](#get-list-of-brokers)Get list of brokers #### curl ```bash curl "$base_uri/brokers" ``` #### NodeJS ```javascript axios .get(`${base_uri}/brokers`) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python res = requests.get(f"{base_uri}/brokers").json() pretty(res) ``` Expected output: ```bash {brokers: [0]} ``` ### [](#create-a-consumer)Create a consumer To retrieve events from a topic using consumers, you must create a consumer and a consumer group, and then subscribe the consumer instance to a topic. Each action involves a different endpoint and method. The first endpoint is: `/consumers/`. For this REST call, the payload is the group information. #### curl ```bash curl -s \ -X POST \ "$base_uri/consumers/test_group" \ -H "Content-Type: application/vnd.kafka.v2+json" \ -d '{ "format":"json", "name":"test_consumer", "auto.offset.reset":"earliest", "auto.commit.enable":"false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }' ``` #### NodeJS ```javascript let payload = { "name": "test_consumer", "format": "json", "auto.offset.reset": "earliest", "auto.commit.enable": "false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }; let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .post(`${base_uri}/consumers/test_group`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group", data=json.dumps({ "name": "test_consumer", "format": "json", "auto.offset.reset": "earliest", "auto.commit.enable": "false", "fetch.min.bytes": "1", "consumer.request.timeout.ms": "10000" }), headers={"Content-Type": "application/vnd.kafka.v2+json"}).json() pretty(res) ``` Expected output: ```bash {"instance_id":"test_consumer","base_uri":"http://127.0.0.1:8082/consumers/test_group/instances/test_consumer"} ``` > 📝 **NOTE** > > - Consumers expire after five minutes of inactivity. To prevent this from happening, try consuming events within a loop. If the consumer has expired, you can create a new one with the same name. > > - The output `base_uri` is the full URL path for this specific consumer instance and differs from the `base_uri` variable used in the code examples. ### [](#subscribe-to-the-topic)Subscribe to the topic After creating the consumer, subscribe to the topic that you created. #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X POST \ "$base_uri/consumers/test_group/instances/test_consumer/subscription"\ -H "Content-Type: application/vnd.kafka.v2+json" \ -d '{ "topics": [ "test_topic" ] }' ``` #### NodeJS ```javascript let payload = { topics: ["test_topic"]}; let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .post(`${base_uri}/consumers/test_group/instances/test_consumer/subscription`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group/instances/test_consumer/subscription", data=json.dumps({"topics": ["test_topic"]}), headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` Expected response is an HTTP 204, without a body. Now you can get the events from `test_topic`. ### [](#retrieve-events)Retrieve events Retrieve the events from the topic: #### curl ```bash curl -s \ "$base_uri/consumers/test_group/instances/test_consumer/records?timeout=1000&max_bytes=100000"\ -H "Accept: application/vnd.kafka.json.v2+json" ``` #### NodeJS ```javascript let options = { headers: { Accept: "application/vnd.kafka.json.v2+json" }, params: { timeout: "1000", max_bytes: "100000", }, }; axios .get(`${base_uri}/consumers/test_group/instances/test_consumer/records`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.get( url=f"{base_uri}/consumers/test_group/instances/test_consumer/records", params={"timeout":1000,"max_bytes":100000}, headers={"Accept": "application/vnd.kafka.json.v2+json"}).json() pretty(res) ``` Expected output: ```bash [{"topic":"test_topic","key":null,"value":"Redpanda","partition":0,"offset":0},{"topic":"test_topic","key":null,"value":"HTTP proxy","partition":1,"offset":0},{"topic":"test_topic","key":null,"value":"Test event","partition":2,"offset":0}] ``` ### [](#get-offsets-from-consumer)Get offsets from consumer #### curl ```bash curl -s \ -X 'GET' \ curl -s -o /dev/null -w "%{http_code}" \ -X 'POST' \ "$base_uri/consumers/test_group/instances/test_consumer/offsets" \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'Content-Type: application/vnd.kafka.v2+json' \ -d '{ "partitions": [ { "topic": "test_topic", "partition": 0 }, { "topic": "test_topic", "partition": 1 }, { "topic": "test_topic", "partition": 2 } ] }' ``` #### Python ```python res = requests.get( url=f"{base_uri}/consumers/test_group/instances/test_consumer/offsets", data=json.dumps( dict(partitions=[ dict(topic="test_topic", partition=p) for p in [0, 1, 2] ])), headers={"Content-Type": "application/vnd.kafka.v2+json"}).json() pretty(res) ``` Expected output: ```bash { "offsets": [{ "topic": "test_topic", "partition": 0, "offset": 0, "metadata": "" },{ "topic": "test_topic", "partition": 1, "offset": 0, "metadata": "" }, { "topic": "test_topic", "partition": 2, "offset": 0, "metadata": "" }] } ``` ### [](#commit-offsets-for-consumer)Commit offsets for consumer After events have been handled by a consumer, the offsets can be committed, so that the consumer group won’t retrieve them again. #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X 'POST' \ "$base_uri/consumers/test_group/instances/test_consumer/offsets" \ -H 'accept: application/vnd.kafka.v2+json' \ -H 'Content-Type: application/vnd.kafka.v2+json' \ -d '{ "partitions": [ { "topic": "test_topic", "partition": 0, "offset": 0 }, { "topic": "test_topic", "partition": 1, "offset": 0 }, { "topic": "test_topic", "partition": 2, "offset": 0 } ] }' ``` #### NodeJS ```javascript let options = { headers: { accept: "application/vnd.kafka.v2+json", "Content-Type": "application/vnd.kafka.v2+json", } }; let payload = { partitions: [ { topic: "test_topic", partition: 0, offset: 0 }, { topic: "test_topic", partition: 1, offset: 0 }, { topic: "test_topic", partition: 2, offset: 0 }, ]}; axios .post(`${base_uri}/consumers/test_group/instances/test_consumer/offsets`, payload, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` Run the application: ```bash node index.js ``` #### Python ```python res = requests.post( url=f"{base_uri}/consumers/test_group/instances/test_consumer/offsets", data=json.dumps( dict(partitions=[ dict(topic="test_topic", partition=p, offset=0) for p in [0, 1, 2] ])), headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` Expected output: none. ### [](#delete-a-consumer)Delete a consumer To remove a consumer from a group, send a DELETE request as shown below: #### curl ```bash curl -s -o /dev/null -w "%{http_code}" \ -X 'DELETE' \ "$base_uri/consumers/test_group/instances/test_consumer" \ -H 'Content-Type: application/vnd.kafka.v2+json' ``` #### NodeJS ```javascript let options = { headers: { "Content-Type": "application/vnd.kafka.v2+json" }}; axios .delete(`${base_uri}/consumers/test_group/instances/test_consumer`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python res = requests.delete( url=f"{base_uri}/consumers/test_group/instances/test_consumer", headers={"Content-Type": "application/vnd.kafka.v2+json"}) ``` ## [](#authenticate-with-http-proxy-2)Authenticate with HTTP Proxy HTTP Proxy supports authentication using SCRAM credentials or OIDC tokens. The authentication method depends on the [`authentication_method`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#http_proxy_auth_method) broker property and the cluster’s [`http_authentication`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#http_authentication) settings. ### [](#scram-authentication-2)SCRAM Authentication If HTTP Proxy is configured to support SASL, you can provide the SCRAM username and password as part of the Basic Authentication header in your request. For example, to list topics as an authenticated user: #### curl ```bash curl -s -u ":" ":8082/topics" ``` #### NodeJS ```javascript let options = { auth: { username: "", password: "" }, }; axios .get(`${base_uri}/topics`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python auth = ("", "") res = requests.get(f"{base_uri}/topics", auth=auth).json() pretty(res) ``` ### [](#oidc-authentication-2)OIDC Authentication If HTTP Proxy is configured to support OIDC, you can provide an OIDC token in the Authorization header. For example: #### curl ```bash curl -s -H "Authorization: Bearer " ":8082/topics" ``` #### NodeJS ```javascript let options = { headers: { Authorization: `Bearer ` }, }; axios .get(`${base_uri}/topics`, options) .then(response => console.log(response.data)) .catch(error => console.error(error)); ``` #### Python ```python headers = {"Authorization": "Bearer "} res = requests.get(f"{base_uri}/topics", headers=headers).json() pretty(res) ``` For details about configuring OIDC authentication, see [OIDC Authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/#oidc-http). ## [](#generate-a-security-report-for-http-proxy)Generate a security report for HTTP Proxy Use the [`/v1/security/report`](https://docs.redpanda.com/api/doc/admin/operation/operation-get_security_report) Admin API endpoint to generate a comprehensive security report for your cluster. This endpoint provides detailed information about TLS configuration, authentication methods, authorization status, and security alerts across all Redpanda interfaces, including HTTP Proxy. Input ```bash curl 'http://localhost:9644/v1/security/report' ``` View output ```bash { "interfaces": { "kafka": [ { "name": "test_kafka_listener", "host": "0.0.0.0", "port": 9092, "advertised_host": "0.0.0.0", "advertised_port": 9092, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_method": "None", "authorization_enabled": false } ], "rpc": { "host": "0.0.0.0", "port": 33145, "advertised_host": "127.0.0.1", "advertised_port": 33145, "tls_enabled": false, "mutual_tls_enabled": false }, "admin": [ { "name": "test_admin_listener", "host": "0.0.0.0", "port": 9644, "tls_enabled": false, "mutual_tls_enabled": false, "authentication_methods": [], "authorization_enabled": false } ] }, "alerts": [ { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_TLS", "description": "\"kafka\" interface \"test_kafka_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHN", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authentication. This is insecure and not recommended." }, { "affected_interface": "kafka", "listener_name": "test_kafka_listener", "issue": "NO_AUTHZ", "description": "\"kafka\" interface \"test_kafka_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "rpc", "issue": "NO_TLS", "description": "\"rpc\" interface is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_TLS", "description": "\"admin\" interface \"test_admin_listener\" is not using TLS. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHZ", "description": "\"admin\" interface \"test_admin_listener\" is not using authorization. This is insecure and not recommended." }, { "affected_interface": "admin", "listener_name": "test_admin_listener", "issue": "NO_AUTHN", "description": "\"admin\" interface \"test_admin_listener\" is not using authentication. This is insecure and not recommended." } ] } ``` ## [](#use-swagger-with-http-proxy)Use Swagger with HTTP Proxy You can use Swagger UI to test and interact with Redpanda HTTP Proxy endpoints. Use Docker to start Swagger UI: ```bash docker run -p 80:8080 -d swaggerapi/swagger-ui ``` Verify that the Swagger container is available: ```bash docker ps ``` Verify that the Docker container has been added and is running: `swaggerapi/swagger-ui` with `Up…` status In a browser, enter `` in the address bar to open the Swagger console. Change the URL to `[http://:8082/v1](http://\:8082/v1)`, and click `Explore` to update the page with Redpanda HTTP Proxy endpoints. You can call the endpoints in any application and language that supports web interactions. ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 78: Kafka Compatibility **URL**: https://docs.redpanda.com/streaming/current/develop/kafka-clients.md --- # Kafka Compatibility > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Kafka Compatibility latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: kafka-clients page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: kafka-clients.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/kafka-clients.adoc description: Kafka clients, version 0.11 or later, are compatible with Redpanda. Validations and exceptions are listed. page-topic-type: reference personas: developer learning-objective-1: Identify which Kafka clients are validated with Redpanda learning-objective-2: Identify Kafka compatibility exceptions when integrating with Redpanda page-git-created-date: "2023-05-30" page-git-modified-date: "2026-08-04" support-status: supported --- Apache Kafka® clients developed for Kafka protocol version 0.11 or later work with Redpanda with minimal or no changes to your application. This page identifies which clients are validated and calls out any exceptions. Use this reference to: - Identify which Kafka clients are validated with Redpanda - Identify Kafka compatibility exceptions when integrating with Redpanda ## [](#kafka-client-compatibility)Kafka client compatibility Redpanda validates the Apache Kafka Java client and a set of widely used non-Java clients, at their current versions that support Kafka 4.x, using the ducktape and chaos test suites. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations, at current client versions. Modern clients auto-negotiate protocol versions or use an earlier protocol version accepted by Redpanda brokers. > 💡 **TIP** > > Always use the latest supported version of a Kafka client. The following clients have been validated with Redpanda. | Language | Client | | --- | --- | | Java | Apache Kafka Java Client | | C/C++ | librdkafka | | Go | franz-goconfluent-kafka-goSarama | | Python | kafka-pythonconfluent-kafka-python | | Rust | kafka-rust | | Node.js | KafkaJSconfluent-kafka-javascript | Clients that have not been validated by Redpanda Data, but use the Kafka protocol, remain compatible with Redpanda subject to the limitations in the next section (particularly those based on librdkafka, such as confluent-kafka-dotnet). If you find a client that does not work with Redpanda, reach out in the [Redpanda community Slack](https://redpanda.com/slack). ## [](#compatibility-exceptions)Compatibility exceptions Redpanda is compatible with the Kafka protocol, with the following exceptions: - Multiple SCRAM mechanisms simultaneously for SASL users are not supported. For example, a user cannot have both a `SCRAM-SHA-256` and a `SCRAM-SHA-512` credential. Redpanda supports only one SASL/SCRAM mechanism per user: either `SCRAM-SHA-256` or `SCRAM-SHA-512`. For details, see [Configure SASL authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/#sasl). - HTTP Proxy (`pandaproxy`): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions. - The Kafka request rate quota (`request_percentage`), which limits the share of broker request-handling capacity a client can consume, is not supported. Redpanda supports byte-rate (`producer_byte_rate`, `consumer_byte_rate`) and topic-mutation (`controller_mutation_rate`) quotas, which you can apply [per user, per client, or per client group](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/#client-throughput-limits). - [KIP-890](https://cwiki.apache.org/confluence/display/KAFKA/KIP-890) (Transactions Server-Side Defense): Redpanda does not implement the server-side portion of KIP-890, which addresses transaction errors specific to Kafka’s replication model. Redpanda’s implementation of transactions is not susceptible to this class of errors. When connecting to Redpanda, Kafka 4.x clients detect that Transactions V2 is unsupported and fall back to the original transaction protocol (per-transaction epoch bumping is part of V2 and does not apply). If you find an unsupported feature or incompatibility, [file an issue](https://github.com/redpanda-data/redpanda/issues/new) with the Redpanda team. --- # Page 79: Topics **URL**: https://docs.redpanda.com/streaming/current/develop/manage-topics.md --- # Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: manage-topics/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: manage-topics/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/manage-topics/index.adoc description: Learn how to manage topics in Redpanda, including creation, configuration, and advanced features. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" support-status: supported --- - [Manage Topics](config-topics/) Learn how to create topics, update topic configurations, and delete topics or records. - [Manage Cloud Topics](cloud-topics/) Cloud Topics are "diskless" Redpanda topics that enable you to store data directly to object storage to trade off latency for lower costs. --- # Page 80: Manage Cloud Topics **URL**: https://docs.redpanda.com/streaming/current/develop/manage-topics/cloud-topics.md --- # Manage Cloud Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Cloud Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: manage-topics/cloud-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: manage-topics/cloud-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/manage-topics/cloud-topics.adoc description: Cloud Topics are "diskless" Redpanda topics that enable you to store data directly to object storage to trade off latency for lower costs. page-topic-type: how-to personas: streaming_developer, platform_admin learning-objective-1: Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics learning-objective-2: Create a Cloud Topic using rpk on a cluster that has cloud storage enabled learning-objective-3: Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs page-git-created-date: "2026-03-31" page-git-modified-date: "2026-07-28" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Starting in v26.1, Redpanda provides [Cloud Topics](https://docs.redpanda.com/streaming/current/reference/glossary/#cloud-topic) to support multi-modal streaming workloads in the most cost-effective way possible: as a per-topic configuration running mixed latency workloads. While standard Redpanda [topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/) that use local storage or Tiered Storage are ideal for latency-sensitive workloads (for example, for audit logs or analytics), Cloud Topics are optimized for latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major consideration that can become the dominant cost driver at high throughput. These workloads can include observability streams, offline analytics, AI/ML model training data feeds, or development environments that have flexible latency requirements. Instead of replicating every byte across expensive network links, Cloud Topics leverage durable, inexpensive cloud storage (S3, ADLS, GCS, MinIO) as the primary mechanism to both replicate data and serve it to consumers. This eliminates over 90% of the cost of replicating data over network links in multi-AZ clusters. The end-to-end latency experienced when using Cloud Topics can range from 500 ms to as high as a few seconds with different object stores. Lower latencies may be achievable in certain environments, but Cloud Topics is optimized for throughput rather than low latency or tightly constrained tail latency. This latency profile is often acceptable for many streaming workloads, and can unlock new streaming use cases that previously were not cost effective. With Cloud Topics, data from the client is not acknowledged until it is uploaded to object storage. This maintains durability in the face of infrastructure failures, but results in an increase in both produce latency and end-to-end latency, driven by both batching of produced data and the inherent latency of the underlying object store. You should generally expect end-to-end latencies of 1-2 seconds with public cloud stores. After reading this page, you will be able to: - Describe the latency and cost trade-offs of Cloud Topics compared to standard Redpanda topics - Create a Cloud Topic using rpk on a cluster that has cloud storage enabled - Identify Cloud Topics limitations and configurations that reduce cross-AZ networking costs ## [](#prerequisites)Prerequisites - [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) v26.1 or later. - [Enable cloud storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#set-up-tiered-storage) on your Redpanda cluster. > 📝 **NOTE** > > If you plan to use Cloud Topics for all new topics in a Redpanda cluster, be sure to set the following cluster-level property: > > ```bash > default_redpanda_storage_mode=cloud > ``` > > This ensures that newly-created Redpanda topics are Cloud Topics by default. For details, see [Enable Tiered Storage for a cluster](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage-for-a-cluster). - [Configure object storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage). - Ensure that you have an Enterprise license. To check your license status, run: ```bash rpk cluster license info ``` ## [](#limitations)Limitations - In Redpanda versions earlier than v26.2, shadow links do not support Cloud Topics. - Once created, a Cloud Topic cannot be converted back to a standard Redpanda topic that uses local storage or Tiered Storage v1. Conversely, existing topics created as local or Tiered Storage v1 topics cannot be converted to Cloud Topics. Starting in Redpanda v26.2, Cloud Topics can be converted to and from Tiered Storage v2 topics. ## [](#create-cloud-topics)Create Cloud Topics Cloud Topics don’t require a separate cluster property to enable them. When cloud storage is enabled for your cluster, you can create Cloud Topics directly. > 📝 **NOTE** > > In Redpanda versions earlier than v26.2, you must also set the `cloud_topics_enabled` cluster property to `true` and restart the cluster. This property is deprecated in v26.2 and later. For details, see [Deprecated features](https://docs.redpanda.com/streaming/current/upgrade/deprecated/). To create a Cloud Topic, set the topic property `redpanda.storage.mode` to `cloud`: ```bash rpk topic create -c redpanda.storage.mode=cloud ``` ```console TOPIC STATUS audit.analytics.may2025 OK ``` You can make a topic a Cloud Topic only at topic creation time. In addition to replication, cross-AZ ingress (producer) and egress (consumer) traffic can also contribute substantially to cloud networking costs. When running multi-AZ clusters in general, Redpanda strongly recommends using [Follower Fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/), which allows consumers to avoid crossing network zones. When possible, you can use [leader pinning](https://docs.redpanda.com/streaming/current/develop/produce-data/leader-pinning/), which positions a topic’s partition leader close to the producers, providing a similar benefit for ingress traffic. These features can add additional savings to the replication cost savings of Cloud Topics. For client-side tuning guidance, see [Configure producers for Cloud Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/configure-producers-for-cloud-topics/). --- # Page 81: Manage Topics **URL**: https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics.md --- # Manage Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: manage-topics/config-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: manage-topics/config-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/manage-topics/config-topics.adoc description: Learn how to create topics, update topic configurations, and delete topics or records. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-08-12" support-status: supported --- Topics provide a way to organize events in a data streaming platform. When you create a topic, the default cluster-level topic configurations are applied using the cluster configuration file, unless you specify different configurations. The following table shows the default cluster-level properties and their equivalent topic-level properties: | Cluster property | Default | Topic property | | --- | --- | --- | | log_cleanup_policy | delete | cleanup.policy | | retention_bytes | null (no limit) | retention.bytes | | log_retention_ms | 604800000 ms (1 week) | retention.ms | | log_segment_ms | null (no limit) | segment.ms | | log_segment_size | 134217728 bytes (128 MiB) | segment.bytes | | log_compression_type | producer | compression.type | | log_message_timestamp_type | CreateTime | message.timestamp.type | | kafka_batch_max_bytes | 1048576 bytes (1 MiB) | max.message.bytes | | write_caching_default | false | write.caching | These default settings are best suited to a one-broker cluster in a development environment. To learn how to modify the default cluster-wide configurations, see [Configure Cluster Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/). Even if you set default values that work for most topics, you may still want to change some properties for a specific topic. > 📝 **NOTE** > > For details about topic properties, see [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/). ## [](#create-a-topic)Create a topic Creating a topic can be as simple as specifying a name for your topic on the command line. For example, to create a topic named `xyz`, run: ```bash rpk topic create xyz ``` This command creates a topic named `xyz` with one partition and one replica, because these are the default values set in the cluster configuration file. Replicas are copies of partitions that are distributed across different brokers, so if one broker goes down, other brokers still have a copy of the data. ### [](#choose-the-number-of-partitions)Choose the number of partitions A partition acts as a log file where topic data is written. Dividing topics into partitions allows producers to write messages in parallel and consumers to read messages in parallel. The higher the number of partitions, the greater the throughput. > 💡 **TIP** > > As a general rule, select a number of partitions that corresponds to the maximum number of consumers in any consumer group that will consume the data. For example, suppose you plan to create a consumer group with 10 consumers. To create topic `xyz` with 10 partitions, run: ```bash rpk topic create xyz -p 10 ``` ### [](#choose-the-replication-factor)Choose the replication factor The default replication factor in the cluster configuration is set to 1. By choosing a replication factor greater than 1, you ensure that each partition has a copy of its data on at least one other broker. One replica acts as the leader, and the other replicas are followers. To specify a replication factor of 3 for topic `xyz`, run: ```bash rpk topic create xyz -r 3 ``` > 📝 **NOTE** > > The replication factor must be an odd number. Redpanda Data recommends a replication factor of 3 for most use cases. Administrators may set a minimum required replication factor for any new topic in the cluster through the cluster-level [`minimum_topic_replications`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#minimum_topic_replications) property. > 💡 **TIP** > > If you enable [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) on a topic, you can then use [topic recovery](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/topic-recovery/) to restore data for a deleted topic. ### [](#choose-a-storage-mode)Choose a storage mode Starting in Redpanda v26.1, you can set the `redpanda.storage.mode` topic property to control how a topic stores data: | Value | Behavior | | --- | --- | | unset (default) | Legacy behavior. Tiered Storage is controlled by the redpanda.remote.read and redpanda.remote.write topic properties and their cluster-level defaults (cloud_storage_enable_remote_read and cloud_storage_enable_remote_write). | | local | Topic data is stored only on the broker’s local disk. Object storage upload is disabled for the topic, regardless of redpanda.remote.read and redpanda.remote.write values. | | tiered | Data is stored on local disk and uploaded to object storage. Enables Tiered Storage for the topic regardless of redpanda.remote.read and redpanda.remote.write values. Starting in Redpanda v26.2, Tiered Storage is available in two versions (tiered_v1 and tiered_v2). See Choose a Tiered Storage version. | | cloud | Data is stored durably in object storage using the Cloud Topics architecture. Local storage is used only as a write buffer. See Manage Cloud Topics. | To set the storage mode at topic creation time: ```bash rpk topic create -c redpanda.storage.mode=tiered ``` When `redpanda.storage.mode` is set to `local`, `tiered`, or `cloud`, the `redpanda.remote.read` and `redpanda.remote.write` topic properties have no effect on the topic. Starting in Redpanda v26.2, the `tiered` value is an alias that resolves to one of two Tiered Storage versions when the topic is created. The [`default_redpanda_storage_mode_tiered_impl`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#default_redpanda_storage_mode_tiered_impl) cluster property selects the version (`tiered_v1` by default). To select the version explicitly for a single topic, set both `redpanda.storage.mode` and `redpanda.storage.mode.impl` when you create the topic: ```bash rpk topic create -c redpanda.storage.mode=tiered -c redpanda.storage.mode.impl=tiered_v2 ``` The `redpanda.storage.mode.impl` property reports the exact implementation of every topic’s storage mode and is read-only after topic creation. For details, see [Choose a Tiered Storage version](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#tiered-storage-versions). To apply a default storage mode to all new topics in a cluster, set the `default_redpanda_storage_mode` cluster property: ```bash rpk cluster config set default_redpanda_storage_mode=tiered ``` To set `local` as the default storage mode for all new topics in a cluster: ```bash rpk cluster config set default_redpanda_storage_mode=local ``` If `default_redpanda_storage_mode` is not configured (the default), new topics use `unset` mode and Tiered Storage behavior is inherited from the cluster-level `cloud_storage_enable_remote_write` and `cloud_storage_enable_remote_read` properties. ## [](#update-topic-configurations)Update topic configurations After you create a topic, you can update the topic property settings for all new data written to it. For example, you can add partitions or change the cleanup policy. ### [](#add-partitions)Add partitions You can assign a certain number of partitions when you create a topic, and add partitions later. For example, suppose you add brokers to your cluster, and you want to take advantage of the additional processing power. To increase the number of partitions for existing topics, run: ```bash rpk topic add-partitions [TOPICS...] --num [#] ``` Note that `--num <#>` is the number of partitions to _add_, not the total number of partitions. > 📝 **NOTE** > > If a topic already has messages and you add partitions, the existing messages won’t be redistributed to the new partitions. If you require messages to be redistributed, then you must create a new topic with the new partition count, then stream the messages from the old topic to the new topic so they are appropriately distributed according to the new partition hashing. ### [](#reduce-the-number-of-partitions)Reduce the number of partitions You cannot reduce the number of partitions on an existing topic. The Kafka API does not support it: a record’s partition is chosen when the record is produced, and records that have already been written stay in the partition where they landed, so removing a partition would orphan its data. Redpanda assigns a record that has a key to a partition by hashing the key, and leaves a record without a key to the producer’s partitioner, which usually spreads such records across all available partitions. To move a topic’s data to fewer partitions, copy it to a new topic and switch your clients over. Before you start, check that your applications can tolerate the following: - **Duplicates**: Redpanda Connect delivers records at least once, so a restart or a retry during the copy can write the same record to the new topic twice. A lag of zero shows only how far the copy has committed, not that the new topic is free of duplicates. Either make your consumers idempotent, or deduplicate on a record ID after the copy. - **Ordering**: records keep their keys, so all records for a key still land on one partition and keep their order relative to each other, but the global order of records across partitions is not preserved. - **Retention**: the copy starts at the oldest record that is still retained. Records that retention or compaction has already removed cannot be copied, and both keep running during the copy, so complete the copy well within the topic’s retention period. - **Consumer offsets**: consumer group offsets are stored per topic, so the offsets your consumers committed on the original topic do not carry over. Each consumer starts from the beginning of the new topic and replays what the copy wrote, unless you set its offsets explicitly with `rpk group seek`. - **Write downtime**: producers must stop writing to the original topic before you switch clients over, so plan a window in which the topic accepts no writes. The following procedure reduces a topic named `orders` from three partitions to one. 1. Check the configuration of the original topic so that you can recreate it. Note the replication factor, and every row whose `SOURCE` is `DYNAMIC_TOPIC_CONFIG`, which is an override you must set on the new topic: ```bash rpk topic describe orders ``` Example output (abbreviated) ```bash SUMMARY ======= NAME orders PARTITIONS 3 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DYNAMIC_TOPIC_CONFIG segment.bytes 134217728 DEFAULT_CONFIG ``` Here, only `retention.ms` is an override. If the topic has Tiered Storage settings, a custom cleanup policy, or other overrides, carry all of them over: a new topic created without them silently falls back to the cluster defaults. 2. Create the new topic with the target number of partitions, the replication factor of the original topic, and each override from the previous step: ```bash rpk topic create orders-reduced --partitions 1 --replicas 1 --topic-config retention.ms=604800000 ``` Example output ```bash TOPIC STATUS orders-reduced OK ``` 3. Copy the data with a [Redpanda Connect](https://docs.redpanda.com/connect/get-started/about/) pipeline. This configuration reads all records that are still available in the original topic and preserves record keys, so records for the same key land on the same partition of the new topic: `reduce-partitions.yaml` ```yaml input: redpanda: seed_brokers: [""] topics: ["orders"] consumer_group: orders-to-orders-reduced start_offset: earliest output: redpanda: seed_brokers: [""] topic: orders-reduced key: ${! @kafka_key } ``` Give the consumer group a name that is unique to this copy, such as `-to-`. `start_offset: earliest` applies only when the group has no committed offset, so a group name that has been used before resumes from where it left off and skips records. ```bash rpk connect run reduce-partitions.yaml ``` Example output ```bash level=info msg="Launching a Redpanda Connect instance, use CTRL+C to close" level=info msg="Output type redpanda is now active" level=info msg="Input type redpanda is now active" ``` 4. Stop the producers that write to the original topic. Leave the pipeline running so that it copies the last records they wrote. 5. Wait for the copy to drain. It is complete when the consumer group reports a `LAG` of `0` for every partition of the original topic: ```bash rpk group describe orders-to-orders-reduced ``` Example output ```bash GROUP orders-to-orders-reduced COORDINATOR-NODE 0 COORDINATOR-PARTITION __consumer_offsets/0 STATE Stable BALANCER cooperative-sticky MEMBERS 1 TOTAL-LAG 0 TOPIC PARTITION CURRENT-OFFSET LOG-START-OFFSET LOG-END-OFFSET LAG MEMBER-ID CLIENT-ID HOST orders 0 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 orders 1 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 orders 2 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 ``` 6. Compare the record counts of the two topics. For each topic, the number of available records is the sum of `HIGH-WATERMARK` minus `LOG-START-OFFSET` across its partitions. A higher count on the new topic means the copy wrote duplicates: ```bash rpk topic describe orders -p rpk topic describe orders-reduced -p ``` Example output ```bash PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 0 1 [0] 0 3 1 0 1 [0] 0 3 2 0 1 [0] 0 3 PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 0 1 [0] 0 9 ``` Nine records across the three original partitions, and the same nine on the single partition of the new topic. 7. Point your producers and consumers at the new topic. Consumers start from the beginning of the new topic unless you set their offsets with `rpk group seek`. 8. Stop the pipeline with Ctrl+C. 9. When you no longer need the original topic, delete it to reclaim storage. See [Delete a topic](#delete-a-topic). > ⚠️ **CAUTION** > > Do not delete the original topic until the new topic holds the data you expect and your consumers are running against it. Deleting a topic deletes its data. ### [](#change-the-replication-factor)Change the replication factor Suppose you create a topic with the default replication factor of 1 (which is specified in the cluster properties configuration file). Now you want to change the replication factor to 3, so you can have two backups of topic data in case a broker goes down. To set the replication factor to 3, run: ```bash rpk topic alter-config [TOPICS...] --set replication.factor=3 ``` > 📝 **NOTE** > > The replication factor can’t exceed the number of Redpanda brokers. If you try to set a replication factor greater than the number of brokers, the request is rejected. ### [](#change-the-storage-mode)Change the storage mode You can change a topic’s `redpanda.storage.mode` after creation, with the following restrictions: | From | To | Permitted | Notes | | --- | --- | --- | --- | | local | tiered (v1) | Yes | Enables Tiered Storage for the topic. Object storage must be configured. | | tiered (v1) | local | With caution | Disables object storage uploads. Redpanda strongly recommends against repeatedly toggling this setting, as it can result in data gaps in Tiered Storage. | | cloud | tiered (v2) | Yes | Starting in Redpanda v26.2. All brokers must run v26.2 or later. | | tiered (v2) | cloud | Yes | Starting in Redpanda v26.2. | | tiered (v1) | tiered (v2) | No | Topics cannot be converted between Tiered Storage versions, in either direction. | | local | tiered (v2) | No | | | Any other mode | cloud | No | Cloud Topics can otherwise only be set at topic creation time. | | cloud | local or tiered (v1) | No | A Cloud Topic cannot be converted to a local or Tiered Storage v1 topic. | When you change the storage mode, you always set the `redpanda.storage.mode` property. The `tiered` value resolves to the version selected by the [`default_redpanda_storage_mode_tiered_impl`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#default_redpanda_storage_mode_tiered_impl) cluster property, so, for example, converting a `cloud` topic to Tiered Storage v2 requires `default_redpanda_storage_mode_tiered_impl=tiered_v2`. The `redpanda.storage.mode.impl` property cannot be changed after topic creation. See [Choose a Tiered Storage version](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#tiered-storage-versions). For example, to transition an existing local topic to Tiered Storage: ```bash rpk topic alter-config --set redpanda.storage.mode=tiered ``` > 📝 **NOTE** > > This example requires [`default_redpanda_storage_mode_tiered_impl`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#default_redpanda_storage_mode_tiered_impl) to be `tiered_v1` (the default). If the cluster default is `tiered_v2`, the request is rejected, because a `local` topic cannot be converted to Tiered Storage v2. ### [](#change-the-cleanup-policy)Change the cleanup policy The cleanup policy determines how to clean up the partition log files when they reach a certain size: - `delete` deletes data based on age or log size. Topics retain all records until then. - `compact` compacts the data by only keeping the latest values for each KEY. - `compact,delete` combines both methods. Unlike compacted topics, which keep only the most recent message for a given key, topics configured with a `delete` cleanup policy provide a running history of all changes for those topics. > ⚠️ **WARNING** > > All topic properties take effect immediately after being set. Do not modify properties on internal Redpanda topics (such as `__consumer_offsets`, `_schemas`, or other system topics) as this can cause cluster instability. For example, to change a topic’s policy to `compact`, run: ```bash rpk topic alter-config [TOPICS…] —-set cleanup.policy=compact ``` For details on compaction in Redpanda, see [Compaction settings](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings/). ### [](#configure-write-caching)Configure write caching Write caching is a relaxed mode of [`acks=all`](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/#acksall) that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. Write caching applies to user topics. It does not apply to transactions or consumer offsets: data written in the context of a transaction and consumer offset commits is always written to disk and fsynced before being acknowledged to the client. > 📝 **NOTE** > > For clusters in [development mode](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda-mode/#development-mode), write caching is enabled by default. For clusters in production mode, it is disabled by default. Only enable write caching on workloads that can tolerate some data loss in the case of multiple, simultaneous broker failures. Leaving write caching disabled safeguards your data against complete data center or availability zone failures. #### [](#configure-at-cluster-level)Configure at cluster level To enable write caching by default in all user topics, set the cluster-level property [`write_caching_default`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#write_caching_default): `rpk cluster config set write_caching_default=true` With `write_caching_default` set to true at the cluster level, Redpanda fsyncs to disk according to [`raft_replica_max_pending_flush_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_replica_max_pending_flush_bytes) and [`raft_replica_max_flush_delay_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_replica_max_flush_delay_ms), whichever is reached first. #### [](#configure-at-topic-level)Configure at topic level To override the cluster-level setting at the topic level, set the topic-level property `write.caching`: `rpk topic alter-config my_topic --set write.caching=true` With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. ### [](#remove-a-configuration-setting)Remove a configuration setting You can remove a configuration that overrides the default setting, and the setting will use the default value again. For example, suppose you altered the cleanup policy to use `compact` instead of the default, `delete`. Now you want to return the policy setting to the default. To remove the configuration setting `cleanup.policy=compact`, run `rpk topic alter-config` with the `--delete` flag: ```bash rpk topic alter-config [TOPICS...] --delete cleanup.policy ``` ## [](#list-topic-configuration-settings)List topic configuration settings To display all the configuration settings for a topic, run: ```bash rpk topic describe -c ``` The `-c` flag limits the command output to just the topic configurations. This command is useful for checking the default configuration settings before you make any changes and for verifying changes after you make them. The following command output displays after running `rpk topic describe test-topic`, where `test-topic` was created with default settings: ```bash rpk topic describe test_topic SUMMARY ======= NAME test_topic PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DYNAMIC_TOPIC_CONFIG compression.type producer DEFAULT_CONFIG max.message.bytes 1048576 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.datapolicy function_name: script_name: DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG redpanda.storage.mode unset DEFAULT_CONFIG redpanda.storage.mode.impl unset DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 1073741824 DEFAULT_CONFIG ``` Suppose you add two partitions, and increase the number of replicas to 3. The new command output confirms the changes in the `SUMMARY` section: SUMMARY ======= NAME test\_topic PARTITIONS 3 REPLICAS 3 ## [](#delete-a-topic)Delete a topic To delete a topic, run: ```bash rpk topic delete ``` When a topic is deleted, its underlying data is deleted, too. To delete multiple topics at a time, provide a space-separated list. For example, to delete two topics named `topic1` and `topic2`, run: ```bash rpk topic delete topic1 topic2 ``` You can also use the `-r` flag to specify one or more regular expressions; then, any topic names that match the pattern you specify are deleted. For example, to delete topics with names that start with “f” and end with “r”, run: ```bash rpk topic delete -r '^f.*' '.*r$' ``` Note that the first regular expression must start with the `^` symbol, and the last expression must end with the `$` symbol. This requirement helps prevent accidental deletions. ## [](#delete-records-from-a-topic)Delete records from a topic Redpanda allows you to delete data from the beginning of a partition up to a specific offset (a monotonically increasing sequence number for records in a partition). Deleting records frees up disk space, which is especially helpful if your producers are pushing more data than anticipated in your retention plan. Delete records when you know that all consumers have read up to that given offset, and the data is no longer needed. There are different ways to delete records from a topic, including using the [`rpk topic trim-prefix`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-trim-prefix/) command, using the `DeleteRecords` Kafka API with Kafka clients, or using Redpanda Console. > 📝 **NOTE** > > - To delete records, `cleanup.policy` must be set to `delete` or `compact,delete`. > > - Object storage is deleted asynchronously. After messages are deleted, the partition’s start offset will have advanced, but garbage collection of deleted segments may not be complete. > > - Similar to Kafka, after deleting records, local storage and object storage may still contain data for deleted offsets. (Redpanda does not truncate segments. Instead, it bumps the start offset, then it attempts to delete as many whole segments as possible.) Data before the new start offset is not visible to clients but could be read by someone with access to the local disk of a Redpanda node. > ⚠️ **WARNING** > > When you delete records from a topic with a timestamp, Redpanda advances the partition start offset to the first record whose timestamp is after the threshold. If record timestamps are not in order with respect to offsets, this may result in unintended deletion of data. Before using a timestamp, verify that timestamps increase in the same order as offsets in the topic to avoid accidental data loss. For example: > > ```bash > rpk topic consume -n 50 --format '%o %d{go[2006-01-02T15:04:05Z07:00]} %k %v' > ``` ## [](#next-steps)Next steps [Configure Producers](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/) ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 82: Configure Producers for Cloud Topics **URL**: https://docs.redpanda.com/streaming/current/develop/manage-topics/configure-producers-for-cloud-topics.md --- # Configure Producers for Cloud Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Producers for Cloud Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: manage-topics/configure-producers-for-cloud-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: manage-topics/configure-producers-for-cloud-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/manage-topics/configure-producers-for-cloud-topics.adoc description: Learn about producer configuration considerations for Cloud Topics. page-topic-type: best-practices, how-to personas: streaming_developer, platform_admin learning-objective-1: Apply producer configuration settings to maximize throughput for Cloud Topics page-git-created-date: "2026-05-18" page-git-modified-date: "2026-05-18" support-status: supported --- This page describes how to tune the client producer for Cloud Topics (note that general producer configuration guidance still applies). See [Configure Producers](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/). With idempotency enabled, Kafka’s protocol allows only five in-flight requests at a time on a single broker connection. This imposes a tight limit on how much data can be in flight at a single time. Over high-latency links, this is already a problem for standard topics. Cloud Topics use a 250ms batching interval at the broker to reduce cloud storage costs, which effectively makes every producer connection a high-latency link. Compared to standard topics, this changes producer tuning priorities. Configuring batch size, linger time, and request size correctly is critical for achieving good single-producer throughput. This section covers the key settings and example values for the most common client libraries. Calculate the maximum throughput for a single producer as follows: ```text broker_count * max_in_flight * request_size / latency_seconds ``` Throughput formula definitions: - `broker_count`: The number of brokers in the cluster. - `max_in_flight`: The maximum number of in-flight requests on a single connection. With an idempotent producer, this is usually five. - `latency_seconds`: The latency of a single produce request. Assuming normal operation, this can be equated to the 250ms as earlier. In practice, it is typically lower, as the timer starts with the first byte arriving. - `request_size`: The size of a single produce request. This is the primary tuning lever because the other factors are effectively constants. Another way to increase throughput in the system is to increase producer count, as doing so naturally increases available parallelism in the system as more connections are opened to the brokers. ## [](#producer-settings)Producer settings Most Kafka client libraries offer an assortment of tunables, with many properties impacting request size. The most important ones are described here, along with an example for the Kafka Java client, along with caveats from other libraries. ### [](#batches-and-requests)Batches and requests A Kafka produce request consists of one or more batches. A batch contains multiple records. Records mostly correspond to application-level messages as passed to the Kafka client API. At the broker level, the unit that matters is the batch, as everything happens at the batch level. Hence, creating the largest possible batches is the most important factor for performance in general, and equally applies for Cloud Topics. A request can contain multiple batches, this can sometimes alleviate the need for massive batches. However, this requires that a producer is producing to multiple partitions and that there are enough partitions per broker to fill the request with batches. As explained in [Java client](#java-client), there are also exceptions to this in some client libraries, so you should not blindly rely on them. ### [](#java-client)Java client For the Java client, Redpanda Data recommends the following as minimal settings: | Setting | Recommended value | | --- | --- | | linger.ms | 10ms | | batch.size | 131072 | | max.request.size | 1048576 (default) | As shown in the preceding formula, with a basic three-broker setup and enough partitions, you can achieve a maximum throughput of ~65 MB/s per producer. If more throughput is needed on a single producer, increase batch and max request size. Setting `linger.ms` to a higher value is recommended, but it’s less critical for Cloud Topics because as soon as there are five requests in flight, messages are force-batched even after crossing the `linger.ms` threshold. ### [](#librdkafka)librdkafka librdkafka is a commonly-used Kafka C library. However, it’s also the backing library for many other Kafka clients like confluent-kafka-python. librdkafka only allows a single batch in a produce request, unlike most other Kafka client libraries. This significantly cuts down how much data is packed into a single request. Thus, it’s important to increase `batch.size` to even higher values, as it is effectively the limiting factor. librdkafka has a default of 1MB, which typically allows for decent throughput. ### [](#idempotency-considerations)idempotency considerations Disabling idempotency allows you to avoid the in-flight limitation and vastly increases the number of concurrent in-flight requests and throughput. However, running without idempotency can result in duplicate and out-of-order messages, which for most applications is a problem, and is not recommended. In cases where message requirements are fairly lax, it can be a viable alternative. ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 83: Produce Data **URL**: https://docs.redpanda.com/streaming/current/develop/produce-data.md --- # Produce Data > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Produce Data latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: produce-data/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: produce-data/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/produce-data/index.adoc description: Learn how to configure producers and idempotent producers. page-git-created-date: "2023-05-30" page-git-modified-date: "2024-07-24" support-status: supported --- - [Configure Producers](configure-producers/) Learn about configuration options for producers, including write caching and acknowledgment settings. - [Idempotent producers](idempotent-producers/) Idempotent producers assign a unique ID to every write request, guaranteeing that each message is recorded only once in the order in which it was sent. - [Configure Leader Pinning](leader-pinning/) Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. --- # Page 84: Configure Producers **URL**: https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers.md --- # Configure Producers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Producers latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: produce-data/configure-producers page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: produce-data/configure-producers.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/produce-data/configure-producers.adoc description: Learn about configuration options for producers, including write caching and acknowledgment settings. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-03-31" support-status: supported --- Producers are client applications that write data to Redpanda in the form of events. Producers communicate with Redpanda through the Kafka API. When a producer publishes a message to a Redpanda cluster, it sends it to a specific partition. Every event consists of a key and value. When selecting which partition to produce to, if the key is blank, then the producer publishes in a round-robin fashion between the topic’s partitions. If a key is provided, then the partition hashes the key using the murmur2 algorithm and modulates across the number of partitions. ## [](#producer-acknowledgment-settings)Producer acknowledgment settings The `acks` property sets the number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. Redpanda guarantees data safety with fsync, which means flushing to disk. - With `acks=all`, every write is fsynced by default. - With other `acks` settings, or with `write_caching_default=true` at the cluster level, Redpanda fsyncs to disk according to `raft_replica_max_pending_flush_bytes` and `raft_replica_max_flush_delay_ms`, whichever is reached first. - With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first. ### [](#acks0)`acks=0` The producer doesn’t wait for acknowledgments from the leader and doesn’t retry sending messages. This increases throughput and lowers latency of the system at the expense of durability and data loss. This option allows a producer to immediately consider a message acknowledged when it is sent to the Redpanda broker. This means that a producer does not have to wait for any response from the Redpanda broker. This is the least safe option, because a leader-broker crash can cause data loss if the data has not yet replicated to the other brokers in the replica set. However, this setting is useful when you want to optimize for the highest throughput and are willing to risk some data loss. Because of the lack of guarantees, this setting is the most network bandwidth-efficient. This is helpful for use cases like IoT/sensor data collection, where updates are periodic or stateless and you can afford some degree of data loss, but you want to gather as much data as possible in a given time interval. ### [](#acks1)`acks=1` The producer waits for an acknowledgment from the leader, but it doesn’t wait for the leader to get acknowledgments from followers. This setting doesn’t prioritize throughput, latency, or durability. Instead, `acks=1` attempts to provide a balance between all of them. Replication is not guaranteed with this setting because it happens in the background, after the leader broker sends an acknowledgment to the producer. This setting could result in data loss if the leader broker crashes before any followers manage to replicate the message or if a majority of replicas go down at the same time before fsyncing the message to the disk. ### [](#acksall)`acks=all` The producer receives an acknowledgment after the majority of (implicitly, all) replicas acknowledge the message. Redpanda guarantees data safety by fsyncing every message to disk before acknowledgement back to clients. This increases durability at the expense of lower throughput and increased latency. Sometimes referred to as `acks = -1`, this option instructs the broker that replication is considered complete when the message has been replicated (and fsynced) to the majority of the brokers responsible for the partition in the cluster. As soon as the fsync call is complete, the message is considered acknowledged and is made visible to readers. > 📝 **NOTE** > > This property has an important distinction compared to Kafka’s behavior. In Kafka, a message is considered acknowledged without the requirement that it has been fsynced. Messages that have not been fsynced to disk may be lost in the event of a broker crash. So when using `acks=all`, the Redpanda default configuration is more resilient than Kafka’s. You can also consider using [write caching](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/#configure-write-caching), which is a relaxed mode of `acks=all` that acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. ### [](#retries)`retries` This property controls the number of times a message is re-sent to the broker if the broker fails to acknowledge it. This is essentially the same as if the client application resends the erroneous message after receiving an error response. The default value of `retries` in most client libraries is 0. This means that if the send fails, the message is not re-sent at all. If you increase this to a higher value, check the `max.in.flight.requests.per.connection` value as well, because leaving that property at its default value can potentially cause ordering issues in the target topic where the messages arrive. This occurs if two batches are sent to a single partition and the first fails and is retired, but the second succeeds so the records in the second batch may appear first. ### [](#max-in-flight-requests-per-connection)`max.in.flight.requests.per.connection` This property controls how many unacknowledged messages can be sent to the broker simultaneously at any given time. The default value is 5 in most client libraries. If you set this to 1, then the producer does not send any more messages until the previous one is either acknowledged or an error happens, which can prompt a retry. If you set this to a value higher than 1, then the producer sends more messages at the same time, which can help increase throughput but adds a risk of message reordering if retries are enabled. When you configure the producer to be [idempotent](https://docs.redpanda.com/streaming/current/develop/produce-data/idempotent-producers/), up to five requests can be guaranteed to be in flight with the order preserved. ### [](#enable-idempotence)`enable.idempotence` To enable idempotence, set `enable.idempotence` to `true` (the default) in your Redpanda configuration. When idempotence is enabled, the producer ensures that exactly one copy of every message is written to the broker. When set to `false`, the producer retries sending a message for any reason (such as transient errors like brokers not being available or not enough replicas exception), and it can lead to duplicates. In most client libraries `enable.idempotence` is set to true by default. Internally, this is implemented using a special identifier that is assigned to every producer (the producer ID or PID). This ID, along with a sequence number, is included in every message sent to the broker. The broker checks if the PID/sequence number combination is larger than the previous one and, if not, it discards the message. To guarantee true idempotent behavior, you must also set `acks=all` to ensure that all brokers record messages in order, even in the event of node failures. In this configuration, both the producer and the broker prefer safety and durability over throughput. Idempotence is only guaranteed within a session. A session starts after the producer is instantiated and a connection is established between the client and the Redpanda broker. When the connection is closed, the session ends. If your application code retries a request, the producer client assigns a new ID to that request, which may lead to duplicate messages. ## [](#message-batching)Message batching Batching is an efficient way to save on both network bandwidth and disk size, because messages can be compressed easier. When a producer prepares to send messages to a broker, it first fills up a buffer. When this buffer is full, the producer compresses (if instructed to do so) and sends out this batch of messages to the broker. The number of batches that can be sent in a single request to the broker is limited by the `max.request.size` property. The number of requests that can simultaneously be in this sending state is controlled by the `max.in.flight.requests.per.connection` value, which defaults to 5 in most client libraries. Tune the batching configuration with the following properties: ### [](#buffer-memory)`buffer.memory` This property controls the total amount of memory available to the producer for buffering. If messages are sent faster than they can be delivered to the broker, the producer application may run out of memory, which causes it to either block subsequent send calls or throw an exception. The `max.block.ms` property controls the amount of time the producer blocks before throwing an exception if it cannot immediately send messages to the broker. ### [](#batch-size)`batch.size` This property controls the maximum size of coupled messages that can be batched together in one request. The producer automatically puts messages being sent to the same partition into one batch. This configuration property is given in bytes, as opposed to the number of messages. When the producer is gathering messages to assign to a batch, at some point it hits this byte-size limit, which triggers it to send the batch to the broker. However, the producer does not necessarily wait (for as much time as set using `linger.ms`) until the batch is full. Sometimes, it can even send single-message batches. This means that setting the batch size too large is not necessarily undesirable, because it won’t cause throttling when sending messages; rather, it only causes increased memory usage. Conversely, setting the batch size too small can cause the producer to send batches of messages faster, which can cause network overhead, meaning a reduced throughput. The default value is usually 16384, but you can set this as low as 0, which turns off batching entirely. ### [](#linger-ms)`linger.ms` This property controls the maximum amount of time the producer waits before sending out a batch of messages, if it is not already full. This means you can somewhat force the producer to make sure that batches are filled as efficiently as possible. If you’re willing to tolerate some latency, setting this value to a number larger than the default of `0` causes the producer to send fewer, more efficient batches of messages. If you set the value to `0`, there is still a high chance messages arrive around the same time to be batched together. ## [](#common-producer-configurations)Common producer configurations ### [](#compression-type)`compression.type` This property controls how the producer should compress a batch of messages before sending it to the broker. The default is `none`, which means the batch of messages is not compressed at all. Compression occurs on full batches, so you can improve batching throughput by setting this property to use one of the available compression algorithms (along with increasing batch size). The available options are: `zstd`, `lz4`, `gzip`, and `snappy`. ### [](#serializers)Serializers Serializers are responsible for converting a message to a byte array. You can influence the speed/memory efficiency of your streaming setup by choosing one of the built-in serializers or writing a custom one. The performance consequences of using serializers is not typically significant. For example, if you opt for the JSON serializer, you have more data to transport with each message because every record contains its schema in a verbose format, which impacts your compression speeds and network throughput. Alternatively, going with AVRO or Protobuf allows you to only define the schema in one place, while also enabling features like schema evolution. ## [](#broker-timestamps)Broker timestamps Redpanda employs a unique strategy to help ensure the accuracy of retention operations. In this strategy, closed segments are only eligible for deletion when the age of all messages in the segment exceeds a configured threshold. However, when a producer sends a message to a topic, the timestamp set by the producer may not accurately reflect the time the message reaches the broker. To address this time skew, each time a producer sends a message to a topic, Redpanda records the broker’s system date and time in the `broker_timestamp` property of the message. This property helps maintain accurate retention policies, even when the message’s creation timestamp deviates from the broker’s time. > 📝 **NOTE** > > Clock synchronization should be monitored by the server owner, as Redpanda does not monitor clock synchronization. While Redpanda does not rely on clocks for correctness, if you are using `LogAppendTime` (server timestamp set by Redpanda), server clocks may affect the time your application sees. See also: - [Set time-based retention](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/#set-time-based-retention) ### [](#configure-broker-timestamp-alerting)Configure broker timestamp alerting Each time a broker receives a message with a skewed timestamp that is outside a configured range, Redpanda increments the [`vectorized_kafka_rpc_produce_bad_create_time`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_kafka_rpc_produce_bad_create_time) metric. Two cluster properties control this range. The minimum accepted value for both of these properties is five minutes. Any attempt to set a value lower than that is rejected by Redpanda. - `log_message_timestamp_alert_before_ms`: Defines the allowed skew before the broker’s time. This check is effectively disabled when the value is set to `null`. Minimum: `300000 ms` (5 minutes), Default: `null`. - `log_message_timestamp_alert_after_ms`: Defines the allowed skew after the broker’s time. There is no way to disable this check. Minimum: `300000 ms` (5 minutes), Default: `7200000 ms` (2 hours). ### [](#disable-broker-timestamp-retention)Disable broker timestamp retention While not advised for typical use, Redpanda lets you override the use of broker timestamps for retention policy with the Admin API. Use the [`activate feature`](https://docs.redpanda.com/api/doc/admin/operation/operation-put_feature) API to disable the `broker_time_based_retention` property. If you disable this feature, make sure to specify your desired timestamp policy. This is stored in the [`log_message_timestamp_type`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_message_timestamp_type) cluster property. The timestamp policy defaults to `CreateTime` (client timestamp set by producer) but may be updated to `LogAppendTime` (server timestamp set by Redpanda). ## [](#producer-optimization-strategies)Producer optimization strategies You can optimize for speed (throughput and latency) or safety (durability and availability) by adjusting properties. Finding the optimal configuration depends on your use case. There are many configuration options within Redpanda. The configuration options mentioned here work best when combined with other broker and consumer configuration options. See also: - [Configure Broker Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/node-property-configuration/) - [Consumer Offsets](https://docs.redpanda.com/streaming/current/develop/consume-data/consumer-offsets/) ### [](#optimize-for-speed)Optimize for speed To get data into Redpanda as quickly as possible, you can maximize latency and throughput in a variety of ways: - Experiment with [acks](#producer-acknowledgment-settings) settings. The quicker a producer receives a reply from the broker that the message has been committed, the sooner it can send the next message, which generally results in higher throughput. Hence, if you set `acks=1`, then the leader broker does not need to wait for replication to occur, and it can reply as soon as it finishes committing the message. This can result in less durability overall. - Enable [write caching](<#Write caching>), which acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to fsync to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write. - Experiment with other component’s properties, like the topic partition size. - Explore how the producer batches messages. Increasing the value of `batch.size` and `linger.ms` can increase throughput by making the producer add more messages into one batch before sending it to the broker and waiting until the batches can properly fill up. This approach negatively impacts latency though. By contrast, if you set `linger.ms` to `0` and `batch.size` to `1`, you can achieve lower latency, but sacrifice throughput. ### [](#optimize-for-safety)Optimize for safety For applications where you must guarantee that there are no lost messages, duplicates, or service downtime, you can use higher durability `acks` settings. If you set `acks=all`, then the producer waits for a majority of replicas to acknowledge the message before it can send the next message, resulting in lower latency, because there is more communication required between brokers. This approach can guarantee higher durability because the message is replicated to all brokers. You can also increase durability by increasing the number of retries the broker can make in case messages are not delivered successfully. The trade-off is that duplicates may enter the system and potentially alter the ordering of messages. ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 85: Idempotent producers **URL**: https://docs.redpanda.com/streaming/current/develop/produce-data/idempotent-producers.md --- # Idempotent producers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Idempotent producers latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: produce-data/idempotent-producers page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: produce-data/idempotent-producers.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/produce-data/idempotent-producers.adoc description: Idempotent producers assign a unique ID to every write request, guaranteeing that each message is recorded only once in the order in which it was sent. page-git-created-date: "2023-05-30" page-git-modified-date: "2025-05-07" support-status: supported --- When a producer writes messages to a topic, each message should be recorded only once in the order in which it was sent. However, network issues such as a connection failure can result in a timeout, which prevents a write request from succeeding. In such cases, the client retries the write request until one of these events occurs: - The client receives an acknowledgment from the broker that the write was successful. - The retry limit is reached. - The message delivery timeout limit is reached. Since there is no way to tell if the initial write request succeeded before the disruption, a retry can result in a duplicate message. A retry can also cause subsequent messages to be written out of order. Idempotent producers prevent this problem by assigning a unique ID to every write request. The request ID consists of the producer ID and a sequence number. The sequence number identifies the order in which each write request was sent. If a retry results in a duplicate message, Redpanda detects and rejects the duplicate message and maintains the original order of the messages. If new write requests continue while a previous request is being retried, the new requests are stored in the client’s memory in the order in which they were sent. The client must also retry these requests once the previous request is successful. ## [](#enable-idempotence-for-producers)Enable idempotence for producers To make producers idempotent, the `enable.idempotence` property must be set to `true` in your producer configuration, as well as in the Redpanda cluster configuration, where it is set to `true` by default. Some Kafka clients have `enable.idempotence` set to `false` by default. In this case, set the property to `true` by following the instructions for your particular client. Idempotence is guaranteed within a session. A session starts once a producer is created and a connection is established between the client and the Kafka broker. > 📝 **NOTE** > > Idempotent producers retry unsuccessful write requests automatically. If you manually retry a write request, the client will assign a new ID to that request, which may lead to duplicate messages. To disable idempotence (and risk duplicate messages as a result of retries), set `enable_idempotence` to `false`. For instructions on how to edit any cluster property, see [Configure cluster properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/). ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 86: Configure Leader Pinning **URL**: https://docs.redpanda.com/streaming/current/develop/produce-data/leader-pinning.md --- # Configure Leader Pinning > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Leader Pinning latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: produce-data/leader-pinning page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: produce-data/leader-pinning.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/produce-data/leader-pinning.adoc description: Learn about Leader Pinning and how to configure a preferred partition leader location based on cloud availability zones or regions. page-topic-type: how-to personas: streaming_developer, platform_admin learning-objective-1: Configure preferred partition leader placement using rack labels learning-objective-2: Configure ordered rack preference for priority-based leader failover learning-objective-3: Identify conditions where Leader Pinning cannot place leaders in preferred racks page-git-created-date: "2024-12-03" page-git-modified-date: "2026-04-08" support-status: supported --- Produce requests that write data to Redpanda topics are routed through the topic partition leader, which syncs messages across its follower replicas. For a Redpanda cluster deployed across multiple availability zones (AZs), Leader Pinning ensures that a topic’s partition leaders are geographically closer to clients, which helps decrease networking costs and guarantees lower latency. If consumers are located in the same preferred region or AZ for Leader Pinning, and you have not set up [follower fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/), Leader Pinning can also help reduce networking costs on consume requests. After reading this page, you will be able to: - Configure preferred partition leader placement using rack labels - Configure ordered rack preference for priority-based leader failover - Identify conditions where Leader Pinning cannot place leaders in preferred racks ## [](#prerequisites)Prerequisites > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Before you can enable Leader Pinning, you must [configure rack awareness](https://docs.redpanda.com/streaming/current/manage/rack-awareness/#configure-rack-awareness) on the cluster. If the `[enable_rack_awareness](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_rack_awareness)` cluster configuration property is set to `false`, Leader Pinning is disabled across the cluster. ## [](#set-leader-rack-preferences)Set leader rack preferences You can configure Leader Pinning at the topic level, the cluster level, or both. Set the topic configuration property to configure individual topics, or set the cluster configuration property to apply a default for all topics. You can also combine both: apply a cluster-wide default, then override specific topics with the topic property. This configuration is based on the following scenario: you have Redpanda deployed in a multi-AZ or multi-region cluster, and you have configured each broker so that the `[rack](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#rack)` configuration property contains racks corresponding to the AZs: - Set the topic configuration property [`redpanda.leaders.preference`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-leaders-preference). This property accepts the following string values: - `none`: Disable Leader Pinning for the topic. - `racks:[,,…​]`: Specify the preferred location (rack) of all topic partition leaders. The list can contain one or more racks, and you can list the racks in any order. Spaces in the list are ignored, for example: `racks:rack1,rack2` and `racks: rack1, rack2` are equivalent. You cannot specify empty racks, for example: `racks: rack1,,rack2`. If you specify multiple racks, Redpanda tries to distribute the partition leader locations equally across brokers in these racks. - `ordered_racks:[,,…​]`: Supported in Redpanda v26.1 or later. Specify the preferred racks in priority order. Redpanda places leaders in the first listed rack when available, failing over to each subsequent rack when higher-priority racks are unavailable. If all listed racks are unavailable, leaders fall back to any other available brokers. Brokers with no rack assignment are treated as lowest priority. Use `ordered_racks` for multi-region deployments with a primary region for leaders and explicit failover to a disaster recovery site. The [`redpanda.leaders.preference`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-leaders-preference) property inherits the default value from the cluster property `default_leaders_preference`. To find the rack identifiers of all brokers, run: ```bash rpk cluster info ``` Expected output ```bash CLUSTER ======= redpanda.be267958-279d-49cd-ae86-98fc7ed2de48 BROKERS ======= ID HOST PORT RACK 0* 54.70.51.189 9092 us-west-2a 1 35.93.178.18 9092 us-west-2b 2 35.91.121.126 9092 us-west-2c ``` To set the topic property: ```bash rpk topic alter-config --set redpanda.leaders.preference=ordered_racks:, ``` - Set the cluster configuration property `[default_leaders_preference](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#default_leaders_preference)`, which specifies the default Leader Pinning configuration for all topics that don’t have `redpanda.leaders.preference` explicitly set. It accepts values in the same format as `redpanda.leaders.preference`, where the default is `none`. This property also affects internal topics, such as `__consumer_offsets` and transaction coordinators. All offset tracking and transaction coordination requests get placed within the preferred regions or AZs for all clients, so you see end-to-end latency and networking cost benefits. To set the cluster property: ```bash rpk cluster config set default_leaders_preference ordered_racks:, ``` If there is more than one broker in the preferred AZ (or AZs), Leader Pinning distributes partition leaders uniformly across brokers in the AZ. ## [](#limitations)Limitations Leader Pinning controls which replica is elected as leader, and does not move replicas to different brokers. If all of a topic’s replicas are on brokers in non-preferred racks, no replica exists in the preferred racks to elect as leader, and Redpanda may elect a non-preferred leader indefinitely. For example, consider a cluster deployed across four racks (A, B, C, D) with Leader Pinning configured as `ordered_racks:A,B,C,D`. With a replication factor of 3, rack awareness can only place replicas in three of the four racks. If the highest-priority rack (A) does not receive a replica, no replica exists there to elect as leader, and Redpanda may elect a non-preferred leader indefinitely. To prevent this scenario: - Enable `[enable_rack_awareness](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_rack_awareness)` to distribute replicas across racks automatically. - Ensure the topic’s replication factor at least equals the total number of racks in the cluster, so every rack, including the highest-priority rack, receives a replica. ## [](#leader-pinning-failover-across-availability-zones)Leader Pinning failover across availability zones If there are three AZs: A, B, and C, and A becomes unavailable, the failover behavior with `racks` is as follows: - The topic with `A` as the preferred leader AZ will have its partition leaders uniformly distributed across B and C. - The topic with `A,B` as the preferred leader AZs will have its partition leaders in B. - The topic with `B` as the preferred leader AZ will have its partition leaders in B as well. ### [](#failover-with-ordered-rack-preference)Failover with ordered rack preference With `ordered_racks`, the failover order follows the configured priority list. Leaders move to the next available rack in the list when higher-priority racks become unavailable. For a topic configured with `ordered_racks:A,B,C`: - The topic with `A` as the first-priority rack will have its partition leaders in A. - If A becomes unavailable, leaders move to B. - If A and B become unavailable, leaders move to C. - If A, B, and C all become unavailable, leaders fall back to any available brokers. If a higher-priority rack recovers and the topic’s replication factor ensures that rack receives a replica, Redpanda automatically moves leaders back to the highest available preferred rack. ## [](#suggested-reading)Suggested reading - For latency-tolerant, high-throughput workloads where cross-AZ networking charges are a major cost driver, also consider [Cloud Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/cloud-topics/) - [Follower Fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/) --- # Page 87: Transactions **URL**: https://docs.redpanda.com/streaming/current/develop/transactions.md --- # Transactions > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Transactions latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: transactions page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: transactions.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/develop/pages/transactions.adoc description: Learn how to use transactions; for example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-06-05" support-status: supported --- Redpanda supports Apache Kafka®-compatible transaction semantics and APIs. For example, you can fetch messages starting from the last consumed offset and transactionally process them one by one, updating the last consumed offset and producing events at the same time. A transaction can span partitions from different topics, and a topic can be deleted while there are active transactions on one or more of its partitions. In-flight transactions can detect deletion events, remove the deleted partitions (and related messages) from the transaction scope, and commit changes to the remaining partitions. If a producer is sending multiple messages to the same or different partitions, and network connectivity or broker failure cause the transaction to fail, then it’s guaranteed that either all messages are written to the partitions or none. This is important for applications that require strict guarantees, like financial services transactions. Transactions guarantee both exactly-once semantics (EOS) and atomicity: - EOS helps developers avoid the anomalies of at-most-once processing (with potential lost events) and at-least-once processing (with potential duplicated events). Redpanda supports EOS when transactions are used in combination with [idempotent producers](https://docs.redpanda.com/streaming/current/develop/produce-data/idempotent-producers/). - Atomicity additionally commits a set of messages across partitions as a unit: either all messages are committed or none. Encapsulated data received or sent across multiple topics in a single operation can only succeed or fail globally. > 📝 **NOTE** > > Atomicity of transactions is not guaranteed when [remote recovery](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#remote-recovery) is used. ## [](#use-transactions)Use transactions By default, the `[enable_transactions](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_transactions)` cluster configuration property is set to true. However, in the following use cases, clients must explicitly use the Transactions API to perform operations within a transaction: - [Atomic (all or nothing) publishing of multiple messages](#atomic-publishing-of-multiple-messages) - [Exactly-once stream processing](#exactly-once-stream-processing) When you use transactions, you must set the [`transactional.id`](https://kafka.apache.org/documentation/#producerconfigs_transactional.id) property in the producer configuration. This property uniquely identifies the producer and enables reliable semantics across multiple producer sessions. It ensures that all transactions issued by a given producer are completed before any new transactions are started. ### [](#atomic-publishing-of-multiple-messages)Atomic publishing of multiple messages A banking IT system with an event-sourcing microservice architecture illustrates why transactions are necessary. In this system, each bank branch is implemented as an independent microservice that manages its own distinct set of accounts. Every branch maintains its own transaction history, stored as a Redpanda partition. When a branch starts, it replays the transaction history to reconstruct its current state. Financial transactions such as money transfers require the following guarantees: - A sender can’t withdraw more than the account withdrawal limit. - A recipient receives exactly the same amount sent. - A transaction is fast and is run at most once. - If a transaction fails, the system rolls back to the initial state. - Without withdrawals and deposits, the amount of money in the system remains constant with any history of money transfers. These requirements are easy to satisfy when the sender and the recipient of a financial transaction are hosted by the same branch. The operation doesn’t leave the consistency domain, and all checks and locks can be performed within a single service (ledger). Things get more complex with cross-branch financial transactions, because they involve several ledgers, and the operations should be performed atomically (all or nothing). The default approach (saga pattern) breaks a transaction into a sequence of reversible idempotent steps; however, this violates the isolation principle and adds complexity, making the application responsible for orchestrating the steps. Redpanda natively supports transactions, so it’s possible to atomically update several ledgers at the same time. For example: Show multi-ledger transaction example: ```java Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); props.put(ProducerConfig.ACKS_CONFIG, "all"); props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "app-id"); Producer producer = null; while (true) { // waiting for somebody to initiate a financial transaction var sender_branch = ...; var sender_account = ...; var recipient_branch = ...; var recipient_account = ...; var amount = 42; if (producer == null) { try { producer = new KafkaProducer<>(props); producer.initTransactions(); } catch (Exception e1) { // TIP: log error for further analysis try { if (producer != null) { producer.close(); } } catch(Exception e2) { } producer = null; // TIP: notify the initiator of a transaction about the failure continue; } } producer.beginTransaction(); try { var f1 = producer.send(new ProducerRecord("ledger", sender_branch, sender_account, "" + (-amount))); var f2 = producer.send(new ProducerRecord("ledger", recipient_branch, recipient_account, "" + amount)); f1.get(); f2.get(); } catch (Exception e1) { // TIP: log error for further analysis try { producer.abortTransaction(); } catch (Exception e2) { // TIP: log error for further analysis try { producer.close(); } catch (Exception e3) { } producer = null; } // TIP: notify the initiator of a transaction about the failure continue; } try { producer.commitTransaction(); } catch (Exception e1) { try { producer.close(); } catch (Exception e3) {} producer = null; // TIP: notify the initiator of a transaction about the failure continue; } // TIP: notify the initiator of a transaction about the success } ``` When a transaction fails before a `commitTransaction` attempt completes, you can assume that it is not executed. When a transaction fails after a `commitTransaction` attempt completes, the true transaction status is unknown. Redpanda only guarantees that there isn’t a partial result: either the transaction is committed and complete, or it is fully rolled back. ### [](#exactly-once-stream-processing)Exactly-once stream processing Redpanda is commonly used as a pipe connecting different applications and storage systems. An application could use an OLTP database and then rely on change data capture to deliver the changes to a data warehouse. Redpanda transactions let you use streams as a smart pipe in your applications, building complex atomic operations that transform, aggregate, or otherwise process data transiting between external applications and storage systems. For example, here is the regular pipe flow: Postgresql -> topic -> warehouse Here is the smart pipe flow, with a transformation in `topic(1) -> topic(2)`: Postgresql -> topic(1) transform topic(2) -> warehouse The transformation reads a record from `topic(1)`, processes it, and writes it to `topic(2)`. Without transactions, an intermittent error can cause a message to be lost or processed several times. With transactions, Redpanda guarantees exactly-once semantics. For example: Show exactly-once processing example: ```java var source = "source-topic"; var target = "target-topic"; Properties pprops = new Properties(); pprops.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); pprops.put(ProducerConfig.ACKS_CONFIG, "all"); pprops.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); pprops.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, UUID.randomUUID().toString()); Properties cprops = new Properties(); cprops.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "..."); cprops.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); cprops.put(ConsumerConfig.GROUP_ID_CONFIG, "app-id"); cprops.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); cprops.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); Consumer consumer = null; Producer producer = null; boolean should_reset = false; while (true) { if (should_reset) { should_reset = false; if (consumer != null) { try { consumer.close(); } catch(Exception e) {} consumer = null; } if (producer != null) { try { producer.close(); } catch (Exception e2) {} producer = null; } } try { if (consumer == null) { consumer = new KafkaConsumer<>(cprops); consumer.subscribe(Collections.singleton(source)); } } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } try { if (producer == null) { producer = new KafkaProducer<>(pprops); producer.initTransactions(); } } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } ConsumerRecords records = null; try { records = consumer.poll(Duration.ofMillis(10000)); } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; continue; } var it = records.iterator(); while (it.hasNext()) { var record = it.next(); // transformation var old_value = record.value(); var new_value = old_value.toUpperCase(); try { producer.beginTransaction(); producer.send(new ProducerRecord(target, record.key(), new_value)); var offsets = new HashMap(); offsets.put(new TopicPartition(source, record.partition()), new OffsetAndMetadata(record.offset() + 1)); producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata()); } catch (Exception e1) { // TIP: log error for further analysis try { producer.abortTransaction(); } catch (Exception e2) { } should_reset = true; break; } try { producer.commitTransaction(); } catch (Exception e1) { // TIP: log error for further analysis should_reset = true; break; } } } ``` #### [](#exactly-once-processing-configuration-requirements)Exactly-once processing configuration requirements Redpanda’s default configuration supports exactly-once processing. To preserve this capability, ensure the following settings are maintained: - `enable_idempotence = true` - `enable_transactions = true` - `transaction_coordinator_delete_retention_ms` is greater than or equal to `transactional_id_expiration_ms` ## [](#best-practices)Best practices To help avoid common pitfalls and optimize performance, consider the following when configuring transactional workloads in Redpanda: ### [](#tune-producer-id-limits)Tune producer ID limits For production environments with heavy producer usage, configure both [`max_concurrent_producer_ids`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#max_concurrent_producer_ids) and [`transactional_id_expiration_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#transactional_id_expiration_ms) to prevent out-of-memory (OOM) crashes. Setting limits on producer IDs helps manage memory usage in high-throughput environments, particularly when using transactions or idempotent producers. If you have\`kafka\_connections\_max\` configured, you can determine an appropriate value for `max_concurrent_producer_ids` based on your connection patterns. - Lower bound: `kafka_connections_max` / `number_of_shards`, assuming each producer connects to only one shard. - Upper bound: `topic_partitions_per_shard` \* `kafka_connections_max`, assuming producers connect to all shards. If `kafka_connections_max` is not configured, estimate the value for `max_concurrent_producer_ids` based on your application patterns. A conservative approach is to start with 1000-5000 per shard, then monitor and adjust as needed. Applications with many partitions per producer typically require higher values, such as 10000 or more per shard. Tune `transactional_id_expiration_ms` based on your application’s transaction patterns. Calculate this value by taking your longest expected transaction time and adding a safety buffer. For example, if transactions typically run for 30 minutes, consider setting this to 2-4 hours. Short-lived transactions can use values between 1-4 hours, while batch processing applications should match their batch interval plus buffer time. Interactive applications may benefit from shorter values to free up memory faster. Client applications should minimize producer ID churn. Reuse producer instances when possible, instead of creating new ones for each operation. Avoid using random transactional IDs, as some Flink configurations do, because this creates excessive producer ID churn. Instead, use consistent transactional IDs that can be resumed across application restarts. Monitor the following metrics to determine if the limit is being reached: - [`vectorized_cluster_producer_state_manager_evicted_producers`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_cluster_producer_state_manager_evicted_producers): Number of evicted producers (should be 0 in steady state) - [`vectorized_cluster_producer_state_manager_producer_manager_total_active_producers`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_cluster_producer_state_manager_producer_manager_total_active_producers): Current number of active producers per shard If `vectorized_cluster_producer_state_manager_evicted_producers` > 0, the shard is exceeding the configured limit. For applications with long-running transactions, ensure [`transactional_id_expiration_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#transactional_id_expiration_ms) accommodates your typical transaction lifetime to avoid premature producer ID expiration. ### [](#configure-transaction-timeouts-and-limits)Configure transaction timeouts and limits - If a consumer is configured to use the read\_committed isolation level, it can only process successfully committed transactions. As a result, an ongoing transaction with a large timeout that becomes stuck could prevent the consumer from processing other committed transactions. To avoid this, don’t set the transaction timeout client setting (`transaction.timeout.ms` in the Kafka Java client implementation) to a value that is too high. The longer the timeout, the longer consumers may be blocked. - When running transactional workloads from clients, tune [`max_transactions_per_coordinator`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#max_transactions_per_coordinator) to match the number of concurrent transactions your clients run (if your client transaction IDs are not reused). The total number of transactions allowed in the cluster at any time is determined by `max_transactions_per_coordinator * transaction_coordinator_partitions` (default is 50 partitions). When the limit is exceeded, Redpanda terminates old sessions. If an idle producer corresponding to a terminated session becomes active and tries to produces again, Redpanda rejects its batches with an `invalid producer epoch` or `invalid_producer_id_mapping` error, depending on where it is in the transaction execution phase. Be aware that if you keep the `transaction_coordinator_partitions` at the default of 50 and your clients create a new ID for every transaction, the total continues to accumulate, which bloats memory. - Transactional metadata is stored in the internal topic `kafka_internal/tx`. Over time, this topic can consume disk space. You can manage its disk usage by tuning the `transaction_coordinator_delete_retention_ms` and `transactional_id_expiration_ms` cluster properties. See also: [Manage Disk Space](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/#manage-transaction-coordinator-disk-usage) - When upgrading a self-managed deployment, make sure to use maintenance mode with a [rolling upgrade](https://docs.redpanda.com/streaming/current/reference/glossary/#rolling-upgrade). ## [](#handle-transaction-failures)Handle transaction failures Different transactions require different approaches to handling failures within the application. Consider the approaches to failed or timed-out transactions in the provided use cases: - Publishing of multiple messages: The request came from outside the system, and it is the application’s responsibility to discover the true status of a timed-out transaction. (This example doesn’t use consumer groups to distribute partitions between consumers.) - Exactly-once streaming (consume-transform-loop): This is a closed system. Upon re-initialization of the consumer and producer, the system automatically discovers the moment it was interrupted and continues from that place. Additionally, this automatically scales by the number of partitions. Run another instance of the application, and it starts processing its share of partitions in the source topic. ## [](#transactions-with-compacted-segments)Transactions with compacted segments Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches). At a cluster-level, compaction is set when [`log_cleanup_policy`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_cleanup_policy) or [`cleanup.policy`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#cleanup-policy) are set to either `compact` or `compact,delete`. Optionally, you can enable removal of transactional control batches (commit and abort markers) during compaction by setting [`log_compaction_tx_batch_removal_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_compaction_tx_batch_removal_enabled) to `true`. When enabled, the topic’s [`delete.retention.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#delete-retention-ms) setting is applied to these markers. For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level. This feature is not applied when Tiered Storage is enabled. See [Transactional control batch removal](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings/#transactional-control-batch-removal). ## [](#suggested-reading)Suggested reading - [Kafka-compatible fast distributed transactions](https://redpanda.com/blog/fast-transactions) ## Suggested labs - [Stream Stock Market Data from a CSV file Using Node.js](https://docs.redpanda.com/labs/clients/stock-market-activity-nodejs/) - [Stream Stock Market Data from a CSV file Using Python](https://docs.redpanda.com/labs/clients/stock-market-activity-python/) - [Build a Chat Room Application with Redpanda and Golang](https://docs.redpanda.com/labs/clients/docker-go/) - [Build a Chat Room Application with Redpanda and Java](https://docs.redpanda.com/labs/clients/docker-java/) - [Build a Chat Room Application with Redpanda and Node.js](https://docs.redpanda.com/labs/clients/docker-nodejs/) - [Build a Chat Room Application with Redpanda and Python](https://docs.redpanda.com/labs/clients/docker-python/) - [Build a Chat Room Application with Redpanda and Rust](https://docs.redpanda.com/labs/clients/docker-rust/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 88: Get Started **URL**: https://docs.redpanda.com/streaming/current/get-started.md --- # Get Started > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Get Started latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/index.adoc description: Get Started index page. page-git-created-date: "2023-05-30" page-git-modified-date: "2023-08-21" support-status: supported --- - [What’s New](release-notes/) Summary of new features and updates. - [Introduction to Redpanda](intro-to-events/) Learn about Redpanda event streaming. - [How Redpanda Works](architecture/) Learn specifics about Redpanda architecture. - [Introduction to Redpanda Console](https://docs.redpanda.com/streaming/current/console/) Learn about Redpanda Console: a web interface for managing and interacting with Redpanda clusters. - [Redpanda Quickstarts](quickstarts/) Get started with Redpanda using these hands-on tutorials. Explore features that demonstrate how Redpanda can power your streaming applications. - [Redpanda Licensing](licensing/) - [Redpanda CLI](rpk/) The `rpk` command line interface tool lets you manage your Redpanda cluster, without the need to run a separate script for each function, as with Apache Kafka. - [Partner Integrations](partner-integration/) Learn about Redpanda integrations built and supported by our partners. --- # Page 89: Specify Admin API Addresses for rpk **URL**: https://docs.redpanda.com/streaming/current/get-started/admin-addresses.md --- # Specify Admin API Addresses for rpk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Specify Admin API Addresses for rpk latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: admin-addresses page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: admin-addresses.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/admin-addresses.adoc description: Learn how and when to specify Redpanda admin addresses for rpk commands, so rpk knows where to run admin-related commands. page-git-created-date: "2024-07-24" page-git-modified-date: "2026-07-28" support-status: supported --- For `rpk` to know where to run admin-related commands, you must provide the admin server addresses for each broker of a Redpanda cluster. You can specify these addresses as IP addresses or as hostnames, using any of these methods: - Command line flag (`-X admin.hosts`) - Environment variable setting (`RPK_ADMIN_HOSTS`) - Configuration file setting in `redpanda.yaml` (`rpk.admin_api.addresses`) Command line flag settings take precedence over environment variable settings and configuration file settings. If the command line does not contain the `-X admin.hosts` settings, the environment variable settings are used. If the environment variables are not set, the values in the configuration file are used. ## [](#command-line-flags)Command line flags Admin API addresses are required for communicating with the Admin API. Provide these addresses with the `—-api-urls` flag for commands related to cluster or user tasks, such as [`rpk cluster health`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-health/) and [`rpk cluster maintenance enable `](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-maintenance/). Note that `rpk cluster info` and `rpk cluster metadata` instead require the [`-X brokers` flag](https://docs.redpanda.com/streaming/current/get-started/broker-admin/). The following table shows which `rpk` commands require the `-X admin.hosts` or `--hosts` flag. | Command | Address flag required | | --- | --- | | rpk security user | -X admin.hosts | | rpk cluster (all except rpk cluster info and rpk cluster metadata) | -X admin.hosts | | rpk redpanda admin (deprecated in 26.2, see Deprecated Features) | --hosts | ## [](#environment-variable-settings)Environment variable settings Environment variable settings last for the duration of the shell session, or until you set the variable to a different setting. Configure the environment variable `RPK_ADMIN_HOSTS`. For example, to configure the addresses to use when running Admin API commands on an external cluster with three brokers: ```bash export RPK_ADMIN_HOSTS="192.168.78.34:9644,192.168.78.35:9644,192.168.78.36:9644" ``` ## [](#configuration-file-settings)Configuration file settings As each Redpanda broker starts up, a `redpanda.yaml` configuration file is automatically generated for that broker. This file contains a section for `rpk` settings, which includes Admin API settings. The `admin_api` section contains the address and port for each admin server. The default address is `0.0.0.0`, and the default port is 9644. You can edit this line and replace it with the Admin API IP addresses. The following example shows the addresses and port numbers for two admin servers. ```yaml rpk: admin_api: - 192.168.72.34:9644 - 192.168.72.35:9644 ``` > 📝 **NOTE** > > If you do not update the default addresses in the `redpanda.yaml` file, you must provide the required addresses on the command line or by setting the corresponding environment variable. --- # Page 90: How Redpanda Works **URL**: https://docs.redpanda.com/streaming/current/get-started/architecture.md --- # How Redpanda Works > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: How Redpanda Works latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: architecture page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: architecture.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/architecture.adoc description: Learn specifics about Redpanda architecture. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-07-17" support-status: supported --- At its core, Redpanda is a fault-tolerant transaction log for storing event streams. Producers and consumers interact with Redpanda using the Kafka API. To achieve high scalability, producers and consumers are fully decoupled. Redpanda provides strong guarantees to producers that events are stored durably within the system, and consumers can subscribe to Redpanda and read the events asynchronously. Redpanda achieves this decoupling by organizing events into topics. Topics represent a logical grouping of events that are written to the same log. A topic can have multiple producers writing events to it and multiple consumers reading events from it. This page provides details about how Redpanda works. For a high-level overview, see [Introduction to Redpanda](https://docs.redpanda.com/streaming/current/get-started/intro-to-events/). ## [](#tiered-storage)Tiered Storage Redpanda Tiered Storage is a multi-tiered object storage solution that provides the ability to offload log segments to object storage in near real time. Tiered Storage can be combined with local storage to provide long-term data retention and disaster recovery on a per-topic basis. Consumers that read from more recent offsets continue to read from local storage, and consumers that read from historical offsets read from object storage, all with the same API. Consumers can read and reread events from any point within the maximum retention period, whether the events reside on local or object storage. As data in object storage grows, the metadata for it grows. To support efficient long-term data retention, Redpanda splits the metadata in object storage, maintaining metadata of only recently-updated segments in memory or local disk, while safely archiving the remaining metadata in object storage and caching it locally on disk. Archived metadata is then loaded only when historical data is accessed. This allows Tiered Storage to handle partitions of virtually any size or retention length. For more information, see [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/). ## [](#partitions)Partitions To scale topics, Redpanda shards them into one or more partitions that are distributed across the nodes in a cluster. This allows for concurrent writing and reading from multiple nodes. When producers write to a topic, they route events to one of the topic’s partitions. Events with the same key (like a stock ticker) are always routed to the same partition, and Redpanda guarantees the order of events at the partition level. Consumers read events from a partition in the order that they were written. If a key is not specified, then events are sent to all topic partitions in a round-robin fashion. ## [](#raft-consensus-algorithm)Raft consensus algorithm Redpanda provides strong guarantees for data safety and fault tolerance. Events written to a topic partition are appended to a log file on disk. They can be replicated to other nodes in the cluster and appended to their copies of the log file on disk to prevent data loss in the event of failure. The [Raft consensus algorithm](https://raft.github.io/) is used for data replication. Every topic partition forms a Raft group consisting of a single elected leader and zero or more followers (as specified by the topic’s replication factor). A Raft group can tolerate ƒ failures given 2ƒ+1 nodes. For example, in a cluster with five nodes and a topic with a replication factor of five, the topic remains fully operational if two nodes fail. Raft is a majority vote algorithm. For a leader to acknowledge that an event has been committed to a partition, a majority of its replicas must have written that event to their copy of the log. When a majority (quorum) of responses have been received, the leader can make the event available to consumers and acknowledge receipt of the event when `acks=all (-1)`. [Producer acknowledgement settings](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/#producer-acknowledgement-settings) define how producers and leaders communicate their status while transferring data. As long as the leader and a majority of the replicas are stable, Redpanda can tolerate disturbances in a minority of the replicas. If [gray failures](https://blog.acolyer.org/2017/06/15/gray-failure-the-achilles-heel-of-cloud-scale-systems/) cause a minority of replicas to respond slower than normal, then the leader does not have to wait for their responses to progress, and any additional latency is not passed on to the clients. The result is that Redpanda is less sensitive to faults and can deliver predictable performance. ## [](#partition-leadership-elections)Partition leadership elections [Raft](https://raft.github.io/) uses a heartbeat mechanism to maintain leader authority and to trigger leader elections. The partition leader sends a heartbeat to all followers every 150 milliseconds to assert its leadership in the current term (an election cycle). If a follower does not receive a heartbeat within the election timeout, it triggers an election to choose a new partition leader. Configure the election timeout with the [`election_timeout_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#election_timeout_ms) cluster property (default: 1500 milliseconds). The follower increments its term and votes for itself to be the leader for that term. It then sends a vote request to the other nodes and waits for one of the following scenarios: - It receives a majority of votes and becomes the leader. Raft guarantees that at most one candidate can be elected the leader for a given term. - Another follower establishes itself as the leader. While waiting for votes, the candidate may receive communication from another node in the group claiming to be the leader. The candidate only accepts the claim if its term is greater than or equal to the candidate’s term; otherwise, the communication is rejected and the candidate continues to wait for votes. - No leader is elected over a period of time. If multiple followers timeout and become election candidates at the same time, it’s possible that no candidate gets a majority of votes. When this happens, each candidate increments its term and triggers a new election round. Raft uses a random timeout between 150-300 milliseconds to ensure that split votes are rare and resolved quickly. As long as there is a timing inequality between heartbeat time, election timeout, and mean time between node failures (MTBF), then Raft can elect and maintain a steady leader and make progress. A leader can maintain its position as long as one of the ten heartbeat messages it sends to all of its followers every 1.5 seconds is received; otherwise, a new leader is elected. If a follower triggers an election, but the incumbent leader subsequently springs back to life and starts sending data again, then it’s too late. As part of the election process, the follower (now an election candidate) incremented the term and rejects requests from the previous term, essentially forcing a leadership change. If a cluster is experiencing wider network infrastructure problems that result in latencies above the heartbeat timeout, then back-to-back election rounds can be triggered. During this period, unstable Raft groups may not be able to form a quorum. This results in partitions rejecting writes, but data previously written to disk is not lost. Redpanda has a Raft-priority implementation that allows the system to settle quickly after network outages. ## [](#controller-partition-and-snapshots)Controller partition and snapshots Redpanda stores metadata update commands (such as creating and deleting topics or users) in a system partition called the controller partition. A new snapshot is created after each controller command is added, or, with rapid updates, after a set period of time (default is 60 seconds). Controller snapshots save the current cluster metadata state to disk, so startup is fast. For example, with a partition that has moved several times, a snapshot can restore the latest state without replaying every move command. Each broker has a snapshot file stored in the controller log directory, such as `/var/lib/redpanda/data/redpanda/controller/0_0/snapshot`. The controller partition is replicated by a Raft group that includes all cluster brokers, and the controller snapshot is the Raft snapshot for this group. Snapshots are hydrated when a broker joins the cluster or restarts. Snapshots are enabled by default for all clusters, both new and upgraded. ## [](#optimized-platform-performance)Optimized platform performance Redpanda is designed to exploit advances in modern hardware, from the network down to the disks. Network bandwidth has increased considerably, especially in object storage, and spinning disks have been replaced by SSD devices that deliver better I/O performance. CPUs are faster too, but this is largely due to the increased core counts as opposed to the increase in single-core speeds. Redpanda has tuners that detect your hardware configuration to automatically optimize itself. Examples of platform and kernel features that Redpanda uses to optimize its performance: - Direct Memory Access (DMA) for disk I/O - Sparse file system support with XFS - Distribution of interrupt request (IRQ) processing between CPU cores - Isolated processes with control groups (cgroups) - Disabled CPU power-saving modes - Upfront memory allocation, partitioned and pinned to CPU cores ## [](#tpc)Thread-per-core model Redpanda implements a thread-per-core programming model through its use of the [Seastar](https://seastar.io/) library. This allows Redpanda to pin each of its application threads to a CPU core to avoid context switching and blocking. It combines this with message passing to asynchronously communicate between the pinned threads. With this, Redpanda avoids the overhead of context switching and expensive locking operations to improve processing performance and efficiency. From a sizing perspective, Redpanda’s ability to efficiently use all available hardware enables it to scale up to get the most out of your infrastructure, before you’re forced to scale out to meet the demands of your workload. Redpanda delivers better performance with a smaller footprint, resulting in reduced operational costs and complexity. ## [](#next-steps)Next steps [Try out Redpanda](https://docs.redpanda.com/streaming/current/get-started/quick-start/), or learn about [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). ## [](#suggested-reading)Suggested reading - [A developer’s guide to Redpanda](https://redpanda.com/blog/data-streaming-with-redpanda) - [How Redpanda’s cloud-first storage model reduces TCO](https://redpanda.com/blog/cloud-native-streaming-data-lower-cost) - [Thread-per-core buffer management for a modern Kafka-API storage system](https://redpanda.com/blog/tpc-buffers?utm_medium=content&utm_assetname=sizing_guide&utm_assettype=report&utm_source=gated_content&utm_campaign=tpc_architecture_blog) ## [](#suggested-videos)Suggested videos - [YouTube - Lightning Talk: Tiered Storage (11:39 mins)](https://www.youtube.com/watch?v=3_Tmdvrp5sU&ab_channel=RedpandaData) - [YouTube - Intro to Redpanda: Thread-per-core architecture in C++ (60 mins)](https://www.youtube.com/watch?v=guoaxRJG8p8&ab_channel=RedpandaData) - [YouTube - Differences between Apache Kafka and Redpanda: Thread per Core Architecture (4:30 mins)](https://www.youtube.com/watch?v=UxM1mn1gwoc&ab_channel=RedpandaData) - [YouTube - Common pitfalls for Redpanda beginners (44:35 mins)](https://www.youtube.com/watch?v=CEVxZznqTDo&ab_channel=RedpandaData) --- # Page 91: Specify Broker Addresses for rpk **URL**: https://docs.redpanda.com/streaming/current/get-started/broker-admin.md --- # Specify Broker Addresses for rpk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Specify Broker Addresses for rpk latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: broker-admin page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: broker-admin.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/broker-admin.adoc description: Learn how and when to specify Redpanda broker addresses for rpk commands, so rpk knows where to run Kafka-related commands. page-git-created-date: "2023-07-24" page-git-modified-date: "2025-05-07" support-status: supported --- For `rpk` to know where to run Kafka-related commands, you must provide the broker addresses for each broker of a Redpanda cluster. You can specify these addresses as IP addresses or as hostnames, using any of these methods: - Command line flag (`-X brokers`) - Environment variable setting (`RPK_BROKERS`) - Configuration file setting in `redpanda.yaml` (`rpk.kafka_api.brokers`) Command line flag settings take precedence over environment variable settings and configuration file settings. If the command line does not contain the `-X brokers` settings, the environment variable settings are used. If the environment variables are not set, the values in the configuration file are used. ## [](#command-line-flags)Command line flags Broker addresses are required for communicating with the Kafka API. Provide these addresses with the `-X brokers` flag for commands related to Kafka broker tasks, such as [`rpk topic create`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-create/), [`rpk topic produce`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-produce/), and [`rpk topic consume`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-consume/). The following table shows which `rpk` commands require the `-X brokers` flag. | Command | Address flag required | | --- | --- | | rpk cluster info | -X brokers | | rpk cluster metadata | -X brokers | | rpk group | -X brokers | | rpk security acl | -X brokers | | rpk topic | -X brokers | ## [](#environment-variable-settings)Environment variable settings Environment variable settings last for the duration of the shell session, or until you set the variable to a different setting. Configure the environment variable `RPK_BROKERS` for broker addresses, so you don’t have to include the `-X brokers` flag each time you run an `rpk` command. For example, to configure three brokers on a single machine running on localhost: ```bash export RPK_BROKERS="192.168.72.34:9092,192.168.72.35:9092,192.168.72.36.9092" ``` ## [](#configuration-file-settings)Configuration file settings As each Redpanda broker starts up, a `redpanda.yaml` configuration file is automatically generated for that broker. This file contains a section for `rpk` settings, which includes Kafka API settings. The `kafka_api` section contains the address and port for each broker. The default address is `0.0.0.0`, and the default port is 9092. You can edit this line and replace it with the IP addresses of your Redpanda brokers. The following example shows the addresses and port numbers for three brokers. ```yaml rpk: kafka_api: brokers: - 192.168.72.34:9092 - 192.168.72.35:9092 - 192.168.72.36.9092 ``` > 📝 **NOTE** > > If you do not update the default addresses in the `redpanda.yaml` file, you must provide the required addresses on the command line or by setting the corresponding environment variable. --- # Page 92: rpk Profiles **URL**: https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile.md --- # rpk Profiles > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: rpk Profiles latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: config-rpk-profile page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: config-rpk-profile.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/config-rpk-profile.adoc description: Use rpk profile to simplify your development experience with multiple Redpanda clusters by saving and reusing configurations for different clusters. page-git-created-date: "2023-07-31" page-git-modified-date: "2025-08-22" support-status: supported --- Use rpk profiles to simplify your development experience using `rpk` with multiple Redpanda clusters by saving and reusing configurations for different clusters. > 💡 **TIP** > > **rpk profiles are the recommended way to configure rpk**. They provide persistent, reusable configurations that work across sessions and are easier to manage than environment variables or command-line flags. > ⚠️ **CAUTION** > > Profile files may contain sensitive information such as passwords or SASL credentials. Do not commit `rpk.yaml` files to version control systems like Git. ## [](#about-rpk-profiles)About rpk profiles An rpk profile contains a reusable configuration for a Redpanda cluster. When running `rpk`, you can create a profile, configure it for a cluster you’re working with, and use it repeatably when running an `rpk` command for the cluster. You can create different profiles for different Redpanda clusters. For example, your local cluster, development cluster, and production cluster can each have their own profile, with all of their information managed locally by rpk. You set a unique name for each profile. A profile saves rpk-specific command properties. For details, see [Specify command properties](https://docs.redpanda.com/streaming/current/get-started/intro-to-rpk/#specify-configuration-properties). All `rpk` commands can read configuration values from a profile. You pass a profile to an `rpk` command by setting the `--profile` flag. For example, the command `rpk topic produce dev-topic --profile dev` gets its configuration from the profile named `dev`. ## [](#quickstart)Quickstart Create a profile with authentication and TLS to quickly set up cluster access instead of using environment variables or connection flags: ```bash rpk profile create \ --set brokers= \ --set admin.hosts= \ --set user= \ --set pass= \ --set sasl.mechanism= \ --set tls.enabled=true \ --description "" ``` Replace `` with your desired SASL mechanism (`SCRAM-SHA-256`, `SCRAM-SHA-512`, or `PLAIN`). When you create a profile, rpk automatically switches to use that profile so you don’t need to pass `--profile` flags every time. Check the active profile: ```bash rpk profile current ``` Now all `rpk` commands use this profile automatically: ```bash rpk topic list rpk topic create ``` You can change profiles by running: ```bash rpk profile use ``` For environment variables and other configuration methods, see [rpk -X options](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-x-options/). ## [](#work-with-rpk-profiles)Work with rpk profiles The primary tasks for working with rpk profiles: - Create one or more profiles. - Choose the profile to use. - Edit or set default values across all profiles and values for a single profile. - Call an `rpk` command with a profile. - Delete unused profiles. ### [](#create-profile)Create profile To create a new profile, run [`rpk profile create`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-create/): ```bash rpk profile create [flags] ``` An rpk profile can be generated from different sources: - A `redpanda.yaml` file, using the `--from-redpanda` flag. - A different rpk profile, using the `--from-profile` flag. After the profile is created, rpk switches to the newly created profile. You can specify the configuration during creation with the `--set [key=value]` flag. To simplify configuration, the `--set` flag supports autocompletion of valid keys, suggesting key names based on their `-X` format. > 📝 **NOTE** > > You should always use and set the `--description` flag to describe your profiles. The description is printed in the output of [`rpk profile list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-list/). Created profiles are stored in an `rpk.yaml` file in a default local OS directory (for example, `~/.config/rpk/` for Linux and `~/Library/Application Support/rpk/` for MacOS). All profiles created by a developer are stored in the same `rpk.yaml` file. ### [](#choose-profile-to-use)Choose profile to use With multiple created profiles, choose the profile to use with [`rpk profile use`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-use/): ```bash rpk profile use ``` ### [](#set-or-edit-configuration-values)Set or edit configuration values You can customize settings for a single profile. To set a profile’s configuration: - Use [`rpk profile set`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-set/) to set `key=value` pairs of configuration options to write to the profile’s section of `rpk.yaml`. - Use [`rpk profile edit`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-edit/) to edit the profile’s section of the `rpk.yaml` file in your default editor. You can configure settings that apply to all profiles. To set these `globals`: - Use [`rpk profile set-globals`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-set-globals/) to set `key=value` pairs to write to the globals section of `rpk.yaml`. - Use [`rpk profile edit-globals`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-edit-globals/) to edit the globals section of the `rpk.yaml` file in your default editor. > 💡 **TIP** > > For a list of all the available properties that can be set in your profile, see [`rpk -X options`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-x-options/). #### [](#customize-command-prompt-per-profile)Customize command prompt per profile A configurable field of an rpk profile is the `prompt` field. It enables the customization of the command prompt for a profile, so information about the in-use profile can be displayed within your command prompt. The format string is intended for a `PS1` prompt. For details on the prompt format string, see the [`rpk profile prompt`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-prompt/) reference. The `rpk profile prompt` command prints the ANSI-escaped text of the `prompt` field for the in-use profile. You can call `rpk profile prompt` in your shell’s (rc) configuration file to assign your `PS1`. For example, to customize your bash prompt for a `dev` rpk profile , first call `rpk profile edit dev` to set its `prompt` field: ```yaml name: dev prompt: hi-red, "[%n]" ``` - `hi-red` sets the text to high-intensity red - `%n` is a variable for the profile name Then in `.bashrc`, set `PS1` to include a call to `rpk profile prompt`: ```bash export PS1='\u@\h\n$(rpk profile prompt)% ' ``` > 📝 **NOTE** > > When setting your `PS1` variable, use single quotation marks and not double quotation marks, because double quotation marks aren’t reevaluated after every command. The resulting prompt looks like this: username@hostname\[dev\]% ### [](#use-profile-with-rpk-command)Use profile with `rpk` command An rpk command that can use a profile supports the `--profile ` flag. When the `--profile` flag is set for an rpk command, the configuration for the cluster that rpk is interfacing with will be read from the named profile. See the [rpk commands reference](https://docs.redpanda.com/streaming/current/reference/rpk/) for commands that support profiles. ### [](#delete-profile)Delete profile To delete a profile, run [`rpk profile delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-delete/). ## [](#related-topics)Related topics For details about all commands for rpk profiles, see the [`rpk profile`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile/) reference page and its sub-pages. --- # Page 93: Docker Compose Labs **URL**: https://docs.redpanda.com/streaming/current/get-started/docker-compose-labs.md --- # Docker Compose Labs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Docker Compose Labs latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: docker-compose-labs page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: docker-compose-labs.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/docker-compose-labs.adoc description: Explore this collection of hands-on labs for deploying and testing Redpanda Streaming using Docker Compose. Whether you're a beginner looking to get started or an experienced user aiming to deepen your knowledge, these labs provide step-by-step instructions and practical examples to enhance your skills. page-git-created-date: "2024-12-12" page-git-modified-date: "2026-05-26" support-status: supported --- Explore this collection of hands-on labs for deploying and testing Redpanda Streaming using Docker Compose. Whether you’re a beginner looking to get started or an experienced user aiming to deepen your knowledge, these labs provide step-by-step instructions and practical examples to enhance your skills. - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) Combine Redpanda Shadowing for data replication with Envoy proxy for transparent client routing during disaster recovery. - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) Integrate Azure Entra ID with Redpanda and Redpanda Console for unified identity using OpenID Connect (OIDC). - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) Migrate data, schemas, and consumer offsets from a source Kafka cluster to a target Redpanda cluster using Redpanda Migrator. - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) Use Redpanda Shadowing to continuously migrate schemas and topic data from a real Confluent deployment into a Redpanda shadow cluster. - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) Manage and monitor applications in Redpanda Console using data from an example e-commerce application called owl shop. - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) Pair Redpanda with MinIO for Tiered Storage and write data in the Iceberg format to enable seamless analytics workflows on data in Redpanda topics. - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) Use Debezium to capture the changes made to a MySQL database in real time and stream them to Redpanda. - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) Use Debezium to capture the changes made to a Postgres database in real time and stream them to Redpanda. - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) Start three Redpanda brokers and Redpanda Console to start developing your application on Redpanda locally. - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) Start a single Redpanda broker and Redpanda Console to start developing your application on Redpanda locally. - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) Build a real-time Jira metrics pipeline using Redpanda Connect and Redpanda to track development performance, SLA compliance, and team productivity. --- # Page 94: Introduction to Redpanda **URL**: https://docs.redpanda.com/streaming/current/get-started/intro-to-events.md --- # Introduction to Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Introduction to Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: intro-to-events page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: intro-to-events.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/intro-to-events.adoc description: Learn about Redpanda event streaming. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-05-26" support-status: supported --- Distributed systems often require data and system updates to happen as quickly as possible. In software architecture, these updates can be handled with either messages or events. - With messages, updates are sent directly from one component to another to trigger an action. - With events, updates indicate that an action occurred at a specific time, and are not directed to a specific recipient. An event is simply a record of something changing state. For example, the event of a credit card transaction includes the product purchased, the payment, the delivery, and the time of the purchase. The event occurred in the purchasing component, but it also impacted the inventory, the payment processing, and the shipping components. In an event-driven architecture, all actions are defined and packaged as events to precisely identify individual actions and how they’re processed throughout the system. Instead of processing updates in consecutive order, event-driven architecture lets components process events at their own pace. This helps developers build fast and scalable systems. ## [](#what-is-redpanda)What is Redpanda? Redpanda is an event streaming platform: it provides the infrastructure for streaming real-time data. Producers are client applications that send data to Redpanda in the form of events. Redpanda safely stores these events in sequence and organizes them into topics, which represent a replayable log of changes in the system. Consumers are client applications that subscribe to Redpanda topics to asynchronously read events. Consumers can store, process, or react to the events. Redpanda decouples producers from consumers to allow for asynchronous event processing, event tracking, event manipulation, and event archiving. Producers and consumers interact with Redpanda using the Apache Kafka® API. ![Producers and consumers in a cluster](https://docs.redpanda.com/streaming/current/shared/_images/cluster.png) | Event-driven architecture (Redpanda) | Message-driven architecture | | --- | --- | | Producers send events to an event processing system (Redpanda) that acknowledges receipt of the write. This guarantees that the write is durable within the system and can be read by multiple consumers. | Producers send messages directly to each consumer. The producer must wait for acknowledgement that the consumer received the message before it can continue with its processes. | Event streaming lets you extract value out of each event by analyzing, mining, or transforming it for insights. You can: - Take one event and consume it in multiple ways. - Replay events from the past and route them to new processes in your application. - Run transformations on the data in real-time or historically. - Integrate with other event processing systems that use the Kafka API. ## [](#redpanda-differentiators)Redpanda differentiators Redpanda is less complex and less costly than any other commercial mission-critical event streaming platform. It’s fast, it’s easy, and it keeps your data safe. - Redpanda is designed for maximum performance on any data streaming workload. It can scale up to use all available resources on a single machine and scale out to distribute performance across multiple nodes. Built on C++, Redpanda delivers greater throughput and up to 10x lower p99 latencies than other platforms. This enables previously unimaginable use cases that require high throughput, low latency, and a minimal hardware footprint. - Redpanda is packaged as a single binary: it doesn’t rely on any external systems. It’s compatible with the Kafka API, so it works with the full ecosystem of tools and integrations built on Kafka. Redpanda can be deployed on bare metal, containers, or virtual machines in a data center or in the cloud. And Redpanda Console makes it easy to set up, manage, and monitor your clusters. Additionally, Tiered Storage lets you offload log segments to object storage in near real-time, providing long-term data retention and topic recovery. - Redpanda uses the [Raft consensus algorithm](https://raft.github.io/) throughout the platform to coordinate writing data to log files and replicating that data across multiple servers. Raft facilitates communication between the nodes in a Redpanda cluster to make sure that they agree on changes and remain in sync, even if a minority of them are in a failure state. This allows Redpanda to tolerate partial environmental failures and deliver predictable performance, even at high loads. - Redpanda provides data sovereignty. With the Bring Your Own Cloud (BYOC) offering, you deploy Redpanda in your own virtual private cloud, and all data is contained in your environment. Redpanda handles provisioning, monitoring, and upgrades, but you manage your streaming data without Redpanda’s control plane ever seeing it. ## [](#redpanda-streaming-versions)Redpanda Streaming versions You can deploy Redpanda in a self-hosted environment (Redpanda Streaming) or as a fully managed cloud service (Redpanda Cloud). Redpanda Streaming version numbers follow the convention AB.C.D, where AB is the two-digit year, C is the feature release, and D is the patch release. For example, version 22.3.1 indicates the first patch release on the third feature release of the year 2022. Patch releases include bug fixes and minor improvements, with no change to user-facing behavior. New and enhanced features are documented with each feature release. Redpanda Cloud releases on a continuous basis and uptakes Redpanda Streaming versions. ## [](#next-steps)Next steps - To spin up a Redpanda cluster to try it out, see [Redpanda Quickstart](https://docs.redpanda.com/streaming/current/get-started/quick-start/). - To learn more about Redpanda, see [How Redpanda Works](https://docs.redpanda.com/streaming/current/get-started/architecture/). - For information about a Redpanda Streaming deployment, see [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/). - For information about a Redpanda Cloud deployment, see [Redpanda Cloud Overview](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/). ## [](#suggested-reading)Suggested reading - [Upgrade your data streaming: a beginner’s guide to Redpanda](https://redpanda.com/blog/data-streaming-with-redpanda) - [Develop real-time apps faster and simpler with Redpanda](https://redpanda.com/blog/real-time-streaming-data-kafka-vs-redpanda?utm_assettype=blog&utm_assetname=redpanda_vs_kafka&utm_source=blog&utm_medium=content&utm_campaign=pillar_streaming_101) - [Free guide - Migrating from Kafka to Redpanda](https://go.redpanda.com/how-to-migrate-from-kafka-to-redpanda?utm_assettype=report&utm_assetname=migration_guide&utm_source=blog&utm_medium=content&utm_campaign=pillar_streaming_101) - [On-demand workshop - Get started with Redpanda](https://go.redpanda.com/virtual-workshop-april-2023?utm_assettype=workshop&utm_assetname=get_started_redpanda&utm_source=blog&utm_medium=content&utm_campaign=pillar_streaming_101) - [Redpanda University - A hands-on guide to Redpanda](https://university.redpanda.com/courses/hands-on-redpanda-getting-started) ## [](#suggested-videos)Suggested videos - [YouTube - Redpanda in a Nutshell (6:36 mins)](https://www.youtube.com/watch?v=FEVL8cLUFOc&ab_channel=RedpandaData) --- # Page 95: Introduction to rpk **URL**: https://docs.redpanda.com/streaming/current/get-started/intro-to-rpk.md --- # Introduction to rpk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Introduction to rpk latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: intro-to-rpk page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: intro-to-rpk.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/intro-to-rpk.adoc description: Learn about rpk and how to use it to interact with your Redpanda cluster. page-git-created-date: "2023-07-24" page-git-modified-date: "2025-08-21" support-status: supported --- The `rpk` command line interface tool is designed to manage your entire Redpanda cluster, without the need to run a separate script for each function, as with Apache Kafka. The `rpk` commands handle everything from configuring brokers to high-level general Redpanda tasks. For example, you can use `rpk` to monitor your cluster’s health, perform tuning, and implement access control lists (ACLs) and other security features. You can also use `rpk` to perform basic streaming tasks, such as creating topics, producing to topics, and consuming from topics. After you install `rpk`, you can use it to: - Manage Redpanda - Set up access control lists (ACLs) and other security features - Create topics, produce to topics, and consume from topics See also: - [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) - [rpk Profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/) - [Redpanda CLI Quickstart](https://docs.redpanda.com/streaming/current/get-started/rpk-quickstart/) ## [](#specify-configuration-properties)Specify configuration properties You can specify `rpk` command properties in the following ways: - Create an [`rpk profile`](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/). - Specify the appropriate flag on the command line. - Define the corresponding [environment variables](#environment-variables). Environment variable settings only last for the duration of a shell session. Command line flag settings take precedence over the corresponding environment variables, and environment variables take precedence over configuration file settings. If a required flag is not specified on the command line, Redpanda searches the environment variable. If the environment variable is not set, the value in the `rpk.yaml` configuration file is used, if that file is available, otherwise the value in the `redpanda.yaml` configuration file is used. > 💡 **TIP** > > If you specify `rpk` command properties in the configuration files or as environment variables, you don’t need to specify them again on the command line. ### [](#common-configuration-properties)Common configuration properties Every `rpk` command supports a set of common configuration properties. You can set one or more options in an `rpk` command by using the `-X` flag: ```bash rpk -X -X ``` Get a list of available options with `-X list`: ```bash rpk -X list ``` Or, get a detailed description about each option with `-X help`: ```bash rpk -X help ``` Every `-X` option can be translated into an environment variable by prefixing it with `RPK_` and replacing periods (`.`) with underscores (`_`). For example, the flag `tls.enabled` has the equivalent environment variable `RPK_TLS_ENABLED`. Some of the common configuration properties apply across all `rpk` commands as defaults. These default properties have keys with names starting with `globals`, and they’re viewable in `rpk -X list` and `rpk -X help`. For more details, see [`rpk -X options`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-x-options/). ### [](#environment-variables)Environment variables `rpk` supports environment variables through `RPK_*` that correspond to `-X` options. For a comprehensive list and configuration examples, see: - [rpk profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/) - Create and manage persistent configurations (recommended) - [rpk -X options](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-x-options/) - Complete configuration reference including environment variables ## [](#next-steps)Next steps - [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) - [rpk Command reference](https://docs.redpanda.com/streaming/current/reference/rpk/) --- # Page 96: Redpanda Licensing **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing.md --- # Redpanda Licensing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Licensing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/index.adoc page-git-created-date: "2024-12-03" page-git-modified-date: "2024-12-03" support-status: supported --- - [Redpanda Licenses and Enterprise Features](overview/) Learn about Redpanda licensing for Redpanda, Redpanda Console, and Redpanda Connect, available in both Community and Enterprise editions. Understand licensing requirements and how to access enterprise features with a valid license key. - [Add an Enterprise Edition License to Redpanda](add-license-redpanda/) Learn how to add an Enterprise Edition license to Redpanda Streaming. - [Add an Enterprise Edition License to Redpanda Console](add-license-console/) Learn how to add a license to Redpanda Console using one of these provided options. - [Check the Status of Licenses](check-status/) Explore the options for checking and monitoring the status of your Redpanda Enterprise Edition license. - [Disable Enterprise Features in Redpanda](disable-enterprise-features/) Disable specific enterprise features in Redpanda to ensure your cluster operates within the scope of the Community Edition without enterprise features. --- # Page 97: Add an Enterprise Edition License to Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-console.md --- # Add an Enterprise Edition License to Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Add an Enterprise Edition License to Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/add-license-console page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/add-license-console.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/add-license-console.adoc description: Learn how to add a license to Redpanda Console using one of these provided options. page-git-created-date: "2025-02-06" page-git-modified-date: "2025-02-06" support-status: supported --- Redpanda Console supports a number of options for adding an Enterprise Edition license. Choose the option that best fits your deployment: | Option | Description | Requirements | | --- | --- | --- | | Upload a license through the UI | Uses the UI to upload a license to Redpanda and Redpanda Console. | Redpanda Console must be connected to a Redpanda cluster. | | Load from the Redpanda cluster at startup (Linux or Kubernetes) | Automatically loads the license from the connected Redpanda cluster during startup, allowing centralized license management. | Redpanda Console must be connected to a Redpanda cluster that has a valid license installed. | | Configure a standalone license for Redpanda Console | Configures Redpanda Console with its own license. This option is best for deployments that are not connected to a Redpanda cluster. | You must add the license key to the Redpanda Console configuration file or an environment variable. | --- # Page 98: Add an Enterprise Edition License to Redpanda **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda.md --- # Add an Enterprise Edition License to Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Add an Enterprise Edition License to Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/add-license-redpanda/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/add-license-redpanda/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/add-license-redpanda/index.adoc description: Learn how to add an Enterprise Edition license to Redpanda Streaming. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- - [Add an Enterprise Edition License to Redpanda in Linux](linux/) Learn how to add or update a Redpanda Enterprise Edition license in a Linux environment. - [Add an Enterprise Edition License to Redpanda in Kubernetes](kubernetes/) Learn how to add or update a Redpanda Enterprise Edition license in a Kubernetes environment. - [Manage Enterprise Edition Licenses through Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/) Learn how to manage Enterprise Edition licenses in Redpanda Console. --- # Page 99: Add an Enterprise Edition License to Redpanda in Kubernetes **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/kubernetes.md --- # Add an Enterprise Edition License to Redpanda in Kubernetes > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Add an Enterprise Edition License to Redpanda in Kubernetes latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/add-license-redpanda/kubernetes page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/add-license-redpanda/kubernetes.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/add-license-redpanda/kubernetes.adoc description: Learn how to add or update a Redpanda Enterprise Edition license in a Kubernetes environment. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- To enable [enterprise features for Redpanda Streaming](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/), you must have an Enterprise Edition license. This guide outlines how to apply or update an Enterprise Edition license for Redpanda Streaming in a Kubernetes environment. ## [](#prerequisites)Prerequisites You must have an Enterprise Edition license. To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). ## [](#add-a-new-license)Add a new license Redpanda supports the following ways to apply a new license: - [Use a Kubernetes Secret to store the license](#secret). - [Provide the license string inline in your Helm values or manifest file](#inline). - [Use Redpanda Console to upload the license to Redpanda](#console). ### [](#secret)Use a Kubernetes Secret You can store the license in a Kubernetes Secret and reference it in your Helm values or manifest file. 1. Download your license file (`redpanda.license`) and create a Kubernetes Secret: ```bash kubectl create secret generic redpanda-license \ --from-file=license=./redpanda.license \ --namespace ``` This command creates a Kubernetes Secret named `redpanda-license` in the specified namespace, containing the license file. 2. Reference the Secret: #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: enterprise: licenseSecretRef: name: redpanda-license key: license ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `redpanda-license.yaml` ```yaml enterprise: licenseSecretRef: name: redpanda-license key: license ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values redpanda-license.yaml --reuse-values ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda \ --namespace \ --create-namespace \ --set enterprise.licenseSecretRef.name=redpanda-license \ --set enterprise.licenseSecretRef.key=license ``` ### [](#inline)Provide the license inline If you prefer to provide the license string directly, you can do so as follows: #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: enterprise: license: ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `redpanda-license.yaml` ```yaml enterprise: license: ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values redpanda-license.yaml --reuse-values ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda \ --namespace \ --create-namespace \ --set enterprise.license= ``` ### [](#console)Use Redpanda Console You can upload a license directly through Redpanda Console. See [Manage Enterprise Edition Licenses through Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/). ## [](#verify-a-license)Verify a license After adding or updating a license, you can use `rpk` to verify that the license was set. ```bash kubectl exec --namespace -c redpanda -- \ rpk cluster license info ``` This command will display the current license details, including the expiration date and whether any enterprise features are active. For example: LICENSE INFORMATION =================== Organization: redpanda Type: enterprise Expires: Oct 24 2027 > 📝 **NOTE** > > Redpanda blocks upgrades to new feature releases if enterprise features are active without a valid license. Ensure compliance by obtaining a license to maintain access to the latest features and updates. ## [](#update-an-existing-license)Update an existing license The process for updating a license depends on how it was originally applied: - [Update the Kubernetes Secret](#secret-update). - [Update the license string inline in your Helm values or manifest file](#inline-update). - [Use Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/) When a new license is uploaded, enterprise features in Redpanda Streaming are unlocked immediately without requiring a cluster restart. However, to unlock enterprise features in Redpanda Console, you must restart the Redpanda Console instance. ### [](#secret-update)Update the Kubernetes Secret If the license is provided through a Kubernetes Secret, follow these steps to update it: 1. Download the updated license file and overwrite the existing `redpanda.license` file. 2. Delete the existing Secret: ```bash kubectl delete secret redpanda-license --namespace ``` 3. Create a new Secret with a **new name** that contains the contents of the updated license: ```bash kubectl create secret generic redpanda-license-updated \ --from-file=license=./redpanda.license \ --namespace ``` 4. Update the Redpanda CRD to use the new Secret. `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: enterprise: licenseSecretRef: name: redpanda-license-updated key: license ``` 5. Apply the changes to the Redpanda CRD: ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` The Redpanda Operator updates the license without restarting the Repanda broker. 6. Check the status of the new license to make sure it was successfully applied: ```bash rpk cluster license info ``` The output displays the following details: Organization: Organization the license was generated for. Type: Type of license. Expires: Expiration date of the license. Version: License schema version. 7. If you use Redpanda Console, delete the Redpanda Console Pods to force Redpanda Console to reload the updated license: ```bash kubectl delete pod $(kubectl get pod --namespace | grep redpanda-console | awk '{print $1}') --namespace ``` ### [](#inline-update)Update the license inline If you applied the license inline, follow these steps to update it: 1. Modify the `enterprise.license` value with the new license string: #### Operator `redpanda-cluster.yaml` ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Redpanda metadata: name: redpanda spec: chartRef: {} clusterSpec: enterprise: license: ``` ```bash kubectl apply -f redpanda-cluster.yaml --namespace ``` #### Helm ##### --values `redpanda-license.yaml` ```yaml enterprise: license: ``` ```bash helm upgrade --install redpanda redpanda/redpanda --namespace --create-namespace \ --values redpanda-license.yaml --reuse-values ``` ##### --set ```bash helm upgrade --install redpanda redpanda/redpanda \ --namespace \ --create-namespace \ --set enterprise.license= ``` 2. Check the status of new license to make sure it was successfully applied: ```bash rpk cluster license info ``` The output displays the following details: Organization: Organization the license was generated for. Type: Type of license:. Expires: Expiration date of the license. Version: License schema version. 3. If you use Redpanda Console, delete the Redpanda Console Pods to force a reload of the updated license: ```bash kubectl delete pod $(kubectl get pod --namespace | grep redpanda-console | awk '{print $1}') --namespace ``` ## [](#next-steps)Next steps [Check the Status of Licenses](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/). --- # Page 100: Add an Enterprise Edition License to Redpanda in Linux **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/linux.md --- # Add an Enterprise Edition License to Redpanda in Linux > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Add an Enterprise Edition License to Redpanda in Linux latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/add-license-redpanda/linux page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/add-license-redpanda/linux.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/add-license-redpanda/linux.adoc description: Learn how to add or update a Redpanda Enterprise Edition license in a Linux environment. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-05-26" support-status: supported --- To enable [enterprise features for Redpanda Streaming](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/), you must have an Enterprise Edition license. This guide outlines how to apply or update an Enterprise Edition license for Redpanda Streaming in a Linux environment. ## [](#prerequisites)Prerequisites - You must have [`rpk` installed](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) and configured to connect to your Redpanda cluster. - You must have an Enterprise Edition license. To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). ## [](#add-a-new-license)Add a new license Redpanda supports the following ways to apply a new license: - [Provide the path to a file containing the license](#file). - [Pass the license string directly](#inline). - [Use Redpanda Console to upload the license to Redpanda](#console). ### [](#file)Apply the license using a license file If you have the license key stored in a file, you can apply it by specifying the file path: ```bash rpk cluster license set --path -X admin.hosts= ``` Replace the following placeholders: - `` with the path to your license file. - `` the Redpanda admin host and port ### [](#inline)Apply the license using an inline license string If you want to provide the license string directly, use the following command: ```bash rpk cluster license set ``` If neither the path nor the license string are provided, `rpk` looks for the license in `/etc/redpanda/redpanda.license`. ### [](#console)Use Redpanda Console You can upload a license directly through Redpanda Console. See [Manage Enterprise Edition Licenses through Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/). ## [](#verify-a-license)Verify a license After adding or updating a license, you can use `rpk` to verify that the license was set. ```bash rpk cluster license info ``` This command will display the current license details, including the expiration date and whether any enterprise features are active. For example: LICENSE INFORMATION =================== Organization: redpanda Type: enterprise Expires: Oct 24 2027 > 📝 **NOTE** > > Redpanda blocks upgrades to new feature releases if enterprise features are active without a valid license. Ensure compliance by obtaining a license to maintain access to the latest features and updates. ## [](#update-an-existing-license)Update an existing license To update an existing license, you can use one the following methods: - [rpk cluster license set](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-license-set/) - [Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/add-license/) When a new license is uploaded, enterprise features in Redpanda Streaming are unlocked immediately without requiring a cluster restart. However, to unlock enterprise features in Redpanda Console, you must restart the Redpanda Console instance. ## [](#next-steps)Next steps [Check the Status of Licenses](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/). ## [](#suggested-reading)Suggested reading [rpk cluster license set](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-license-set/) --- # Page 101: Check the Status of Licenses **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/check-status.md --- # Check the Status of Licenses > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Check the Status of Licenses latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/check-status/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/check-status/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/check-status/index.adoc description: Explore the options for checking and monitoring the status of your Redpanda Enterprise Edition license. page-git-created-date: "2024-12-17" page-git-modified-date: "2024-12-17" support-status: supported --- To track license expiration and ensure compliance with [Redpanda licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/), you can monitor the status of your Enterprise Edition license using a variety of tools, such as `rpk`, Redpanda Console, and observability tools such as Prometheus metrics. - [Check License Status and Feature Usage with `rpk`](rpk/) Learn how to monitor the status of an Enterprise Edition license and enterprise feature usage with `rpk`. - [Check License Status and Feature Usage with Redpanda Operator](redpanda-operator/) Learn how to monitor the status of an Enterprise Edition license and enterprise feature usage in Kubernetes using the Redpanda Operator. - [Check License Status in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/check-license/) Learn how to check the status of your Redpanda Enterprise Edition license using the Redpanda Console. This topic includes steps to check license details and understand license warnings. - [Monitor a License Using Metrics](metrics/) Learn how to monitor the expiration status of your Redpanda Enterprise license using the `redpanda_cluster_features_enterprise_license_expiry_sec` metric. Set up alerts and integrate dashboards to ensure proactive license management. --- # Page 102: Monitor a License Using Metrics **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/metrics.md --- # Monitor a License Using Metrics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Monitor a License Using Metrics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/check-status/metrics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/check-status/metrics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/check-status/metrics.adoc description: Learn how to monitor the expiration status of your Redpanda Enterprise license using the redpanda_cluster_features_enterprise_license_expiry_sec metric. Set up alerts and integrate dashboards to ensure proactive license management. page-git-created-date: "2024-12-17" page-git-modified-date: "2026-08-13" support-status: supported --- Redpanda exposes the [`redpanda_cluster_features_enterprise_license_expiry_sec`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_cluster_features_enterprise_license_expiry_sec) metric to help you track the time remaining before your license expires. Example metric output ```promql # HELP redpanda_cluster_features_enterprise_license_expiry_sec Seconds remaining until the enterprise license expires. # TYPE redpanda_cluster_features_enterprise_license_expiry_sec gauge redpanda_cluster_features_enterprise_license_expiry_sec 2592000 ``` In this example, the metric indicates 2,592,000 seconds (30 days) remaining until the license expires. ## [](#prerequisites)Prerequisites Redpanda exposes metrics in the [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/) through an HTTP `/metrics` endpoint. For help setting up Prometheus monitoring, see [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/) or [Monitor Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/). ## [](#set-up-alerts)Set up alerts To avoid violating the terms of the enterprise license, configure alerts when the expiration time falls below a certain threshold. For example, to set up alerts with Prometheus: ```yaml groups: - name: Redpanda License Alerts rules: - alert: RedpandaLicenseExpiringSoon expr: redpanda_cluster_features_enterprise_license_expiry_sec < 604800 for: 1h labels: severity: warning annotations: summary: "Redpanda license is expiring in less than 7 days" description: "The Redpanda Enterprise license will expire soon. Renew the license to avoid feature restrictions." ``` ## [](#integrate-with-dashboards)Integrate with dashboards Use visualization tools like Grafana to create a dashboard for Redpanda license monitoring: - Create a gauge to display the time remaining. - Add an alert panel for approaching expiration thresholds. ## [](#suggested-reading)Suggested reading - [Generate Grafana dashboard](https://docs.redpanda.com/streaming/current/manage/monitoring/#generate-grafana-dashboard). - [Generate Grafana dashboard in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/monitoring/k-monitor-redpanda/#generate-grafana-dashboard). --- # Page 103: Check License Status and Feature Usage with Redpanda Operator **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/redpanda-operator.md --- # Check License Status and Feature Usage with Redpanda Operator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Check License Status and Feature Usage with Redpanda Operator latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/check-status/redpanda-operator page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/check-status/redpanda-operator.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/check-status/redpanda-operator.adoc description: Learn how to monitor the status of an Enterprise Edition license and enterprise feature usage in Kubernetes using the Redpanda Operator. page-git-created-date: "2024-12-17" page-git-modified-date: "2024-12-17" support-status: supported --- If your Redpanda cluster is deployed in Kubernetes using the Redpanda Operator, you can monitor the license status from the Redpanda custom resource. The operator integrates license monitoring into the resource’s `status.conditions` and `status.license` fields, allowing you to get the license state with `kubectl`. These fields provide a detailed overview of your Redpanda license, including its status, expiration, and enterprise features currently in use. 1. Identify your Redpanda resources: ```bash kubectl get redpanda -A ``` Example output: NAMESPACE NAME LICENSE READY STATUS redpanda cluster Valid True Redpanda reconciliation succeeded The `License` field indicates whether the license is valid. Possible values include: - `Valid`: The license is valid. - `Expired`: The license has expired. - `Not Present`: No license is applied. 2. List detailed information about the license on a particular cluster: ```bash kubectl get redpanda -o jsonpath='{.status.license}' ``` Replace `` with the name of your Redpanda resource, such as `example-redpanda`. The output depends on the license status. It can include the following: > 📝 **NOTE** > > If the license is in violation, you must either obtain a valid license or disable the enterprise features in use to ensure compliance. For instructions, see [Disable Enterprise Features in Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/disable-enterprise-features/). ## [](#examples)Examples This section provides examples of what the operator reports depending on the license status. Valid license: ```json { "license": { "status": "valid", "violation": false, "inUseFeatures": ["partition_auto_balancing_continuous"], "organization": "Devex", "type": "enterprise", "expiration": "2025-10-11T00:00:00Z" } } ``` Without a license: ```json { "license": { "status": "not_present", "violation": false, "inUseFeatures": ["partition_auto_balancing_continuous"] } } ``` Expired license: ```json { "license": { "status": "expired", "violation": true, "inUseFeatures": ["partition_auto_balancing_continuous"], } } ``` ## [](#trigger-license-checks)Trigger license checks The Redpanda Operator performs a license check during reconciliation, which is triggered in the following cases: - If changes are made to the Redpanda resource or its status. - If changes are made to resources managed by the operator, such as the StatefulSet resources for brokers or Deployment resources for Redpanda Console. - If no changes occur, the operator reconciles every 10 hours (default cache resync interval). - If you force reconciliation by making a no-op change, such as adding an annotation, to the Redpanda resource. ## [](#suggested-reading)Suggested reading - [Monitor a License Using Metrics](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/metrics/) - [Check License Status and Feature Usage with `rpk`](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/rpk/) --- # Page 104: Check License Status and Feature Usage with rpk **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/rpk.md --- # Check License Status and Feature Usage with `rpk` > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Check License Status and Feature Usage with rpk latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/check-status/rpk page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/check-status/rpk.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/check-status/rpk.adoc description: Learn how to monitor the status of an Enterprise Edition license and enterprise feature usage with rpk. page-git-created-date: "2024-12-17" page-git-modified-date: "2026-08-20" support-status: supported --- To check the status of your license, use the `rpk cluster license info` command. Starting from version [24.2.8](https://github.com/redpanda-data/redpanda/releases/tag/v24.2.8), this command provides a detailed overview of your Redpanda license, including its status, expiration, and a list of currently used enterprise features. 1. Make sure that [`rpk` is installed](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) and configured to connect to your cluster’s Admin API endpoint. 2. Get the details about your cluster’s license: ```bash rpk cluster license info ``` The command displays the license information in a user-friendly format. The output depends on the license status. It can include the following: > 📝 **NOTE** > > If the license is in violation, you must either obtain a valid license or disable the enterprise features in use to ensure compliance. For instructions, see [Disable Enterprise Features in Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/disable-enterprise-features/). If the license is within 30 days of expiration, a warning is logged. `rpk` displays warnings when you execute `rpk` commands that use the Admin API in the following scenarios: - **License violation**: When enterprise features are enabled without a valid license. - **Trial expiration**: When enterprise features are enabled and a trial license expires in less than 15 days. - **Enterprise expiration**: When enterprise features are enabled and an enterprise license is expired. ## [](#examples)Examples This section provides examples of what `rpk` reports depending on the license status. Valid license: LICENSE INFORMATION =================== License status: valid License violation: false Enterprise features in use: \[partition\_auto\_balancing\_continuous\] Organization: Devex Type: enterprise Expires: Oct 11 2025 Without a license: LICENSE INFORMATION =================== License status: not\_present License violation: false Enterprise features in use: \[partition\_auto\_balancing\_continuous\] Expired license: WARNING: The following enterprise features are being used in your Redpanda cluster: \[partition\_auto\_balancing\_continuous\]. These features require a license. LICENSE INFORMATION =================== License status: expired License violation: true Enterprise features in use: \[partition\_auto\_balancing\_continuous\] ## [](#feature-names-in-the-output)Feature names in the output The `Enterprise features in use` list reports internal feature names. The following table maps each reported name to its documented feature: | Reported name | Documented feature | | --- | --- | | audit_logging | Audit Logging | | cloud_storage | Remote Read ReplicasTopic RecoveryTiered StorageWhole Cluster Restore (WCR) | | partition_auto_balancing_continuous | Continuous Data Balancing | | core_balancing_continuous | Continuous Intra-Broker Partition Balancing | | gssapi | Kerberos Authentication | | oidc | OAUTHBEARER/OIDC Authentication | | schema_id_validation | Server-Side Schema ID Validation | | rbac | Group-Based Access Control (GBAC)Role-Based Access Control (RBAC) | | fips | FIPS Compliance | | datalake_iceberg | Iceberg Topics | | leadership_pinning | Leader Pinning | | shadow_linking | Shadowing | | topic_deletion_disabled | Topic Deletion Control | ## [](#change-the-output-format)Change the output format Different output formats can be useful depending on your scenario. For example, if you are writing scripts or automating license monitoring, you may prefer the JSON format, as it’s easily parsed by tools like `jq` or integrated into monitoring systems. To get the license information in another format, use the `--format` flag: ```bash rpk cluster license info --format ``` Replace `` with one of the available formats, such as `json` or `yaml`. For more formats, see [rpk cluster license info](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-license-info/). ## [](#suggested-reading)Suggested reading - [Monitor a License Using Metrics](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/metrics/) - [Check License Status and Feature Usage with Redpanda Operator](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/redpanda-operator/) --- # Page 105: Disable Enterprise Features in Redpanda **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/disable-enterprise-features.md --- # Disable Enterprise Features in Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Disable Enterprise Features in Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/disable-enterprise-features page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/disable-enterprise-features.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/disable-enterprise-features.adoc description: Disable specific enterprise features in Redpanda to ensure your cluster operates within the scope of the Community Edition without enterprise features. page-git-created-date: "2024-12-03" page-git-modified-date: "2024-12-03" support-status: supported --- Enterprise features in Redpanda are available only in the Enterprise Edition and require a valid license. If your cluster has enterprise features enabled without a valid license, it is essential to either upload a valid license or disable these features to maintain compliance with Redpanda licensing terms. ## [](#prerequisites)Prerequisites Before you begin, consider the following: - Take a backup of your current configuration to allow rollback if needed. - Disabling enterprise features may affect cluster performance, security, or functionality. Test these changes in a staging environment before applying them to production. - If you need enterprise features, consider purchasing a valid license to continue using them. To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). ## [](#check-for-enterprise-features-in-violation)Check for enterprise features in violation To check the status of your license, use the `rpk cluster license info` command. This command provides a detailed overview of your Redpanda license, including whether your cluster has enterprise features enabled without a valid license. > 📝 **NOTE** > > This command reports license violations only if enterprise features in Redpanda are enabled without a valid license. It does not report license violations for enterprise features in [Redpanda Connect](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/#connect) or [Redpanda Console](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/#console). 1. Ensure that [`rpk` is installed](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) and configured to connect to your cluster’s Admin API endpoint. 2. Get the details about your cluster’s license: ```bash rpk cluster license info ``` If the `license violation` status is `true`, you must either obtain a valid license or disable the enterprise features in use to ensure compliance. ## [](#disable-enterprise-features)Disable enterprise features To disable specific enterprise features, refer to the following table: > 📝 **NOTE** > > These instructions apply to bare-metal deployments on Linux. If you are running Redpanda in a different environment, such as Kubernetes or Docker, the way you disable features may vary. | Feature | Action to Disable | | --- | --- | | Audit Logging | Set the cluster config audit_enabled to false:rpk cluster config set audit_enabled false | | Continuous Data Balancing | Set the cluster config partition_autobalancing_mode to node_add:rpk cluster config set partition_autobalancing_mode node_add | | Continuous Intra-Broker Partition Balancing (core_balancing_continuous) | Set the cluster config core_balancing_continuous to false:rpk cluster config set core_balancing_continuous false | | FIPS Compliance | Set the node config fips_mode to disabled:rpk node config set fips_mode disabled | | Kerberos authentication | Remove GSSAPI from the cluster config sasl_mechanisms:rpk cluster config set sasl_mechanisms | | Leader Pinning | Set default_leaders_preference to none:rpk cluster config set default_leaders_preference none | | OAUTHBEARER/OIDC authentication | Remove OIDC from the cluster config sasl_mechanisms and http_authentication:rpk cluster config set sasl_mechanisms rpk cluster config set http_authentication | | Remote Read Replicas | Set the cluster config cloud_storage_enable_remote_read to false:rpk cluster config set cloud_storage_enable_remote_read false | | Role-Based Access Control (RBAC) | Use rpk security role delete to delete all configured roles:rpk security role list rpk security role delete | | Server-Side Schema ID Validation | Set the cluster config enable_schema_id_validation to false:rpk cluster config set enable_schema_id_validation false | | Tiered Storage | Set the cluster config cloud_storage_enabled to false:rpk cluster config set cloud_storage_enabled false | ## [](#verify-the-license-status)Verify the license status When all required changes are made, confirm that the `license violation` status is now `false`. ```bash rpk cluster license info ``` ## [](#suggested-reading)Suggested reading For more information about licensing, see [Redpanda Licenses and Enterprise Features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/). --- # Page 106: Redpanda Licenses and Enterprise Features **URL**: https://docs.redpanda.com/streaming/current/get-started/licensing/overview.md --- # Redpanda Licenses and Enterprise Features > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Licenses and Enterprise Features latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: licensing/overview page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: licensing/overview.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/licensing/overview.adoc description: Learn about Redpanda licensing for Redpanda, Redpanda Console, and Redpanda Connect, available in both Community and Enterprise editions. Understand licensing requirements and how to access enterprise features with a valid license key. page-git-created-date: "2024-12-03" page-git-modified-date: "2026-08-20" support-status: supported --- Redpanda, Redpanda Console, and Redpanda Connect are available in community and enterprise editions. Each product has a single binary that supports both editions. - Redpanda Community Edition is free and source-available on GitHub: - [Redpanda](https://github.com/redpanda-data/redpanda) - [Redpanda Console](https://github.com/redpanda-data/console) - [Redpanda Connect](https://github.com/redpanda-data/connect) - Redpanda Enterprise Edition requires a license key and includes additional features. > 💡 **TIP: Ready to try Redpanda Enterprise Edition?** > > Go to the Redpanda Streaming quickstart to set up a local Redpanda cluster and try the Enterprise Edition free for up to 60 days. > 📝 **NOTE** > > Redpanda Cloud is a managed deployment of Redpanda Enterprise Edition. To learn more about Redpanda Cloud, see the [Redpanda Cloud Overview](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/). ## [](#community-edition)Community Edition The Community Edition is licensed with the Redpanda [Business Source License](https://github.com/redpanda-data/redpanda/blob/dev/licenses/bsl.md) (BSL). These core features are free and source-available. ### [](#restrictions-and-licensing-terms)Restrictions and licensing terms - Users cannot provide Redpanda as a commercial streaming or queuing service to others. - The BSL code converts to Apache 2.0 licensing four years after each code merge. ## [](#enterprise-edition)Enterprise Edition The Enterprise Edition is licensed with the [Redpanda Community License](https://github.com/redpanda-data/redpanda/blob/dev/licenses/rcl.md) (RCL). It includes the free features licensed under the Redpanda BSL, as well as enterprise features. You can also generate a [Trial license](#trial-license) for the Enterprise Edition. ### [](#trial-license)Trial license All new Redpanda clusters automatically receive a trial license valid for 30 days. To extend this trial for 30 days, use the [`rpk generate license`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-generate/rpk-generate-license/) command, or [request a new trial license key](https://redpanda.com/try-enterprise). This extended evaluation period begins after you apply the trial license to your cluster using the `--apply` flag. After this period expires, inactive enterprise features are disabled, and active features enter a restricted state. > ❗ **IMPORTANT** > > Only one trial license is permitted per email and business domain. Attempts to generate multiple trial licenses from the same email or business domain result in a license error. To get a permanent license, contact [Redpanda Sales](https://www.redpanda.com/contact). To generate and apply a 30-day trial license for Enterprise Edition: ```bash rpk generate license --apply First Name: Last Name: Company name: Business Email: ``` > 📝 **NOTE** > > To activate the trial license, you _must_ apply it to your cluster using the `--apply` flag. The trial license is saved in your working directory or the specified path, based on the `--path` flag. Example output: ```bash Successfully saved license to "/home//code/rp/redpanda/src/go/rpk/redpanda.license". Upload this license in Redpanda Console, or run: rpk cluster license set --path /home//rp/redpanda/src/go/rpk/redpanda.license This license expires on 2025-05-07. For more information, see: xref:streaming:get-started:licensing/overview.adoc#license-keys[License keys] ``` ### [](#license-keys)License keys Enterprise features require an Enterprise Edition license key, sometimes called enterprise license, license key, or license. - **Redpanda**: Starting with version 24.3, new Redpanda clusters automatically receive a trial license that’s valid for 30 days, allowing unrestricted use of enterprise features. This evaluation period begins when the cluster is created for the first time. After this period expires, inactive enterprise features are disabled, and active features enter a restricted state. To extend your trial license see [Trial license](#trial-license), [request a new trial license key](https://redpanda.com/try-enterprise), or [upgrade to Redpanda Enterprise](https://redpanda.com/upgrade). Redpanda blocks upgrades to new feature releases if enterprise features are active without a valid license. Ensure compliance by obtaining a license to maintain access to the latest features and updates. > ❗ **IMPORTANT** > > To avoid startup issues with Redpanda Console when a trial or Enterprise license expires, use Redpanda Console v2.8.3 or later with clusters running Redpanda 24.3 or later. - **Redpanda Connect**: To evaluate enterprise features in Redpanda Connect, you must [apply a trial license key](https://docs.redpanda.com/connect/get-started/licensing/#apply-a-license-key-to-redpanda-connect). After the 30-day evaluation period, you are blocked from using enterprise connectors unless you [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade). ### [](#self-managed)Redpanda Enterprise Edition features The following table lists the enterprise features for Redpanda and how Redpanda behaves upon license expiration when each enterprise feature is enabled. - **Upon license expiration**: - The cluster continues to operate without data loss, but the further use of enterprise features is restricted (see the [table 1](#redpanda-enterprise-features)). - Configuration of enterprise features remains unchanged, allowing you to add a new license and continue using enterprise features as before expiration. For instructions, see [Add an Enterprise Edition License to Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) - **After license expiration**: You cannot enable enterprise features without a valid license. | Feature | Description | Behavior Upon Expiration | | --- | --- | --- | | Audit Logging | Records detailed logs of cluster activities for compliance and monitoring. | Read access to the audit log topic is denied, but logging continues. | | Cloud Topics | A Redpanda topic type that uses durable object storage as the primary backing store instead of local disk replication. | New Cloud Topics cannot be created. Existing Cloud Topics cannot be modified, including adding or modifying partitions. Major upgrades are blocked when in a violation state. | | Continuous Data Balancing | Automatically balances partitions across a cluster to optimize resource use and performance.Continuous Data Balancing is enabled by default for all new clusters with valid licenses. | Continuous balancing is disabled, reverting to node_add setting that balances partitions only after a broker is added to the cluster. | | Continuous Intra-Broker Partition Balancing (core_balancing_continuous) | Balances partition replicas across CPU cores in an individual broker to optimize disk space usage.Continuous Intra-Broker Partition Balancing is enabled by default for all new clusters with a valid license. | Continuous Intra-Broker Partition Balancing is disabled. | | Fetch Read Coalescing | Shares one read result across concurrent fetches of the same data, reducing read CPU and fetch-response memory under high consumer fan-out. | It will switch off automatically, similar to the other balancing features. | | FIPS Compliance | Enables compliance with FIPS security standards for cryptography. | No change. | | Group-Based Access Control (GBAC) | Manages permissions using OIDC group memberships for ACLs and role assignments. | ACLs with Group: principals cannot be created. Existing group ACLs continue to be evaluated and can be deleted. | | Iceberg Topics | Enables Iceberg integration for Redpanda topics. | Topics cannot be created or modified with the redpanda.iceberg.mode property. | | Kerberos Authentication | Provides secure Kerberos-based authentication. | No change. | | Leader Pinning | Specifies the set of availability zones where the leaders of partitions of a given topic should be located. | Leader Pinning is disabled on all topics. | | OAUTHBEARER/OIDC Authentication | Allows for OAUTHBEARER and OpenID Connect (OIDC) authentication. | No change. | | Remote Read Replicas | Enables remote clusters to read data stored in object storage for disaster recovery. | Remote Read Replica topics cannot be created or modified. | | Role-Based Access Control (RBAC) | Manages user roles and permissions within the cluster. | Roles and ACLs associated with roles cannot be created or modified. Role deletion is allowed. | | Schema Registry Authorization | Manages ACLs for Redpanda Schema Registry resources within the cluster. | You can no longer enable schema_registry_enable_authorization, nor can you create or modify schema ACLs. | | Server-Side Schema ID Validation | Validates schema IDs server-side to ensure schema compatibility. With schema ID validation, records associated with unregistered schemas are detected and dropped by a broker rather than a consumer. | Topics with schema validation settings cannot be created or modified. | | Shadowing | Provides enterprise-grade disaster recovery through asynchronous, offset-preserving replication between distinct Redpanda clusters for cross-region data protection. | New shadow links cannot be created. Existing shadow links continue operating and can be updated. | | Tiered Storage | Enables data storage in cloud object storage for long-term retention and retrieval. | Topics cannot be created or modified to enable Tiered Storage features. Additional partitions cannot be added to topics with Tiered Storage properties enabled. | | Topic Deletion Control (delete_topic_enable) | When set to false, prevents all users, including superusers, from deleting topics through the Kafka DeleteTopics API. This property provides a cluster-wide safety guard against accidental topic deletion. | Topic deletion reverts to enabled (true). | | Topic Recovery | Allows restoring a single topic from Tiered Storage using remote recovery properties. | You cannot create topics with the redpanda.remote.recovery=true property or perform topic recovery operations. To proceed, add a valid license to the target cluster. Without a valid license, topic recovery is blocked. | | Whole Cluster Restore (WCR) | Enables the recovery of cluster data from a source cluster’s snapshot. | If the license is expired, you cannot perform WCR. To proceed, add a valid license to the target cluster. If the source cluster has an expired license, the target cluster inherits the restriction until a valid license is applied. | ### [](#console)Redpanda Console enterprise features The following enterprise features for Redpanda Console are activated with a valid Enterprise Edition license key: | Feature | Description | Restrictions Without Valid License | | --- | --- | --- | | Authentication for Redpanda Console | Enables authentication for Redpanda Console, including secure login through OIDC and OAuth 2.0 SSO. | All pages are redirected to the license expiration landing page, and all other access is restricted. | | Authorization (RBAC) for Redpanda Console | Manages user roles and permissions for accessing features within Redpanda and Redpanda Console. | All pages are redirected to the license expiration landing page, and all other access is restricted. | | Debug bundle generation | Enables generating and downloading debug bundles in Redpanda Console for comprehensive cluster diagnostics. | All pages are redirected to the license expiration landing page, and all other access is restricted. | | Reassign Partitions | Enables the ability to move partitions between brokers. | All pages are redirected to the license expiration landing page, and all other access is restricted. | ### [](#connect)Redpanda Connect enterprise features The following enterprise features are available with a valid Enterprise Edition license. | Feature | Description | Restrictions Without Valid License | | --- | --- | --- | | Allow or deny lists | Limit the Redpanda Connect components that users can run within data pipelines on a Redpanda Connect instance. | No change. | | Enterprise connectors | Additional inputs, outputs, and processors available only to enterprise customers. | All enterprise connectors are blocked. | | FIPS compliance for Redpanda Connect | Run Redpanda Connect using a FIPS-compliant version of rpk, the Redpanda command-line interface (CLI). | No change. | | Redpanda Connect configuration service | A configuration block that you can use to send logs and status events to a topic on a Redpanda cluster. | No change. | | Secrets management | Retrieve secrets values from a remote system, such as a secret management solution, without setting environment variables. | No change. | ### [](#operator)Redpanda Operator enterprise features The Redpanda Operator gates the following Kubernetes features on a valid Enterprise Edition license. In both cases, the operator reads the license from a Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart. | Feature | Description | Restrictions Without Valid License | | --- | --- | --- | | Redpanda Connect pipelines (Pipeline resource) (beta) | Run Redpanda Connect pipelines declaratively through the operator’s Connect controller. Requires a license that includes the Redpanda Connect product. | When the license is missing, expired, invalid, or does not include the Redpanda Connect product, the operator stops creating and updating pipeline workloads: new Pipeline resources report License: False in their status and are never deployed, and existing pipelines keep running with their last-applied configuration but no longer receive updates. Applying a valid license resumes reconciliation automatically, without recreating any Pipeline resources. | | Stretch Clusters (StretchCluster resource) | A single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability, managed by the multicluster Redpanda Operator. | The multicluster operator requires a valid license to start, so stretch clusters cannot be deployed or managed without one. On a running stretch cluster, the operator continuously reports license state through the StretchCluster resource’s LicenseValid status condition and status.licenseStatus, including the expiration date and any enterprise features in violation. When the license expires, the stretch cluster keeps running and its brokers' enterprise features follow the restrictions in table 1. | ## [](#manage-licenses-for-redpanda)Manage licenses for Redpanda Redpanda offers multiple ways to manage your license depending on your deployment method. Proper license management ensures access to advanced enterprise features and avoids restrictions when licenses are invalid or expired. See [Add an Enterprise Edition License to Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/). ## [](#manage-licenses-for-redpanda-console)Manage licenses for Redpanda Console Redpanda Console offers two methods for applying or updating a license, depending on your setup: - If you use Redpanda Console without connecting it to a Redpanda cluster, you can [configure the license through the local configuration file or environment variables](https://docs.redpanda.com/streaming/current/console/config/enterprise-license/). This method allows you to add a license for Redpanda Console independently of the Redpanda cluster. - If Redpanda Console is connected to a Redpanda cluster, you can [upload a license through the Redpanda Console UI](https://docs.redpanda.com/streaming/current/console/ui/add-license/). This method allows you to manage and update licenses for both Redpanda Console and the connected Redpanda cluster. ## [](#manage-licenses-for-redpanda-connect)Manage licenses for Redpanda Connect Redpanda Connect offers multiple ways to apply or update your license. See [Apply a license key to Redpanda Connect](https://docs.redpanda.com/connect/get-started/licensing/#apply-a-license-key-to-redpanda-connect). ## [](#manage-licenses-for-the-redpanda-operator)Manage licenses for the Redpanda Operator The Redpanda Operator reads its license from a Kubernetes Secret referenced by the `enterprise.licenseSecretRef` values in the operator Helm chart: ```bash kubectl create secret generic redpanda-license \ --from-file=license= \ --namespace helm upgrade --install redpanda-operator redpanda/operator \ --namespace \ --set enterprise.licenseSecretRef.name=redpanda-license \ --set enterprise.licenseSecretRef.key=license ``` This operator-level license gates the operator’s own enterprise features, such as the [multicluster operator and the Connect controller](#operator). The clusters that the operator manages are licensed independently: set the `enterprise.license` or `enterprise.licenseSecretRef` field in a Redpanda or StretchCluster resource, and the operator applies that license to the cluster for you. See [Add an Enterprise Edition License to Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/kubernetes/). ## [](#next-steps)Next steps - [Add an Enterprise Edition License to Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/add-license-redpanda/) - [Check the Status of Licenses](https://docs.redpanda.com/streaming/current/get-started/licensing/check-status/) --- # Page 107: Partner Integrations **URL**: https://docs.redpanda.com/streaming/current/get-started/partner-integration.md --- # Partner Integrations > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Partner Integrations latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: partner-integration page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: partner-integration.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/partner-integration.adoc description: Learn about Redpanda integrations built and supported by our partners. page-git-created-date: "2024-05-21" page-git-modified-date: "2026-08-13" support-status: supported --- Learn about Redpanda integrations built and supported by our partners. | Partner | Description | More information | | --- | --- | --- | | Superstream | Superstream optimizes and improves Redpanda (and other Kafka platforms) for cost reduction, increased reliability, and improved visibility. | Superstream for Redpanda | | Aklivity Zilla | Zilla is a multi-protocol proxy that abstracts Redpanda for non-native clients, such as browsers and IoT devices, by exposing Redpanda topics using user-defined REST, Server-Sent Events (SSE), MQTT, or gRPC API entry points. | Modern Eventing with CQRS, Redpanda and Zilla | | Bytewax | Bytewax is an open source framework and distributed stream processing engine in Python. | Enriching streaming data with Bytewax and Redpanda | | ClickHouse | ClickHouse is a high-performance, column-oriented SQL database management system (DBMS) for online analytical processing (OLAP). | Building an OLAP database with ClickHouse and Redpanda | | Conduktor | Conduktor provides simple, flexible, and powerful tooling for Kafka developers and infrastructure. | Conduktor & Redpanda: Best of breed Kafka experience | | Decodable | Decodable is a real-time data processing platform powered by Apache Flink and Debezium. | Decodable + Redpanda | | ElastiFlow | ElastiFlow captures and analyzes flow and SNMP data to provide detailed insights into network performance and security. | Leveraging Redpanda for Enhanced Network Observability: ElastiFlow Integration | | Materialize | Materialize is a data warehouse purpose-built for operational workloads where an analytical data warehouse would be too slow, and a stream processor would be too complicated. | Ingesting data from Redpanda with Materialize | | PeerDB | PeerDB provides a fast, simple, and cost-effective way to replicate data from Postgres to data warehouses, queues and storage. | Quickstart guide | | Pinecone | Pinecone is a vector database for building accurate and performant AI applications at scale. The Pinecone connector for Redpanda Connect provides a production-ready integration from many existing data sources through simple YAML configuration. | Redpanda Connect integration | | RisingWave | RisingWave is a distributed SQL streaming database that enables simple, efficient, and reliable processing of streaming data. | Ingesting data from Redpanda with Risingwave | | Timeplus | Timeplus is a stream processor that provides powerful end-to-end capabilities, leveraging the open source streaming engine Proton. | Realizing low latency streaming analytics with Timeplus and Redpanda | | Tinybird | Tinybird is a data platform for data and engineering teams to solve complex real-time, operational, and user-facing analytics use cases at any scale. | Building a complete IoT backend with Redpanda and Tinybird | | Quix | Quix is a complete platform for building, deploying, and monitoring stream processing pipelines in Python. | Integrating Redpanda with Quix | | Yugabyte | YugabyteDB is an open-source, distributed SQL database that combines the capabilities of relational databases with the scalability of NoSQL systems. | How to Integrate Yugabyte CDC Connector with Redpanda | ## [](#how-to-contribute-to-this-page)How to contribute to this page To request a partner integration with Redpanda Data, reach out to ([partners@redpanda.com](mailto:partners@redpanda.com\)). Provide a link to your product documentation or a blogpost explaining how your product integrates with Redpanda. After meeting these requirements, you can [contribute to this page](https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/partner-integration.adoc). --- # Page 108: Quickstart for Redpanda Self-Managed Data Platform **URL**: https://docs.redpanda.com/streaming/current/get-started/quick-start.md --- # Quickstart for Redpanda Self-Managed Data Platform > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Quickstart for Redpanda Self-Managed Data Platform latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: quick-start page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: quick-start.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/quick-start.adoc description: Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-08-13" support-status: supported --- Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. Redpanda Data Platform is a modern streaming platform, compatible with Kafka APIs, designed for speed, simplicity, and efficiency. In this quickstart, you: - Deploy a three-broker Redpanda Streaming cluster. - Explore streaming data in Redpanda Console. - Learn the basics of streaming with the Redpanda CLI (`rpk`). - Deploy a streaming pipeline with Redpanda Connect. - Learn how to extend your trial license for an additional 30 days. > 📝 **NOTE** > > This quickstart uses Docker to run Redpanda, which is only for development and testing purposes. For production deployments, see the [Linux deployment guides](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/) or the [Kubernetes deployment guides](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/). To download the Redpanda binary, see [GitHub](https://github.com/redpanda-data/redpanda/releases/latest). > > **Looking for a managed solution?** You can also get started quickly with a hosted Redpanda cluster by signing up for [Redpanda Cloud](https://cloud.redpanda.com). ## [](#enterprise-features)Enterprise features The Enterprise Edition of Redpanda adds advanced capabilities to help scale, manage, optimize, and secure your cluster. All new Redpanda clusters include a built-in 30-day Enterprise Edition license, so you can evaluate enterprise features. Some features highlighted in this quickstart require an enterprise license: - **Redpanda Console Authentication**: Securely control login access to Redpanda Console. - **Audit Logging**: Track and monitor all user actions in a Redpanda topic. - **Continuous Data Balancing**: Automatically balance data across brokers to optimize performance. - **Tiered Storage**: Lower storage costs by offloading log segments to object storage. Your trial license also includes additional enterprise features. For more information, see [Redpanda Licenses and Enterprise Features](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/). > 💡 **TIP: Want a longer trial?** > > [Sign up for an extended trial license](https://cloud.redpanda.com/try-enterprise). ## [](#prerequisites)Prerequisites You need the following tools and settings before proceeding: - **tar (GNU tar)**: On most Linux distributions, GNU tar is preinstalled. On macOS, install GNU tar using Homebrew and ensure it’s used instead of the BSD version: ```bash brew install gnu-tar ``` - **Docker Compose**: For installation instructions, see the Docker Compose documentation: [Docker Compose documentation](https://docs.docker.com/compose/install/). To verify you have it installed, run: ```bash docker compose version ``` You should see the installed Docker Compose version. - **Docker Desktop (macOS/Windows)**: Ensure that Docker Desktop’s file sharing is enabled for the directory where you plan to run the quickstart. If your working directory isn’t shared, containers won’t be able to see your files. - **Available memory**: Make sure your machine has at least 4 GB of free memory before starting the containers. If your resource limits are too low, some containers may hang or fail to start. > 📝 **NOTE: Windows users** > > If you’re on **Windows**, use a compatible terminal that supports Unix-like commands (for example, PowerShell or Windows Terminal). Alternatively, you can use Windows Subsystem for Linux (WSL). For WSL installation instructions, see: [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install). ## [](#deploy-redpanda)Deploy Redpanda To download, extract, and start Redpanda in Docker, run: ```bash mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose && docker compose up -d (3) ``` | 1 | Create and navigate to the redpanda-quickstart directory. | | --- | --- | | 2 | Download and extract the archive. Explore the downloaded filesdocker-compose.ymlname: redpanda-quickstart-multi-broker networks: redpanda_network: driver: bridge volumes: redpanda-0: null redpanda-1: null redpanda-2: null minio: null services: ################## # Redpanda Brokers # ################## redpanda-0: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 # Address the broker advertises to clients that connect to the Kafka API. # Use the internal addresses to connect to the Redpanda brokers # from inside the same Docker network. # Use the external addresses to connect to the Redpanda brokers # from outside the Docker network. - --advertise-kafka-addr internal://redpanda-0:9092,external://localhost:19092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082 # Address the broker advertises to clients that connect to the HTTP Proxy. - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://localhost:18082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081 # Redpanda brokers use the RPC API to communicate with each other internally. - --rpc-addr redpanda-0:33145 - --advertise-rpc-addr redpanda-0:33145 # Mode dev-container uses well-known configuration properties for development in containers. - --mode dev-container # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system. - --smp 1 - --default-log-level=info image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 container_name: redpanda-0 # Sets the username and password of the bootstrap SCRAM superuser # See https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#bootstrap-a-user-account environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-0:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 18081:18081 - 18082:18082 - 19092:19092 - 19644:9644 healthcheck: test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"] interval: 10s timeout: 15s retries: 10 depends_on: minio: condition: service_healthy redpanda-1: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:29092 - --advertise-kafka-addr internal://redpanda-1:9092,external://localhost:29092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:28082 - --advertise-pandaproxy-addr internal://redpanda-1:8082,external://localhost:28082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:28081 - --rpc-addr redpanda-1:33145 - --advertise-rpc-addr redpanda-1:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 container_name: redpanda-1 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-1:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 28081:28081 - 28082:28082 - 29092:29092 - 29644:9644 healthcheck: test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"] interval: 10s timeout: 15s retries: 10 depends_on: - redpanda-0 - minio redpanda-2: command: - redpanda - start - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:39092 - --advertise-kafka-addr internal://redpanda-2:9092,external://localhost:39092 - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:38082 - --advertise-pandaproxy-addr internal://redpanda-2:8082,external://localhost:38082 - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:38081 - --rpc-addr redpanda-2:33145 - --advertise-rpc-addr redpanda-2:33145 - --mode dev-container - --smp 1 - --default-log-level=info - --seeds redpanda-0:33145 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 container_name: redpanda-2 environment: RP_BOOTSTRAP_USER: "superuser:secretpassword" volumes: - redpanda-2:/var/lib/redpanda/data - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml networks: - redpanda_network ports: - 38081:38081 - 38082:38082 - 39092:39092 - 39644:9644 healthcheck: test: ["CMD", "rpk", "cluster", "info", "-X", "user=superuser", "-X", "pass=secretpassword"] interval: 10s timeout: 15s retries: 10 depends_on: - redpanda-0 - minio #################### # Redpanda Console # #################### console: container_name: redpanda-console image: docker.redpanda.com/redpandadata/console:v3.11.0 networks: - redpanda_network entrypoint: /bin/sh command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml && /app/console' volumes: - ./config:/tmp/config/ environment: CONFIG_FILEPATH: ${CONFIG_FILEPATH:-/tmp/config.yml} CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] sasl: enabled: true impersonateUser: true schemaRegistry: enabled: true urls: ["http://redpanda-0:8081","http://redpanda-1:8081","http://redpanda-2:8081"] authentication: impersonateUser: true redpanda: adminApi: enabled: true urls: ["http://redpanda-0:9644","http://redpanda-1:9644","http://redpanda-2:9644"] authentication: basic: username: superuser password: secretpassword impersonateUser: false console: # Configures Redpanda Console to fetch topic documentation from GitHub and display it in the UI. # See https://docs.redpanda.com/current/console/config/topic-documentation/ # # Disabled by default so local `docker compose up` works without a # GitHub token. redpanda-data/docs is private: with topic docs enabled # and an empty token, Console fails cloning the repo and never binds # :8080. To enable locally (or in CI), set: # REDPANDA_TOPIC_DOCS_ENABLED=true # REDPANDA_GITHUB_TOKEN= # (CI sets both via the actions bot token - see test-docs.yml.) topicDocumentation: enabled: ${REDPANDA_TOPIC_DOCS_ENABLED:-false} git: enabled: true repository: url: https://github.com/redpanda-data/docs branch: main baseDirectory: tests/docker-compose basicAuth: enabled: true username: x-access-token password: ${REDPANDA_GITHUB_TOKEN} authentication: jwtSigningKey: vazxnT+ZHtxKslK6QlDGovcYnSjTk/lKMmZ+mHrBVE+YdVDkLgSuP6AszAKe9Gvq basic: enabled: true authorization: roleBindings: - roleName: admin users: - loginType: basic name: superuser ports: - 8080:8080 depends_on: redpanda-0: condition: service_healthy createtopic: condition: service_completed_successfully registerschema: condition: service_completed_successfully deploytransform: condition: service_completed_successfully #################### # Redpanda Connect # #################### connect: container_name: redpanda-connect image: docker.redpanda.com/redpandadata/connect networks: - redpanda_network entrypoint: /bin/sh depends_on: redpanda-0: condition: service_healthy command: -c 'echo "$$CONNECT_CFG_FILE" > /tmp/connect.yml; /redpanda-connect -c /tmp/connect.yml' environment: # This Redpanda Connect configuration creates fake data, # processes it, and writes the output to a set of topics. # # Input: # - Uses Redpanda Connect's generate input to generate fake data. # See https://docs.redpanda.com/redpanda-connect/components/inputs/generate/ # Pipeline: # - Bloblang mapping to batch each input and map 1 message to 'logins' # topic, and a random number (1-3) of messages to 'transaction' topic # - Unarchive processor to parse the JSON array and extract each # element into its own message. # See https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/ # Output: # - kafka_franz output to write the messages to the Redpanda brokers. # See https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz/ CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser #################### # rpk container to create the edu-filtered-domains topic # # See https://docs.redpanda.com/current/reference/rpk/rpk-topic/rpk-topic-create/ #################### createtopic: command: - topic - create - edu-filtered-domains # Idempotent on restart when volumes are kept; without this, TOPIC_ALREADY_EXISTS # exits 1 and blocks Console / deploytransform / consumergroup # (service_completed_successfully). - --if-not-exists - -X user=superuser - -X pass=secretpassword - -X brokers=redpanda-0:9092 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to register the schema # # See https://docs.redpanda.com/current/manage/schema-reg/schema-reg-api/ #################### registerschema: command: - registry - schema - create - transactions - --schema - /etc/redpanda/transactions-schema.json - -X user=superuser - -X pass=secretpassword - -X registry.hosts=redpanda-0:8081 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 # Mount the local directory that contains your schema to the container. volumes: - ./transactions-schema.json:/etc/redpanda/transactions-schema.json networks: - redpanda_network depends_on: redpanda-0: condition: service_healthy #################### # rpk container to deploy a consumer group # # See https://docs.redpanda.com/current/reference/rpk/rpk-topic/rpk-topic-consume/ #################### consumergroup: command: - topic - consume - transactions - --group - transactions-consumer - -X user=superuser - -X pass=secretpassword - -X brokers=redpanda-0:9092 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 networks: - redpanda_network depends_on: createtopic: condition: service_completed_successfully deploytransform: condition: service_completed_successfully #################### # rpk container to deploy the pre-built data transform # # See https://docs.redpanda.com/current/develop/data-transforms/deploy/ #################### deploytransform: command: - transform - deploy - --file=/etc/redpanda/regex.wasm - --name=regex - --input-topic=logins - --output-topic=edu-filtered-domains - --var=PATTERN="[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.edu" - --var=MATCH_VALUE=true - -X user=superuser - -X pass=secretpassword - -X admin.hosts=redpanda-0:9644 image: docker.redpanda.com/redpandadata/redpanda:v26.2.2 volumes: - ./transform/regex.wasm:/etc/redpanda/regex.wasm networks: - redpanda_network depends_on: createtopic: condition: service_completed_successfully #################### # MinIO for Tiered Storage # # See https://min.io/ # # NOTE: MinIO is included in this quickstart for development and evaluation purposes only. # It is not supported for production deployments of Redpanda. # # For production environments, use one of the supported object storage providers: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/requirements/#object-storage-providers-for-tiered-storage #################### minio: container_name: minio image: minio/minio:RELEASE.2025-05-24T17-08-30Z command: server --console-address ":9001" /data ports: - 9000:9000 - 9001:9001 environment: MINIO_ROOT_USER: minio MINIO_ROOT_PASSWORD: redpandaTieredStorage7 MINIO_SERVER_URL: "http://minio:9000" MINIO_REGION_NAME: local MINIO_DOMAIN: minio volumes: - minio:/data networks: redpanda_network: aliases: - redpanda.minio healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/ready"] interval: 10s timeout: 5s retries: 3 mc: depends_on: minio: condition: service_healthy image: minio/mc:RELEASE.2025-05-21T01-59-54Z container_name: mc networks: - redpanda_network environment: - AWS_ACCESS_KEY_ID=minio - AWS_SECRET_ACCESS_KEY=redpandaTieredStorage7 - AWS_REGION=local entrypoint: > /bin/sh -c " until (/usr/bin/mc alias set minio http://minio:9000 minio redpandaTieredStorage7) do echo '...waiting...' && sleep 1; done; /usr/bin/mc mb minio/redpanda; /usr/bin/mc policy set public minio/redpanda; tail -f /dev/null "rpk-profile.yaml# This file configures `rpk` to connect to a remote Redpanda cluster running in the same local network as `rpk`. # Configuration for connecting to the Kafka API of the Redpanda cluster. kafka_api: # SASL (Simple Authentication and Security Layer) settings for authentication. sasl: user: superuser # The username used for authentication password: secretpassword # The password associated with the username mechanism: scram-sha-256 # Authentication mechanism; SCRAM-SHA-256 provides secure password-based authentication # List of Kafka brokers in the Redpanda cluster. # These brokers ensure high availability and fault tolerance for Kafka-based communication. brokers: - 127.0.0.1:19092 # Broker 1: Accessible on localhost, port 19092 - 127.0.0.1:29092 # Broker 2: Accessible on localhost, port 29092 - 127.0.0.1:39092 # Broker 3: Accessible on localhost, port 39092 # Configuration for connecting to the Redpanda Admin API. # The Admin API allows you to perform administrative tasks such as managing configurations, monitoring, and scaling. admin_api: # List of Admin API endpoints for managing the cluster. addresses: - 127.0.0.1:19644 # Admin API for Broker 1: Accessible on localhost, port 19644 - 127.0.0.1:29644 # Admin API for Broker 2: Accessible on localhost, port 29644 - 127.0.0.1:39644 # Admin API for Broker 3: Accessible on localhost, port 39644 # Configuration for connecting to the Redpanda Schema Registry API. schema_registry: # List of Schema Registry API endpoints. addresses: - 127.0.0.1:18081 # Schema Registry API for Broker 1: Accessible on localhost, port 18081 - 127.0.0.1:28081 # Schema Registry API for Broker 2: Accessible on localhost, port 28081 - 127.0.0.1:38081 # Schema Registry API for Broker 3: Accessible on localhost, port 38081bootstrap.yml# ================================================================= # This file defines initial cluster properties for a Redpanda cluster. # Some of these settings are intended for quickstart development and evaluation # and are not suitable for production environments. # # For more information on bootstrap files, see: # https://docs.redpanda.com/current/deploy/deployment-option/self-hosted/manual/production/production-deployment/#configure-a-bootstrap-file # ================================================================= # # Enable SASL authentication for the Kafka and Admin APIs. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#admin_api_require_auth admin_api_require_auth: true # At least one superuser is required to be able to create other SASL users # https://docs.redpanda.com/current/reference/properties/cluster-properties/#superusers superusers: - superuser # https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_sasl enable_sasl: true # Allow topics to be created on first access. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#auto_create_topics_enabled auto_create_topics_enabled: true # Enable data transforms. # https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/ data_transforms_enabled: true # Enable audit logging (enterprise feature). # https://docs.redpanda.com/current/manage/audit-logging/ audit_enabled: true # Enable Tiered Storage (enterprise feature). # https://docs.redpanda.com/current/manage/tiered-storage/ cloud_storage_enabled: true cloud_storage_region: local cloud_storage_access_key: minio cloud_storage_secret_key: redpandaTieredStorage7 cloud_storage_api_endpoint: minio cloud_storage_api_endpoint_port: 9000 cloud_storage_disable_tls: true cloud_storage_bucket: redpanda # Forces segments to be uploaded to Tiered Storage faster for the purposes of the quickstart # https://docs.redpanda.com/current/reference/properties/object-storage-properties/#cloud_storage_segment_max_upload_interval_sec cloud_storage_segment_max_upload_interval_sec: 60 # Continuous Data Balancing (enterprise feature) continuously monitors your node and rack availability and disk usage. This enables self-healing clusters that dynamically balance partitions, ensuring smooth operations and optimal cluster performance. # https://docs.redpanda.com/current/manage/cluster-maintenance/continuous-data-balancing/ partition_autobalancing_mode: continuous # Enable Redpanda to collect consumer group metrics. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_consumer_group_metrics enable_consumer_group_metrics: - "group" - "partition" - "consumer_lag" # Lower the interval for the quickstart # https://docs.redpanda.com/current/reference/properties/cluster-properties/#consumer_group_lag_collection_interval_sec consumer_group_lag_collection_interval_sec: 60 # Enable Redpanda to collect host metrics. # https://docs.redpanda.com/current/reference/properties/cluster-properties/#enable_host_metrics enable_host_metrics: true | | 3 | Start Docker Compose. | When the containers are running, you have: - A Redpanda cluster with a 30-day free trial license. - A running data streaming pipeline generating data into the `logins` and `transactions` topics. - A Wasm data transform processing data from `logins` into `edu-filtered-domains`. - A secured instance of Redpanda Console with login authentication. ## [](#explore-redpanda-console)Explore Redpanda Console Redpanda Console is a developer-friendly web UI for managing and debugging your Redpanda cluster and your applications. This section provides practical examples and scenarios to help you understand how to leverage Redpanda Console for different use cases, including data observability, Redpanda management, access control, and connectivity. Redpanda Console was deployed as part of the `docker-compose.yml` file and is running locally on port 8080. ### [](#log-in-to-redpanda-console)Log in to Redpanda Console To start using Redpanda Console, you first need to log in with the credentials of a user. Redpanda is configured with a bootstrap SCRAM user called `superuser`, so you can log into Redpanda Console as that user (enterprise feature). This user has admin access to Redpanda and Redpanda Console. 1. Open your web browser and go to [http://localhost:8080/login](http://localhost:8080/login). 2. Enter the following credentials: - Username: ```none superuser ``` - Password: ```none secretpassword ``` Leave the SASL mechanism as **SCRAM-SHA-256**. You should now see an overview of your cluster’s status, health, and brokers. ![overview](https://docs.redpanda.com/streaming/current/console/_images/overview.png) To view details about a specific broker, click **View** at the end of the row in the **Broker Details** table. ![broker overview](https://docs.redpanda.com/streaming/current/console/_images/broker-overview.png) See also: [Authentication in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/security/authentication/). ### [](#view-topics-and-filter-messages)View topics and filter messages This quickstart deployment comes with two pre-configured topics, `logins` and `transactions`, actively receiving data from a Redpanda Connect pipeline. These topics simulate real-world data flows, enabling you to explore and interact with Redpanda Console features. You can filter messages in topics using several methods: - Text search: Search for specific text within the message. - Partition offset filters: Set the start and end offsets to narrow down the message range. - JavaScript filters: Use JavaScript function bodies to create complex filters that run server-side, allowing you to filter through topics with millions of records. Suppose you’re asked to find all transactions related to the `.edu` domain. You can use a JavaScript filter to display only messages that include email addresses in that domain. 1. In the menu, click **Topics**. 2. Click the **transactions** topic. When the topic’s page opens, the **Messages** tab is selected by default. 3. Click **Add filter** > **JavaScript Filter**. 4. Give your filter a name, such as "Find .edu domains". ![js filter](https://docs.redpanda.com/streaming/current/console/_images/js-filter.png) 5. Replace the default JavaScript code with the following: ```js return value.email.includes(".edu"); ``` 6. Click **Save** to apply the filter. You should see only messages that include the specific domain. By default, the filter runs on the newest 50 messages. You can run the filter on older messages by changing the start offset. For example, to start from the oldest messages, select **Beginning**. See also: [Filter Messages with JavaScript in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/programmable-push-filters/). ### [](#explore-schema-registry)Explore Schema Registry On the **Schema Registry** page, you see an overview of your schemas. You can create, manage, and inspect your schemas without leaving Redpanda Console. ![schema reg](https://docs.redpanda.com/streaming/current/console/_images/schema-reg.png) See also: [Use Schema Registry in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/schema-reg/). ### [](#configure-access-control-for-redpanda)Configure access control for Redpanda Managing access control lists (ACLs) can be complex, but Redpanda Console simplifies this with an intuitive interface. This quickstart deployment includes a bootstrapped superuser with full administrative privileges. The superuser has unrestricted access to all topics, consumer groups, transactions, and cluster-level operations. By default, the credentials for the superuser are: - Username: `superuser` - Password: `secretpassword` For improved security and role-based access control (RBAC), you should create additional users with restricted permissions as needed. On the **Security** page, you can: - Create, view, and edit ACLs for topics, consumer groups, transactions, and the entire cluster. - Quickly understand what each principal (user) is authorized to do. Suppose you’re onboarding a new team member who needs access to specific topics. You can use Redpanda Console to ensure users have the right permissions to perform their tasks. 1. In the menu, click **Security**. 2. On the **Users** tab, click **Create user**. 3. Enter "Sasha" in the username field. 4. Enter "sashapassword" in the password field. 5. Click **Create**. ![user](https://docs.redpanda.com/streaming/current/console/_images/user.png) Click **Done**, and you see a new user called Sasha. This user has no permissions yet. To set permissions on the `transactions` topic: 1. From **Security** in the left navigation menu, go to **Roles**. 2. Click **Create role**, enter "transaction-managers" as the role name, and click **Create**. The role’s detail page opens with empty **ACLs** and **Principals** sections. 3. Under **ACLs**, click **\+ Add ACL** and define a rule: - **Resource Type**: Topic - **Pattern Type**: Literal - **Resource Name**: `transactions` - **Operation**: All - **Permission**: Allow - **Host**: `*` Then click **Add ACL**. 4. Under **Principals**, in the **Add a principal** search, select **Sasha**. The assignment takes effect immediately. Now Sasha has full access only to the topic called transactions. To test these permissions, use `rpk` to connect to Redpanda as the user Sasha. 1. Try to access the `logins` topic as Sasha: ```bash docker exec -it redpanda-0 rpk topic describe logins \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 ``` You are not authorized to view the topic. SUMMARY ======= NAME logins PARTITIONS 0 ERROR TOPIC\_AUTHORIZATION\_FAILED: Not authorized to access topics: \[Topic authorization failed.\] CONFIGS ======= config response contained error: TOPIC\_AUTHORIZATION\_FAILED: Not authorized to access topics: \[Topic authorization failed.\] 2. Now try to access the `transactions` topic as Sasha: ```bash docker exec -it redpanda-0 rpk topic describe transactions \ -X user=Sasha \ -X pass=sashapassword \ -X sasl.mechanism=SCRAM-SHA-256 \ -X brokers=redpanda-0:9092 ``` You have access to this topic. Example output SUMMARY ======= NAME transactions PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT\_CONFIG compression.type producer DEFAULT\_CONFIG delete.retention.ms -1 DEFAULT\_CONFIG flush.bytes 262144 DEFAULT\_CONFIG flush.ms 100 DEFAULT\_CONFIG initial.retention.local.target.bytes -1 DEFAULT\_CONFIG initial.retention.local.target.ms -1 DEFAULT\_CONFIG max.message.bytes 1048576 DEFAULT\_CONFIG message.timestamp.type CreateTime DEFAULT\_CONFIG redpanda.iceberg.delete true DEFAULT\_CONFIG redpanda.iceberg.mode disabled DEFAULT\_CONFIG redpanda.leaders.preference none DEFAULT\_CONFIG redpanda.remote.delete true DEFAULT\_CONFIG redpanda.remote.read true DEFAULT\_CONFIG redpanda.remote.write true DEFAULT\_CONFIG retention.bytes -1 DEFAULT\_CONFIG retention.local.target.bytes -1 DEFAULT\_CONFIG retention.local.target.ms 86400000 DEFAULT\_CONFIG retention.ms 604800000 DEFAULT\_CONFIG segment.bytes 134217728 DEFAULT\_CONFIG segment.ms 1209600000 DEFAULT\_CONFIG write.caching true DEFAULT\_CONFIG ### [](#explore-data-transforms)Explore Data transforms Data transforms let you run common data streaming tasks on the Redpanda broker, like filtering, scrubbing, and transcoding. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. This quickstart deployment comes with one transform function called `regex` running in your cluster. Its job is to find records in the `logins` topic that contain email addresses with the `.edu` domain and add those to a new topic called `edu-filtered-domains`. In the menu, click **Transforms**. On the **Transforms** page, you see your transform. You can use Redpanda Console to manage and monitor your transforms. > 📝 **NOTE** > > The source code for the data transform function is in the [`docker-compose/transform/transform.go` file](https://docs.redpanda.com/streaming/current/test-resources/_attachments/docker-compose/transform/transform.go) that you downloaded. This file is commented to explain how it works. ![transforms](https://docs.redpanda.com/streaming/current/console/_images/transforms.png) See also: - [Manage Data Transforms in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/data-transforms/) - [Data Transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/) ### [](#view-audit-logs)View audit logs Audit logs provide a record of all user actions. You can use these logs to track changes, troubleshoot issues, and maintain compliance with your organization’s security policies. 1. In the menu, click **Topics**. 2. Click the **Show internal topics** checkbox. 3. Click the **\_redpanda.audit\_log** topic in the table. ![audit logs](https://docs.redpanda.com/streaming/current/console/_images/audit-logs.png) See also: [Audit Logging](https://docs.redpanda.com/streaming/current/manage/audit-logging/). ## [](#start-streaming)Start streaming To start building a basic streaming application, you can use the Redpanda CLI (`rpk`) to create a topic, produce messages to it, and consume messages from it. Each Redpanda broker comes preinstalled with `rpk`, so you can use it inside one of the Redpanda broker’s Docker containers. For simplicity, this quickstart uses the bootstrapped superuser (`superuser`) to perform all operations. The superuser has full administrative privileges, making it convenient for setup and exploration. However, to enhance security in a production environment, Redpanda Data recommends creating users with restricted permissions for each task. To use `rpk` inside the Redpanda broker’s Docker container: 1. Get information about the cluster: ```bash docker exec -it redpanda-0 rpk cluster info -X user=superuser -X pass=secretpassword ``` 2. Create a topic called **chat-room**: ```bash docker exec -it redpanda-0 rpk topic create chat-room \ --replicas 3 \ (1) --topic-config redpanda.remote.read=true \ (2) --topic-config redpanda.remote.write=true \ (3) -X user=superuser \ -X pass=secretpassword ``` | 1 | Set a replication factor of 3 to replicate the topic across all 3 brokers. This replication factor provides high availability and data durability. For more details, see Choose the replication factor. | | --- | --- | | 2 | Enable remote reads for this topic to read offloaded records from object storage. | | 3 | Enable remote writes for this topic to offload older records to object storage. For more details, see Use Tiered Storage.Output:TOPIC STATUS chat-room OK | 3. Produce a message to the topic: ```bash docker exec -it redpanda-0 rpk topic produce chat-room -X user=superuser -X pass=secretpassword ``` 4. Enter a message, then press Enter: ```text Pandas are fabulous! ``` Example output: Produced to partition 0 at offset 0 with timestamp 1663282629789. 5. Press Ctrl+C to finish producing messages to the topic. 6. Consume one message from the topic: ```bash docker exec -it redpanda-0 rpk topic consume chat-room --num 1 -X user=superuser -X pass=secretpassword ``` Your message is displayed along with its metadata: ```json { "topic": "chat-room", "value": "Pandas are fabulous!", "timestamp": 1663282629789, "partition": 0, "offset": 0 } ``` ### [](#connect-to-the-cluster-externally)Connect to the cluster externally It’s often more practical to connect to a remote cluster from your local machine with `rpk`. The `docker-compose.yml` file configured the containers to expose ports on your localhost, so you can communicate with the cluster outside the Docker network. To do so, download `rpk` on your local machine and configure it to connect to your cluster on the exposed ports. 1. Install `rpk` on your local machine: #### Linux > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. ##### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ##### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### macOS 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` 2. Create a profile to configure `rpk` to connect to your cluster: ```bash rpk profile create quickstart --from-profile rpk-profile.yaml ``` > 📝 **NOTE** > > The profile is configured in the [`docker-compose/rpk-profile.yaml` file](https://docs.redpanda.com/streaming/current/test-resources/_attachments/docker-compose/rpk-profile.yaml) that you downloaded. This file is commented to explain how it works. 3. Get information about the cluster to test the connection: ```bash rpk cluster info ``` > 📝 **NOTE** > > The Redpanda broker returns a list of all broker addresses, so `rpk` can communicate with all brokers directly. Each broker returns its configured `advertise-*` address that matches the port to which the initial connection has been made. You can see examples of these addresses in the Docker Compose file that you downloaded. ### [](#view-offloaded-data-in-tiered-storage)View offloaded data in Tiered Storage Tiered Storage keeps local disk usage stable and performance consistent by offloading older data to object storage. You can also use Tiered Storage to recover clusters or topics, mount and unmount topics from a cluster, and more. > 📝 **NOTE** > > Redpanda supports Amazon S3, Azure ADLS, and Google GCS as object storage providers. For this quickstart, Redpanda uses MinIO as the object storage provider. **MinIO is not supported for production deployments**. 1. Check the size of remote and local disks for the `chat-room` topic you created earlier: ```bash rpk cluster logdirs describe --topics chat-room ``` BROKER DIR TOPIC PARTITION SIZE ERROR 0 /var/lib/redpanda/data chat-room 0 611 0 remote://redpanda chat-room 0 319 1 /var/lib/redpanda/data chat-room 0 611 1 remote://redpanda chat-room 0 319 2 /var/lib/redpanda/data chat-room 0 611 2 remote://redpanda chat-room 0 319 - Local storage: `/var/lib/redpanda/data` holds recent data. - Remote storage: `remote://redpanda` holds offloaded data. 2. Open MinIO at [http://localhost:9001/browser](http://localhost:9001/browser) to view your data stored in the S3-compatible object store. Login credentials: - Username: ```none minio ``` - Password: ```none redpandaTieredStorage7 ``` See also: [Use Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/). ## [](#deploy-a-pipeline-with-redpanda-connect)Deploy a pipeline with Redpanda Connect Redpanda Connect is a powerful tool that allows you to build and manage data streaming pipelines. It supports hundreds of connectors for various systems, enabling seamless data ingestion, transformation, and routing between different sources and sinks. Using declarative YAML configuration files and the `rpk` CLI, you can quickly set up and deploy complex data pipelines in Redpanda. 1. Deploy the example pipeline: ```bash rpk connect run ../generate-profiles.yaml ``` This pipeline is configured to generate fake user profiles and produce them to the `profiles` topic in your Redpanda cluster. It’ll run until you stop it. > 📝 **NOTE** > > The pipeline configuration is in the [`docker-compose/generate-profiles.yaml` file](https://docs.redpanda.com/streaming/current/test-resources/_attachments/docker-compose/generate-profiles.yaml) that you downloaded. This file is commented to explain how it works. 2. After a few seconds, stop the pipeline by pressing Ctrl+C. 3. Consume 10 messages from the `profiles` topic to confirm that the pipeline produced data to that topic. ```bash rpk topic consume profiles --num 10 ``` ## [](#extend-your-30-day-trial)Extend your 30-day trial Your initial trial license is valid for 30 days. To extend it by another 30 days: 1. [Sign up for a new trial license](https://cloud.redpanda.com/try-enterprise). 2. Upload the new license to your running cluster using [`rpk cluster license set`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-license-set/): ```bash rpk cluster license set ``` Replace the `` placeholder with your license. This command applies the new license to your Redpanda cluster, extending your access to enterprise features for another 30 days. 3. Restart Redpanda Console so that it can fetch the new license from the Redpanda brokers: ```bash docker restart redpanda-console ``` > 📝 **NOTE** > > You can extend the trial license only once. > 💡 **TIP: Ready to go into production?** > > [Upgrade to Redpanda Enterprise](https://www.redpanda.com/upgrade). See also: [Redpanda Licensing](https://docs.redpanda.com/streaming/current/get-started/licensing/). ## [](#clean-up)Clean up To reset your environment, shut down the running processes and delete the containers: ```bash docker compose down ``` To delete the volumes along with all your cluster data: ```bash docker compose down -v ``` ## [](#customize-the-docker-quickstart)Customize the Docker quickstart To customize this quickstart, you can configure Redpanda, Redpanda Console, or Redpanda Connect using the provided Docker Compose file. This allows you to tailor the services to fit your specific requirements. ### [](#redpanda-broker-properties)Redpanda broker properties To configure the Redpanda services with [broker configuration properties](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/), pass properties with the `--set` option in the `redpanda start` command in your Docker Compose file. **Example:** `docker-compose.yml` ```yaml redpanda-0: (1) command: (2) - redpanda - start - --set pandaproxy_client.retries=6 (3) ``` | 1 | The service name in Docker Compose. | | --- | --- | | 2 | Overrides the default command for the Redpanda container. | | 3 | Sets the pandaproxy_client.retries broker configuration property to 6. | ### [](#redpanda-cluster-properties)Redpanda cluster properties Cluster-level configurations are defined in the `bootstrap.yml` file. This file is included in your Docker Compose setup. It contains essential settings that apply to the entire Redpanda cluster. You can find all cluster properties in [Cluster Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/). **Example:** `docker-compose.yml` ```yaml redpanda: volumes: - ./bootstrap.yml:/etc/redpanda/.bootstrap.yaml (1) ``` | 1 | Mounts the bootstrap.yml file from your local parent directory to the container’s /etc/redpanda/ directory. | | --- | --- | ### [](#redpanda-console-configuration)Redpanda Console configuration Redpanda Console configuration is managed with the environment variable `CONSOLE_CONFIG_FILE`. You can specify the path to your configuration file in the Docker Compose file. **Example:** `docker-compose.yml` ```yaml console: environment: CONSOLE_CONFIG_FILE: | # Configure a connection to the Redpanda cluster # See https://docs.redpanda.com/current/console/config/connect-to-redpanda/ kafka: brokers: ["redpanda-0:9092"] ``` ### [](#redpanda-connect-configuration)Redpanda Connect configuration Existing data pipelines are configured using the `CONNECT_CFG_FILE` environment variable in the `connect` service. This environment variable contains YAML configuration settings for Redpanda Connect. **Example:** `docker-compose.yml` ```yaml connect: environment: CONNECT_CFG_FILE: | input: generate: interval: 1s mapping: | let first_name = fake("first_name") let last_name = fake("last_name") root.user_id = counter() root.name = $$first_name + " " + $$last_name root.email = ($$first_name.slice(0,1) + $$last_name + "@" + fake("domain_name")).lowercase() root.ip = fake("ipv4") root.login_time = now() pipeline: processors: - mapping: | root = range(0, random_int(min:2, max:4)).map_each(cust -> this) - unarchive: format: "json_array" - mapping: | if batch_index() == 0 { meta topic = "logins" root = this } else { meta topic = "transactions" root.user_id = this.user_id root.email = this.email root.index = batch_index() - 1 root.product_url = fake("url") root.price = fake("amount_with_currency") root.timestamp = now() } output: kafka_franz: seed_brokers: [ "redpanda-0:9092" ] topic: $${! metadata("topic") } sasl: - mechanism: SCRAM-SHA-256 password: secretpassword username: superuser ``` To create new data pipelines, run `rpk connect run`, specifying the path to your own YAML configuration file: ```bash rpk connect run ``` ## [](#next-steps)Next steps [Contact Redpanda](https://www.redpanda.com/contact) to discuss using Redpanda Enterprise Edition in production. - [Try more examples in Redpanda Labs](https://docs.redpanda.com/labs/) - [Learn more about enterprise features for Redpanda](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/) - [Deploy for development or production](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/) ## [](#suggested-reading)Suggested reading - [Redpanda Connect Documentation](https://docs.redpanda.com/connect/home/) - [rpk Commands](https://docs.redpanda.com/streaming/current/reference/rpk/) - [Introduction to Redpanda Console](https://docs.redpanda.com/streaming/current/console/) - Docker images - [Docker images for Redpanda Console](https://hub.docker.com/r/redpandadata/console/tags) - [Docker images for Redpanda](https://hub.docker.com/r/redpandadata/redpanda) --- # Page 109: Redpanda Quickstarts **URL**: https://docs.redpanda.com/streaming/current/get-started/quickstarts.md --- # Redpanda Quickstarts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Quickstarts latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: quickstarts page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: quickstarts.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/quickstarts.adoc description: Get started with Redpanda using these hands-on tutorials. Explore features that demonstrate how Redpanda can power your streaming applications. page-git-created-date: "2024-12-03" page-git-modified-date: "2024-12-03" support-status: supported --- - [Quickstart for Redpanda Self-Managed Data Platform](https://docs.redpanda.com/streaming/current/get-started/quick-start/) Learn how to quickly start working with a local Redpanda cluster that comes with a free 30-day license for Enterprise Edition. You can also extend the trial license for a further 30 days. - [Redpanda CLI Quickstart](https://docs.redpanda.com/streaming/current/get-started/rpk-quickstart/) Quickly become familiar with `rpk` commands for basic Redpanda tasks, including creating, producing to, describing, and deleting topics, as well as consuming records and managing consumer groups. - [Docker Compose Labs](https://docs.redpanda.com/streaming/current/get-started/docker-compose-labs/) Explore this collection of hands-on labs for deploying and testing Redpanda Streaming using Docker Compose. Whether you're a beginner looking to get started or an experienced user aiming to deepen your knowledge, these labs provide step-by-step instructions and practical examples to enhance your skills. --- # Page 110: What’s New **URL**: https://docs.redpanda.com/streaming/current/get-started/release-notes.md --- # What’s New > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: What’s New latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: release-notes/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: release-notes/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/release-notes/index.adoc description: Summary of new features and updates. page-git-created-date: "2025-04-07" page-git-modified-date: "2025-04-07" support-status: supported --- - [What’s New in Redpanda](redpanda/) Summary of new features and updates in this Redpanda release. - [What’s New in the Redpanda Operator](operator/) Summary of new features and updates in the Redpanda Operator. - [What’s New in the Helm Charts](helm-charts/) Summary of new features and updates in the Helm charts for Redpanda and Redpanda Console. --- # Page 111: What’s New in the Helm Charts **URL**: https://docs.redpanda.com/streaming/current/get-started/release-notes/helm-charts.md --- # What’s New in the Helm Charts > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: What’s New in the Helm Charts latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: release-notes/helm-charts page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: release-notes/helm-charts.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/release-notes/helm-charts.adoc description: Summary of new features and updates in the Helm charts for Redpanda and Redpanda Console. page-git-created-date: "2025-04-07" page-git-modified-date: "2026-03-31" support-status: supported --- This topic includes new content and significant changes in the Redpanda and Redpanda Console Helm charts. For a complete list of all updates, see: - [Changelog for the Redpanda chart](https://github.com/redpanda-data/redpanda-operator/blob/v26.2.2/charts/redpanda/CHANGELOG.md). - [Changelog for the Redpanda Console chart](https://github.com/redpanda-data/redpanda-operator/blob/v26.2.2/charts/console/CHANGELOG.md). See also: - [What’s New in Redpanda](https://docs.redpanda.com/streaming/current/get-started/release-notes/redpanda/) - [Kubernetes Compatibility](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/) - [Upgrade Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/) ## [](#console-chart-v26-1-x)Console chart v26.1.x [Changelog](https://github.com/redpanda-data/redpanda-operator/blob/release/v26.1.x/charts/console/CHANGELOG.md). ### [](#prometheus-servicemonitor)Prometheus ServiceMonitor The Console Helm chart now supports deploying a Prometheus `ServiceMonitor` using `monitoring.enabled`, `monitoring.scrapeInterval`, and `monitoring.labels`. See [Prometheus ServiceMonitor](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy/#prometheus-servicemonitor). ## [](#redpanda-chart-v26-1-x)Redpanda chart v26.1.x [Changelog](https://github.com/redpanda-data/redpanda-operator/blob/release/v26.1.x/charts/redpanda/CHANGELOG.md). ### [](#config-watcher-sidecar-resource-configuration)Config-watcher sidecar resource configuration You can now configure explicit CPU and memory resource requests and limits for the config-watcher sidecar using `statefulset.sideCars.configWatcher.resources`. This is required in namespaces that enforce LimitRange or ResourceQuota policies. See [Configure config-watcher sidecar resources](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-resources/#config-watcher). --- # Page 112: What’s New in the Redpanda Operator **URL**: https://docs.redpanda.com/streaming/current/get-started/release-notes/operator.md --- # What’s New in the Redpanda Operator > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: What’s New in the Redpanda Operator latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: release-notes/operator page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: release-notes/operator.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/release-notes/operator.adoc description: Summary of new features and updates in the Redpanda Operator. page-git-created-date: "2025-04-07" page-git-modified-date: "2026-07-28" support-status: supported --- This topic includes new content and significant changes in the Redpanda Operator. For a complete list of all updates, see the [Redpanda Operator changelog](https://github.com/redpanda-data/redpanda-operator/blob/v26.2.2/operator/CHANGELOG.md). See also: - [What’s New in Redpanda](https://docs.redpanda.com/streaming/current/get-started/release-notes/redpanda/) - [Kubernetes Compatibility](https://docs.redpanda.com/streaming/current/upgrade/k-compatibility/) - [Upgrade Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/) ## [](#redpanda-operator-v26-2-x)Redpanda Operator v26.2.x [Changelog](https://github.com/redpanda-data/redpanda-operator/blob/release/v26.2.x/operator/CHANGELOG.md) ### [](#redpanda-connect-pipelines-beta)Redpanda Connect pipelines (beta) The Redpanda Operator introduces the Pipeline custom resource, which runs [Redpanda Connect](https://docs.redpanda.com/connect/home/) pipelines as declaratively managed Kubernetes workloads. You define the pipeline configuration in the resource, and the operator deploys it, lints it before startup, rolls it automatically on configuration and credential changes, and reports health through status conditions. A pipeline can reference a Redpanda resource and a User resource, and the operator injects broker addresses, TLS material, and SASL credentials into the rendered configuration for you. Pipelines also implement the Kubernetes scale subresource, so you can scale them with `kubectl scale`, a HorizontalPodAutoscaler, or KEDA, including on consumer group lag and on the metrics Redpanda Connect emits. The Pipeline resource is in beta and requires an Enterprise Edition license. Enable it with the operator chart’s `connectController.enabled=true` setting. - [Run Redpanda Connect Pipelines in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-connect-pipelines/): Deploy pipelines, connect them to clusters, and inject secrets and cloud IAM identities. - [Autoscale Redpanda Connect Pipelines](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-autoscale-connect-pipelines/): Step-by-step HPA and KEDA autoscaling. ### [](#stretch-clusters)Stretch Clusters The Redpanda Operator introduces Stretch Clusters: a single logical Redpanda cluster distributed across multiple Kubernetes clusters for multi-region high availability. Operators in each Kubernetes cluster coordinate through Raft consensus, and the new StretchCluster and RedpandaBrokerPool custom resources define the cluster and its per-cluster broker groups. The release also adds the [`rpk k8s` plugin](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s/), with [`rpk k8s multicluster`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster/) commands to bootstrap TLS certificates and peer configuration (`bootstrap`), check deployment health (`status`), and collect diagnostics bundles (`bundle`). For deployment steps, see [Deploy a Stretch Cluster on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). ### [](#gateway-api-support-for-redpanda-console-beta)Gateway API support for Redpanda Console (beta) The Console custom resource and the Redpanda Console Helm chart can expose Redpanda Console through a Kubernetes Gateway API HTTPRoute as an alternative to Ingress. You reference an existing Gateway in `gateway.parentRefs`, and the operator or chart renders and manages the HTTPRoute for you. You can enable only one of `gateway` or `ingress` at a time. See [Expose Redpanda Console with the Gateway API](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy/#expose-console-gateway-api). ### [](#safer-rolling-restarts-with-per-broker-health-probes)Safer rolling restarts with per-broker health probes In Redpanda Operator v26.2.1 or later, the operator gates each step of a rolling restart with Redpanda’s per-broker restart probes instead of a single cluster-wide health check. Before restarting a broker, the operator confirms that the restart cannot lose data acknowledged with `acks=1`, make partitions unavailable, or block `acks=all` produce requests. After each restart, the operator waits until the broker reports that it has fully resynced before restarting the next one, because a Pod can pass its readiness probe while the broker is still replaying partition state. The probes require Redpanda 25.1 or later. On earlier versions, the operator falls back to the cluster-wide health check. See [How the Redpanda Operator gates each restart](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-rolling-restart/#operator-restart-gates). ### [](#broker-pool-resources-by-deployment-type)Broker pool resources by deployment type The new RedpandaBrokerPool custom resource manages broker pools for Stretch Clusters only. The RedpandaBrokerPool CRD is installed only in multicluster mode, and its controller does not reconcile pools for standalone Redpanda resources. For standalone clusters, the [NodePool custom resource](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-node-pools/) (beta) remains the way to manage broker pools. Existing NodePool configurations continue to work. ## [](#redpanda-operator-v26-1-x)Redpanda Operator v26.1.x [Changelog](https://github.com/redpanda-data/redpanda-operator/blob/release/v26.1.x/operator/CHANGELOG.md) ### [](#nodepool-crd-beta)NodePool CRD (beta) The Redpanda Operator introduces the NodePool custom resource, which lets you manage groups of Redpanda brokers as independent units within a single cluster. Each NodePool creates its own StatefulSet, giving you fine-grained control over broker placement, resources, and lifecycle. NodePools enable blue/green node pool migrations: you create a new NodePool on the target Kubernetes node pool, and the operator automatically replicates data and decommissions old brokers with zero downtime. This replaces the manual, broker-by-broker migration process. > 📝 **NOTE** > > The NodePool CRD is in beta. You must enable it with `crds.experimental: true` and the `--enable-v2-nodepools` operator flag. - [Node Pools](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-node-pools/) — Learn how NodePools work and how to enable them. - [Migrate Node Pools](https://docs.redpanda.com/streaming/current/upgrade/k-migrate-node-pools/) — Step-by-step blue/green migration using the NodePool CRD. - [Migrate an existing cluster to NodePools](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-node-pools/#migrate-to-nodepools) — Transition from operator-managed replicas to NodePool-managed brokers. ### [](#prometheus-servicemonitor-for-console)Prometheus ServiceMonitor for Console The Console custom resource supports a `monitoring` configuration that deploys a Prometheus `ServiceMonitor` to automatically discover and scrape Console metrics. See [Prometheus ServiceMonitor](https://docs.redpanda.com/streaming/current/deploy/console/kubernetes/deploy/#prometheus-servicemonitor). ### [](#schema-registry-acls)Schema Registry ACLs The Redpanda custom resource supports configuring Schema Registry ACLs through the `schemaRegistry.authenticationMethod` field. This enables fine-grained access control for Schema Registry operations in clusters deployed with Kubernetes. When you enable Schema Registry authentication, you can control which users and service accounts can register, modify, and read schemas. See [Configure Authentication for Redpanda in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authentication/k-authentication/) for configuration details. ### [](#cloud-topics-for-kubernetes)Cloud Topics for Kubernetes The Redpanda custom resource supports Cloud Topics in Kubernetes deployments. You can configure topics to use cloud storage as the primary backing store by setting the appropriate storage mode properties. Cloud Topics in Kubernetes provide the same cost savings and architectural benefits as self-managed deployments. You configure them declaratively through the Redpanda custom resource. For setup instructions, see [Cloud Topics for Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/cloud-topics/k-cloud-topics/). ### [](#group-based-access-control-gbac)Group-based access control (GBAC) The Redpanda Operator supports group-based access control (GBAC) for Kubernetes deployments with OIDC authentication. You can assign roles and ACLs to OIDC groups, and users automatically inherit permissions from their group memberships. GBAC simplifies permission management in Kubernetes environments by integrating with your identity provider’s group structure. See [Group-based access control for Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/security/authorization/k-group-controller/) for configuration and usage details. --- # Page 113: What’s New in Redpanda **URL**: https://docs.redpanda.com/streaming/current/get-started/release-notes/redpanda.md --- # What’s New in Redpanda > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: What’s New in Redpanda latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: release-notes/redpanda page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: release-notes/redpanda.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/release-notes/redpanda.adoc description: Summary of new features and updates in this Redpanda release. page-git-created-date: "2025-04-07" page-git-modified-date: "2026-08-25" support-status: supported --- This topic includes new content added in version 26.2. For a complete list of all product updates, see the [Redpanda release notes](https://github.com/redpanda-data/redpanda/releases/). See also: - [What’s New in Redpanda Cloud](https://docs.redpanda.com/cloud-data-platform/get-started/whats-new-cloud/) - [Redpanda Cloud vs Self-Managed feature compatibility](https://docs.redpanda.com/cloud-data-platform/get-started/cloud-overview/#redpanda-cloud-vs-self-managed-feature-compatibility) ## [](#shadowing-migrate-schemas-from-confluent-schema-registry)Shadowing: migrate schemas from Confluent Schema Registry Shadowing can now replicate schemas over the Schema Registry HTTP API, including from a Confluent Schema Registry. Both Confluent Cloud and Confluent Platform are supported. A shadow link continuously replicates subjects, versions, and compatibility settings into the shadow cluster’s Schema Registry, so your applications keep working after cutover without a separate schema migration step. You can also replicate only selected contexts or subjects, and map source contexts to different destination contexts. See [Migrate Schemas from Confluent](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/). ## [](#shadowing-role-synchronization)Shadowing: role synchronization Shadow links can now replicate cluster roles from the source cluster to the shadow cluster, alongside topics, ACLs, consumer group offsets, and Schema Registry data. This ensures equivalent RBAC is in place on the shadow cluster prior to failover. Configure role filters to select which roles to sync, in the shadow link configuration file or in Redpanda Console v3.11.0 or later. See [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#role-filtering). ## [](#shadowing-support-for-cloud-topics)Shadowing support for Cloud Topics Shadow links now support Cloud Topics, so you can include Cloud Topics in disaster recovery replication alongside standard topics. See [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/). ## [](#kafka-4-x-client-compatibility)Kafka 4.x client compatibility Redpanda version 26.2 validates a broader set of non-Java Apache Kafka® clients at their current versions that support Kafka 4.x, in addition to the Apache Kafka Java client. Newly validated clients include `confluent-kafka-go` and Sarama for Go, and `kafka-python` and `confluent-kafka-python` for Python. Validation confirms connectivity and correctness across core Kafka APIs, such as produce, consume, and transaction operations. See [Kafka Compatibility](https://docs.redpanda.com/streaming/current/develop/kafka-clients/) for the full list of validated clients and known exceptions. ## [](#deferred-upgrade-finalization)Deferred upgrade finalization You can now keep a feature-release upgrade open, soak testing the new version with a guaranteed rollback path, and finalize the upgrade when you are ready. For clusters in a Shadowing topology, new guidance covers upgrade order: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. See [Defer Upgrade Finalization](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/) and [Upgrade Clusters in a Shadowing Topology](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/upgrade/). ## [](#fetch-read-coalescing)Fetch read coalescing When many consumers fetch the same partitions, each broker repeats the same read work for every consumer. Fetch read coalescing merges these redundant reads on each shard, reducing read CPU and memory consumption under high consumer fan-out. This Enterprise feature is disabled by default: enable it with the `kafka_fetch_read_coalescing_enabled` cluster property. See [Fetch Read Coalescing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/fetch-read-coalescing/). ## [](#per-entity-client-quota-metrics)Per-entity client quota metrics Aggregate quota metrics can confirm that throttling is happening at a broker level, but don’t reveal which user, client, or group of clients is _utilizing_ a quota, or being throttled. Redpanda can now label throttle-time and throughput metrics with the identity of the throttled entity, so you can identify throttled users and clients by name and right-size quota values based on observed use. These metrics are disabled by default: enable them with the `kafka_per_entity_quota_metrics` cluster property. See [Manage Throughput](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/#track-quota-use-per-entity). ## [](#cloud-topics-enabled-by-default)Cloud Topics enabled by default Cloud Topics no longer require a separate cluster property. When cloud storage is enabled for your cluster ([`cloud_storage_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_enabled) set to `true`), you can create Cloud Topics immediately by setting the storage mode during topic creation. The `cloud_topics_enabled` cluster property is deprecated. See [Manage Cloud Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/cloud-topics/). ## [](#schema-registry-context-for-iceberg-translation)Schema Registry context for Iceberg translation Bind a topic’s Iceberg translation to a specific [Schema Registry context](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/) using the new `redpanda.schema.registry.context` topic property, instead of always using the default context. This is useful for multi-tenant or per-environment schema isolation. See [Resolve schemas within a Schema Registry context](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#resolve-schemas-within-a-context). ## [](#schema-registry-contexts-serdes-client-support)Schema Registry contexts: SerDes client support Any SerDes client, in any language, can now target a [Schema Registry context](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/) through its base URL alone: point the client’s `schema.registry.url` at `http://:8081/contexts/{context}`. All endpoints that SerDes clients use are available under the `/contexts/{context}/` prefix, and schema ID lookups are scoped to the context automatically. Previously, only the Java Confluent SerDes could target non-default contexts. Schema Registry contexts are also enabled by default in this release: `schema_registry_enable_qualified_subjects` now defaults to `true`. This applies to existing clusters when they upgrade, unless the property was explicitly set to `false` before the upgrade. If any of your subject names start with `:.`, review the [upgrade considerations](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/#upgrade-considerations) before upgrading. See [Use contexts with SerDes clients](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/#serdes-clients). ## [](#iceberg-key-and-header-translation)Iceberg key and header translation Independently control how Redpanda translates a record’s key, value, and headers into the Iceberg table with a new section-based syntax for the `redpanda.iceberg.mode` topic property. You can decode keys and values using a schema registered in the Schema Registry, store keys or values as UTF-8 strings, or decode header values from raw bytes to strings. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes continue to work as shorthand for common combinations. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#configure-key-value-and-header-translation). ## [](#iceberg-health-monitoring)Iceberg health monitoring Monitor the health of Iceberg topics with a new Admin API endpoint, including connectivity to the external REST catalog and per-partition commit lag. Use it to test the catalog connection and find the partitions with the highest commit lag. See [Monitor Iceberg health](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/#monitor-iceberg-health). ## [](#cluster-health-metrics)Cluster health metrics Redpanda can now export a cluster health summary as Prometheus metrics, so you can alert on overall cluster health, such as brokers down, leaderless partitions, or a missing controller, directly from your metrics system instead of polling [`rpk cluster health`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-health/). The metrics mirror the output of `rpk cluster health` and are disabled by default. To enable them, set [`health_monitor_metrics_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#health_monitor_metrics_enabled) to `true` and restart your brokers. See [Cluster health](https://docs.redpanda.com/streaming/current/manage/monitoring/#cluster-health) for the available metrics and query patterns. ## [](#tiered-storage-v2-beta)Tiered Storage v2 (beta) Tiered Storage v2 is a beta feature. It is not supported for production deployments. Beta features are available for users to test and [provide feedback](https://redpandacommunity.slack.com/). Tiered Storage is now available in two versions. Tiered Storage v2 (`tiered_v2`) adds full compaction support for topics stored in object storage: compaction runs on the data in object storage rather than on local data only, eventually retains only the latest value for each key across the entire partition, and removes tombstone records after the retention period configured by `delete.retention.ms`. Topics using Tiered Storage v2 can also be converted to and from Cloud Topics. You enable both versions the same way, by setting `redpanda.storage.mode=tiered` on a topic. The [`default_redpanda_storage_mode_tiered_impl`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#default_redpanda_storage_mode_tiered_impl) cluster property selects the version for new topics (`tiered_v1` remains the default), and existing topics keep the version they were created with. See [Choose a Tiered Storage version](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#tiered-storage-versions). ## [](#redpanda-cli-rpk)Redpanda CLI (rpk) Redpanda 26.2 introduces the following rpk changes: - **Deferred upgrade finalization commands**: The new [`rpk cluster upgrade`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-upgrade/) command group adds `status` and `finalize`, so you can check whether an upgrade is pending finalization and finalize it when you are ready. See [Defer Upgrade Finalization](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/). - **`rpk redpanda admin` commands moved to `rpk cluster`**: The old commands keep working in this release as deprecated aliases, and each prints its replacement when you run it. See [Deprecated Features](https://docs.redpanda.com/streaming/current/upgrade/deprecated/) for the full command migration table. - **New `k8s` managed plugin**: The [`rpk k8s`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s/) plugin adds the [`rpk k8s multicluster`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster/) commands to bootstrap TLS certificates and peer configuration, check deployment health, and collect diagnostics bundles for [Stretch Clusters](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). Install it with `rpk k8s install`. - **New `check` managed plugin**: The [`rpk check`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check/) plugin runs production readiness checks for a Redpanda deployment. Install it with [`rpk check install`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check-install/). ### [](#v26-2-1)v26.2.1 #### [](#new-commands)New commands - [`rpk check`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check/) - Run production readiness checks for a Redpanda deployment - [`rpk check install`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check-install/) - Install Redpanda Check - [`rpk check uninstall`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check-uninstall/) - Uninstall the Redpanda Check plugin - [`rpk check upgrade`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-check/rpk-check-upgrade/) - Upgrade to the latest Redpanda Check version - [`rpk cluster brokers`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-brokers/) - Manage Redpanda cluster brokers - [`rpk cluster brokers decommission`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-brokers-decommission/) - Decommission the given broker. - [`rpk cluster brokers decommission-status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-brokers-decommission-status/) - Show the progress of a node decommissioning. - [`rpk cluster brokers recommission`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-brokers-recommission/) - Recommission the given broker if it is still decommissioning. - [`rpk cluster loggers`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-loggers/) - List and configure Redpanda broker loggers - [`rpk cluster loggers list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-loggers-list/) - List loggers available on a Redpanda broker. - [`rpk cluster loggers set`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-loggers-set/) - Set broker logger’s log level. - [`rpk cluster upgrade`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-upgrade/) - Check and finalize major-version upgrades. - [`rpk cluster upgrade finalize`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-upgrade-finalize/) - Finalize a deferred cluster upgrade. - [`rpk cluster upgrade status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-upgrade-status/) - Show the cluster upgrade finalization status. - [`rpk k8s`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s/) - Interact with Redpanda clusters running on Kubernetes - [`rpk k8s install`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-install/) - Install the Redpanda Kubernetes plugin. - [`rpk k8s multicluster`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster/) - Commands for bootstrapping and managing the multicluster operators that run a Redpanda Stretch Cluster: a single logical Redpanda cluster distributed across multiple Kubernetes clusters. - [`rpk k8s multicluster bootstrap`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-bootstrap/) - Bootstrap the TLS certificates and kubeconfig secrets that the multicluster operators of a Redpanda Stretch Cluster need to communicate across Kubernetes clusters. - [`rpk k8s multicluster bundle`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-bundle/) - Collects environment data from each Kubernetes cluster in a Redpanda Stretch Cluster deployment and packages it into a ZIP file for support. - [`rpk k8s multicluster status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-multicluster-status/) - Checks the health of the multicluster operators that run a Redpanda Stretch Cluster: each cluster’s operator pod health, raft consensus state, TLS certificate validity, and cross-cluster consistency. - [`rpk k8s uninstall`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-uninstall/) - Uninstall the Redpanda Kubernetes plugin. - [`rpk k8s upgrade`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-upgrade/) - Upgrade to the latest Redpanda Kubernetes plugin version - [`rpk k8s version`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-k8s/rpk-k8s-version/) - Print build information #### [](#new-flags)New flags - `rpk cloud auth list`: Added `--format` - [`rpk cluster config list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-config-list/): Added `--node-id` - [`rpk cluster config status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-config-status/): Added `--format` - [`rpk cluster info`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-info/): Added `--detailed`, `--include-decommissioned` - [`rpk cluster logdirs describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-logdirs-describe/): Added `--format` - [`rpk cluster maintenance status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-maintenance-status/): Added `--format` - [`rpk cluster partitions balancer-status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-partitions-balancer-status/): Added `--format` - [`rpk cluster partitions move-cancel`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-partitions-move-cancel/): Added `--format` - [`rpk cluster partitions move-status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-partitions-move-status/): Added `--format` - [`rpk group delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-delete/): Added `--format` - [`rpk group list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-list/): Added `--format` - [`rpk group offset-delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-offset-delete/): Added `--format` - [`rpk group seek`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-seek/): Added `--format` - [`rpk plugin list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-plugin/rpk-plugin-list/): Added `--format` - [`rpk profile list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-profile/rpk-profile-list/): Added `--format` - [`rpk security acl delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-security/rpk-security-acl-delete/): Added `--format` - `rpk security secret list`: Added `--format` - [`rpk shadow describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-describe/): Added `--print-role` - [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/): Added `--print-registry` - [`rpk topic add-partitions`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-add-partitions/): Added `--format` - [`rpk topic alter-config`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-alter-config/): Added `--format` - [`rpk topic delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-delete/): Added `--format` #### [](#changed-defaults)Changed defaults - [`rpk container start`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container-start/): `--console-image` default changed from `redpandadata/console:v3.7.1` to `redpandadata/console:v3.8.0` #### [](#deprecated-commands)Deprecated commands - `rpk redpanda admin`: Use [`rpk cluster`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster/) instead. Includes its subcommands: `rpk redpanda admin brokers`, `rpk redpanda admin brokers decommission`, `rpk redpanda admin brokers decommission-status`, `rpk redpanda admin brokers list`, `rpk redpanda admin brokers recommission`, `rpk redpanda admin config`, `rpk redpanda admin config log-level`, `rpk redpanda admin config log-level set`, `rpk redpanda admin config print`, `rpk redpanda admin partitions`, `rpk redpanda admin partitions list`. == New configuration properties **Shadowing:** - [`schema_registry_sync_memory_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#schema_registry_sync_memory_bytes): Maximum bytes of schema bodies held in memory while a schema registry link reconciles from the source - [`schema_registry_sync_parallelism`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#schema_registry_sync_parallelism): Maximum number of schemas imported concurrently while a schema registry link reconciles from the source - [`shadow_link_failover_batch_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#shadow_link_failover_batch_size): Maximum number of mirror topics in a single batched failover controller command **Schema Registry:** - [`schema_registry_use_rpc`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#schema_registry_use_rpc): Use internal Redpanda RPCs for Schema Registry internal topic I/O **Metrics and monitoring:** - [`health_monitor_metrics_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#health_monitor_metrics_enabled): Export the cluster health summary as `cluster_health_*` metrics - [`kafka_per_entity_quota_metrics`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_per_entity_quota_metrics): Label quota throttle-time and throughput metrics with the identity of the throttled entity - [`kafka_handler_latency_all`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_handler_latency_all): Enable latency histograms for all Kafka API handlers, instead of only the most important ones **Performance:** - [`kafka_fetch_read_coalescing_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_fetch_read_coalescing_enabled): Coalesce concurrent fetches of the same partition offset into one shared read - [`code_hugepages_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#code_hugepages_enabled): Map the Redpanda binary into hugepages **Tiered Storage:** - [`default_redpanda_storage_mode_tiered_impl`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#default_redpanda_storage_mode_tiered_impl): Select Tiered Storage v1 or v2 for new topics that use the `tiered` storage mode - [`redpanda.storage.mode.impl`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-storage-mode-impl): Report the Tiered Storage version of a topic, and optionally select it at topic creation **Iceberg:** - [`redpanda.schema.registry.context`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-schema-registry-context): Bind a topic’s Iceberg translation to a specific Schema Registry context **Cloud Topics:** > 📝 **NOTE** > > Cloud Topics requires an Enterprise license. For more information, contact [Redpanda sales](https://redpanda.com/try-redpanda?section=enterprise). - [`cloud_topics_compaction_disabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_compaction_disabled): Disable compaction of cloud topics - [`cloud_topics_l1_reader_cache_eviction_timeout_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_l1_reader_cache_eviction_timeout_ms): Time after which idle L1 readers are evicted from the per-shard reader cache - [`cloud_topics_l1_reader_cache_max_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_l1_reader_cache_max_size): Maximum number of L1 readers cached per shard - [`cloud_topics_l1_streaming_read_chunk_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_l1_streaming_read_chunk_size): Maximum bytes buffered in memory by a cache-bypassing L1 streaming read - [`cloud_topics_leveling_disabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_leveling_disabled): Disable leveling of cloud topics - [`cloud_topics_leveling_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_leveling_interval_ms): How often to scan managed cloud-topic partitions for leveling work - [`cloud_topics_leveling_max_range_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_leveling_max_range_bytes): Maximum total bytes per leveling job - [`cloud_topics_leveling_min_extent_size_ratio`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_leveling_min_extent_size_ratio): Size ratio below which an L1 extent is considered undersized and eligible for leveling - [`cloud_topics_max_concurrent_leveling_jobs_per_shard`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_max_concurrent_leveling_jobs_per_shard): Maximum number of leveling jobs that may run concurrently on a single shard - [`cloud_topics_metastore_block_cache_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_block_cache_size): Size of the per-database uncompressed-block cache for the L1 metastore LSM database - [`cloud_topics_metastore_max_pre_open_fibers`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_max_pre_open_fibers): Number of fibers used to pre-open metastore SST files at database open time - [`cloud_topics_metastore_retry_timeout_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_retry_timeout_ms): Overall deadline for retrying an L1 metastore operation on transport errors - [`cloud_topics_metastore_rpc_timeout_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_rpc_timeout_ms): Timeout for a single L1 metastore RPC to the metastore partition leader - [`cloud_topics_metastore_sst_chunk_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_sst_chunk_size): Size of the byte ranges used to read metastore SST files from object storage - [`cloud_topics_metastore_write_buffer_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_topics_metastore_write_buffer_size): Memtable size for the L1 metastore LSM database **Object storage I/O:** - [`cloud_io_admission_control_policy`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_io_admission_control_policy): Admission policy for object storage I/O - [`cloud_io_admission_control_reservation`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#cloud_io_admission_control_reservation): Per-group reserved slots for the `reservation` admission control policy ## [](#changed-property-defaults)Changed property defaults - [`min_cleanable_dirty_ratio`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#min_cleanable_dirty_ratio) and the equivalent [`min.cleanable.dirty.ratio`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#min-cleanable-dirty-ratio) topic property: the default increased from `0.2` to `0.5`. Compaction now waits for a higher proportion of dirty bytes before a partition becomes eligible, reducing background compaction work on write-heavy clusters. - [`schema_registry_enable_qualified_subjects`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#schema_registry_enable_qualified_subjects): the default changed from `false` to `true`. Schema Registry now accepts context-qualified subject names by default. The new default applies to existing clusters when they upgrade, unless the property was explicitly set to `false` before the upgrade. See the [upgrade considerations](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/#upgrade-considerations). ## [](#deprecated-properties)Deprecated properties - `cloud_topics_enabled`: Cloud Topics no longer require a separate enablement property and can be created whenever object storage is enabled for the cluster. - `space_management_enable_override`: now formally deprecated. This property has been ignored since v23.3. See [Deprecated features](https://docs.redpanda.com/streaming/current/upgrade/deprecated/) for the complete list of deprecated properties. ## [](#properties-not-available-in-this-release)Properties not available in this release The following cluster properties were introduced in Redpanda v26.1.14 but are not included in v26.2.1. They are available starting in v26.2.2: - `log_eviction_exempt_topics` - `schema_registry_deferred_recovery` - `schema_registry_replay_on_startup` If you upgrade from v26.1.14 to v26.2.1, any values you set for these properties are not honored until you upgrade to v26.2.2. ## [](#simplified-cluster-administration-commands-rpk-redpanda-admin-deprecated)Simplified cluster administration commands: `rpk redpanda admin` deprecated Cluster administration moves fully under `rpk cluster`: broker decommissioning, recommissioning, logger control, and per-broker configuration and partition listings now live alongside the other cluster commands, and `rpk cluster info -b --detailed` reports broker membership, liveness, and versions. The `rpk redpanda admin` command tree is deprecated and hidden from help output. The old commands still work and print a pointer to their replacements, but they are planned for removal in a future major release. See [Deprecated Features](https://docs.redpanda.com/streaming/current/upgrade/deprecated/) for the full command migration table. --- # Page 114: Install or Update rpk **URL**: https://docs.redpanda.com/streaming/current/get-started/rpk-install.md --- # Install or Update rpk > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Install or Update rpk latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: rpk-install page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: rpk-install.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/rpk-install.adoc description: Install or update rpk to interact with Redpanda from the command line. page-git-created-date: "2023-05-30" page-git-modified-date: "2026-08-13" support-status: supported --- The `rpk` tool is a single binary application that provides a way to interact with your Redpanda clusters from the command line. For example, you can use `rpk` to do the following: - Monitor your cluster’s health - Create, produce, and consume from topics - Set up access control lists (ACLs) and other security features For Redpanda Streaming deployments, the `rpk` binary is automatically installed on each Redpanda broker, so you can use the locally installed `rpk` binary to communicate with the local Redpanda cluster. You can also install `rpk` on your local machine as a standalone binary. With this setup, you can connect to a Redpanda cluster on your local machine and to an external one on a remote server. If you use `rpk` as a standalone binary to communicate with a Redpanda cluster, your installed version of `rpk` must match the version of Redpanda running in your cluster. ## [](#check-rpk-version)Check rpk version To check your current version of the rpk binary, run `rpk --version`. The following example lists the latest version of `rpk`. If your installed version is lower than this latest version, then update `rpk`. For a list of versions, see [Redpanda releases](https://github.com/redpanda-data/redpanda/releases/). ```bash rpk --version ``` ```bash rpk version 26.2.2 (rev fd30e45) ``` ## [](#install-or-update-rpk-on-linux)Install or update rpk on Linux To install, or update to, the latest version of `rpk` for Linux, run: ### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. To install, or update to, a version other than the latest, run: ### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` ### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` ## [](#install-or-update-rpk-on-macos)Install or update rpk on macOS ### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` ### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. #### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` #### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ## [](#upgrade-rpk-plugins)Upgrade rpk plugins Plugins, such as `rpk connect` and `rpk ai`, are separate binaries that `rpk` downloads and runs on your behalf. Upgrading `rpk` does not upgrade its plugins. An installed plugin stays at its version until you upgrade it, even across `rpk` upgrades. To see which plugins are installed: ```bash rpk plugin list --local ``` To upgrade a plugin, run its `upgrade` command. For example: ```bash rpk connect upgrade ``` > 📝 **NOTE** > > Some plugins install themselves the first time you run one of their commands. For example, `rpk connect run` downloads the latest Redpanda Connect if it is not already installed. This happens only on first use. After that, the plugin never updates itself. ## [](#next-steps)Next steps For the complete list of `rpk` commands and their syntax, see the [rpk reference](https://docs.redpanda.com/streaming/current/reference/rpk/). Explore `rpk` with the [Redpanda CLI Quickstart](https://docs.redpanda.com/streaming/current/get-started/rpk-quickstart/). --- # Page 115: Redpanda CLI Quickstart **URL**: https://docs.redpanda.com/streaming/current/get-started/rpk-quickstart.md --- # Redpanda CLI Quickstart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda CLI Quickstart latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: rpk-quickstart page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: rpk-quickstart.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/rpk-quickstart.adoc description: Quickly become familiar with rpk commands for basic Redpanda tasks, including creating, producing to, describing, and deleting topics, as well as consuming records and managing consumer groups. page-git-created-date: "2025-01-28" page-git-modified-date: "2025-02-17" support-status: supported --- This guide shows how to run the Redpanda CLI, `rpk`, for basic Redpanda tasks, including creating, producing to, describing, and deleting topics, as well as consuming records and managing consumer groups. Follow these examples to quickly become familiar with `rpk` commands. Consider creating an `rpk` profile to simplify your development experience with multiple Redpanda clusters by saving and reusing configurations for different clusters. For more information, see [About rpk profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/#about-rpk-profiles). ## [](#prerequisites)Prerequisites - A running Redpanda cluster. - The `rpk` CLI installed. See [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/). ## [](#create-a-topic)Create a topic To start streaming data, first create a topic as the destination for your records: ```bash rpk topic create tutorial ``` Output: ```bash TOPIC STATUS tutorial OK ``` See [rpk topic create](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-create/). ## [](#produce-records-to-a-topic)Produce records to a topic Produce records to the topic. Downstream consumers will then be able to read these records. To exit the producer session, press `Ctrl+C`: ```bash rpk topic produce tutorial ``` Additional input: ```bash hello world ``` Output: ```bash Produced to partition 0 at offset 0 with timestamp 1734640650348. Produced to partition 0 at offset 1 with timestamp 1734640653558. ``` See [rpk topic produce](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-produce/). ## [](#get-a-description-of-a-topic)Get a description of a topic Check the topic’s configuration and status to ensure that it’s ready for use: ```bash rpk topic describe tutorial ``` Output: ```bash SUMMARY ======= NAME tutorial PARTITIONS 1 REPLICAS 1 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT_CONFIG compression.type producer DEFAULT_CONFIG delete.retention.ms -1 DEFAULT_CONFIG flush.bytes 262144 DEFAULT_CONFIG flush.ms 100 DEFAULT_CONFIG initial.retention.local.target.bytes -1 DEFAULT_CONFIG initial.retention.local.target.ms -1 DEFAULT_CONFIG max.message.bytes 1048576 DEFAULT_CONFIG message.timestamp.type CreateTime DEFAULT_CONFIG redpanda.iceberg.delete true DEFAULT_CONFIG redpanda.iceberg.mode disabled DEFAULT_CONFIG redpanda.leaders.preference none DEFAULT_CONFIG redpanda.remote.delete true DEFAULT_CONFIG redpanda.remote.read false DEFAULT_CONFIG redpanda.remote.write false DEFAULT_CONFIG retention.bytes -1 DEFAULT_CONFIG retention.local.target.bytes -1 DEFAULT_CONFIG retention.local.target.ms 86400000 DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG segment.bytes 134217728 DEFAULT_CONFIG segment.ms 1209600000 DEFAULT_CONFIG write.caching true DEFAULT_CONFIG ``` See [rpk topic describe](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-describe/). ## [](#consume-records-from-a-topic)Consume records from a topic Consume records from the topic: ```bash rpk topic consume tutorial ``` Output: ```json { "topic": "tutorial", "value": "hello", "timestamp": 1678807229837, "partition": 0, "offset": 0 } { "topic": "tutorial", "value": "world", "timestamp": 1678807232413, "partition": 0, "offset": 1 } ``` Consume from an offset, where `2` is not inclusive: ```bash rpk topic consume tutorial --offset 0:2 ``` Output: ```json { "topic": "tutorial", "value": "hello", "timestamp": 1678807229837, "partition": 0, "offset": 0 } { "topic": "tutorial", "value": "world", "timestamp": 1678807232413, "partition": 0, "offset": 1 } ``` See [rpk topic consume](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-consume/). ## [](#create-a-consumer-group-and-consume-topics)Create a consumer group and consume topics Organize consumers into groups to share workloads and balance consumption: ```bash rpk topic consume tutorial --group tutorial-group ``` > 📝 **NOTE** > > The consumer group is created when you start consuming from the topic. Output: ```json { "topic": "tutorial", "value": "hello", "timestamp": 1734640650348, "partition": 0, "offset": 0 } { "topic": "tutorial", "value": "world", "timestamp": 1734640653558, "partition": 0, "offset": 1 } ``` See [rpk topic consume](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-consume/). ## [](#list-all-consumer-groups)List all consumer groups List available consumer groups in your cluster: ```bash rpk group list ``` Output: ```bash BROKER GROUP STATE 0 tutorial-group Empty ``` See [rpk group list](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-list/). ## [](#get-a-description-of-a-consumer-group)Get a description of a consumer group View details about the consumer group’s state, coordinator, members, and offsets: ```bash rpk group describe tutorial-group ``` Output: ```bash GROUP tutorial-group COORDINATOR 0 STATE Empty BALANCER MEMBERS 0 TOTAL-LAG 0 TOPIC PARTITION CURRENT-OFFSET LOG-START-OFFSET LOG-END-OFFSET LAG MEMBER-ID CLIENT-ID HOST tutorial 0 2 0 2 0 ``` See [rpk group describe](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-describe/). ## [](#delete-a-consumer-group)Delete a consumer group Clean up by removing the `tutorial-group` consumer group: ```bash rpk group delete tutorial-group ``` Output: ```bash GROUP STATUS tutorial-group OK ``` See [rpk group delete](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-delete/). ## [](#delete-a-topic)Delete a topic Clean up by removing the `tutorial` topic: ```bash rpk topic delete tutorial ``` Output: ```bash TOPIC STATUS tutorial OK ``` See [rpk topic delete](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-delete/). ## [](#next-steps)Next steps - To generate a profile to save and reuse configurations for different Redpanda clusters, see [About rpk profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/#about-rpk-profiles). - For the complete list of `rpk` commands and their syntax, see the [rpk Command Reference](https://docs.redpanda.com/streaming/current/reference/rpk/). --- # Page 116: Redpanda CLI **URL**: https://docs.redpanda.com/streaming/current/get-started/rpk.md --- # Redpanda CLI > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda CLI latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: rpk/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: rpk/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/get-started/pages/rpk/index.adoc description: The rpk command line interface tool lets you manage your Redpanda cluster, without the need to run a separate script for each function, as with Apache Kafka. page-git-created-date: "2023-07-26" page-git-modified-date: "2024-02-26" support-status: supported --- - [Introduction to rpk](https://docs.redpanda.com/streaming/current/get-started/intro-to-rpk/) Learn about `rpk` and how to use it to interact with your Redpanda cluster. - [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) Install or update `rpk` to interact with Redpanda from the command line. - [rpk Profiles](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/) Use `rpk profile` to simplify your development experience with multiple Redpanda clusters by saving and reusing configurations for different clusters. - [Specify Broker Addresses for rpk](https://docs.redpanda.com/streaming/current/get-started/broker-admin/) Learn how and when to specify Redpanda broker addresses for `rpk` commands, so `rpk` knows where to run Kafka-related commands. - [Specify Admin API Addresses for rpk](https://docs.redpanda.com/streaming/current/get-started/admin-addresses/) Learn how and when to specify Redpanda admin addresses for `rpk` commands, so `rpk` knows where to run admin-related commands. --- # Page 117: Manage **URL**: https://docs.redpanda.com/streaming/current/manage.md --- # Manage > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/index.adoc description: Manage Redpanda. page-git-created-date: "2023-05-30" page-git-modified-date: "2024-02-26" support-status: supported --- - [Manage Redpanda in Kubernetes](kubernetes/) Learn how to manage Redpanda in Kubernetes. - [Cluster Maintenance](cluster-maintenance/) Learn about cluster balancing, rolling upgrades, disk space management, and cluster diagnostics. - [Security](security/) Learn how to configure authentication, authorization, encryption, listeners, and other security features. - [Tiered Storage](tiered-storage-linux/) Tiered Storage helps to lower storage costs by offloading log segments to object storage. - [Integrate Redpanda with Iceberg](iceberg/) Generate Iceberg tables for your Redpanda topics for data lakehouse access. - [Schema Registry](schema-reg/) Redpanda's Schema Registry provides the interface to store and manage event schemas. - [High Availability](high-availability/) Learn about the trade-offs with different high availability configurations. - [Disaster Recovery](disaster-recovery/) Learn about Shadowing with cross-region replication for disaster recovery. - [Remote Read Replicas](remote-read-replicas/) Learn how to create a Remote Read Replica topic, which is a read-only topic that mirrors a topic on a different cluster. - [Recovery Mode](recovery-mode/) Recovery mode starts Redpanda with limited functionality and disables partitions so you can repair a failed cluster. - [Enable Rack Awareness](rack-awareness/) Enable rack awareness to place partition replicas across different failure zones. - [Raft Group Reconfiguration](raft-group-reconfiguration/) Learn how the Redpanda Raft group protocol provides consistency and availability during reconfiguration. - [Optimize I/O](io-optimization/) Learn how to optimize I/O performance. - [Redpanda Console](console/) Learn how to manage Redpanda using Redpanda Console. - [Manage Redpanda using the Admin API](use-admin-api/) Manage components of a Redpanda cluster, such as individual brokers and partition leadership. The Redpanda Admin API also allows you to perform operations that are specific to Redpanda Streaming and cannot be done using the standard Kafka API. - [Monitor Redpanda](monitoring/) Metrics to monitor the health of your system to predict issues and optimize performance. --- # Page 118: Audit Logging **URL**: https://docs.redpanda.com/streaming/current/manage/audit-logging.md --- # Audit Logging > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Audit Logging latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: audit-logging page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: audit-logging.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/audit-logging.adoc description: Learn how to use Redpanda's audit logging capabilities. page-git-created-date: "2023-12-22" page-git-modified-date: "2025-07-31" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Many scenarios for streaming data include the need for fine-grained auditing of user activity related to the system. This is especially true for regulated industries such as finance, healthcare, and the public sector. Complying with [PCI DSS v4](https://www.pcisecuritystandards.org/document_library/?document=pci_dss) standards, for example, requires verbose and detailed activity auditing, alerting, and analysis capabilities. Redpanda’s auditing capabilities support recording both administrative and operational interactions with topics and with users. Redpanda complies with the Open Cybersecurity Schema Framework (OCSF), providing a predictable and extensible solution that works seamlessly with industry standard tools. With audit logging enabled, there should be no noticeable changes in performance other than slightly elevated CPU usage. > 📝 **NOTE** > > Audit logging is configured at the cluster level. Redpanda supports excluding specific topics or principals from auditing to help reduce noise in the log. Audit logging is disabled by default. ## [](#audit-log-flow)Audit log flow The Redpanda audit log mechanism functions similar to the Kafka flow. When a user interacts with another user or with a topic, Redpanda writes an event to a specialized audit topic. The audit topic is immutable. Only Redpanda can write to it. Users are prevented from writing to the audit topic directly and the Kafka API cannot create or delete it. ![Audit log flow](https://docs.redpanda.com/streaming/current/shared/_images/audit-logging-flow.png) By default, any management and authentication actions performed on the cluster yield messages written to the audit log topic that are retained for seven days. Interactions with all topics by all principals are audited. Actions performed using the Kafka API and Admin API are all audited, as are actions performed directly through `rpk`. Messages recorded to the audit log topic comply with the [open cybersecurity schema framework](https://schema.ocsf.io/). Any number of analytics frameworks, such as Splunk or Sumo Logic, can receive and process these messages. Using an open standard ensures Redpanda’s audit logs coexist with those produced by other IT assets, powering holistic monitoring and analysis of your assets. ## [](#audit-log-configuration-options)Audit log configuration options Redpanda’s audit logging mechanism supports several options to control the volume and availability of audit records. Configuration is applied at the cluster level. - [`audit_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_enabled): Boolean value to enable audit logging. When you set this to `true`, Redpanda checks for an existing topic named `_redpanda.audit_log`. If none is found, Redpanda automatically creates one for you. Default: `false`. - [`audit_log_num_partitions`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_log_num_partitions): Integer value defining the number of partitions used by a newly created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for an existing audit log topic. Default: `12`. - [`audit_log_replication_factor`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_log_replication_factor): Optional Integer value defining the replication factor for a newly created audit log topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. If a value is not provided, Redpanda uses the `internal_topic_replication_factor` cluster property value. Default: `null`. - [`audit_client_max_buffer_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_client_max_buffer_size): Integer value defining the number of bytes allocated by the internal audit client for audit messages. When changing this, you must disable audit logging and then re-enable it for the change to take effect. Consider increasing this if your system generates a very large number of audit records in a short amount of time. Default: `16777216`. - [`audit_queue_max_buffer_size_per_shard`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_queue_max_buffer_size_per_shard): Integer value defining the maximum amount of memory in bytes used by the audit buffer in each shard. When this size is reached, requests to log additional audit messages return a non-retryable error. You must restart the cluster when changing this value. Default: `1048576`. - [`audit_enabled_event_types`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_enabled_event_types): List of strings in JSON style identifying the event types to include in the audit log. This may include any of the following: `management, produce, consume, describe, heartbeat, authenticate, schema_registry, admin`. Default: `'["management","authenticate","admin"]'`. - [`audit_excluded_topics`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_excluded_topics): List of strings in JSON style identifying the topics the audit logging system should ignore. This list cannot include the `_redpanda.audit_log` topic. Redpanda rejects the command if you do attempt to include that topic. Default: `null`. - [`audit_queue_drain_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_queue_drain_interval_ms): Internally, Redpanda batches audit log messages in memory and periodically writes them to the audit log topic. This defines the period in milliseconds between draining this queue to the audit log topic. Longer intervals may help prevent duplicate messages, especially in high throughput scenarios, but they also increase the risk of data loss during hard shutdowns where the queue is lost. Default: `500`. - [`audit_excluded_principals`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#audit_excluded_principals): List of strings in JSON style identifying the principals the audit logging system should ignore. Principals can be listed as `User:name` or `name`, both are accepted. Default: `null`. Even though audited event messages are stored to a specialized immutable topic, standard topic settings still apply. For example, you can apply the same Tiered Storage, retention time, and replication settings available to normal topics. These particular options are important for controlling the amount of disk space utilized by your audit topics. > ❗ **IMPORTANT** > > You must configure certain audit logging properties before enabling audit logging because these settings impact the creation of the `_redpanda.audit_log` topic itself. These properties include: `audit_log_num_partitions` and `audit_log_replication_factor`. The Kafka API allows you to add partitions or alter the replication factor after enabling audit logging, but Redpanda prevents you from altering these two configuration values directly. ## [](#audit-logging-event-types)Audit logging event types Redpanda’s auditable events fall into one of eight different event types. The APIs associated with each event type are as follows. | Audit event type | Associated APIs | | --- | --- | | management | AlterPartitionReassignmentsCreateACLsCreatePartitionsCreateTopicsDeleteAclsDeleteGroupsDeleteRecordsDeleteTopicsIncrementalAlterconfigsOffsetDelete | | produce | AddPartitionsToTxnEndTxnInitProducerIdProduce | | consume | AddOffsetsToTxnFetchJoinGroupLeaveGroupListOffsetOffsetCommitSyncGroupTxOffsetCommit | | describe | DescribeAclsDescribeConfigsDescribeGroupsDescribeLogDirsFindCoordinatorListGroupsListPartitionReassignmentsMetadataOffsetForLeaderEpochDescribeProducersDescribeTransationsListTransactions | | heartbeat | Heartbeat | | authenticate | All authentication events | | schema_registry | All Schema Registry API calls | | admin | All Admin API calls | ### [](#logged-events)Logged events The following table identifies the data logging level for each audit event entry. > 📝 **NOTE** > > The Included column captures whether the event itself is included (for example, successful and failed access attempts), or whether a piece of data is included in the event itself (for example, Source IP address). | Data Logging Level | Audit Event | Included? | Details | | --- | --- | --- | --- | | System Level | Date and time stamp for each entry | Yes | time field on each event | | Successful and failed access attempts | Yes | The status_id field shows success/failure for all access attempts for which auditing is enabled | | User ID | Yes | user.name | | User group memberships | Yes | user.groups field with type idp_group. Included in authentication events for OIDC users and in authorization events when a group ACL matches. See Configure Group-Based Access Control. | | User connect and disconnect time | No | Connect and disconnect time may be inferred from the presence or absence of activity. | | Password change | Yes | For SCRAM users managed through Redpanda core, the Admin API call associated with the password change is logged. Note that this does not cover users synced from external IdPs, such as through OIDC. | | Changes of security settings | Yes | For example, ACL creation is logged (kafka create_acls), and cluster configuration changes are logged (Admin API events) | | Successful and failed attempts to add/remove users from the system | Yes | See Password change | | Failed attempts to access system data | Yes | Generally, access attempts are logged. For example, kafka produce and consume calls are audited. | | Failed attempts to access critical directories and files | No | Not applicable | | Date and time of system start-up and shut-down | Yes | An application lifecycle event is logged when the broker starts/stops | | Use of external/peripheral devices | No | Not applicable | | Application Level | Transaction date and time | Yes | time field on each event | | User ID | Yes | user.name | | Source IP address | Yes | src_endpoint.ip field | | Type of transaction | Yes | api.operation shows the relevant Kafka API accessed or HTTP method used (Admin/Schema Registry) | | Transaction data | Yes | resources[] for Kafka ops; http_request (headers, URL) for HTTP endpoints | | Transaction result (success/fail) and reason of failure | Partial | Auditing occurs at request time (start of event). Authentication failures are captured with status_id and status_detail fields. However, downstream operation outcomes (whether the actual Kafka/Admin API/Schema Registry operation succeeded or failed) are not included in audit logs. | | Transaction number | No | Redpanda auditing does not assign a unique number to each request | | Failed attempts to access system data | Yes | Access attempts are logged through various audit event types | | Failed attempts to access critical directories and files | No | Not applicable | | Use of external/peripheral devices | Yes | Device access events are captured when applicable | | Network Level | Source IP address (IPv4 or IPv6) | Yes | src_endpoint.ip field | | Destination IP address (IPv4 or IPv6) | Yes | dst_endpoint.ip field | | Protocol/Ports | Yes | service.name shows the service/protocol used (kafka/http). http_request.url.scheme shows http/https for http requests. dst_endpoint.port may be indicative of the protocol. | | Timestamp | Yes | time field | | Host name | Yes | url.hostname: HTTP endpoints (Admin API / Schema Registry) include the HTTP hostname. Not included for Kafka (only source/destination IP). | | VLAN ID | No | Not applicable | | MAC address | No | Not applicable | ## [](#enable-audit-logging)Enable audit logging All audit log settings are applied at the cluster level. Use `rpk cluster config` to configure audit logs. Some options require a cluster restart. You can verify this using `rpk cluster config status`. Some key tuning recommendations for your audit logging settings include: - To change the number of partitions or the replication factor for your audit log topic, set the `audit_log_num_partitions` and `audit_log_replication_factor` properties, respectively. - Choose the type of events needed by setting `audit_enabled_event_types` to the desired list of event categories. Keep this as restrictive as possible based on your compliance and security needs to avoid excessive noise in your audit logs. - Identify non-sensitive topics so that you can exclude them from auditing. Specify this list of topics in `audit_excluded_topics`. - Identify non-sensitive principals so that you can exclude them from auditing. Specify this list of principals in `audit_excluded_principals`. This command accepts names as `name` or `User:name`. - Set `audit_enabled` to `true`. - [Optimize costs for audit logging](#optimize-costs-for-audit-logging). The sequence of commands in `rpk` for this audit log configuration is: rpk cluster config set audit\_log\_num\_partitions 6 rpk cluster config set audit\_log\_replication\_factor 5 rpk cluster config set audit\_enabled\_event\_types '\["management","describe","authenticate"\]' rpk cluster config set audit\_excluded\_topics '\["topic1","topic2"\]' rpk cluster config set audit\_excluded\_principals '\["User:principal1", "principal2"\]' rpk cluster config set audit\_enabled true rpk topic alter-config \_redpanda.audit\_log --set retention.ms=259200000 ## [](#optimize-costs-for-audit-logging)Optimize costs for audit logging When enabled, audit logging can quickly generate a very large amount of data, especially if all event types are selected. Proper configuration of audit logging is critical to avoid filling your disk or using excess Tiered Storage. The configuration options available help ensure your audit logs contain only the volume of data necessary to meet your regulatory or legal requirements. With audit logging, the pattern of message generation may be very different from your typical sources of data. These messages reflect usage of your system as opposed to the operational data your topics typically process. As a result, your retention, replication, and Tiered Storage requirements may differ from your other topics. A typical scenario with audit logging is to route the messages to an analytics platform like Splunk. If your retention period is too long, you may find that you are storing excessive amounts of replicated messages in both Redpanda and in your analytics suite. Identifying the right balance of retention and replication settings minimizes this duplication while retaining your data in a system that provides actionable intelligence. Assess the retention needs for your audit logs. You may not need to keep the logs for the default seven days. This is controlled by setting [`retention.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#retention-ms) for the `_redpanda.audit_log` topic or by setting [`log_retention_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_retention_ms) at the cluster level. ## [](#next-steps)Next steps [See samples of audit log messages](audit-log-samples/) ## [](#suggested-reading)Suggested reading - [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/) - [Manage Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/) ## Suggested labs - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 119: Sample Audit Log Messages **URL**: https://docs.redpanda.com/streaming/current/manage/audit-logging/audit-log-samples.md --- # Sample Audit Log Messages > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Sample Audit Log Messages latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: audit-logging/audit-log-samples page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: audit-logging/audit-log-samples.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/audit-logging/audit-log-samples.adoc description: Sample Redpanda audit log messages. page-git-created-date: "2023-12-22" page-git-modified-date: "2026-08-13" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Redpanda’s audit logs comply with version 1.0.0 of the [Open Cybersecurity Schema Framework (OCSF)](https://github.com/ocsf). This provides a predictable and extensible solution that works seamlessly with industry standard tools. This page aggregates several sample log files covering a range of scenarios. ## [](#standard-ocsf-messages)Standard OCSF messages Redpanda produces the following standard OCSF class messages: - Authentication (3002) for all authentication events - Application Lifecycle (6002) for when the audit system is enabled or disabled or when Redpanda starts or stops (if auditing is enabled when Redpanda starts or stops) - API Activity (6003) for any access to the Kafka API, Admin API, or Schema Registry Refer to the [OCSF Schema Definition](https://schema.ocsf.io/) for the field definitions for each event class. ## [](#authentication-events)Authentication events These messages illustrate various scenarios around successful and unsuccessful authentication events. Authentication successful This scenario shows the message resulting from an admin using rpk with successful authentication. This is an authentication type event. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", // This is the Node ID of the broker that produced this audit event "uid": "2", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700533469078, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-SCRAM", "auth_protocol_id": 99, // This is the IP address of the Kafka broker that received the authorization request "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, // Name of the Redpanda kafka server "svc_name": "kafka rpc protocol" }, // Indicates that credentials were not encrypted using TLS "is_cleartext": true, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, // This is the IP address of the client that generated the authorization request "src_endpoint": { "ip": "127.0.0.1", // This is the client ID of the kafka client "name": "rpk", "port": 42906 }, "status_id": 1, "user": { "name": "user", "type_id": 1 } } ``` Authentication successful (OIDC with group claims) This scenario shows a successful OIDC authentication event that includes the user’s IdP group memberships in the `user.groups` field. Group memberships are extracted from the OIDC token and included in all authentication events for OIDC users. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v26.1.1" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700533469078, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-OAUTHBEARER", "auth_protocol_id": 99, "dst_endpoint": { "ip": "127.0.0.1", "port": 9092, "svc_name": "kafka rpc protocol" }, "is_cleartext": false, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, "src_endpoint": { "ip": "10.0.1.50", "name": "kafka-client", "port": 48210 }, "status_id": 1, // IdP group memberships extracted from the OIDC token "user": { "name": "alice@example.com", "type_id": 1, "groups": [ {"type": "idp_group", "name": "engineering"}, {"type": "idp_group", "name": "analytics"} ] } } ``` Authentication failed This scenario illustrates a common failure where a user entered the wrong credentials. This is an authentication type event. ```json { "category_uid": 3, "class_uid": 3002, "metadata": { "product": { "name": "Redpanda", "uid": "1", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "version": "1.0.0" }, "severity_id": 1, "time": 1700534756350, "type_uid": 300201, "activity_id": 1, "auth_protocol": "SASL-SCRAM", "auth_protocol_id": 99, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, "is_cleartext": true, "is_mfa": false, "service": { "name": "kafka rpc protocol" }, "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 45236 }, "status_id": 2, "status_detail": "SASL authentication failed: security: Invalid credentials", "user": { "name": "admin", "type_id": 1 } } ``` ## [](#kafka-api-events)Kafka API events The Redpanda Kafka API offers a wide array of options for interacting with your Redpanda clusters. Following are examples of messages from common interactions with the API. Create ACL entry This example illustrates an ACL update that also requires a superuser authentication. It lists the edited ACL and the updated permissions. This is a management type event. ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "time": 1700533393776, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", // This shows a superuser level authorization "policy": { "desc": "superuser", "name": "aclAuthorization" } } ], "user": { "name": "admin", "type_id": 2 } }, "api": { // The API operation performed "operation": "create_acls", "service": { "name": "kafka rpc protocol" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, // List of resources accessed "resources": [ // The created ACL { "name": "create acl", "type": "acl_binding", "data": { "resource_type": "topic", "resource_name": "*", "pattern_type": "literal", "acl_principal": "{type user name user}", "acl_host": "{{any_host}}", "acl_operation": "all", "acl_permission": "allow" } }, // Below indicates that the user had cluster level authorization { "name": "kafka-cluster", "type": "cluster" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 50276 }, "status_id": 1, "unmapped": { // Provides a more parsable output of how the // authorization decision was made "authorization_metadata": { "acl_authorization": { "host": "", "op": "", "permission_type": "AUTHORIZED", "principal": "" }, "resource": { "name": "", "pattern": "", "type": "" } } } } ``` Authorization matched on a group ACL This example shows an API Activity (6003) where the authorization decision matched an ALLOW ACL on a `Group:` principal. The `actor.user.groups` field includes the matched group with type `idp_group`, and the `authorization_metadata` shows the group ACL that granted access. See [Group-Based Access Control](https://docs.redpanda.com/streaming/current/manage/security/authorization/gbac/). ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v26.1.0" }, "version": "1.0.0" }, "severity_id": 1, "time": 1774544504327, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", "policy": { "desc": "acl: {principal type {group} name {/sales} host {{any_host}} op all perm allow}, resource: type {topic} name {sales-topic} pattern {literal}", "name": "aclAuthorization" } } ], // The matched group appears in the user's groups field "user": { "name": "alice", "type_id": 1, "groups": [ { "type": "idp_group", "name": "/sales" } ] } }, "api": { "operation": "produce", "service": { "name": "kafka rpc protocol" } }, "dst_endpoint": { "ip": "127.0.1.1", "port": 9092, "svc_name": "kafka rpc protocol" }, "resources": [ { "name": "sales-topic", "type": "topic" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rdkafka", "port": 42728 }, "status_id": 1, "unmapped": { "authorization_metadata": { "acl_authorization": { "host": "{{any_host}}", "op": "all", "permission_type": "allow", "principal": "type {group} name {/sales}" }, "resource": { "name": "sales-topic", "pattern": "literal", "type": "topic" } } } } ``` Metadata request (with counts) This shows a message for a scenario where a user requests a set of metadata using rpk. It provides detailed information on the type of request and the information sent to the user. This is a describe type event. ```json { "category_uid": 6, "class_uid": 6003, // If present, indicates that >1 of the same authz check was performed // within the period of the audit log collecting entries // This provides start and end time (the time period these events were // observed) "count": 2, "end_time": 1700533480725, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "start_time": 1700533480724, "time": 1700533480724, "type_uid": 600303, "activity_id": 3, "actor": { "authorizations": [ { "decision": "authorized", // Represents a policy for a non-super user "policy": { "desc": "acl: {principal {type user name user} host {{any_host}} op all perm allow}, resource: type {topic} name {*} pattern {literal}", "name": "aclAuthorization" } } ], "user": { "name": "user", "type_id": 1 } }, "api": { "operation": "metadata", "service": { "name": "kafka rpc protocol" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 19092, "svc_name": "kafka rpc protocol" }, "resources": [ // The topics accessed { "name": "test", "type": "topic" } ], "src_endpoint": { "ip": "127.0.0.1", "name": "rpk", "port": 53602 }, "status_id": 1, "unmapped": { "authorization_metadata": { "acl_authorization": { "host": "{{any_host}}", "op": "all", "permission_type": "allow", "principal": "{type user name user}" }, "resource": { "name": "*", "pattern": "literal", "type": "topic" } } } } ``` ## [](#admin-api-events)Admin API events The following examples show audit messages related to use of the Redpanda Admin API. Requesting cluster configurations as a superuser This example shows the log message when you use the Admin API to retrieve the cluster configurations in a zipped archive. Note that a user must authenticate with the superuser role to perform this action. ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "uid": "2", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "time": 1700575714976, "type_uid": 600302, "activity_id": 2, "actor": { "authorizations": [ { "decision": "authorized", "policy": { "desc": "", "name": "Admin httpd authorizer" } } ], "user": { "name": "admin", "type_id": 2 } }, "api": { "operation": "GET", "service": { "name": "Redpanda Admin HTTP Server" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 9644, "svc_name": "Redpanda Admin HTTP Server" }, "http_request": { "http_headers": [ { "name": "Accept-Encoding", "value": "gzip" }, { "name": "Accept", "value": "application/json" }, { "name": "Content-Type", "value": "application/json" }, { "name": "User-Agent", "value": "Go-http-client/1.1" }, { "name": "Authorization", "value": "******" }, { "name": "Host", "value": "127.0.0.1:9644" } ], "http_method": "GET", "url": { "hostname": "127.0.0.1:9644", "path": "/v1/cluster_config?include_defaults=true", "port": 9644, "scheme": "http", "url_string": "http://127.0.0.1:9644/v1/cluster_config?include_defaults=true" }, "user_agent": "Go-http-client/1.1", "version": "1.1" }, "src_endpoint": { "ip": "127.0.0.1", "port": 44150 }, "status_id": 1, "unmapped": {} } ``` Unauthorized user requesting cluster configurations Similar to the previous example, this example illustrates a user requesting cluster configurations as a zip archive. Unlike the previous example, however, the user in this case is not authorized to retrieve this information. ```json { "category_uid": 6, "class_uid": 6003, "metadata": { "product": { "name": "Redpanda", "uid": "0", "vendor_name": "Redpanda Data, Inc.", "version": "v23.3.0-dev-2457-g76dc896f8c" }, "profiles": [ "cloud" ], "version": "1.0.0" }, "severity_id": 1, "time": 1700576203097, "type_uid": 600302, "activity_id": 2, "actor": { "authorizations": [ { "decision": "denied", "policy": { "desc": "Forbidden (superuser role required)", "name": "Admin httpd authorizer" } } ], "user": { "name": "user", "type_id": 1 } }, "api": { "operation": "GET", "service": { "name": "Redpanda Admin HTTP Server" } }, "cloud": { "provider": "" }, "dst_endpoint": { "ip": "127.0.0.1", "port": 9644, "svc_name": "Redpanda Admin HTTP Server" }, "http_request": { "http_headers": [ { "name": "Accept-Encoding", "value": "gzip" }, { "name": "Accept", "value": "application/json" }, { "name": "Content-Type", "value": "application/json" }, { "name": "User-Agent", "value": "Go-http-client/1.1" }, { "name": "Authorization", "value": "******" }, { "name": "Host", "value": "127.0.0.1:9644" } ], "http_method": "GET", "url": { "hostname": "127.0.0.1:9644", "path": "/v1/cluster_config?include_defaults=true", "port": 9644, "scheme": "http", "url_string": "http://127.0.0.1:9644/v1/cluster_config?include_defaults=true" }, "user_agent": "Go-http-client/1.1", "version": "1.1" }, "src_endpoint": { "ip": "127.0.0.1", "port": 53296 }, "status_id": 2, "unmapped": {} } ``` ## Suggested labs - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 120: Cluster Maintenance **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance.md --- # Cluster Maintenance > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Cluster Maintenance latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/index.adoc description: Learn about cluster balancing, rolling upgrades, disk space management, and cluster diagnostics. page-git-created-date: "2023-06-02" page-git-modified-date: "2024-02-26" support-status: supported --- - [Configure Cluster Properties](cluster-property-configuration/) Learn how to configure cluster properties. - [Configure Broker Properties](node-property-configuration/) Learn how to configure broker properties with the `redpanda.yaml` file. - [Configure Topic Properties](topic-property-configuration/) Learn how to configure topic properties to control Redpanda's behavior for individual topics, including retention, cleanup policies, and Tiered Storage settings. - [Cluster Balancing](cluster-balancing/) Learn about the different tools Redpanda provides for balanced clusters. - [Configure Continuous Data Balancing](continuous-data-balancing/) Continuous Data Balancing simplifies operations with self-healing clusters that dynamically balance partitions. - [Decommission Brokers](decommission-brokers/) Remove a broker so that it is no longer considered part of the cluster. - [Maintenance Mode](https://docs.redpanda.com/streaming/current/manage/node-management/) Enable maintenance mode to temporarily take a broker offline, for example during a rolling upgrade. - [Perform a Rolling Restart](rolling-restart/) Learn how to perform a rolling restart of your Redpanda cluster. - [Audit Logging](https://docs.redpanda.com/streaming/current/manage/audit-logging/) Learn how to use Redpanda's audit logging capabilities. - [Manage Disk Space](disk-utilization/) Redpanda provides several ways to manage disk space to ensure the stability of a cluster. - [About Client Throughput Quotas](about-throughput-quotas/) Understand how Redpanda's user-based and client ID-based throughput quotas work, including entity hierarchy, precedence rules, and quota tracking behavior. - [Manage Throughput](manage-throughput/) Configure broker-wide and client-specific throughput quotas to prevent resource exhaustion and noisy-neighbor issues. - [Fetch Read Coalescing](fetch-read-coalescing/) Reduce redundant read CPU and fetch-response memory under high consumer fan-out by sharing one read result across concurrent fetches of the same data. - [Compaction Settings](compaction-settings/) Redpanda's approach to compaction and options for configuring it. - [Configure Client Connections](configure-client-connections/) Learn about guidelines for configuring client connections in Redpanda clusters for optimal availability. - [Forced partition recovery](partition-recovery/) Recover a single partition using the Admin API. - [Node-wise Partition Recovery](nodewise-partition-recovery/) Feature to recover partitions that have lost a majority of replicas. --- # Page 121: About Client Throughput Quotas **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/about-throughput-quotas.md --- # About Client Throughput Quotas > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: About Client Throughput Quotas latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/about-throughput-quotas page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/about-throughput-quotas.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/about-throughput-quotas.adoc description: Understand how Redpanda's user-based and client ID-based throughput quotas work, including entity hierarchy, precedence rules, and quota tracking behavior. page-topic-type: concepts personas: platform_admin, developer learning-objective-1: Describe the difference between user-based and client ID-based quotas learning-objective-2: Determine which quota type to use for your use case learning-objective-3: Explain quota precedence rules and how Redpanda tracks quota usage page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" support-status: supported --- Redpanda uses throughput quotas to limit the rate of produce and consume requests from clients. Understanding how quotas work helps you prevent individual clients from disproportionately consuming resources and causing performance degradation for other clients (also known as the "noisy-neighbor" problem), and ensure fair resource sharing across users and applications. After reading this page, you will be able to: - Describe the difference between user-based and client ID-based quotas - Determine which quota type to use for your use case - Explain quota precedence rules and how Redpanda tracks quota usage To configure and manage throughput quotas, see [Manage Throughput](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/). ## [](#throughput-control-overview)Throughput control overview Redpanda provides two ways to control throughput: - Broker-wide limits: Configured using cluster properties. For details, see [Broker-wide throughput limits](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/#broker-wide-throughput-limits). - Client throughput quotas: Configured using the Kafka API. Client quotas enable per-user and per-client rate limiting with fine-grained control through entity hierarchy and precedence rules. This page focuses on client quotas. ## [](#supported-quota-types)Supported quota types Redpanda supports three Kafka API-based quota types: | Quota type | Description | | --- | --- | | producer_byte_rate | Limit throughput of produce requests (bytes per second) | | consumer_byte_rate | Limit throughput of fetch requests (bytes per second) | | controller_mutation_rate | Limit rate of topic mutation requests (partitions created or deleted per second) | All quota types can be applied to groups of client connections based on user principals, client IDs, or combinations of both. ## [](#quota-entities)Quota entities Redpanda uses two pieces of identifying information from each client connection to determine which quota applies: - Client ID: An ID that clients self-declare. Quotas can target an exact client ID (`client-id`) or a prefix (`client-id-prefix`). Multiple client connections that share a client ID or ID prefix are grouped into a single quota entity. - User [principal](https://docs.redpanda.com/streaming/current/reference/glossary/#principal): An authenticated identity verified through SASL, mTLS, or OIDC. Connections that share the same user are considered one entity. You can configure quotas that target either entity type, or combine both for fine-grained control. ### [](#client-id-based-quotas)Client ID-based quotas Client ID-based quotas apply to clients identified by their `client-id` field, which is set by the client application. The client ID is typically a configurable property when you create a client with Kafka libraries. When using client ID-based quotas, multiple clients using the same client ID share the same quota tracking. Client ID-based quotas rely on clients honestly reporting their identity and correctly setting the `client-id` property. This makes client ID-based quotas unsuitable for guaranteeing isolation between tenants. Use client ID-based quotas when: - Authentication is not enabled. - Grouping by application or service name is sufficient. - You operate a single-tenant environment where all clients are trusted. - You need simple rate limiting without user-level isolation. ### [](#user-based-quotas)User-based quotas > ❗ **IMPORTANT** > > User-based quotas require [authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/) to be enabled on your cluster. User-based quotas apply to authenticated user principals. Each user has a separate quota, providing a way to limit the impact of individual users on the cluster. User-based quotas rely on Redpanda’s authentication system to verify user identity. The user principal is extracted from SASL credentials, mTLS certificates, or OIDC tokens and cannot be forged by clients. Use user-based quotas when: - You operate a multi-tenant environment, such as SaaS platforms or enterprises with departments. - You require isolation between users or tenants, to avoid noisy-neighbor issues. - You need per-user billing or metering. ### [](#combined-user-and-client-quotas)Combined user and client quotas You can combine user and client identities for fine-grained control over specific (user, client) combinations. Use combined quotas when: - You need fine-grained control, for example: user `alice` using a specific application. - Different rate limits apply to different apps used by the same user. For example, `alice`'s `payment-processor` gets 10 MB/s, but `alice`'s `analytics-consumer` gets 50 MB/s. See [Quota precedence and tracking](#quota-precedence-and-tracking) for examples. ## [](#quota-precedence-and-tracking)Quota precedence and tracking When a request arrives, Redpanda resolves which quota to apply by matching the request’s authenticated user principal and client ID against configured quotas. Redpanda applies the most specific match, using the precedence order in the following table (highest priority first). The precedence level that matches also determines how quota usage is tracked. Redpanda tracks quota usage using a tracker key that determines which connections share the same quota bucket. How connections are grouped into buckets depends on the type of entity the quota targets. To get independent quota tracking per user and client ID combination, configure quotas that include both dimensions, such as `/config/users//clients/` or `/config/users//clients/`. | Level | Match type | Config path | Tracker key | Isolation behavior | | --- | --- | --- | --- | --- | | 1 | Exact user + exact client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 2 | Exact user + client prefix | /config/users//client-id-prefix/ | (user, client-id-prefix) | Clients matching the prefix share tracking within that user | | 3 | Exact user + default client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 4 | Exact user only | /config/users/ | user | All clients for that user share a single tracking bucket | | 5 | Default user + exact client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 6 | Default user + client prefix | /config/users//client-id-prefix/ | (user, client-id-prefix) | Clients matching the prefix share tracking within each user | | 7 | Default user + default client | /config/users//clients/ | (user, client-id) | Each unique (user, client-id) pair tracked independently | | 8 | Default user only | /config/users/ | user | All clients for each user share a single tracking bucket (per user) | | 9 | Exact client only | /config/clients/ | client-id | All users with that client ID share a single tracking bucket | | 10 | Client prefix only | /config/client-id-prefix/ | client-id-prefix | All clients matching the prefix share a single bucket across all users | | 11 | Default client only | /config/clients/ | client-id | Each unique client ID tracked independently | | 12 | No quota configured | N/A | N/A | No tracking / unlimited throughput | > ❗ **IMPORTANT** > > The `` entity matches any user or client that doesn’t have a more specific quota configured. This is different from an empty/unauthenticated user (`user=""`), or undeclared client ID (`client-id=""`), which are treated as specific entities. ### [](#unauthenticated-connections)Unauthenticated connections Unauthenticated connections have an empty user principal (`user=""`) and are not treated as `user=`. Unauthenticated connections: - Fall back to client-only quotas. - Have unlimited throughput only if no client-only quota matches. ### [](#example-precedence-resolution)Example: Precedence resolution Given these configured quotas: ```bash rpk cluster quotas alter --add consumer_byte_rate=5000000 --name user=alice --name client-id=app-1 rpk cluster quotas alter --add consumer_byte_rate=10000000 --name user=alice rpk cluster quotas alter --add consumer_byte_rate=20000000 --name client-id=app-1 ``` | User + Client ID | Precedence match | | --- | --- | | user=alice, client-id=app-1 | Level 1: Exact user + exact client | | user=alice, client-id=app-2 | Level 4: Exact user only | | user=bob, client-id=app-1 | Level 9: Exact client only | | user=bob, client-id=app-2 | Level 12: No quota configured | When no quota matches (level 12), the connection is not throttled. ### [](#example-user-only-quota)Example: User-only quota If you configure a 10 MB/s produce quota for user `alice`: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice ``` Then `alice` connecting with client ID `app-1` and `alice` connecting with client ID `app-2` share the same 10 MB/s produce limit. To give each of `alice`'s clients an independent 10 MB/s limit, configure: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --name user=alice --default client-id ``` ### [](#example-user-default-quota)Example: User default quota If you configure a default 10 MB/s produce quota for all users: ```bash rpk cluster quotas alter --add producer_byte_rate=10000000 --default user ``` This quota applies to all users who don’t have a more specific quota configured. Each user is tracked independently: `alice` gets her own 10 MB/s bucket, `bob` gets his own 10 MB/s bucket, and so on. Within each user, all client ID values share that user’s bucket. `alice` connecting with client ID `app-1` and `alice` connecting with client ID `app-2` share the same 10 MB/s produce limit, while `bob`'s connections have a separate 10 MB/s limit. ## [](#throttling-enforcement)Throughput throttling enforcement > 📝 **NOTE** > > As of v24.2, Redpanda enforces all throughput limits per broker, including client throughput. Redpanda enforces throughput limits by applying backpressure to clients. When a connection exceeds its throughput limit, Redpanda throttles the connection to bring the rate back within the allowed level: 1. Redpanda adds a `throttle_time_ms` field to responses, indicating how long the client should wait. 2. If the client doesn’t honor the throttle time, Redpanda inserts delays on the connection’s next read operation. The throttling delay may not exceed the limit set by the `max_kafka_throttle_delay_ms` tunable property. ## [](#default-behavior)Default behavior Quotas are opt-in restrictions and not enforced by default. When no quotas are configured, clients have unlimited throughput. ## [](#next-steps)Next steps - [Configure throughput quotas](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/) - [Enable authentication for user-based quotas](https://docs.redpanda.com/streaming/current/manage/security/authentication/) --- # Page 122: Cluster Balancing **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-balancing.md --- # Cluster Balancing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Cluster Balancing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/cluster-balancing page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/cluster-balancing.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/cluster-balancing.adoc description: Learn about the different tools Redpanda provides for balanced clusters. page-git-created-date: "2023-06-02" page-git-modified-date: "2026-04-28" support-status: supported --- Cluster balancing is crucial for optimal performance. Unbalanced clusters can saturate resources on one or more brokers, impacting throughput and latency. Furthermore, a cluster with replicas on a down broker risks availability loss if more brokers fail, and a cluster that keeps losing brokers without healing eventually risks data loss. Redpanda provides various topic-aware tools to balance clusters for best performance. | Topic-aware data balancer | Description | | --- | --- | | Partition leadership balancing | This balancer transfers the leadership of a broker’s partitions to other replicas to avoid topic leadership hotspots on one or a few specific brokers in your cluster.The partition leader regularly sends heartbeats to its followers. If a follower does not receive a heartbeat within a timeout, it triggers a new leader election. Redpanda also provides leadership balancing when brokers are added or decommissioned. | | Partition replica balancing | This balancer moves partition replicas to avoid topic replica hotspots on one or a few specific brokers in your cluster.Redpanda prioritizes balancing a topic’s partition replica count evenly across all brokers while it’s balancing the cluster’s overall partition count. Because different topics in a cluster can have vastly different load profiles, this better distributes the workload evenly across brokers.Redpanda provides partition replica balancing when brokers are added or decommissioned. | | Intra-broker partition balancing | This balancer moves partition replicas across CPU cores in an individual broker. Redpanda maintains balanced partition replica assignments between cores to avoid topic hotspots on one or a few specific cores within a broker.Continuous intra-broker partition balancing (core_balancing_continuous) requires an enterprise license. | | Continuous Data Balancing | This balancer monitors broker and rack availability, as well as disk usage, to avoid topic hotspots when moving data off brokers with fuller disks. Continuous Data Balancing enables self-healing clusters that dynamically balance partitions. It also ensures adherence to rack-aware replica placement policy and self-heals after rack (or availability zone) failure or replacement. This balancer does not keep the relative fullness of each broker within a defined range, it just prevents hitting the fullness threshold of each individual broker.Continuous Data Balancing requires an enterprise license. | > 📝 **NOTE** > > If a topic already has messages and you add partitions, the existing messages won’t be redistributed to the new partitions. If you require messages to be redistributed, then you must create a new topic with the new partition count, then stream the messages from the old topic to the new topic so they are appropriately distributed according to the new partition hashing. ## [](#partition-leadership-balancing)Partition leadership balancing Every Redpanda topic partition forms a Raft group with a single elected leader. This leader manages all writes for the partition. Raft uses a heartbeat mechanism to maintain leadership authority and to initiate leader elections. The partition leader regularly sends heartbeats ([`raft_heartbeat_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_heartbeat_interval_ms)) to its followers. If a follower does not receive a heartbeat within a timeout ([`raft_heartbeat_timeout_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_heartbeat_timeout_ms)), it triggers a new leader election. For more information, see [Raft consensus algorithm](https://docs.redpanda.com/streaming/current/get-started/architecture/#raft-consensus-algorithm) and [partition leadership elections](https://docs.redpanda.com/streaming/current/get-started/architecture/#partition-leadership-elections). By default, Redpanda enables topic-aware leadership balancing with the [`enable_leader_balancer`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_leader_balancer) property. Automatic partition leadership balancing improves cluster performance by transferring partition leadership from one broker to other replicas. This transfer changes where data is written to first, but leadership transfer does not involve any data transfer. > 📝 **NOTE** > > In addition to the periodic heartbeat, leadership balancing can also occur when a [broker restarts](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/#impact-of-broker-restarts) or when the controller leader changes (such as when a controller partition changes leader). The _controller leader_ manages the entire cluster. For example, when decommissioning a broker, the controller leader creates a reallocation plan for all partition replicas allocated to that broker. The _partition leader_ then handles the reconfiguration for its Raft group. ### [](#manually-change-leadership)Manually change leadership Despite an even distribution of leaders, sometimes the write pattern is not even across topics, and a set of traffic-heavy partitions could land on one broker and cause a latency spike. For information about metrics to monitor, see [Partition health](https://docs.redpanda.com/streaming/current/manage/monitoring/#partition-health). To manually change leadership, use the Admin API: ```bash curl -X POST http://:9644/v1/partitions/kafka///transfer_leadership?target= ``` For example, to change leadership to broker 2 for partition 0 on topic `test`: ```bash curl -X POST "http://localhost:9644/v1/partitions/kafka/test/0/transfer_leadership?target=2" ``` > 📝 **NOTE** > > In Kubernetes, run the `transfer_leadership` request on the Pod that is running the current partition leader. ## [](#partition-replica-balancing)Partition replica balancing While leadership balancing doesn’t move any data, partition balancing does move partition replicas to alleviate disk pressure and maintain the configured replication factor across brokers and the additional redundancy across failure domains (such as racks). Depending on the data volume, this process may take some time. Partition balancing is invoked periodically as determined by the [`partition_autobalancing_tick_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#partition_autobalancing_tick_interval_ms) property. For predictable and stable performance, Redpanda ensures an even distribution of a topic’s partition replicas across all brokers in a cluster. It allocates partitions to random healthy brokers to prevent topic hotspots, without waiting for a batch of moves to finish before scheduling the next batch. Different topics in a cluster can have vastly different load profiles, but partitions of a single topic are often similar. To distribute the workload of heavily-used topics, Redpanda prioritizes an equal count of a topic’s partition replicas on each broker while it aims for an equal overall partition count on each broker. For example, suppose you have a Redpanda cluster that has 3 brokers and 1 topic, `light`, with 100 partitions and replication factor=3. Partitions are balanced, so each broker hosts 100 replicas of partitions of topic `light`. You expand the cluster with 3 new brokers and, before partition balancing moves partitions to the new broker, you create another topic, `heavy`, which serves 10x more produce/consume requests than `light`. The `heavy` topic also has 100 partitions and replication factor=3. With topic-aware partition balancing, when creating `heavy`, Redpanda prioritizes an equal count of partitions of that topic: brokers 1, 2, and 3 host 100 replicas of `light` and 50 replicas of `heavy`, and brokers 4, 5, and 6 host 50 replicas of `heavy`. Partition balancing then kicks in and evens out the distribution of replicas of `light`, so that there are 50 replicas of `light` and 50 replicas of `heavy` on each broker. This topic-aware partition balancing is enabled by default with the [`partition_autobalancing_topic_aware`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#partition_autobalancing_topic_aware) property. Redpanda supports flexible use of network bandwidth for replicating under-replicated partitions. For example, if only one partition is moving, it can use the entire bandwidth for the broker. Redpanda detects which shards are idle, so other shards can essentially steal bandwidth from them. Total bandwidth is controlled by the [`raft_learner_recovery_rate`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_learner_recovery_rate) property. Redpanda’s default partition balancing includes the following: - When a broker is added to the cluster, some replicas are moved from other brokers to the new broker to take advantage of the additional capacity. - When a broker is down for a configured timeout, existing online replicas are used to construct a replacement replica on a new broker. - When a broker’s free storage space drops below its low disk space threshold, some of the replicas from the broker with low disk space are moved to other brokers. Monitoring unavailable brokers lets Redpanda self-heal clusters by moving partitions from a failed broker to a healthy broker. Monitoring low disk space lets Redpanda distribute partitions across brokers with enough disk space. If free disk space reaches a critically low level, Redpanda blocks clients from producing. For information about the disk space threshold and alert, see [Handle full disks](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/#handle-full-disks). ### [](#partition_autobalancing_mode)Partition balancing settings The [`partition_autobalancing_mode`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#partition_autobalancing_mode) cluster property controls when and how Redpanda automatically rebalances partition replicas across brokers. To check the current value: ```bash rpk cluster config get partition_autobalancing_mode ``` To change the value: ```bash rpk cluster config set partition_autobalancing_mode ``` | Setting | Description | | --- | --- | | node_add | Partition balancing happens when brokers (nodes) are added. To avoid hotspots, Redpanda allocates partitions to random healthy brokers.This is the default setting for clusters without an enterprise license. | | continuous | Redpanda continuously monitors the cluster for broker failures and high disk usage and automatically redistributes partitions to maintain optimal performance and availability. It also monitors rack availability after failures, and for a given partition, it tries to move excess replicas from racks that have more than one replica to racks where there are none. See Configure Continuous Data Balancing.This is the default setting for clusters with an enterprise license. An enterprise license is required. | | off | All partition balancing from Redpanda is turned off.This mode is not recommended for production clusters. Only set to off if you need to move partitions manually. | ## [](#intra-broker-partition-balancing)Intra-broker partition balancing In Redpanda, every partition replica is assigned to a CPU core on a broker. While Redpanda’s default [partition balancing](#partition-replica-balancing) monitors cluster-level events, such as the addition of new brokers or broker failure to balance partition assignments, it does not account for the distribution of partitions _within_ an individual broker. Topic-aware intra-broker partition balancing allows for dynamically reassigning partitions within a broker. Redpanda prioritizes an even distribution of a topic’s partition replicas across all cores in a broker. If a broker’s core count changes, when the broker starts back up, Redpanda can check partition assignments across the broker’s cores and reassign partitions, so that a balanced assignment is maintained across all cores. Redpanda can also check partition assignments when partitions are added to or removed from a broker, and rebalance the remaining partitions between cores. To determine when to use intra-broker partition balancing, use the public metrics for CPU usage described in the [Monitoring](https://docs.redpanda.com/streaming/current/manage/monitoring/#cpu-usage) guide. Configure the following properties to trigger intra-broker partition balancing: | Cluster configuration property | Description | | --- | --- | | core_balancing_on_core_count_change | Set to true to rebalance partition assignments across cores after broker startup, if core count increases or decreases. Default value: true. | | core_balancing_continuous | Set to true to rebalance partition assignments across cores in runtime, for example when partitions are moved to or away from brokers. Default value: false.This requires an enterprise license. | You can also manually trigger intra-broker partition balancing with the Admin API: ```bash curl -X POST http://localhost:9644/v1/partitions/rebalance_cores ``` To check the new partition assignments, make a GET request to the `/v1/partitions` Admin API endpoint: ```bash curl http://localhost:9644/v1/partitions ``` ## [](#manually-move-partitions)Manually move partitions As an alternative to Redpanda partition balancing, you can change partition assignments explicitly with `rpk cluster partitions move`. To reassign partitions with `rpk`: 1. Set the `partition_autobalancing_mode` property to `off`. If Redpanda partition balancing is enabled, Redpanda may change partition assignments regardless of what you do with `rpk`. ```bash rpk cluster config set partition_autobalancing_mode off ``` 2. Show initial replica sets. For example, for topic `test`: ```bash rpk topic describe test -p PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 1 1 [1 2 3] 0 645 1 1 1 [0 1 2] 0 682 2 3 1 [0 1 3] 0 672 ``` 3. Change partition assignments. For example, to change the replica set of partition 1 from `[0 1 2]` to `[3 1 2]`, and to change the replica set of partition 2 from `[0 1 3]` to `[2 1 3]`, run: ```bash rpk cluster partitions move test -p 1:3,1,2 -p 2:2,1,3 NAMESPACE TOPIC PARTITION OLD-REPLICAS NEW-REPLICAS ERROR kafka test 1 [0-1, 1-1, 2-0] [1-1, 2-0, 3-0] kafka test 2 [0-0, 1-0, 3-1] [1-0, 2-0, 3-1] Successfully began 2 partition movement(s). Check the movement status with 'rpk cluster partitions move-status' or see new assignments with 'rpk topic describe -p TOPIC'. ``` or ```bash rpk cluster partitions move -p test/1:3,1,2 -p test/2:2,1,3 ``` 4. Verify that the reassignment is complete with `move-status`: ```bash rpk cluster partitions move-status ONGOING PARTITION MOVEMENTS =========================== NAMESPACE-TOPIC PARTITION MOVING-FROM MOVING-TO COMPLETION-% PARTITION-SIZE BYTES-MOVED BYTES-REMAINING kafka/test 1 [0 1 2] [1 2 3] 57 87369012 50426326 36942686 kafka/test 2 [0 1 3] [1 2 3] 52 83407045 43817575 39589470 ``` Alternatively, run `rpk topic describe` again to show your reassigned replica sets: ```bash rpk topic describe test -p PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 1 2 [1 2 3] 0 645 1 1 2 [1 2 3] 0 682 2 3 1 [1 2 3] 0 672 ``` To cancel all in-progress partition reassignments, run `move-cancel`: ```bash rpk cluster partitions move-cancel ``` To cancel specific movements to or from a given node, run: ```bash rpk cluster partitions move-cancel --node 2 ``` > 📝 **NOTE** > > If you prefer, Redpanda also supports the use of the `AlterPartitionAssignments` Kafka API and using standard kafka tools such as `kafka-reassign-partitions.sh`. ## [](#differences-in-partition-balancing-between-redpanda-and-kafka)Differences in partition balancing between Redpanda and Kafka - In a partition reassignment, you must provide the broker ID for each replica. Kafka validates the broker ID for any new replica that wasn’t in the previous replica set against the list of alive brokers. Redpanda validates all replicas against the list of alive brokers. - When there are two identical partition reassignment requests, Kafka cancels the first one without returning an error code, while Redpanda rejects the second one with `Partition configuration update in progress` or `update_in_progress`. - In Kafka, attempts to add partitions to a topic during in-progress reassignments result in a `reassignment_in_progress` error, while Redpanda successfully adds partitions to the topic. - Kafka doesn’t support shard-level (core) partition assignments, but Redpanda does. For help specifying a shard for partition assignments, see `rpk cluster partitions move --help`. ## [](#assign-partitions-at-topic-creation)Assign partitions at topic creation To manually assign partitions at topic creation, run: ```bash kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --topic custom-assignment --replica-assignment 0:1:2,0:1:2,0:1:2 ``` ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 123: Configure Cluster Properties **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration.md --- # Configure Cluster Properties > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Cluster Properties latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/cluster-property-configuration page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/cluster-property-configuration.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/cluster-property-configuration.adoc description: Learn how to configure cluster properties. page-git-created-date: "2023-06-02" page-git-modified-date: "2025-08-15" support-status: supported --- When you install Redpanda software, cluster configuration properties are automatically set to the default values. Examples of cluster properties include Kafka front-end settings, authentication settings, and settings for enabling features, like Tiered Storage and rack awareness. Redpanda includes cluster properties, broker properties, and topic properties. Some properties can be configured at either the cluster level or the topic level, such as `retention_bytes` (cluster-level) and `retention.bytes` (topic-level). Topic-level properties override cluster-level properties. Most Redpanda configuration properties are cluster properties. | Cluster properties | Broker properties | | --- | --- | | Cluster properties are stored internally and automatically replicated across all nodes, which ensures that each broker is in sync. | Broker properties are stored in the redpanda.yaml file located in the /etc/redpanda directory for each broker. | See also: - [Cluster configuration properties](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/) - [Broker configuration properties](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) and [Configure broker properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/node-property-configuration/) ## [](#edit-cluster-properties)Edit cluster properties > 📝 **NOTE** > > For Kubernetes deployments, see [Configure Cluster Properties in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-configure-helm-chart/). To change any property settings, edit the configuration from the command line using your default text editor. As you make changes, the Redpanda Admin API verifies that the new value is valid. For example, if you change `fetch_max_bytes` from the default of `57671680` to `5o` (using the letter “o” by mistake), the system displays the following message: ```bash PROPERTY PRIOR NEW fetch_max_bytes 57671680 5o Validation errors: * fetch_max_bytes: expected type integer No changes were made. ``` After you save your configuration changes, the new values are automatically applied and a new version number is generated. Any subsequent edits start with the most recent version of the configuration. > 📝 **NOTE** > > Some properties require that you restart the cluster for a change to take effect. The default configuration includes this information in the descriptions for these properties. To assign new values to cluster properties: 1. Open a terminal window and log in to a broker on your cluster. 2. Run `rpk cluster config edit`. To edit tuning properties, run `rpk cluster config edit --all` instead. 3. Edit the configuration file and set values for the [properties](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/) you want to change. 4. Save the file and quit the editor. 5. Run `rpk cluster config status` to see whether the cluster requires a restart. If necessary, you see a message like this: ```bash $ rpk cluster config status NODE CONFIG-VERSION NEEDS-RESTART INVALID UNKNOWN 1 4 true [] [] ``` 6. If necessary, restart the cluster. When you finish your edits, the system updates the configuration and displays a message that lists which property settings were changed, along with their prior and new values. The message also includes the new version number of the configuration. For example: ```bash PROPERTY PRIOR NEW tx_timeout_delay_ms 1000 2000 Successfully updated configuration. New configuration version is 2. ``` > 📝 **NOTE** > > You can also change property values using the `rpk cluster config set` command, but this method does not display the current setting or the description. ## [](#view-current-value-of-a-property)View current value of a property To see the current value of a property, run `rpk cluster config get `. For example: ```bash $ rpk cluster config get log_compression_type producer ``` ## [](#copy-configurations-to-other-clusters)Copy configurations to other clusters Use the `export` option to save the current cluster configuration to a file. You can then copy this file to other clusters, so they can use the same configuration. 1. Export the current configuration settings to a YAML file by running `rpk cluster config export --filename .yaml`. To store the configuration file outside your current working directory, use the full pathname for `--filename`; otherwise, supply the filename to store the file in your current working directory. 2. Copy `.yaml` to the other cluster. 3. Log in to the other cluster, and import the file with the saved configuration by running `rpk cluster config import --filename .yaml`. This command applies the property settings in `.yaml` to all nodes in the cluster. > ⚠️ **CAUTION** > > Redpanda does not support importing cluster-specific identification (such as `cluster_id`) with this command. ## [](#remove-cluster-properties-from-redpanda-yaml)Remove cluster properties from redpanda.yaml If you have a `redpanda.yaml` file that contains both cluster properties and broker properties, the cluster properties are ignored. To remove cluster properties from the `redpanda.yaml` file, run: ```bash rpk cluster config lint ``` This avoids the issue of referring to a previous version or custom configuration. ## [](#suggested-reading)Suggested reading - [rpk cluster config](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-config/) - [Using Raft to centralize cluster configuration in Redpanda](https://redpanda.com/blog/raft-centralized-cluster-configuration-improvements/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 124: Compaction Settings **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings.md --- # Compaction Settings > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Compaction Settings latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/compaction-settings page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/compaction-settings.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/compaction-settings.adoc description: Redpanda's approach to compaction and options for configuring it. page-git-created-date: "2023-12-22" page-git-modified-date: "2026-07-28" support-status: supported --- Configure compaction for your cluster to optimize storage utilization. ## [](#redpanda-compaction-overview)Redpanda compaction overview Compaction is an optional mechanism intended to reduce the storage needs of Redpanda topics. You can enable compaction through configuration of a cluster or topic’s cleanup policy. When compaction is enabled as part of the cleanup policy, a background process executes on a pre-set interval to perform compaction operations. When triggered for a partition, the process purges older versions of records for a given key and only retains the most recent record in that partition. This is done by analyzing closed segments in the partition, copying the most recent records for each key into a new segment, then deleting the source segments. ![Example of topic compaction](https://docs.redpanda.com/streaming/current/shared/_images/compaction-example.png) This diagram illustrates a compacted topic. Imagine a remote sensor network that uses image recognition to track appearances of red pandas in a geographic area. The sensor network employs special devices that send records to a topic when they detect one. You might enable compaction to reduce topic storage while still maintaining a record in the topic of the last time each device saw a red panda, perhaps to see if they stop frequenting a given area. The left side of the diagram shows all records sent across the topic. The right side illustrates the results of compaction; older records for certain keys are deleted from the log. > 📝 **NOTE** > > If your application requires consuming every record for a given key, consider using the `delete` [cleanup policy](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/#change-the-cleanup-policy) instead. > ❗ **IMPORTANT** > > When using [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/), compaction functions at the local storage level. As long as a segment remains in local storage, its records are eligible for compaction. Once a segment is uploaded to object storage and removed from local storage it is not retrieved for further compaction operations. A key may therefore appear in multiple segments between Tiered Storage and local storage. While compaction reduces storage needs, Redpanda’s compaction (just like Kafka’s) does not guarantee perfect de-duplication of a topic. It represents a best effort mechanism to reduce storage needs but duplicates of a key may still exist within a topic. Compaction is not a complete topic operation, either, since it operates on subsets of each partition within the topic. ## [](#configure-a-cleanup-policy)Configure a cleanup policy A compaction policy may be applied to a cluster or to an individual topic. If both are set, the topic-level policy overrides the cluster-level policy. The cluster-level [`log_cleanup_policy`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_cleanup_policy) and the topic-level [`cleanup.policy`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#cleanup-policy) support the following three options: - `delete`: Records are deleted from the topic once the specified retention period (time and/or size allocations) is exceeded. This is the default mechanism and is analogous to disabling compaction. - `compact`: This triggers only cleanup of records with multiple versions. A record that represents the only version for a given key is not deleted. - `compact,delete`: This combines both policies, deleting records exceeding the retention period while compacting multiple versions of records. > ⚠️ **WARNING** > > All topic properties take effect immediately after being set. Do not modify properties on internal Redpanda topics (such as `__consumer_offsets`, `_schemas`, or other system topics) as this can cause cluster instability. ## [](#tune-log-compaction-with-a-dirty-ratio-threshold)Tune log compaction with a dirty ratio threshold Use the dirty ratio to control when log compaction runs in compacted topics. The dirty ratio is the size of dirty segments divided by the total size of closed segments. Dirty segments are closed but un-compacted, meaning they may still contain duplicate keys that exist earlier in the log. ```none dirty_ratio = dirty_segment_bytes / total_closed_segment_bytes ``` Where: - **Dirty segments** are closed segments that may contain duplicate keys that haven’t yet been compacted. - **Closed segments** are all finalized segments in the log. ### [](#configuration-options)Configuration options | Property | Scope | Description | | --- | --- | --- | | min_cleanable_dirty_ratio | Cluster | The minimum ratio between the number of bytes in dirty segments and the total number of bytes in closed segments that must be reached before a partition’s log is eligible for compaction in a compact topic. | | min.cleanable.dirty.ratio | Topic | Topic-level override of the cluster-wide dirty ratio threshold. | | log_compaction_interval_ms | Cluster | Compaction frequency in milliseconds. | | max_compaction_lag_ms | Cluster | The maximum amount of time in milliseconds that a message remains ineligible for compaction. Use to guarantee the maximum delay between the time a message is written and the time the message becomes eligible for compaction. This setting is useful for ensuring that messages are compacted within a predictable timeframe. | | min_compaction_lag_ms | Cluster | The minimum time in milliseconds that a message remains uncompacted in the log. Use to guarantee the minimum length of time that must pass after a message is written before it could be compacted. For example, to provide a lower bound on how long each message will remain in the (uncompacted) head. | | max.compaction.lag.ms | Topic | The maximum amount of time in milliseconds that a message remains ineligible for compaction. Use to guarantee the maximum delay between the time a message is written and the time the message becomes eligible for compaction. | | min.compaction.lag.ms | Topic | The minimum time in milliseconds that a message remains uncompacted in the log. Use to guarantee the minimum length of time that must pass after a message is written before it could be compacted. For example, to provide a lower bound on how long each message will remain in the (uncompacted) head. | Redpanda runs a scan every `log_compaction_interval_ms`. During each scan: - Logs are evaluated for compaction eligibility using their dirty ratio. - Only logs with a dirty ratio greater than the configured threshold are compacted. - Logs are compacted in descending order of dirty ratio to maximize efficiency. ### [](#use-cases-for-dirty-ratio-based-compaction)Use cases for dirty ratio-based compaction | Use Case | Recommended Setting | | --- | --- | | High-throughput topics with frequent key overwrites | Lower min_cleanable_dirty_ratio to enable more aggressive compaction. | | Topics with large segment sizes or expensive I/O | Raise min_cleanable_dirty_ratio to defer compaction until it is more efficient. | | Topics requiring custom tuning | Use min.cleanable.dirty.ratio to override the cluster setting on specific topics. | ## [](#tombstone-record-removal)Tombstone record removal Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone contains a key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted. You can specify how long Redpanda keeps these tombstones for compacted topics using both a cluster configuration property `[tombstone_retention_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#tombstone_retention_ms)` and a topic configuration property [`delete.retention.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#delete-retention-ms). If both are set, the topic-level tombstone retention policy overrides the cluster-level policy. > 📝 **NOTE** > > Redpanda does not remove tombstone records for compacted topics that use Tiered Storage v1. Compacted topics that use [Tiered Storage v2](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#tiered-storage-versions), available starting in Redpanda v26.2, support tombstone removal, configured with `delete.retention.ms`. > > The following restrictions apply to Tiered Storage v1. They do not apply to topics that use Tiered Storage v2: > > - You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`. > > - On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`. To set the cluster-level tombstone retention policy, run the command: ```bash rpk cluster config set tombstone_retention_ms=100 ``` You can unset the tombstone retention policy for a topic so it inherits the cluster-wide default policy: ```bash rpk topic alter-config --delete delete.retention.ms ``` To override the cluster-wide default for a specific topic: ```bash rpk topic alter-config --set delete.retention.ms=5 ``` To disable tombstone removal for a specific topic: ```bash rpk topic alter-config --set delete.retention.ms=-1 ``` Redpanda removes tombstones as follows: - For topics with a `compact` only cleanup policy: Tombstones are removed when the topic exceeds the tombstone retention limit. The `delete.retention.ms` or `tombstone_retention_ms` values therefore also set the time bound that a consumer has in order to see a complete view of the log with tombstones present before they are removed. - For topics with a `compact,delete` cleanup policy: Both the tombstone retention policy and standard garbage collection can remove tombstone records. If obtaining a complete snapshot of the log, including tombstone records, is important to your consumers, set the tombstone retention value such that consumers have enough time for their reads to complete before tombstones are removed. Consumers may not see tombstones if their reads take longer than `delete.retention.ms` and `tombstone_retention_ms`. The trade-offs to ensuring tombstone visibility to consumers are increased disk usage and potentially slower compaction. On the other hand, if more frequent cleanup of tombstones is important for optimizing workloads and space management, consider setting a shorter tombstone retention, for example the typical default of 24 hours (86400000 ms). Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging. ## [](#transactional-control-batch-removal)Transactional control batch removal Transactional workloads write control batches (commit and abort markers) to the log. By default, these markers are retained indefinitely. You can enable their removal during compaction by setting [`log_compaction_tx_batch_removal_enabled`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_compaction_tx_batch_removal_enabled) to `true`: ```bash rpk cluster config set log_compaction_tx_batch_removal_enabled=true ``` When enabled, the [`delete.retention.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#delete-retention-ms) setting is applied to transactional control batches, removing them after the retention period. This uses the same coordinated compaction mechanism as tombstone removal. > 📝 **NOTE** > > For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level to enable removal of transactional control batches. Consider enabling this feature if you have compacted topics with heavy transactional workloads and observe disk usage from accumulated transaction markers. ## [](#compaction-policy-settings)Compaction policy settings The various cleanup policy settings rely on proper tuning of a cluster’s compaction and retention policy options. The applicable settings are: - [`log_compaction_interval`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_compaction_interval_ms): Defines the compaction frequency in milliseconds. (default: 10,000ms) - [`min_cleanable_dirty_ratio`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#min_cleanable_dirty_ratio): Minimum dirty ratio a log must exceed to be eligible for compaction. - [`compaction_ctrl_backlog_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_backlog_size): Defines the size for the compaction backlog of the backlog controller. (default: 10% of disk capacity) - [`compaction_ctrl_min_shares`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_min_shares): Defines the minimum number of I/O and CPU shares the compaction process can use. (default: 10) - [`compaction_ctrl_max_shares`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_max_shares): Defines the maximum number of I/O and CPU shares the compaction process can use. (default: 1,000) - [`storage_compaction_index_memory`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#storage_compaction_index_memory): Defines the amount of memory in bytes that each shard may use for creating the compaction index. This index optimizes execution during compaction operations. (default: 128 MiB) - `storage_compaction_key_map_memory`: Defines the amount of memory in bytes that each shard may use when creating the key map for a partition during compaction operations. The compaction process uses this key map to de-dupe keys within the compacted segments. (default: 128 MiB) - [`compacted_log_segment_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compacted_log_segment_size): Defines the base size for a compacted log segment in bytes. (default: 268435456 \[256 MiB\]) - [`max_compacted_log_segment_size`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#max_compacted_log_segment_size): Defines the maximum size after consolidation for a compacted log segment in bytes. (default: 5368709120 \[5 GiB\]) > 📝 **NOTE** > > Additional [tunable properties](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/) are available but should only be used with direction from Redpanda support. These properties include [`compaction_ctrl_p_coeff`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_p_coeff), [`compaction_ctrl_i_coeff`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_i_coeff), [`compaction_ctrl_d_coeff`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_d_coeff), and [`compaction_ctrl_update_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#compaction_ctrl_update_interval_ms). ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 125: Configure Client Connections **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/configure-client-connections.md --- # Configure Client Connections > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Client Connections latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/configure-client-connections page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/configure-client-connections.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/configure-client-connections.adoc description: Learn about guidelines for configuring client connections in Redpanda clusters for optimal availability. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-11-20" support-status: supported --- Optimize the availability of your clusters by configuring and tuning properties. > 💡 **TIP** > > Before you configure connection limits or reconnection settings, start by gathering detailed data about your client connections. > > - Internal metrics that follow the `vectorized_kafka_rpc_.*connect*` naming pattern provide details on Kafka client connection activity. For example, [`vectorized_kafka_rpc_active_connections`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_kafka_rpc_active_connections) reports the current number of active connections. > > - For Redpanda v25.3 and later, use [`rpk cluster connections list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-connections-list/) or the Admin API ListKafkaConnections endpoint to identify: > > - Which clients and applications are connected > > - Long-lived connections and long-running requests > > - Connections with no activity > > - Whether any clients are causing excessive load > > > By reviewing connection details, you can make informed decisions about tuning connection limits and troubleshooting issues. > > > See also: [ListKafkaConnections reference](https://docs.redpanda.com/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections), [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/#throughput) ## [](#limit-client-connections)Limit client connections To mitigate the risk of a client creating too many connections and using too many system resources, you can configure a Redpanda cluster to impose limits on the number of client connections that can be created. The following Redpanda cluster properties limit the number of connections: - [`kafka_connections_max_per_ip`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_connections_max_per_ip): Similar to Kafka’s `max.connections.per.ip`, this sets the maximum number of connections accepted per IP address by a broker. - [`kafka_connections_max_overrides`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_connections_max_overrides): A list of IP addresses for which `kafka_connections_max_per_ip` is overridden and doesn’t apply. - [`kafka_connections_max`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_connections_max): Similar to Kafka’s `max.connections`, this sets the maximum number of connections per broker. Redpanda also provides properties to manage the rate of connection creation: - [`kafka_connection_rate_limit`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_connection_rate_limit): This property limits the maximum rate of connections created per second. It applies to each CPU core. - [`kafka_connection_rate_limit_overrides`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#kafka_connection_rate_limit_overrides): A list of IP addresses for which `kafka_connection_rate_limit` is overridden and doesn’t apply. > 📝 **NOTE** > > - These connection limit properties are disabled by default. You must manually enable them. > > - The total number of connections is not equal to the number of clients, because a client can open multiple connections. As a conservative estimate, for a cluster with N brokers, plan for N + 2 connections per client. ## [](#configure-client-reconnections)Configure client reconnections You can configure the Kafka client backoff and retry properties to change the default behavior of the clients to suit your failure requirements. Set the following Kafka client properties on your application’s producer or consumer to manage client reconnections: - `reconnect.backoff.ms`: Amount of time to wait before attempting to reconnect to the broker. The default is 50 milliseconds. - `reconnect.backoff.max.ms`: Maximum amount of time in milliseconds to wait when reconnecting to a broker. The backoff increases exponentially for each consecutive connection failure, up to this maximum. The default is 1000 milliseconds (1 second). Additionally, you can use Kafka properties to control message retry behavior. Delivery fails when either the delivery timeout or the number of retries is met. - `delivery.timeout.ms`: Amount of time for message delivery, so messages are not retried forever. The default is 120000 milliseconds (2 minutes). - `retries`: Number of times a producer can retry sending a message before marking it as failed. The default value is 2147483647 for Kafka >= 2.1, or 0 for Kafka <= 2.0. - `retry.backoff.ms`: Amount of time to wait before attempting to retry a failed request to a given topic partition. The default is 100 milliseconds. ## [](#see-also)See also - [Configure Producers](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/) - [Manage Throughput](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput/) --- # Page 126: Configure Continuous Data Balancing **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing.md --- # Configure Continuous Data Balancing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Continuous Data Balancing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/continuous-data-balancing page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/continuous-data-balancing.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/continuous-data-balancing.adoc description: Continuous Data Balancing simplifies operations with self-healing clusters that dynamically balance partitions. page-topic-type: how-to personas: infrastructure_operator learning-objective-1: Enable Continuous Data Balancing on a Redpanda cluster learning-objective-2: Check data balancing status using rpk learning-objective-3: Cancel partition balancing moves for a specific node page-git-created-date: "2023-06-02" page-git-modified-date: "2026-03-31" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Continuous Data Balancing continuously monitors your node and rack availability and disk usage, dynamically balancing partitions to maintain smooth operations and optimal cluster performance. Continuous Data Balancing also maintains the configured replication level, even after infrastructure failure. Node availability has the highest priority in data balancing. After a rack (with all nodes belonging to it) becomes unavailable, Redpanda moves partition replicas to the remaining nodes. This violates the rack awareness constraint. After the rack (or a replacement rack) becomes available, Redpanda repairs the constraint by moving excess replicas from racks that have more than one replica to the newly-available rack. After reading this page, you will be able to: - Enable Continuous Data Balancing on a Redpanda cluster - Check data balancing status using rpk - Cancel partition balancing moves for a specific node ## [](#set-continuous-data-balancing-properties)Set Continuous Data Balancing properties To enable Continuous Data Balancing, set the `partition_autobalancing_mode` property to `continuous`. Customize the following properties to monitor node availability and disk usage. | Property | Description | | --- | --- | | partition_autobalancing_node_availability_timeout_sec | When a node is unreachable for the specified amount of time, Redpanda acts as if the node had been decommissioned: rebalancing begins, re-creating all of its replicas on other nodes in the cluster.The node remains part of the cluster and can rejoin when it comes back online. A node that was actually decommissioned is removed from the cluster.Default is 900 seconds (15 minutes). | | partition_autobalancing_node_autodecommission_timeout_sec | When a node is unavailable for this timeout duration, Redpanda automatically and permanently decommissions the node. This property only applies when partition_autobalancing_mode is set to continuous. Unlike partition_autobalancing_node_availability_timeout_sec, which moves partitions while keeping the node in the cluster, this property removes the node from the cluster entirely. A decommissioned node cannot rejoin the cluster.Only one node is decommissioned at a time. If a decommission is already in progress, automatic decommission does not trigger until it completes. If the decommission stalls (for example, because the node holds the only replica of a partition), manual intervention is required. See Node-wise Partition Recovery.By default, this property is null and automatic decommission is disabled. | | partition_autobalancing_max_disk_usage_percent | When a node fills up to this disk usage percentage, Redpanda starts moving replicas off the node to other nodes with disk utilization below the percentage.Default is 80%. | For the other `partition_autobalancing_mode` options, see [Cluster balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-balancing/). ## [](#use-data-balancing-commands)Use data balancing commands Use the following `rpk` commands to monitor and control data balancing. ### [](#check-data-balancing-status)Check data balancing status To see the status, run: ```bash rpk cluster partitions balancer-status ``` This shows the time since the last data balancing, the number of replica movements in progress, the nodes that are unavailable, and the nodes that are over the disk space threshold (default = 80%). It also returns a data balancing status: `off`, `ready`, `starting`, `in-progress`, or `stalled`. If the command reports a `stalled` status, verify: - Are there enough healthy nodes? For example, in a three node cluster, no movements are possible for partitions with three replicas. - Does the cluster have sufficient space? Partitions are not moved if all nodes in the cluster are utilizing more than their disk space threshold. - Do all partitions have quorum? Partitions are not moved if the majority of its replicas are down. - Are any nodes in maintenance mode? Partitions are not moved if a node is in maintenance mode. ### [](#cancel-data-balancing-moves)Cancel data balancing moves To cancel the current partition balancing moves, run: ```bash rpk cluster partitions movement-cancel ``` To cancel partition moves on a specific node, use the `--node` flag. For example: ```bash rpk cluster partitions movement-cancel --node 1 ``` > 📝 **NOTE** > > If continuous balancing is still enabled and the cluster remains unbalanced, Redpanda schedules another partition balancing round. To stop all balancing, first set `partition_autobalancing_mode` to `off`, then cancel the current data balancing moves. ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 127: Decommission Brokers **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/decommission-brokers.md --- # Decommission Brokers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Decommission Brokers latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/decommission-brokers page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/decommission-brokers.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/decommission-brokers.adoc description: Remove a broker so that it is no longer considered part of the cluster. page-git-created-date: "2023-08-03" page-git-modified-date: "2026-08-13" support-status: supported --- When you decommission a broker, its partition replicas are reallocated across the remaining brokers and it is removed from the cluster. You may want to decommission a broker in the following circumstances: - The broker has lost its storage and you need a new broker with a new node ID (broker ID). - You are replacing a broker, for example to upgrade the Linux kernel or to replace the hardware. - You are removing a broker to decrease the size of the cluster. > ⚠️ **CAUTION** > > When a broker is decommissioned, it cannot rejoin the cluster. If a broker with the same ID tries to rejoin the cluster, it is rejected. ## [](#decommissioning-methods)Decommissioning methods There are two ways to decommission brokers in Redpanda: - Manual decommissioning (described in this guide): Use `rpk` commands to explicitly decommission a broker when you need full control over the timing and selection of brokers to remove. - Automatic decommissioning: When [Continuous Data Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/) is enabled, you can configure the [partition\_autobalancing\_node\_autodecommission\_timeout\_sec](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/#partition_autobalancing_node_autodecommission_timeout_sec) property to automatically decommission brokers that remain unavailable for a specified duration. Both methods permanently remove the broker from the cluster. Decommissioned brokers cannot rejoin. > 📝 **NOTE** > > Entering [maintenance mode](https://docs.redpanda.com/streaming/current/manage/node-management/) before decommissioning is optional. Decommissioning drains partition leadership gracefully on its own. ## [](#what-happens-when-a-broker-is-decommissioned)What happens when a broker is decommissioned? When a broker is decommissioned, the controller leader creates a reallocation plan for all partition replicas that are allocated to that broker. By default, this reallocation is done in batches of 50 to avoid overwhelming the remaining brokers with Raft recovery. See [`partition_autobalancing_concurrent_moves`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#partition_autobalancing_concurrent_moves). The reallocation of each partition is translated into a Raft group reconfiguration and executed by the controller leader. The partition leader then handles the reconfiguration for its Raft group. After the reallocation for a partition is complete, it is recorded in the controller log and the status is updated in the topic tables of each broker. The decommissioning process is successful only when all partition reallocations have been completed successfully. The controller leader polls for the status of all the partition-level reallocations to ensure that everything completes as expected. During the decommissioning process, new partitions are not allocated to the broker that is being decommissioned. After all the reallocations have been completed successfully, the broker is removed from the cluster. > 📝 **NOTE** > > The decommissioning process is designed to tolerate controller leadership transfers. > 💡 **TIP** > > This guide uses `jq` to make parsing JSON output easier. For additional details, see [jq downloads](https://stedolan.github.io/jq/download/). ## [](#should-you-decommission-brokers)Should you decommission brokers? There are several considerations for determining your cluster’s minimum broker count, and whether or not to decommission any brokers. For the purposes of this section, the focus is on a cluster with seven brokers. In subsequent sections, the output from the given commands provides additional details to help you determine the minimum number of brokers. ### [](#availability)Availability You should have a sufficient number of brokers to properly span across each rack or availability zone. Run the following command to determine whether rack awareness is enabled in your cluster: ```bash rpk cluster config get enable_rack_awareness ``` true When enabled, you can view which rack each broker is assigned to by running the following command: ```bash rpk cluster info ``` Example output ```bash CLUSTER ======= redpanda.560e2403-3fd6-448c-b720-7b456d0aa78c BROKERS ======= ID HOST PORT RACK 0 redpanda-0.testcluster.local 32180 A 1 redpanda-1.testcluster.local 32180 A 4 redpanda-3.testcluster.local 32180 B 5* redpanda-2.testcluster.local 32180 B 6 redpanda-4.testcluster.local 32180 C 8 redpanda-6.testcluster.local 32180 C 9 redpanda-5.testcluster.local 32180 D ``` The output shows four racks (A/B/C/D), so you might want to have at least four brokers to make use of all racks. Rack awareness is just one aspect of availability. Check out [High Availability](https://docs.redpanda.com/streaming/current/manage/high-availability/) for details on deploying Redpanda for high availability. ### [](#cost)Cost Infrastructure costs increase with each broker because each broker requires a dedicated node (instance), so adding a broker means an additional instance cost. For example, if the instance cost is $1925 per month in a cluster with seven brokers, the instance cost for each broker is $275. Reducing the number of brokers from seven to five would save $550 per month ($275 x 2), and reducing it to three brokers would save $1100 per month. You must also consider other costs, but they won’t be as impacted by changing the broker count. ### [](#data-retention)Data retention Local data retention is determined by the storage capability of each broker and how much data is being produced over a given period (that is, producer throughput). When decommissioning, storage capability must take into account both the free storage space and amount of space already used by existing partitions. Run the following command to determine how much storage is being used (in bytes) on each broker: ```bash rpk cluster logdirs describe --aggregate-into broker ``` Example output ```bash BROKER SIZE ERROR 0 263882790656 1 256177979648 2 257698037504 3 259934992896 4 254087316992 5 258369126144 6 255227998208 ``` The example output shows that each broker contains roughly 240GB of data, which means scaling down to five brokers would require each broker to have at least 337GB to hold current data. Throughput is the primary measurement required to calculate future data storage requirements. In the example cluster there is a throughput of 200MB/sec, which means it will generate 0.72TB/hour (or 17.28TB/day, or 120.96TB/wk). Divide this amount by the target number of brokers to get an estimate of how much storage is needed to retain that much data for various periods of time: | Retention | Disk size (on each of the 5 brokers) | | --- | --- | | 30mins | (200MB/sec * 30mins * 1.1) = 0.396TB / 5 brokers = 79.2GB | | 6hrs | (200MB/sec * 6hrs * 1.1) = = 4.752TB / 5 brokers = 950.4GB | | 1d | (200MB/sec * 1d * 1.1) = 19.008TB / 5 brokers = 3.8TB | | 3d | (200MB/sec * 3d * 1.1) = 57.024TB / 5 brokers = 11.4TB | In the example cluster, only 6 hours of data locally must be retained (any older data is moved to Tiered Storage with a retention of 1 year). So each broker should have available storage of around 1.2TB, taking into account both throughput and current data. Cost and use case requirements dictate how much to spend on local disk capacity. Tiered Storage can help to both decrease costs and expand data retention capabilities. For details, see [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/). > 📝 **NOTE** > > At this point in the example, it remains unclear whether or not it makes sense to scale down to five brokers. Current calculations are based on five brokers. You can consider other broker counts later as needed. > > Additionally, assumptions have been made regarding a constant throughput and perfect data balancing. Throughput fluctuates across all partitions, which causes data imbalance. The calculations above attempt to accommodate for this by padding disk size by 1%. You can increase this buffer (for example, in the case of expected hot spot partitions). For details on sizing, see [Sizing Guidelines](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/sizing/). ### [](#durability)Durability The brokers in a Redpanda cluster are part of a Raft group that requires sufficient brokers to form a quorum-based majority (minimally, three brokers). Each topic’s partitions are also Raft groups, so your cluster also needs to have at least as many brokers as the lowest replication factor across all topics. One way to find the max replication factor across all topics in a cluster is to run the following command: ```bash rpk topic list | tail -n +2 | awk '{print $3}' | sort -n | tail -1 ``` 5 In this example the highest replication factor is 5, which means at least 5 brokers are required in this cluster. Generally, a cluster can withstand a higher number of brokers going down if there are more brokers in the cluster. For details, see [Raft consensus algorithm](https://docs.redpanda.com/streaming/current/get-started/architecture/#raft-consensus-algorithm). ### [](#partition-count)Partition count It is a best practice to make sure the total partition count does not exceed 1K per core. This max partition count depends on many other factors (such as memory per core, CPU performance, throughput, and latency requirements). Exceeding 1K partitions per core can lead to increased latency, increased number of partition leadership elections, and general reduced stability. Run the following command to get the total partition count: ```bash curl -sk http://:/v1/partitions/local_summary | jq .count ``` 3018 To determine the number of cores that are available across the remaining brokers: ```bash rpk cluster info -b --detailed ``` Example output ```bash ID HOST PORT RACK CORES MEMBERSHIP IS-ALIVE VERSION UUID 0 redpanda-0.testcluster.local 32180 A 8 active true 25.2.13 24c08934-94f6-478c-b57d-45239f452488 1 redpanda-1.testcluster.local 32180 B 8 active true 25.2.13 7f3c1c6e-2f4d-4c8a-9c6e-0a8a6d8b2b61 2 redpanda-2.testcluster.local 32180 C 8 active true 25.2.13 b2e8d4a9-91c1-4b55-9f6a-3f7a2d3c5e44 3 redpanda-3.testcluster.local 32180 A 8 active true 25.2.13 c9a6f7d2-5e3b-4f8c-8d1a-6e2b4a9f7c31 4 redpanda-4.testcluster.local 32180 B 8 active true 25.2.13 4e8b1c3a-9d7f-4a6e-b2c5-8f6d1a3e9b74 5 redpanda-5.testcluster.local 32180 C 8 active true 25.2.13 a1f9c6b4-3d2e-4a8f-9e5b-7c6d8a1b2f93 6 redpanda-6.testcluster.local 32180 A 8 active true 25.2.13 e6b4d2a8-5f1c-4e9b-8a3d-9c7f1b6a5e42 ``` In this example each broker has 8 cores available. If you plan to scale down to five brokers, then you would have 40 cores available, which means that your cluster is limited by core count to 40K partitions (well above the current 3018 partitions). > 📝 **NOTE** > > To best ensure the stability of the cluster, stay under 50K partitions per cluster. ### [](#decommission-assessment)Decommission assessment The considerations tested above yield the following: - At least four brokers are required based on availability. - Cost is not a limiting factor in this example, but lower cost (and lower broker count) is always best. - At least 1.2TB of data resides on each broker (if spread across five brokers). This falls within the 1.5TB of local storage available in this example. - At least five brokers are required based on the highest replication factor across all topics. - At 3018 partitions, the partition count is so low as to not be a determining factor in broker count (a single broker in this example environment could handle many more partitions). So the primary limitation consideration is the replication factor of five, meaning that you could scale down to five brokers at minimum. ## [](#decommission-a-broker)Decommission a broker 1. List your brokers and their associated broker IDs: ```bash rpk cluster info \ -X brokers=: ``` 2. Decommission the broker with your selected broker ID: ```bash rpk cluster brokers decommission \ -X admin.hosts=: ``` > 📝 **NOTE** > > If the broker is not running, add the `--skip-liveness-check` flag. Without this flag, the command fails when it cannot verify the version of the offline broker. The command returns `Success, broker decommission started.` when the cluster accepts the request. Decommissioning continues in the background while partitions move off the broker. You can monitor the decommissioning status to follow its progress. 3. Monitor the decommissioning status: ```bash rpk cluster brokers decommission-status \ -X admin.hosts=: ``` The output uses cached cluster health data that is refreshed every 10 seconds. When the completion column for all rows is 100%, the broker is decommissioned. > ❗ **IMPORTANT** > > If you add a new broker, make sure to give it a unique ID. Do not reuse the ID of the decommissioned broker. ## [](#troubleshooting)Troubleshooting If the decommissioning process is not making progress, investigate the following potential issues: - **Absence of a controller leader or partition leader**: The controller leader serves as the orchestrator for decommissioning. Additionally, if one of the partitions undergoing reconfiguration does not have a leader, the reconfiguration process may stall. Make sure that an elected leader is present for all partitions. - **Bandwidth limitations for partition recovery**: Try increasing the value of [`raft_learner_recovery_rate`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#raft_learner_recovery_rate), and monitor the status using the [`redpanda_raft_recovery_partition_movement_available_bandwidth`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_raft_recovery_partition_movement_available_bandwidth) metric. If these steps do not allow the decommissioning process to complete, enable `TRACE` level logging on the controller leader to investigate any other issues. ## [](#suggested-reading)Suggested reading - [`rpk cluster brokers decommission`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-brokers-decommission/) - [Engineering a more robust Raft group reconfiguration](https://redpanda.com/blog/raft-protocol-reconfiguration-solution) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 128: Manage Disk Space **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization.md --- # Manage Disk Space > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Disk Space latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/disk-utilization page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/disk-utilization.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/disk-utilization.adoc description: Redpanda provides several ways to manage disk space to ensure the stability of a cluster. page-git-created-date: "2023-06-02" page-git-modified-date: "2026-05-26" support-status: supported --- It is important to manage the local disk space in Redpanda to ensure the stability of a cluster. If a node’s disk space reaches a critically-low level, then Redpanda blocks clients from writing new data. If a node runs out of disk space, then the Redpanda process terminates. This has a significant impact on performance, as client connections and topic data concentrates on fewer nodes. Redpanda provides several ways to manage disk space, with varying degrees of flexibility and control over what data is removed from local disk and when. - Redpanda space management, when used with [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/), treats local disks as a cache. - For more granularity, you can configure topic-level retention policies to manage log cleanup based on partition size or age. - You can configure storage thresholds to alert you when disk space is running low. - You can enable [Continuous Data Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/) to ensure well-balanced disk usage across the cluster. - You can create a ballast file to allow fast recovery from a full disk. ## [](#configure-message-retention)Configure message retention By default, all topics on Redpanda Streaming clusters retain 24 hours of data on local disk, while Redpanda Cloud topics retain 6 hours of data. Redpanda makes use of dynamic [space management](#space_management) strategies to save on disk space. If data is written fast enough, however, it’s possible to exhaust local disk space even when using Tiered Storage. Proper configuration of message retention properties for your use case can prevent this from happening. Retention properties control the minimum length of time messages are kept on disk before they’re deleted or compacted. Setting message retention properties is the best way to prevent old messages from accumulating on disk to the point that the disk becomes full. You can configure retention properties to delete messages based on the following conditions: - Message age is exceeded. - Aggregate message size in the topic is exceeded. - A combination of message age and aggregate size, triggered when either is exceeded. You may set retention properties at the topic level or the cluster level. If a value isn’t specified for the topic, then the topic uses the value for the cluster. Note that cluster-level property names use snake-case, while topic-level properties use dots. > 📝 **NOTE** > > Although retention policy is set at a cluster or topic level, it applies to each partition in a topic independently. Within a partition, only closed segments are eligible for deletion or compaction. When a time-based policy is set, all messages within the segment must exceed the set limit. For example, assume the `retention.ms` for a topic is 300,000 ms (5 minutes). If the `segment.ms` is 1,800,000ms (30 minutes) then messages will remain for a minimum of 30 minutes while the segment containing them remains open. In this scenario, five minutes after a new segment begins, the retention policy would trigger deletion of the closed segment. | Retention property | Cluster level | Topic level(overrides cluster configuration) | | --- | --- | --- | | Time-based | log_retention_msDefault - 604800000 | retention.msNo default | | Size-based | retention_bytesNo default | retention.bytesNo default | | Time-based (with Tiered Storage enabled) | retention_local_target_ms_defaultDefault - 86400000 | retention.local.target.msNo default | | Size-based (with Tiered Storage enabled) | retention_local_target_bytes_defaultDefault - null | retention.local.target.bytesNo default | | Segment lifetime | log_segment_msNo default | segment.msNo default | | Segment size | log_segment_sizeDefault - 1073741824 | segment.bytesNo default | Data expires from object storage following both `retention.ms` and `retention.bytes`. For example, if `retention.bytes` is set to 10 GiB, then every partition in the topic has a limit of 10 GiB storage. When `retention.bytes` is exceeded by data in object storage, the data in object storage is trimmed, even if `retention.ms` is not yet exceeded. With Tiered Storage enabled, data expires from local storage following `retention.local.target.ms` or `retention.local.target.bytes`. ![Time-based segment rolling](https://docs.redpanda.com/streaming/current/shared/_images/segment-rolling-size-and-time.png) Retention policy functions by deleting or compacting closed segments. The segment lifetime and segment size configurations help ensure new segments are created regularly within each partition. This illustration shows how Redpanda creates new segments based on size and time. After the limit for a segment is reached, whether it’s size- or time-based, Redpanda closes the segment and begins filling a new segment. If the limits are set too high, the segment may fill available disk space before closing and therefore never become eligible for deletion or compaction. If the values are set too low, your partitions will have a large number of segments that must be checked each time deletion or compaction processes execute, having a potential adverse impact on system resource utilization. > 📝 **NOTE** > > Both size-based and time-based retention policies are applied simultaneously. It’s possible for your size-based property to override your time-based property, or vice versa. For example, if your size-based property requires removing one segment, and your time-based property requires removing three segments, then three segments are removed. Size-based properties reclaim disk space as close as possible to the maximum size, without exceeding the limit. Redpanda runs a log cleanup process in the background to apply these policy settings. If you start to run out of disk space, adjusting your retention properties is an excellent way to reduce the amount of disk space used. See also: - [Manage local capacity for Tiered Storage topics](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#manage-local-capacity-for-tiered-storage-topics) - [Delete records from a topic](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/#delete-records-from-a-topic) ### [](#set-time-based-retention)Set time-based retention Redpanda enforces time-based retention to manage log segment deletion according to record timestamps. Messages are eligible for deletion when their age exceeds the value specified in `log_retention_ms` (the cluster-level property) or `retention.ms` (the topic-level property). Only closed segments are eligible for deletion and all messages in a closed segment must exceed the age limit before Redpanda considers the segment for cleanup. If `retention.ms` is not set at the topic level, the topic inherits the `log_retention_ms` setting. Starting in v25.3, Redpanda validates timestamps using the properties `log_message_timestamp_before_max_ms` and `log_message_timestamp_after_max_ms`. These settings ensure that records are only accepted if their timestamps are within an allowed range, and prevent retention issues caused by records with timestamps that are too far in the past or future. By validating directly from record batches, Redpanda ensures: - Accurate, timestamp-based retention across clusters - Predictable and consistent data deletion behavior - Proper retention alignment for replicated and disaster recovery use cases Clusters running versions prior to v25.3 may continue to use the broker timestamp (`broker_timestamp`), which is populated when the broker first receives the record. You can also manually opt in clusters running versions prior to v25.3 to this new `max_timestamp` behavior using the Admin API: ```bash curl --request PUT http://localhost:9644/v1/features/validated_batch_timestamps -d '{"state":"active"}' ``` > ❗ **IMPORTANT** > > When manually opting in clusters running versions prior to v25.3, it is particularly important for users of MirrorMaker2 and Shadowing to enable this feature flag on their destination clusters. Without this feature flag, active clusters created by MirrorMaker2 that are running versions prior to v25.3 will have retention enforced from the date of cluster creation rather than the date of the data. You can see this if you compare the data present on the original cluster versus the newly-replicated cluster, as it will show a difference in retention enforcement. > 📝 **NOTE** > > Make sure that all of the produced data sent to `redpanda` pre-validation checks is correct with respect to having properly set `log_message_timestamp_before_max_ms` and `log_message_timestamp_after_max_ms` in each record batch. If these timestamps are missing or incorrect: > > - Data reclaimed earlier than you expect when using `broker_timestamp`\-based retention due to unset or low `max_timestamp` values > > - Retention could be blocked due to high `max_timestamp` at some point in the future. To set retention time for a single topic, use `retention.ms`, which overrides `log_retention_ms`. - `retention.ms` - Topic-level property that specifies how long a message stays on disk before it’s deleted. To minimize the likelihood of out-of-disk outages, set `retention.ms` to `86400000`, which is one day. There is no default. To set `retention.ms` on an individual topic: ```bash rpk topic alter-config --set retention.ms= ``` - `log_retention_ms` - Cluster-level property that specifies how long a message stays on disk before it’s deleted. To minimize the likelihood of out-of-disk outages, set `log_retention_ms` to `86400000`, which is one day. The default is `604800000`, which is one week. > ⚠️ **CAUTION** > > Do not set `log_retention_ms` to `-1` unless you’re using [remote write with Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#remote-write) to upload segments to object storage. Setting it to `-1` configures indefinite retention, which can fill disk space. See also: - [Broker timestamps](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/#broker-timestamps) ### [](#set-size-based-retention)Set size-based retention Messages are eligible for deletion after the storage size of the partition containing them exceeds the value specified in `retention_bytes` (the cluster-level property) or `retention.bytes` (the topic-level property). If `retention.bytes` is not set at the topic level, the topic inherits the `retention_bytes` setting. Segments are deleted in chronological order until the partition is back under the specified size limit. - `retention.bytes` - Topic-level property that specifies the maximum size of a partition. There is no default. To set `retention.bytes`: ```bash rpk topic alter-config --set retention.bytes= ``` - `retention_bytes` - Cluster-level property that specifies the maximum size of a partition. Set this to a value that is lower than the disk capacity, or a fraction of the disk capacity based on the number of partitions per topic. For example, if you have one partition, `retention_bytes` can be 80% of the disk size. If you have 10 partitions, it can be 80% of the disk size divided by 10. The default is `null`, which means that retention based on topic size is disabled. To set `retention_bytes`: ```bash rpk cluster config set retention_bytes ``` ## [](#configure-offset-retention)Configure offset retention Redpanda supports consumer group offset retention through both periodic offset expiration and the Kafka OffsetDelete API. For periodic offset expiration, set the retention duration of consumer group offsets and the check period. Redpanda identifies offsets that are expired and removes them to reclaim storage. For a consumer group, the retention timeout starts from when the group becomes empty as a consequence of losing all its consumers. For a standalone consumer, the retention timeout starts from the time of the last commit. Once elapsed, an offset is considered to be expired and is discarded. | Property | Description | | --- | --- | | group_offset_retention_check_ms | Period at which Redpanda checks for expired consumer group offsets. | | group_offset_retention_sec | Retention duration of consumer group offsets. | | legacy_group_offset_retention_enabled | Enable group offset retention for Redpanda clusters upgraded from versions prior to v23.1. | Redpanda supports group offset deletion with the Kafka OffsetDelete API through rpk with the [`rpk group offset-delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-group/rpk-group-offset-delete/) command. The offset delete API provides finer control over culling consumer offsets. For example, it enables the manual removal of offsets that are tracked by Redpanda within the `__consumer_offsets` topic. The offsets requested to be removed will be removed only if either the group in question is in a dead state, or the partitions being deleted have no active subscriptions. ## [](#manage-transaction-coordinator-disk-usage)Manage transaction coordinator disk usage Redpanda uses the internal topic `kafka_internal/tx` to store transaction metadata for exactly-once and transactional producers. The log files contain all historical transactions, both committed and current open ones. Over time, this topic can consume excessive disk space in niche use cases that generate a large number of transactional sessions. You can manage the disk usage of `kafka_internal/tx` by tuning the following cluster properties: - `[transaction_coordinator_delete_retention_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#transaction_coordinator_delete_retention_ms)`. Default: `604800000` (7 days). - `[transactional_id_expiration_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#transactional_id_expiration_ms)`. Default: `604800000` (7 days). To mitigate unbounded growth of `kafka_internal/tx` disk usage and manage its storage consumption more effectively, [monitor your storage metrics](#monitor-disk-space) and lower the values of the relevant properties as needed. To adjust these properties, run: ```bash rpk cluster config set transaction_coordinator_delete_retention_ms= transactional_id_expiration_ms= ``` ## [](#configure-segment-size)Configure segment size The `log_segment_size` property specifies the size of each log segment within the partition. Redpanda closes segments after they exceed this size and messages begin filling a new segment. ![Cluster message retention hierarchy](https://docs.redpanda.com/streaming/current/shared/_images/cluster-message-retention.png) To set `log_segment_size`: ```bash rpk cluster config set log_segment_size ``` If you know which topics will receive more data, it’s best to specify the size for each topic. To configure log segment size on a topic: ```bash rpk topic alter-config --set segment.bytes= ``` ### [](#segment-size-for-compacted-topics)Segment size for compacted topics Compaction, or key-based retention, saves space by retaining at least the most recent value for a message key within a topic partition’s log and discarding older values. Compaction runs periodically in the background in a best effort fashion, and it doesn’t guarantee that there are no duplicate values per key. When compaction is configured, topics take their size from `compacted_log_segment_size`. The `log_segment_size` property does not apply to compacted topics. When compaction executes, one or more segments are merged into one new compacted segment. The old segments are deleted. The size of the initial segments are controlled by `segment.bytes`. The `max_compacted_log_segment_size` property controls how many segments are merged together. For example, if you set `segment.bytes` to 128 MB, but leave `max_compacted_log_segment_size` at 5 GB, fresh segments are 128 MB but merged segments may grow up to 5 GB after compaction. Redpanda periodically performs compaction in the background. The compaction period is configured by the cluster property [log\_compaction\_interval\_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_compaction_interval_ms). Keep in mind that very large segments delay, or possibly prevent, compaction. A very large active segment cannot be cleaned up or compacted until it is closed, and very large closed segments require significant memory and CPU to process for compaction. Very small segments increase the frequency of processing for applying compaction and resource limits. To calculate an upper limit on segment size, divide the disk size by the number of partitions. For example, if you have a 128 GB disk and 1000 partitions, the upper limit of the segment size is `134217728`. Default is `1073741824`. For details about how to modify cluster configuration properties, see [Cluster configuration](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/). For further information on how compaction works, see [Compaction tuning](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/compaction-settings/). ### [](#log-rolling)Log rolling Writing data for a topic usually spans multiple log segments. An **active segment** of a topic is a log segment that is being written to. As data of a topic is written and an active segment becomes full (reaches `log_segment_size`), it’s closed and changed to read-only mode. A new segment is created and set to read-write mode, and it becomes the active segment. **Log rolling** is the rotation between segments to create a new active segment. Configurable timeouts can also trigger log rolling. This is useful when applying topic retention limits within a known fixed duration. A log rolling timeout starts from the first write to an active segment. When a timeout elapses before the segment is full, the segment is rolled. The timeouts are configured with cluster-level and topic-level properties: - [log\_segment\_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_segment_ms) (or `log.roll.ms`) is a cluster property that configures the default segment rolling timeout for all topics of a cluster. To set `log_segment_ms` for all topics of a cluster for a duration in milliseconds: ```bash rpk cluster config set log_segment_ms ``` - `segment.ms` is a topic-level property that configures the default segment rolling timeout for one topic. It’s not set by default. If set, it overrides `log_segment_ms`. To set `segment.ms` for a topic: ```bash rpk topic alter-config --set segment.ms= ``` - [log\_segment\_ms\_min](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_segment_ms_min) and [log\_segment\_ms\_max](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#log_segment_ms_max) are cluster-level properties that configure the lower and upper limits, respectively, of log rolling timeouts. ## [](#space_management)Space management > 📝 **NOTE** > > Space management only works when [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) is enabled on all topics. Space management and the housekeeping process only considers removing data that is safely stored in Tiered Storage. The goal of space management is to utilize the local disk space as a cache. It works alongside [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) to provide faster access to recent data, while making sure that local disk space is managed in accordance with retention policies and other processes, such as [Continuous Data Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/continuous-data-balancing/) and [decommissioning](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/decommission-brokers/). Space management divides the disk space into different areas that can be managed separately: - Reserved disk space (`disk_reservation_percent`) is the area of disk space that Redpanda does not use. - As the disk space used by the other areas grows to their target sizes, the reserved space provides buffer space to avoid free disk space alerts. - SSDs that run near capacity can experience performance degradation, so this buffer space prevents the disks from running at capacity. - Cache storage (the minimum of `cloud_storage_cache_size_percent` or `cloud_storage_cache_size`) is the maximum size of the [disk cache](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#caching) used by Tiered Storage. As the cache reaches its limit, new data added to the cache removes old data from the cache. - Log storage (the minimum of `retention_local_target_capacity_percent` or `retention_local_target_capacity_bytes`) is the area of disk space used for topic data. This is typically 70-80% of total disk space. ![Redpanda disk storage categories](https://docs.redpanda.com/streaming/current/shared/_images/disk_storage.png) Log segment eviction occurs in each of the following phases. As soon as log storage usage falls below the target, the eviction process ends. > ❗ **IMPORTANT** > > Redpanda’s space management features are enabled with the [`space_management_enable`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#space_management_enable) parameter. As of Redpanda **v23.3.2**, all new clusters default this value to `true`. When upgrading from older versions, ensure this parameter is set to `true` if you wish to make use of space management as described here. Alternatively, if you wish to explicitly disable these features, set this property to `false`. See also: [Object storage housekeeping](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#object-storage-housekeeping) ### [](#phases-of-data-removal)Phases of data removal #### [](#phase-1-follow-retention-policy)Phase 1: Follow retention policy A housekeeping process in Redpanda periodically performs compaction and removes partition data that has expired according to your retention policy. This applies to both Tiered Storage and non-Tiered Storage topics. Space management attempts to apply retention to partitions in the order that removes the largest amount of data. - When `retention_local_strict` is false (default), the housekeeping process removes data above the configured log storage reservation. - When `retention_local_strict` is true, the housekeeping process uses local retention settings to select what data to remove. > 📝 **NOTE** > > The `retention_local_strict` property is set to true in clusters upgraded from release `23.1` and earlier. #### [](#phase-2-trim-to-local-retention)Phase 2: Trim to local retention This phase removes partition data that exceeds the effective local retention policy. This includes the explicit retention settings applied to a topic, as well as the cluster-level defaults, which are assigned to any topic that does not have explicit topic-level overrides. - When `retention_local_strict` is false (default), the retention policy was met in the previous phase, so no more data is removed. - When `retention_local_strict` is true, the housekeeping process removes data fairly across all topics until each topic reaches its local retention target. After this phase completes, all partitions are operating at a size that reflects their effective local retention target. The next phase starts to override the local retention settings to remove more data. #### [](#phase-3-trim-data-with-default-local-retention-settings)Phase 3: Trim data with default local retention settings For topics with the default local retention settings, this phase removes partition data to a _low-space_ level, which is a configured size of two log segments that provide minimal space for partition operation. The housekeeping process only considers removing data that is safely stored in Tiered Storage. #### [](#phase-4-trim-data-with-explicitly-configured-retention-settings)Phase 4: Trim data with explicitly-configured retention settings For topics with explicitly-configured retention settings, this phase removes data down to the _low-space_ level of two log segments. #### [](#phase-5-trim-to-active-latest-segment)Phase 5: Trim to active (latest) segment The final phase trims all topics down to their last active segment. Data in the active log segment cannot be removed, and it is not available for reclaim until it is rolled, which occurs when it reaches `segment.bytes` or when `segment.ms` time expires. ## [](#monitor-disk-space)Monitor disk space You can check your total disk size and free space by viewing the metrics: - `redpanda_storage_disk_total_bytes` - `redpanda_storage_disk_free_bytes` Redpanda monitors disk space and updates these metrics and the `storage_space_alert` status based on your full disk alert threshold. You can check the alert status with the `redpanda_storage_disk_free_space_alert` metric. The alert values are: - 0 = No alert - 1 = Low free space alert - 2 = Out of space (degraded, external writes are rejected) ## [](#set-free-disk-space-thresholds)Set free disk space thresholds You can set a soft limit for a minimum free disk space alert. This soft limit generates an error message and affects the value of the [`redpanda_storage_disk_free_space_alert`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_storage_disk_free_space_alert) metric. You can also set a hard limit for minimum disk space, after which Redpanda enters a degraded performance state. You set the thresholds for these values by configuring the following properties, which you can set on any data disk (one drive per node): | Property | Description | | --- | --- | | storage_space_alert_free_threshold_bytes | Minimum free disk space threshold, in bytes, for generating a low disk space alert. | | storage_space_alert_free_threshold_percent | Minimum free disk space allowed, in percentage of total available space for that drive, for generating a low disk space alert. | | storage_min_free_bytes | Disk space threshold beyond which a degraded performance state is entered. | > 📝 **NOTE** > > The alert threshold can be set in either bytes or percentage of total space. To disable one threshold in favor of the other, set it to zero. When a disk exceeds the configured alert threshold, Redpanda updates the [`redpanda_storage_disk_free_space_alert`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_storage_disk_free_space_alert) metric to `1`, indicating low free space, and writes an error level `storage space alert` message to the service log. The message looks like the following: ```bash ERROR 2023-12-08 15:07:45,716 [shard 0] cluster - storage space alert: free space at 25.574% on /var/lib/redpanda/data: 96.732GiB total, 24.739GiB free, min. free 0.000bytes. Be sure to adjust retention policies as needed to avoid running out of space. ``` If you continue to exhaust disk space and reach the `storage_min_free_bytes` value, the `redpanda_storage_disk_free_space_alert` metric changes to `2`, indicating Redpanda is in a degraded performance state. See [Handle full disks](#handle-full-disks) for more information on addressing this situation. Once disk space is freed, Redpanda updates the `redpanda_storage_disk_free_space_alert` metric accordingly. ## [](#handle-full-disks)Handle full disks If you exceed your low disk space threshold, Redpanda blocks clients from producing. In that state, Redpanda returns errors to external writers, but it still allows internal write traffic, such as replication and rebalancing. The [`storage_min_free_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#storage_min_free_bytes) tunable configuration property sets the low disk space threshold—​the hard limit—​for this write rejection. The default value is 5 GiB, which means that when any broker’s free space falls below 5 GiB, Redpanda rejects writes to all brokers. ## [](#create-a-ballast-file)Create a ballast file A ballast file is an empty file that takes up disk space. If Redpanda runs out of disk space and becomes unavailable, you can delete the ballast file as a last resort. This clears up some space and gives you time to delete topics or records and change your retention properties. To create a ballast file, set the following properties in the rpk section of the `redpanda.yaml` file: ```yaml rpk: tune_ballast_file: true ballast_file_path: "/var/lib/redpanda/data/ballast" ballast_file_size: "1GiB" ``` Run `rpk` to create the ballast file: ```bash rpk redpanda tune ballast_file ``` | Property | Description | | --- | --- | | tune_ballast_file | Set to true to enable ballast file creation. Default is false. | | ballast_file_path | You can change the location of the ballast file, but it must be on the same mount point as the Redpanda data directory. Default is /var/lib/redpanda/data/ballast. | | ballast_file_size | Increase the ballast file size if it is a very high-throughput cluster. Decrease the ballast file size if you have very little storage space. The ballast file should be large enough to give you time to delete data and reconfigure retention properties if Redpanda crashes, but small enough that you don’t waste disk space. In general, set this to approximately 10 times the size of the largest segment, to have enough space to compact that topic. Default is 1 GiB. | ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 129: Fetch Read Coalescing **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/fetch-read-coalescing.md --- # Fetch Read Coalescing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Fetch Read Coalescing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/fetch-read-coalescing page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/fetch-read-coalescing.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc description: Reduce redundant read CPU and fetch-response memory under high consumer fan-out by sharing one read result across concurrent fetches of the same data. page-git-created-date: "2026-07-24" page-git-modified-date: "2026-07-28" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output. Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. When all N consumers fetch the same partition at the same offset with the same fetch settings, read CPU and fetch-response memory drop from roughly N times to one. Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Enable it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization. See [Evaluate coalescing effectiveness](#evaluate-coalescing-effectiveness) for how to measure whether it helps your workload. ## [](#prerequisites)Prerequisites - A valid [Redpanda Enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). ## [](#how-fetch-read-coalescing-works)How fetch read coalescing works Redpanda coalesces fetch reads only when they request the same data in the same way: the same partition and offset, with the same isolation level and fetch size (`max_bytes`). When matching reads arrive concurrently, or back-to-back while a previous result is still in use, the broker performs the read once and shares the single result with every requester. Reads that differ in any of these properties run independently. If a shared read fails, all coalesced fetches receive the error. Coalesced consumers share one response buffer instead of each holding its own copy, which is where the memory savings come from. A completed result is retained only while at least one fetch still references it, so the coalescer never pins memory on its own. Coalescing is scoped per shard: it collapses only the fan-out that lands on the same shard. It composes with [follower fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/) rather than replacing it: follower fetching spreads consumers across replicas to distribute the read load, and coalescing removes the redundancy within each shard. ## [](#limitations)Limitations Before enabling fetch read coalescing, familiarize yourself with the following limitations: - **Only identical reads coalesce.** Consumers reading the same data but with a different `max_bytes`, a different offset, or a different isolation level do not share a read. The benefit scales with how closely the concurrent fetches match: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. - **Obligatory and strict reads are grouped apart.** Redpanda distinguishes reads that must return at least one batch regardless of the byte limit (obligatory) from reads that strictly honor `max_bytes` (strict), and neither serves the other. A single partition and offset fetched both ways at the same time incurs one duplicate read by design. - **Retention is best-effort.** Completed results are only reusable by later readers while a fetch still references them. The coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. - **Per-shard scope.** Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it. - **De-duplication adds per-read overhead.** The de-duplication logic runs on every fetch read, whether or not reads coalesce. At low fan-out (for example, one or two consumers per partition), this overhead can increase reactor utilization while saving little to nothing, so only enable coalescing for high fan-out workloads. ## [](#enable-fetch-read-coalescing)Enable fetch read coalescing Enable coalescing with the `kafka_fetch_read_coalescing_enabled` cluster property: ```bash rpk cluster config set kafka_fetch_read_coalescing_enabled true ``` This is a runtime change, and no restart is required. The coalescing cache is fixed-size (about 0.5 MB per shard) and is allocated only while the feature is enabled. There is no separate sizing property. Setting the property back to `false` disables coalescing and clears the per-shard cache immediately. ## [](#evaluate-coalescing-effectiveness)Evaluate coalescing effectiveness The coalescer exports four counters on the internal `/metrics` endpoint, one series per shard. The metrics are registered only when internal metrics are enabled (that is, when `disable_metrics` is set to `false`, the default). | Metric | Description | | --- | --- | | vectorized_kafka_fetch_read_coalescer_insertions_total | Fetch reads that missed and performed a new read. | | vectorized_kafka_fetch_read_coalescer_reinsertions_total | Fetch reads that re-read an existing stale or expired entry. | | vectorized_kafka_fetch_read_coalescer_ready_hits_total | Fetch reads served from a retained, already-completed read. | | vectorized_kafka_fetch_read_coalescer_inflight_hits_total | Fetch reads served by awaiting an in-flight read. | The effectiveness of coalescing is the ratio of hits (reads avoided) to insertions (reads actually performed): coalescing hit ratio = (ready\_hits\_total + inflight\_hits\_total) / (insertions\_total + reinsertions\_total + ready\_hits\_total + inflight\_hits\_total) A ratio near 0 means little fan-out is being collapsed: the reads aren’t aligning on the same partition, offset, and fetch settings, and the workload is unlikely to benefit from coalescing. A high ratio means the coalescer is absorbing most of the fan-out. ## [](#suggested-reading)Suggested reading - [Follower Fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/) - [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 130: Manage Throughput **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/manage-throughput.md --- # Manage Throughput > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Manage Throughput latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/manage-throughput page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/manage-throughput.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/manage-throughput.adoc description: Configure broker-wide and client-specific throughput quotas to prevent resource exhaustion and noisy-neighbor issues. page-topic-type: how-to personas: platform_admin, developer learning-objective-1: Set user-based throughput quotas learning-objective-2: Set client ID-based quotas learning-objective-3: Monitor quota usage and throttling behavior page-git-created-date: "2023-08-03" page-git-modified-date: "2026-07-28" support-status: supported --- Redpanda throttles throughput on ingress and egress independently, and you can configure limits at the broker and client levels. This prevents clients from causing unbounded network and disk usage on brokers. You can configure limits at two levels: - Broker limits: These apply to all clients connected to the broker and restrict total traffic on the broker. See [Broker-wide throughput limits](#broker-wide-throughput-limits). - Client limits: These apply to authenticated users or clients defined by their client ID. You can manage client quotas with [`rpk cluster quotas`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-quotas/), with Redpanda Console, or with the Kafka API. When no quotas apply, the client has unlimited throughput. After reading this page, you will be able to: - Set user-based throughput quotas - Set client ID-based quotas - Monitor quota usage and throttling behavior ## [](#view-connected-client-details)View connected client details Before configuring throughput quotas, check the [current produce and consume throughput](https://docs.redpanda.com/streaming/current/manage/monitoring/#throughput) of a client. Use the [`rpk cluster connections list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-connections-list/) command or the [ListKafkaConnections](https://docs.redpanda.com/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-clusterservice-listkafkaconnections) Admin API endpoint to view detailed information about active Kafka client connections. For example, to view a cluster’s connected clients in order of highest current produce throughput, run: ### rpk ```bash rpk cluster connections list --order-by="recent_request_statistics.produce_bytes desc" ``` ```bash UID STATE USER CLIENT-ID IP:PORT NODE SHARD OPEN-TIME IDLE PROD-TPUT/SEC FETCH-TPUT/SEC REQS/MIN b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55012 0 0 9s 0s 78.9MB 0B 292 36338ca5-86b7-4478-ad23-32d49cfaef61 OPEN UNAUTHENTICATED rpk 127.0.0.1:49722 0 0 13s 13.694243104s 0B 0B 1 7e277ef6-0176-4007-b100-6581bfde570f OPEN UNAUTHENTICATED rpk 127.0.0.1:49736 0 0 13s 10.093957335s 0B 0B 2 567d9918-d3dc-4c74-ab5d-85f70cd3ee35 OPEN UNAUTHENTICATED rpk 127.0.0.1:49748 0 0 13s 0.591413542s 0B 0B 5 08616f21-08f9-46e7-8f06-964bd8240d9b OPEN UNAUTHENTICATED rpk 127.0.0.1:49764 0 0 13s 10.094602845s 0B 0B 2 e4d5b57e-5c76-4975-ada8-17a88d68a62d OPEN UNAUTHENTICATED rpk 127.0.0.1:54992 0 0 10s 0.302090085s 0B 14.5MB 27 b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55002 0 0 8s 7.743592270s 0B 0B 1 62fde947-411d-4ea8-9461-3becc2631b46 CLOSED UNAUTHENTICATED rpk 127.0.0.1:48578 0 0 26s 0.000737836s 0B 0B 1 95387e2e-2ec4-4040-aa5e-4257a3efa1a2 CLOSED UNAUTHENTICATED rpk 127.0.0.1:48564 0 0 26s 0.208180826s 0B 0B 1 ``` ### curl ```bash curl -s -X POST \ --header "Content-Type: application/json" \ --data '{ "filter": "", "order_by": "recent_request_statistics.produce_bytes desc" }' \ "localhost:9644/redpanda.core.admin.v2.ClusterService/ListKafkaConnections" ``` Show example API response ```json { "connections": [ { "nodeId": 0, "shardId": 0, "uid": "b20601a3-624c-4a8c-ab88-717643f01d56", "state": "KAFKA_CONNECTION_STATE_OPEN", "openTime": "2025-10-15T14:15:15.755065000Z", "closeTime": "1970-01-01T00:00:00.000000000Z", "authenticationInfo": { "state": "AUTHENTICATION_STATE_UNAUTHENTICATED", "mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED", "userPrincipal": "" }, "listenerName": "", "tlsInfo": { "enabled": false }, "source": { "ipAddress": "127.0.0.1", "port": 55012 }, "clientId": "perf-producer-client", "clientSoftwareName": "apache-kafka-java", "clientSoftwareVersion": "3.9.0", "transactionalId": "my-tx-id", "groupId": "", "groupInstanceId": "", "groupMemberId": "", "apiVersions": { "18": 4, "22": 3, "3": 12, "24": 3, "0": 7 }, "idleDuration": "0s", "inFlightRequests": { "sampledInFlightRequests": [ { "apiKey": 0, "inFlightDuration": "0.000406892s" } ], "hasMoreRequests": false }, "totalRequestStatistics": { "produceBytes": "78927173", "fetchBytes": "0", "requestCount": "4853", "produceBatchCount": "4849" }, "recentRequestStatistics": { "produceBytes": "78927173", "fetchBytes": "0", "requestCount": "4853", "produceBatchCount": "4849" } }, ... ] "totalSize": "9" } ``` To view connections for a specific client, you can use a filter expression: ### rpk ```bash rpk cluster connections list --client-id="perf-producer-client" ``` ```bash UID STATE USER CLIENT-ID IP:PORT NODE SHARD OPEN-TIME IDLE PROD-TPUT/SEC FETCH-TPUT/SEC REQS/MIN b41584f3-2662-4185-a4b8-0d8510f5c780 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55002 0 0 8s 7.743592270s 0B 0B 1 b20601a3-624c-4a8c-ab88-717643f01d56 OPEN UNAUTHENTICATED perf-producer-client 127.0.0.1:55012 0 0 9s 0s 78.9MB 0B 292 ``` The `USER` field in the connection list shows the authenticated principal. Unauthenticated connections show `UNAUTHENTICATED`, which corresponds to an empty user principal (`user=""`) in quota configurations, not `user=`. ### curl ```bash curl -s -X POST \ --header "Content-Type: application/json" \ --data '{ "filter": "client_id = \"perf-producer-client\"" }' \ "localhost:9644/redpanda.core.admin.v2.ClusterService/ListKafkaConnections" ``` Show example API response ```json { "connections": [ { "nodeId": 0, "shardId": 0, "uid": "b41584f3-2662-4185-a4b8-0d8510f5c780", "state": "KAFKA_CONNECTION_STATE_OPEN", "openTime": "2025-10-15T14:15:15.219538000Z", "closeTime": "1970-01-01T00:00:00.000000000Z", "authenticationInfo": { "state": "AUTHENTICATION_STATE_UNAUTHENTICATED", "mechanism": "AUTHENTICATION_MECHANISM_UNSPECIFIED", "userPrincipal": "" }, "listenerName": "", "tlsInfo": { "enabled": false }, "source": { "ipAddress": "127.0.0.1", "port": 55002 }, "clientId": "perf-producer-client", "clientSoftwareName": "apache-kafka-java", "clientSoftwareVersion": "3.9.0", "transactionalId": "", "groupId": "", "groupInstanceId": "", "groupMemberId": "", "apiVersions": { "18": 4, "3": 12, "10": 4 }, "idleDuration": "7.743592270s", "inFlightRequests": { "sampledInFlightRequests": [], "hasMoreRequests": false }, "totalRequestStatistics": { "produceBytes": "0", "fetchBytes": "0", "requestCount": "3", "produceBatchCount": "0" }, "recentRequestStatistics": { "produceBytes": "0", "fetchBytes": "0", "requestCount": "3", "produceBatchCount": "0" } }, ... ], "totalSize": "2" } ``` To view connections for a specific authenticated user: ```bash rpk cluster connections list --user alice ``` This shows all connections from user `alice`, useful for monitoring clients that are subject to user-based quotas. ## [](#broker-wide-throughput-limits)Broker-wide throughput limits Broker-wide throughput limits account for all Kafka API traffic going into or out of the broker, as data is produced to or consumed from a topic. The limit values represent the allowed rate of data in bytes per second passing through in each direction. Redpanda also provides administrators the ability to exclude clients from throughput throttling and to fine-tune which Kafka request types are subject to throttling limits. ### [](#broker-wide-throughput-limit-properties)Broker-wide throughput limit properties The properties for broker-wide throughput quota balancing are configured at the cluster level, for all brokers in a cluster: | Cluster configuration property | Description | | --- | --- | | kafka_throughput_limit_node_in_bps | A broker’s total throughput limit for ingress Kafka traffic. | | kafka_throughput_limit_node_out_bps | A broker’s total throughput limit for egress Kafka traffic. | | kafka_throughput_control | List of clients for whom broker-wide limits do not apply. | | kafka_throughput_controlled_api_keys | Kafka request types subject to broker-wide throughput limits; defaults to produce and fetch. | | max_kafka_throttle_delay_ms | Maximum delay inserted in the data path of Kafka API requests to throttle them down. Setting this lower than the Kafka client timeout helps ensure throttling alone does not cause client timeouts. | > 📝 **NOTE** > > By default, both `kafka_throughput_limit_node_in_bps` and `kafka_throughput_limit_node_out_bps` are disabled, and no throughput limits are applied. You must manually set them to enable throughput throttling. To set broker-wide throughput limits, use [`rpk cluster config set`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-config-set/) to configure the cluster properties: ```bash # Set ingress limit to 100 MB/s per broker rpk cluster config set kafka_throughput_limit_node_in_bps 100000000 # Set egress limit to 200 MB/s per broker rpk cluster config set kafka_throughput_limit_node_out_bps 200000000 ``` ## [](#client-throughput-limits)Client throughput limits Redpanda provides configurable throughput quotas for individual clients or authenticated users. Quotas are managed through the Kafka-compatible AlterClientQuotas and DescribeClientQuotas APIs, accessible with `rpk`, Redpanda Console, or Kafka client libraries. Redpanda supports two types of client throughput quotas: - Client ID-based quotas: Limit throughput based on the self-declared `client-id` field. - User-based quotas: Limit throughput based on authenticated user [principal](https://docs.redpanda.com/streaming/current/reference/glossary/#principal). Requires [authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/). You can also combine both types for fine-grained control (for example, limiting a specific user when using a specific client application). For conceptual information about quota types, entity hierarchy, precedence rules, and how Redpanda tracks and enforces quotas through throttling, see [About Client Throughput Quotas](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/about-throughput-quotas/). ### [](#set-user-based-quotas)Set user-based quotas > ❗ **IMPORTANT** > > User-based quotas require authentication to be enabled. To set up authentication, see [Configure Authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/). #### [](#quota-for-a-specific-user)Quota for a specific user To limit throughput for a specific authenticated user across all clients: ```bash rpk cluster quotas alter --add producer_byte_rate=2000000 --name user=alice ``` This limits user `alice` to 2 MB/s for produce requests regardless of the client ID used. To view quotas for a user: ```bash rpk cluster quotas describe --name user=alice ``` Expected output: ```bash user=alice producer_byte_rate=2000000 ``` #### [](#default-quota-for-all-users)Default quota for all users To set a fallback quota for any user without a more specific quota: ```bash rpk cluster quotas alter --add consumer_byte_rate=5000000 --default user ``` This applies a 5 MB/s fetch quota to all authenticated users who don’t have a more specific quota configured. ### [](#remove-a-user-quota)Remove a user quota To remove a quota for a specific user: ```bash rpk cluster quotas alter --delete consumer_byte_rate --name user=alice ``` To remove all quotas for a user: ```bash rpk cluster quotas delete --name user=alice ``` ### [](#set-client-id-based-quotas)Set client ID-based quotas Client ID-based quotas apply to all users using a specific client ID. These quotas do not require authentication. Because the client ID is self-declared, client ID-based quotas are not suitable for guaranteeing isolation between tenants. For multi-tenant environments, Redpanda recommends user-based quotas for per-tenant isolation. #### [](#individual-client-id-throughput-limit)Individual client ID throughput limit To view current throughput quotas set through the Kafka API, run [`rpk cluster quotas describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-quotas-describe/). For example, to see the quotas for client ID `consumer-1`: ```bash rpk cluster quotas describe --name client-id=consumer-1 ``` ```bash client-id=consumer-1 producer_byte_rate=140000 ``` To set a throughput quota for a single client, use the [`rpk cluster quotas alter`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-quotas-alter/) command. ```bash rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id=consumer-1 ``` ```bash ENTITY STATUS client-id=consumer-1 OK ``` #### [](#group-of-clients-throughput-limit)Group of clients throughput limit Alternatively, you can view or configure throughput quotas for a group of clients based on a match on client ID prefix. The following example sets the `consumer_byte_rate` quota to client IDs prefixed with `consumer-`: ```bash rpk cluster quotas alter --add consumer_byte_rate=200000 --name client-id-prefix=consumer- ``` > 📝 **NOTE** > > A `client-id-prefix` quota group is not related to Kafka consumer groups. The client ID is an application-defined identifier sent with every request. Client libraries typically default to their own name (such as `kgo`, `rdkafka`, `sarama`, or `perf-producer-client`), but applications can set it using the [`client.id`](https://kafka.apache.org/documentation/#consumerconfigs_client.id) configuration property. This makes prefix-based quotas useful for grouping related applications (for example, `inventory-service-` to match `inventory-service-1`, `inventory-service-2`, etc.). #### [](#default-client-throughput-limit)Default client throughput limit You can apply default throughput limits to clients. Redpanda applies the default limits if no quotas are configured for a specific client ID or prefix. To specify a produce quota of 1 GB/s through the Kafka API (applies across all produce requests to a single broker), run: ```bash rpk cluster quotas alter --default client-id --add producer_byte_rate=1000000000 ``` ### [](#set-combined-user-and-client-quotas)Set combined user and client quotas You can set quotas for specific (user, client ID) combinations for fine-grained control. #### [](#user-with-specific-client)User with specific client To limit a specific user when using a specific client: ```bash rpk cluster quotas alter --add consumer_byte_rate=1000000 --name user=alice --name client-id=consumer-1 ``` User `alice` using `client-id=consumer-1` is limited to a 1 MB/s fetch rate. The same user with a different client ID would use a different quota (or fall back to less specific matches). To view combined quotas: ```bash rpk cluster quotas describe --name user=alice --name client-id=consumer-1 ``` #### [](#user-with-client-prefix)User with client prefix To set a shared quota for a user across multiple clients matching a prefix: ```bash rpk cluster quotas alter --add producer_byte_rate=3000000 --name user=bob --name client-id-prefix=app- ``` All clients used by user `bob` with a client ID starting with `app-` share a combined 3 MB/s produce quota. #### [](#default-user-with-specific-client)Default user with specific client To set a quota for a specific client across all users: ```bash rpk cluster quotas alter --add producer_byte_rate=500000 --default user --name client-id=payment-processor ``` Any user using `client-id=payment-processor` is limited to a 500 KB/s produce rate, unless they have a more specific quota configured. ### [](#bulk-manage-client-throughput-limits)Bulk manage client throughput limits To more easily manage multiple quotas, you can use the `cluster quotas describe` and [`cluster quotas import`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-quotas-import/) commands to do a bulk export and update. For example, to export all client quotas in JSON format: ```bash rpk cluster quotas describe --format json ``` `rpk cluster quotas import` accepts the output string from `rpk cluster quotas describe --format `: ```bash rpk cluster quotas import --from '{"quotas":[{"entity":[{"name":"analytics-consumer","type":"client-id"}],"values":[{"key":"consumer_byte_rate","values":"10000000"}]},{"entity":[{"name":"analytics-","type":"client-id-prefix"}],"values":[{"key":"producer_byte_rate","values":"10000000"},{"key":"consumer_byte_rate","values":"5000000"}]}]}' ``` You can also save the JSON or YAML output to a file and pass the file path in the `--from` flag. ### [](#view-throughput-limits-in-redpanda-console)View throughput limits in Redpanda Console You can also use Redpanda Console to view enforced limits. In the side menu, go to **Quotas**. ### [](#monitor-client-throughput)Monitor client throughput The following metrics provide insights into client throughput quota usage: - Client quota throughput per rule and quota type: - `/public_metrics` - [`redpanda_kafka_quotas_client_quota_throughput`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throughput) - `/metrics` - [`vectorized_kafka_quotas_client_quota_throughput`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_kafka_quotas_client_quota_throughput) - Client quota throttling delay per rule and quota type, in seconds: - `/public_metrics` - [`redpanda_kafka_quotas_client_quota_throttle_time`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throttle_time) - `/metrics` - [`vectorized_kafka_quotas_client_quota_throttle_time`](https://docs.redpanda.com/streaming/current/reference/internal-metrics-reference/#vectorized_kafka_quotas_client_quota_throttle_time) To identify which clients are actively connected and generating traffic, see [View connected client details](#view-connected-client-details). Quota metrics use the `redpanda_quota_rule` label to identify which quota was applied to a request. The label distinguishes between different entity types (user, client, or combinations). See the label values in [`redpanda_kafka_quotas_client_quota_throughput`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_kafka_quotas_client_quota_throughput). The `kafka_quotas` logger provides details at the trace level on client quota throttling: ```bash TRACE 2024-06-14 15:36:05,240 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns TRACE 2024-06-14 15:36:05,240 [shard 2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:51256] throttle request:{snc:0, client:184}, enforce:{snc:-365123762, client:-365123762}, key:0, request_size:1316 TRACE 2024-06-14 15:37:44,835 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 119, delay:0ns, capped_delay:0ns TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - quota_manager.cc:361 - request: ctx:{quota_type: produce_quota, client_id: {rpk}}, key:k_client_id{rpk}, value:{limit: {1111}, rule: kafka_client_default}, bytes: 1316, delay:184518451ns, capped_delay:184518451ns TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context.cc:605 - [127.0.0.1:58636] throttle request:{snc:0, client:184}, enforce:{snc:-14359, client:-14359}, key:0, request_size:1316 ``` #### [](#track-quota-use-per-entity)Track quota use per entity When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening at a broker level, but they cannot reveal which user, client, or group of clients is utilizing a quota or being throttled. Per-entity quota metrics answer these questions. Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each entity actually uses. Use these metrics to identify throttled users and clients by name and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them, run: ```bash rpk cluster config set kafka_per_entity_quota_metrics true ``` The change takes effect without a broker restart. When enabled, Redpanda exposes two additional counters: - Total per-entity quota throttling delay, in milliseconds: - `/public_metrics` - `redpanda_kafka_quotas_client_quota_throttle_time_ms_by_entity` - `/metrics` - `vectorized_kafka_quotas_client_quota_throttle_time_ms_by_entity` - Per-entity quota throughput (bytes for produce and fetch quotas, partition mutations for partition mutation quotas): - `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` - `/metrics` - `vectorized_kafka_quotas_client_quota_throughput_by_entity` Each series is labeled with the entity identity and the quota type: - `redpanda_quota_type`: Always present. One of `produce_quota`, `fetch_quota`, or `partition_mutation_quota`. - `redpanda_quota_user`: The user principal, for user-based quotas. - `redpanda_quota_client_id`: The client ID, for client ID-based quotas. - `redpanda_quota_group_name`: The client ID prefix, for quotas that apply to a [group of clients](#group-of-clients-throughput-limit). Combined quotas, such as a user with a specific client ID, include each matching label on the same series. To keep metric cardinality bounded, Redpanda registers a per-entity series only while an entity is actively being throttled, and removes the series after the entity has been idle. An entity appears in these metrics only after it has been throttled at least once. From that point, Redpanda records the entity’s throughput on every request, not only on throttled requests, so a Prometheus `rate()` query reflects the entity’s actual throughput. To measure how much of its quota an entity is using, divide the entity’s throughput rate by its configured limit. Quota limits apply per broker, so aggregate by broker as well as by entity to keep the results comparable with the configured limits. For example, the following query returns the produce throughput rate for each throttled client ID on each broker: ```promql sum by (redpanda_quota_client_id, pod) ( rate(redpanda_kafka_quotas_client_quota_throughput_by_entity{redpanda_quota_type="produce_quota"}[5m]) ) ``` Compare the result with the entity’s configured limit from [`rpk cluster quotas describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-quotas-describe/). A ratio close to 1 means the client is saturating its quota and its requests are being delayed. ## [](#see-also)See also - [About Client Throughput Quotas](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/about-throughput-quotas/) - [Configure Client Connections](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/configure-client-connections/) - [Configure Authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/) --- # Page 131: Configure Broker Properties **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/node-property-configuration.md --- # Configure Broker Properties > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Broker Properties latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/node-property-configuration page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/node-property-configuration.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/node-property-configuration.adoc description: Learn how to configure broker properties with the redpanda.yaml file. page-git-created-date: "2023-08-17" page-git-modified-date: "2026-08-13" support-status: supported --- When you install Redpanda, a `redpanda.yaml` file is installed on each broker in `/etc/redpanda`. This file contains broker configuration properties. A broker property is one that can be set differently from broker to broker, such as [`data_directory`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#data_directory) or [`node_id`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/#node_id) (when specified). See [Broker Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) for a list of broker properties, their descriptions, and their default values. The default `redpanda.yaml` file groups broker properties into categories: - `pandaproxy` - Properties for the Redpanda HTTP Proxy - `redpanda` - Runtime configuration properties, such as the cluster member IP addresses and the data directory - `rpk` - Properties that determine how `rpk` starts Redpanda - `schema registry` - Properties related to storage, retrieval, and compatibility of the schemas ## [](#set-broker-configuration-properties)Set broker configuration properties The `redpanda.yaml` file rarely needs to be edited after the system is installed, but you can choose to change broker configuration property values. > 📝 **NOTE** > > The broker configuration property [`node_id`](https://docs.redpanda.com/streaming/current/reference/properties/broker-properties/) is immutable. To ensure safe operations, omit the `node_id` field from `redpanda.yaml` and allow Redpanda to assign it automatically. For more information, see [Do not configure broker IDs](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/#do-not-configure-broker-ids). To change a broker property setting: 1. Open a terminal window and navigate to the broker where you want to change a property setting. 2. Go to the `/etc/redpanda` directory. 3. Open the `redpanda.yaml` file. 4. Change values for properties as needed. 5. Save the file and close the editor. 6. Restart the broker to apply the changes. When Redpanda starts up and reads the `redpanda.yaml` file, it checks that each property setting has a valid value. If the file contains an invalid property setting, Redpanda logs an error and refuses to start. ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 132: Node-wise Partition Recovery **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/nodewise-partition-recovery.md --- # Node-wise Partition Recovery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Node-wise Partition Recovery latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/nodewise-partition-recovery page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/nodewise-partition-recovery.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/nodewise-partition-recovery.adoc description: Feature to recover partitions that have lost a majority of replicas. page-git-created-date: "2024-02-06" page-git-modified-date: "2024-12-03" support-status: supported --- Multi-broker or entire [AZ](https://docs.redpanda.com/streaming/current/reference/glossary/#availability-zone-az) failures (especially in cloud environments), along with some forms of human error, can result in ‘stuck’ partitions where there are fewer replicas than required to make a quorum. In such failure scenarios, some data loss may be unavoidable. Node-wise partition recovery provides a way to unsafely recover at least a portion of your data using remaining replicas, which are moved off of target brokers and allocated to healthy ones. In one step, this process repairs partitions while draining the target brokers of all partition replicas. This topic helps admins understand what they can or cannot recover using node-wise partition recovery. > ❗ **IMPORTANT** > > Only use this operation as a last-resort measure when all other recovery options have failed. In some cases, there may be no remaining replicas for the partitions on the dead brokers. This recovery method is intended for scenarios where you have already experienced data loss, with the goal being to stop the loss of additional data. ## [](#perform-the-recovery-operation)Perform the recovery operation To start node-wise partition recovery, run `rpk cluster partitions unsafe-recover`. For example: `rpk cluster partitions unsafe-recover --from-nodes 1,3,5` This command includes a prompt to confirm the generated recovery plan, as it is a destructive operation. When you run node-wise partition recovery, the partitions on the broker are rebuilt on a best-effort basis. When there are zero surviving partition replicas, such as a topic with a replication factor of 1 (`RF=1`), partition recovery rebuilds empty partitions with no data (although you may be able to recover the partition from Tiered Storage), allowing producers to continue writing to the partition even though no data can be recovered in such situations. The `--from-nodes` flag accepts a comma-separated list of the brokers' node IDs you wish to recover the data from. This example performs recovery operations on nodes 1, 3, and 5. Redpanda assesses these brokers to identify which partitions lack a majority. It then creates a plan to recover the impacted partitions and prompts you for confirmation. You must respond `yes` to continue with recovery. The `--dry` flag performs a dry run and allows you to view the recovery plan with no risk to your cluster. > 📝 **NOTE** > > When running node-wise partition recovery, it’s possible that there may be more recent data (a higher offset) available in Tiered Storage if: > > - Raft replication was stuck or slow before the node failure > > - Zero live replicas remain in the cluster (because the partition had a replication factor of one, `RF=1`) > > > For topics configured to use Tiered Storage, Redpanda also attempts to recover partition data from object storage, recovering the latest offset available for a partition in either storage tier (local or object storage). This allows for the maximum amount of data to be recovered in all cases, even for topics with a replication factor of 1, where no replicas remain in local storage. The recovery operation can take some time to complete, especially for a large amount of data. To monitor the status of the recovery operation in real-time, run: `rpk cluster partitions balancer-status` ## [](#example-recovery-operations)Example recovery operations The following example shows the node-wise partition recovery process in action: $ rpk cluster partitions unsafe-recover --from-nodes 1 NAMESPACE TOPIC PARTITION REPLICA-CORE DEAD-NODES kafka bar 0 \[1-1\] \[1\] ? Confirm recovery from these nodes? Yes Executing recovery plan... Successfully queued the recovery plan, you may check the status by running 'rpk cluster partitions balancer-status' $ rpk cluster partitions balancer-status Status: ready Seconds Since Last Tick: 26 Current Reassignment Count: 0 Partitions Pending Recovery (1): \[kafka/bar/0\] The following example shows the status of moved partitions: $ rpk cluster partitions move-status PARTITION MOVEMENTS =================== NAMESPACE-TOPIC PARTITION MOVING-FROM MOVING-TO COMPLETION-% PARTITION-SIZE BYTES-MOVED BYTES-REMAINING kafka/prod\_tests 4 \[045\] \[045\] 0 56204032205 0 56204032205 kafka/prod\_tests 7 \[045\] \[045\] 0 64607340009 0 64607340009 kafka/prod\_tests 12 \[014\] \[014\] 0 29074311639 0 29074311639 kafka/prod\_tests 20 \[014\] \[014\] 0 29673620476 0 29673620476 kafka/prod\_tests 22 \[045\] \[045\] 0 28471089141 0 28471089141 kafka/prod\_tests 23 \[045\] \[045\] 0 29692435312 0 29692435312 kafka/prod\_tests 31 \[014\] \[014\] 0 66982232299 0 66982232299 kafka/prod\_tests 33 \[014\] \[014\] 0 46329276747 0 46329276747 --- # Page 133: Forced partition recovery **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/partition-recovery.md --- # Forced partition recovery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Forced partition recovery latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/partition-recovery page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/partition-recovery.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/partition-recovery.adoc description: Recover a single partition using the Admin API. page-git-created-date: "2024-02-12" page-git-modified-date: "2025-08-20" support-status: supported --- You can use the Redpanda Admin API to recover a partition that is unavailable and has lost a majority of its replicas. This can occur when the partition replicas have lost [Raft](https://raft.github.io/) consensus, for instance if brokers in a Raft group fail, preventing the group from reaching a majority and electing a new leader. Redpanda performs forced partition recovery by promoting the best available replica to leader, making the partition available for produce and consume. Typically, Redpanda chooses the replica with the highest offset. > ⚠️ **CAUTION** > > Forced partition recovery allows some potential data loss on the partition if the best available replica is out of sync, ending up in an unclean leader election. Use this operation with caution, and only when brokers have failed beyond recovery to the point that the remaining replicas cannot form a majority. > 📝 **NOTE** > > If you want to instead force recover all partitions in bulk from a set of failed brokers, use [nodewise recovery](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/nodewise-partition-recovery/). ## [](#use-the-admin-api-to-recover-a-partition)Use the Admin API to recover a partition The following examples assume that partition 0 in topic `test` is unavailable and its replicas cannot form a majority to elect a leader. 1. Call the [`/partitions/kafka/`](https://docs.redpanda.com/api/doc/admin/operation/operation-get_topic_partitions) endpoint to determine the current broker and shard assignments of the partition replicas. ```bash curl http://localhost:9644/v1/partitions/kafka/test ``` ```bash [ { "ns": "kafka", "topic": "test", "partition_id": 0, "status": "done", "leader_id": 1, "raft_group_id": 1, "replicas": [ { "node_id": 1, "core": 1 }, { "node_id": 2, "core": 1 }, { "node_id": 3, "core": 1 } ] } ] ``` 2. In this scenario, brokers 2 and 3 have failed and you want to move the replicas from those brokers to brokers 4 and 5, which are healthy. Make a POST request to the `/debug/partitions/kafka///force_replicas` endpoint: ```bash curl -X POST http://localhost:9644/v1/debug/partitions/kafka/test/0/force_replicas \ -H 'Content-Type: application/json' -d '{ [ { "node_id": 1, "core": 1 }, { "node_id": 4, "core": 1 }, { "node_id": 5, "core": 0 } ] }' ``` The request body includes the broker ID and the CPU core (shard ID) for the replica on broker 1, to hydrate the new replicas assigned to brokers 4 and 5 on cores 1 and 0 respectively. If there are `n` CPU cores on the machine, the value of `core` can be within the range `[0, n-1]`. You may use a random value within the range, or the least loaded shard. See the [public metrics reference](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/) for metrics regarding CPU usage, and [Redpanda Admin API](https://docs.redpanda.com/api/doc/admin/operation/operation-force_update_partition_replicas) for additional detail. ## [](#suggested-reading)Suggested reading - [Partition leadership elections](https://docs.redpanda.com/streaming/current/get-started/architecture/#partition-leadership-elections) --- # Page 134: Perform a Rolling Restart **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/rolling-restart.md --- # Perform a Rolling Restart > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Perform a Rolling Restart latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/rolling-restart page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/rolling-restart.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/rolling-restart.adoc description: Learn how to perform a rolling restart of your Redpanda cluster. page-git-created-date: "2023-12-07" page-git-modified-date: "2025-07-31" support-status: supported --- A rolling restart involves restarting one broker at a time while the remaining brokers in your cluster continue running. This is to minimize downtime during a full cluster restart. You should perform a rolling restart during operations such as configuration updates that require a restart, version upgrades, or cluster maintenance. A rolling restart involves putting a broker into and out of [maintenance mode](https://docs.redpanda.com/streaming/current/manage/node-management/), and then repeating the process on the next broker in the cluster. Placing brokers into maintenance mode ensures a smooth restart of your cluster while reducing the risk of interruption or degradation in service. When a broker is placed into maintenance mode, it reassigns its partition leadership to other brokers for all topics that have a replication factor greater than one. Reassigning partition leadership involves _draining_ leadership from the broker and _transferring_ that leadership to another broker. 1. Check for topics that have a replication factor greater than one. If you have topics with `replication.factor=1`, and if you have sufficient disk space, Redpanda Data recommends temporarily increasing the replication factor. This can help limit outages for these topics during the rolling restart. Do this before the restart to make sure there’s time for the data to replicate to other brokers. For more information, see [Change topic replication factor](https://docs.redpanda.com/streaming/current/migrate/data-migration/#change-topic-replication-factor). 2. Ensure that all brokers are active before restarting: ```bash rpk cluster info -b --detailed ``` All brokers should show `active` for `MEMBERSHIP` and `true` for `IS-ALIVE`: Example output ```none ID HOST PORT RACK CORES MEMBERSHIP IS-ALIVE VERSION UUID 0 redpanda-0.testcluster.local 32180 A 8 active true 25.2.13 24c08934-94f6-478c-b57d-45239f452488 1 redpanda-1.testcluster.local 32180 B 8 active true 25.2.13 7f3c1c6e-2f4d-4c8a-9c6e-0a8a6d8b2b61 2 redpanda-2.testcluster.local 32180 C 8 active true 25.2.13 b2e8d4a9-91c1-4b55-9f6a-3f7a2d3c5e44 ``` ## [](#perform-a-rolling-restart)Perform a rolling restart ### [](#enable-maintenance-mode)Enable maintenance mode 1. Check that all brokers are healthy: ```bash rpk cluster health ``` Example output: ```bash CLUSTER HEALTH OVERVIEW ======================= Healthy: true (1) Controller ID: 0 All nodes: [0 1 2] (2) Nodes down: [] (3) Leaderless partitions: [] (3) Under-replicated partitions: [1] (3) ``` | 1 | The cluster is either healthy (true) or unhealthy (false). | | --- | --- | | 2 | The node IDs of all brokers in the cluster. | | 3 | If the cluster is unhealthy, these fields will contain data. | 2. Optional: You can use the Admin API (default port: 9644) to perform additional checks for potential risks with restarting a specific broker. ```bash curl -X GET "http://:/v1/broker/pre_restart_probe" | jq . ``` Example output: ```json // Returns tuples of partitions (in the format {namespace}/{topic_name}/{partition_id}) affected by the broker restart. { "risks": { "rf1_offline": [ "kafka/topic_a/0" ], "full_acks_produce_unavailable": [], "unavailable": [], "acks1_data_loss": [] } } ``` In this example, the restart probe indicates that there is an under-replicated partition `kafka/topic_a/0` (with a replication factor of 1) at risk of going offline if the broker is restarted. See the [Admin API reference](https://docs.redpanda.com/api/doc/admin/operation/operation-pre_restart_probe) for more details on the restart probe endpoint. 3. Select a broker and place it into maintenance mode: ```bash rpk cluster maintenance enable --wait ``` The `--wait` option tells the command to wait until a given broker, 0 in this example, finishes draining all partitions it originally served. After the partition draining completes, the command completes. Expected output: Successfully enabled maintenance mode for node 0 Waiting for node to drain... 4. Verify that the broker is in maintenance mode: ```bash rpk cluster maintenance status ``` Expected output: NODE-ID DRAINING FINISHED ERRORS PARTITIONS ELIGIBLE TRANSFERRING FAILED 0 true true false 3 0 2 0 1 false false false 0 0 0 0 2 false false false 0 0 0 0 The `Finished` column should read `true` for the broker that you put into maintenance mode. 5. Validate the health of the cluster again: ```bash rpk cluster health --watch --exit-when-healthy ``` The combination of the `--watch` and `--exit-when-healthy` flags tell rpk to monitor the cluster health and exit only when the cluster is back in a healthy state. > 📝 **NOTE** > > You can also evaluate [metrics](https://docs.redpanda.com/streaming/current/manage/monitoring/) to determine cluster health. If the cluster has any issues, take the broker out of maintenance mode by running the following command before retrying the rolling restart: > > ```bash > rpk cluster maintenance disable > ``` ### [](#check-metrics)Check metrics Before continuing with the restart, check these important metrics to make sure the cluster is healthy and working as expected. | Metric Name | Description | Recommendations | | --- | --- | --- | | redpanda_kafka_under_replicated_replicas | Measures the number of under-replicated Kafka replicas. Non-zero: Replication lagging. Zero: All replicas replicated. | Pause restart if non-zero. | | redpanda_cluster_unavailable_partitions | Represents the number of partitions that are currently unavailable. Value of zero indicates all partitions are available. Non-zero indicates the respective count of unavailable partitions. | Ensure metric shows zero unavailable partitions before restart. | | redpanda_rpc_received_bytes and redpanda_rpc_sent_bytes | Total bytes processed for Kafka requests. | Ensure produce and consume rate for each broker recovers to its pre-restart value. | | redpanda_kafka_request_latency_seconds | Latency for processing Kafka requests. Indicates the delay between a Kafka request being initiated and completed. | Ensure the p99 histogram value recovers to its pre-restart level. | | redpanda_rpc_request_latency_seconds | Latency for processing RPC requests. Shows the delay between an RPC request initiation and completion. | Ensure the p99 histogram value recovers to its pre-restart level. | | redpanda_cpu_busy_seconds_total | CPU utilization for a given second. The value is a decimal between 0.0 and 1.0. A value of 1.0 means that the CPU was busy for the entire second, operating at 100% capacity. A value of 0.5 implies the CPU was busy for half the time (or 500 milliseconds) in the given second. A value of 0.0 indicates that the CPU was idle and not busy during the entire second. | If you’re seeing high values consistently, investigate the reasons. It could be due to high traffic or other system bottlenecks. | ### [](#restart-the-broker)Restart the broker To ensure proper ownership and permissions, restart the broker’s Redpanda service using `systemctl`: ```bash sudo systemctl restart redpanda ``` ### [](#disable-maintenance-mode)Disable maintenance mode 1. Take the broker out of maintenance mode: ```bash rpk cluster maintenance disable ``` Expected output: Successfully disabled maintenance mode for node 0 2. Ensure that the broker is no longer in maintenance mode: ```bash rpk cluster maintenance status ``` Expected output: ```none NODE-ID DRAINING FINISHED ERRORS PARTITIONS ELIGIBLE TRANSFERRING FAILED 0 false false false 0 0 0 0 1 false false false 0 0 0 0 2 false false false 0 0 0 0 ``` ### [](#post-restart-tasks)Post-restart tasks To verify that the cluster is running properly, run: ```bash rpk cluster health ``` To view additional information about your brokers, run: ```bash rpk cluster info -b --detailed ``` You can also use the [Admin API](https://docs.redpanda.com/api/doc/admin/operation/operation-post_restart_probe) to check how much each broker has progressed in recovering its workloads: ```bash curl -X GET "http://:/v1/broker/post_restart_probe" ``` Example output: ```json // Returns the load already reclaimed by broker, as a percentage of in-sync replicas { "load_reclaimed_pc": 66 } ``` ## [](#impact-of-broker-restarts)Impact of broker restarts When brokers restart, clients may experience higher latency, nodes may experience CPU spikes when the broker becomes available again, and you may receive alerts about under-replicated partitions. Topics that weren’t using replication (that is, topics that had `replication.factor=1`) will be unavailable. ### [](#temporary-increase-in-latency-on-clients-producers-and-consumers)Temporary increase in latency on clients (producers and consumers) When you restart one or more brokers in a cluster, clients (consumers and producers) may experience higher latency due to partition leadership reassignment. Because clients must communicate with the leader of a partition, they may send a request to a broker whose leadership has been transferred, and receive `NOT_LEADER_FOR_PARTITION`. In this case, clients must request metadata from the cluster to find out the address of the new leader. Clients refresh their metadata periodically, or when the client receives some retryable errors that indicate that the metadata may be stale. For example: 1. Broker A shuts down. 2. Client sends a request to broker A, and receives `NOT_LEADER_FOR_PARTITION`. 3. Client requests metadata, and learns that the new leader is broker B. 4. Client sends the request to broker B. ### [](#cpu-spikes-upon-broker-restart)CPU spikes upon broker restart When a restarted broker becomes available again, you may see your nodes' CPU usage increase temporarily. This temporary increase in CPU usage is due to the cluster rebalancing the partition replicas. ### [](#under-replicated-partitions)Under-replicated partitions When a broker is in maintenance mode, Redpanda continues to replicate updates to that broker. When a broker is taken offline during a restart, partitions with replicas on the broker could become out of sync until it is brought back online. Once the broker is available again, data is copied to its under-replicated replicas until all affected partitions are in sync with the partition leader. ## [](#suggested-reading)Suggested reading - [Monitor Redpanda](https://docs.redpanda.com/streaming/current/manage/monitoring/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 135: Configure Topic Properties **URL**: https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/topic-property-configuration.md --- # Configure Topic Properties > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Topic Properties latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: cluster-maintenance/topic-property-configuration page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: cluster-maintenance/topic-property-configuration.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/cluster-maintenance/topic-property-configuration.adoc description: Learn how to configure topic properties to control Redpanda's behavior for individual topics, including retention, cleanup policies, and Tiered Storage settings. page-git-created-date: "2025-09-03" page-git-modified-date: "2026-06-05" support-status: supported --- Topic properties control Redpanda’s behavior for individual topics, including data retention, cleanup policies, compression settings, and Tiered Storage configurations. These properties let you customize how Redpanda stores, processes, and manages data for each topic, overriding cluster-wide defaults when needed. Redpanda stores topic properties as metadata associated with each topic and replicates them across the cluster to ensure consistency. Many topic properties correspond to [cluster properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/) that establish default values for all topics. When you set a topic property, it overrides the corresponding cluster default for that specific topic. > ⚠️ **WARNING** > > All topic properties take effect immediately after being set. Do not modify properties on internal Redpanda topics (such as `__consumer_offsets`, `_schemas`, or other system topics) as this can cause cluster instability. For a complete reference of available topic properties and their corresponding cluster properties, see [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/). ## [](#configuration-methods)Configuration methods You can configure topic properties through multiple interfaces: - **rpk commands** - Use [`rpk topic create`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-create/) with the `-c` flag to set properties during topic creation, or [`rpk topic alter-config`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-alter-config/) to modify existing topics. - **Kafka Admin API** - Any Kafka-compatible client can use the standard Admin API to configure topic properties. - **Kubernetes Topic resources** - In Kubernetes deployments, configure topic properties using the `additionalConfig` field in Topic resources. See [Manage Topics in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/). - **Redpanda Console** - Use the web interface to [edit topic configuration](https://docs.redpanda.com/streaming/current/console/ui/edit-topic-configuration/) through a graphical interface. ## [](#verify-topic-property-configuration)Verify topic property configuration Use `rpk topic describe ` to view topic properties and their configuration sources: ```none rpk topic describe my-topic ``` The output shows two sections: ```none SUMMARY ======= NAME my-topic PARTITIONS 3 REPLICAS 3 CONFIGS ======= KEY VALUE SOURCE cleanup.policy delete DEFAULT_CONFIG compression.type producer DEFAULT_CONFIG retention.ms 604800000 DEFAULT_CONFIG write.caching true DYNAMIC_TOPIC_CONFIG ``` The `SOURCE` column indicates how each property is configured: - `DEFAULT_CONFIG` - Redpanda’s default value - `DYNAMIC_TOPIC_CONFIG` - User-configured value The replication factor appears as REPLICAS in the SUMMARY section, not as replication.factor in the CONFIGS list. However, you need to use the `replication.factor` key when modifying the value with `rpk topic alter-config`. For partition-level details, add the `-p` flag: ```none rpk topic describe my-topic -p ``` This shows a different output focused on partition information: ```none PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 1 2 [1 2 3] 0 6 1 2 4 [1 2 3] 0 10 2 3 1 [1 2 3] 0 8 ``` ## [](#examples)Examples The following examples show how to configure topic-level properties. Set a topic-level property for a topic to override the value of corresponding cluster property. ### [](#create-topic-with-topic-properties)Create topic with topic properties To set topic properties when creating a topic, use the [rpk topic create](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-create/) command with the `-c` option. For example, to create a topic with the `cleanup.policy` property set to `compact`: #### Local ```bash rpk topic create -c cleanup.policy=compact ``` #### Kubernetes ```bash kubectl exec -- rpk topic create -c cleanup.policy=compact ``` To configure multiple properties for a topic, use the `-c` option for each property. For example, to create a topic with all necessary properties for Tiered Storage: #### Local ```bash rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true ``` #### Kubernetes ```bash kubectl exec -- rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true ``` ### [](#modify-topic-properties)Modify topic properties To modify topic properties of an existing topic, use the [rpk topic alter-config](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-topic/rpk-topic-alter-config/) command. For example, to modify a topic’s `retention.ms` property: #### Local ```bash rpk topic alter-config --set retention.ms= ``` #### Kubernetes ```bash kubectl exec -- rpk topic alter-config --set retention.ms= ``` ### [](#configure-topic-properties-with-kubernetes)Configure topic properties with Kubernetes In Kubernetes deployments, configure topic properties using the `additionalConfig` field in Topic resources: ```yaml apiVersion: cluster.redpanda.com/v1alpha2 kind: Topic metadata: name: example-topic spec: partitions: 3 replicationFactor: 3 additionalConfig: cleanup.policy: "compact" retention.ms: "604800000" segment.ms: "86400000" ``` Apply the configuration: ```bash kubectl apply -f topic-config.yaml ``` ## [](#common-topic-property-categories)Common topic property categories The most commonly configured topic properties fall into these main categories: ### [](#disk-space-management)Disk space management Redpanda manages disk space through two main mechanisms: **compaction** (removing duplicate keys) and **retention** (removing old data). Choose your cleanup strategy with [`cleanup.policy`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#cleanup-policy): - `delete` - Remove old data based on time or size limits - `compact` - Keep only the latest value for each key - `compact,delete` - Combine both strategies ### [](#compaction)Compaction When using `cleanup.policy=compact` or `cleanup.policy=compact,delete`, configure: - [`min.cleanable.dirty.ratio`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#min-cleanable-dirty-ratio) - Control when compaction triggers based on dirty data ratio - [`max.compaction.lag.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#max-compaction-lag-ms) - Set maximum time before compaction is forced - [`min.compaction.lag.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#min-compaction-lag-ms) - Set minimum time before compaction can occur ### [](#retention)Retention When using `cleanup.policy=delete` or `cleanup.policy=compact,delete`, configure: - [`retention.bytes`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#retention-bytes) - Maximum size before cleanup (size-based retention) - [`retention.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#retention-ms) - Maximum age before cleanup (time-based retention) - [`segment.bytes`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#segment-bytes) - Control how frequently cleanup can occur by setting segment size ### [](#performance)Performance Essential performance tuning properties: - [`write.caching`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#write-caching) - Cache writes for lower latency with `acks=all` - [`max.message.bytes`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#max-message-bytes) - Set maximum message size - [`replication.factor`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#replication-factor) - Number of replicas for durability vs. performance For complete details about all available topic properties, see [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/). ## [](#related-topics)Related topics - [Topic Configuration Properties](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/) - Complete reference of all available topic properties - [Configure Cluster Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/) - Configure cluster-wide defaults - [Manage Topics](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/) - Create and manage topics - [Manage Topics in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/k-manage-topics/) - Topic management in Kubernetes deployments - [Edit Topic Configuration in Redpanda Console](https://docs.redpanda.com/streaming/current/console/ui/edit-topic-configuration/) - Graphical topic configuration ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Enable Unified Identity with Azure Entra ID for Redpanda and Redpanda Console](https://docs.redpanda.com/labs/docker-compose/oidc/) - [Owl Shop Example Application in Docker](https://docs.redpanda.com/labs/docker-compose/owl-shop/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 136: Redpanda Console **URL**: https://docs.redpanda.com/streaming/current/manage/console.md --- # Redpanda Console > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Redpanda Console latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: console/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: console/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/console/index.adoc description: Learn how to manage Redpanda using Redpanda Console. page-git-created-date: "2023-06-02" page-git-modified-date: "2025-05-23" support-status: supported --- - [Configure Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/configure-console/) Learn how to configure Redpanda Console using environment variables, YAML files, or command-line arguments. - [Add a License Key to Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/enterprise-license/) Learn how to apply or update a license key to Redpanda Console. - [Configure Redpanda Console to Connect to a Redpanda Cluster](https://docs.redpanda.com/streaming/current/console/config/connect-to-redpanda/) Learn how to configure Redpanda Console to connect to a Redpanda cluster and ensure communication with your Redpanda brokers. - [Redpanda Console Security](https://docs.redpanda.com/streaming/current/console/config/security/) Learn about security topics for Redpanda Console. - [HTTP Path Rewrites in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/http-path-rewrites/) Learn how to configure Redpanda Console to work with your URL path rewrites, particularly when hosted under a subpath. - [Configure Message Deserialization in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/deserialization/) Learn how to configure Redpanda Console to use Schema Registry, Protobuf files, and other deserialization methods to ensure your data is correctly interpreted and displayed. - [Enable Topic Documentation in Redpanda Console](https://docs.redpanda.com/streaming/current/console/config/topic-documentation/) Learn how to embed your Kafka topic documentation into the Redpanda Console UI by linking a Git repository that contains the topic documentation files. - [Redpanda Console Telemetry](https://docs.redpanda.com/streaming/current/console/config/analytics/) Learn what telemetry Redpanda Console collects by default, how it is handled, and how to disable it. - [Connect Redpanda Console to Kafka Connect Clusters](https://docs.redpanda.com/streaming/current/console/config/kafka-connect/) Learn how to connect one or more Kafka Connect clusters with Redpanda Console. --- # Page 137: Disaster Recovery **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery.md --- # Disaster Recovery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Disaster Recovery latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/index.adoc description: Learn about Shadowing with cross-region replication for disaster recovery. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-11-19" support-status: supported --- - [Shadowing](shadowing/) Set up disaster recovery for Redpanda clusters using Shadowing for cross-region replication. - [Whole Cluster Restore](whole-cluster-restore/) Restore a failed cluster, including its metadata. - [Topic Recovery](topic-recovery/) Restore a single topic from object storage. --- # Page 138: Shadowing **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing.md --- # Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Shadowing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/index.adoc description: Set up disaster recovery for Redpanda clusters using Shadowing for cross-region replication. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-08-13" support-status: supported --- These topics cover Shadowing concepts and the configuration workflow for Linux deployments. If you deploy Redpanda in Kubernetes, see [Shadowing in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/) to configure shadow links with the Redpanda Operator or Helm chart. - [Shadowing Overview](overview/) Learn about disaster recovery using Shadowing for cross-region replication. - [Configure Shadowing](setup/) Set up Shadowing for disaster recovery, including cross-region replication, data filters, networking, and authentication. - [Migrate Schemas from Confluent Schema Registry](migrate-schemas-confluent/) Replicate subjects, versions, and compatibility settings from a Confluent Schema Registry into a Redpanda shadow cluster. - [Monitor Shadowing](monitor/) Monitor Shadowing health with status commands, metrics, and best practices for tracking replication performance. - [Failover](failover/) Learn how failover can transform shadow topics into fully writable resources during disasters. - [Failover Runbook](failover-runbook/) Step-by-step emergency guide for failing over Redpanda shadow links during disasters. - [Upgrade Clusters in a Shadowing Topology](upgrade/) Upgrade clusters in a Shadowing topology safely: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies. --- # Page 139: Failover Runbook **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook.md --- # Failover Runbook > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Failover Runbook latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/failover-runbook page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/failover-runbook.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/failover-runbook.adoc description: Step-by-step emergency guide for failing over Redpanda shadow links during disasters. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-07-17" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide provides step-by-step procedures for emergency failover when your primary Redpanda cluster becomes unavailable. Follow these procedures only during active disasters when immediate failover is required. > 📝 **NOTE** > > If you’re running Redpanda in Kubernetes, see [Kubernetes Failover Runbook](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-failover-runbook/) for Kubernetes-specific emergency procedures. > ❗ **IMPORTANT** > > This is an emergency procedure. For planned failover testing or day-to-day shadow link management, see [Failover](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover/). Ensure you have completed the [disaster readiness checklist](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#disaster-readiness-checklist) before an emergency occurs. ## [](#emergency-failover-procedure)Emergency failover procedure Follow these steps during an active disaster: 1. [Assess the situation](#assess-situation) 2. [Verify shadow cluster status](#verify-shadow-status) 3. [Document current state](#document-state) 4. [Initiate failover](#initiate-failover) 5. [Monitor failover progress](#monitor-progress) 6. [Update application configuration](#update-applications) 7. [Verify application functionality](#verify-functionality) 8. [Clean up and stabilize](#cleanup-stabilize) ### [](#assess-situation)Assess the situation Confirm that failover is necessary: ```bash # Check if the primary cluster is responding rpk cluster info --brokers prod-cluster-1.example.com:9092,prod-cluster-2.example.com:9092 # If primary cluster is down, check shadow cluster health rpk cluster info --brokers shadow-cluster-1.example.com:9092,shadow-cluster-2.example.com:9092 ``` **Decision point**: If the primary cluster is responsive, consider whether failover is actually needed. Partial outages may not require full disaster recovery. **Examples that require full failover:** - Primary cluster is completely unreachable (network partition, regional outage) - Multiple broker failures preventing writes to critical topics - Data center failure affecting majority of brokers - Persistent authentication or authorization failures across the cluster **Examples that may NOT require failover:** - Single broker failure with sufficient replicas remaining - Temporary network connectivity issues affecting some clients - High latency or performance degradation (but cluster still functional) - Non-critical topic or partition unavailability ### [](#verify-shadow-status)Verify shadow cluster status Check the health of your shadow links: ```bash # List all shadow links rpk shadow list # Check the configuration of your shadow link rpk shadow describe # Check the status of your disaster recovery link rpk shadow status ``` For detailed command options, see [`rpk shadow list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-list/), [`rpk shadow describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-describe/), and [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/). Verify that the following conditions exist before proceeding with failover: - Shadow link state should be `ACTIVE`. - Topics should be in `ACTIVE` state (not `FAULTED`). - Replication lag should be reasonable for your RPO requirements. #### [](#understanding-replication-lag)Understanding replication lag Use [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/) or the [Data Plane API](https://docs.redpanda.com/api/doc/cloud-dataplane/operation/operation-shadowlinkservice_listshadowlinktopics) to check lag, which shows the message count difference between source and shadow partitions: - **Acceptable lag examples**: 0-1000 messages for low-throughput topics, 0-10000 messages for high-throughput topics - **Concerning lag examples**: Growing lag over 50,000 messages, or lag that continuously increases without recovering - **Critical lag examples**: Lag exceeding your data loss tolerance (for example, if you can only afford to lose 1 minute of data, lag should represent less than 1 minute of typical message volume) ### [](#document-state)Document current state Record the current lag and status before proceeding: ```bash # Capture current status for post-mortem analysis rpk shadow status > failover-status-$(date +%Y%m%d-%H%M%S).log ``` The partition information shows the following: | Field | Description | | --- | --- | | SRC_LSO | Source partition last stable offset | | SRC_HWM | Source partition high watermark | | DST_HWM | Shadow (destination) partition high watermark | | Lag | Message count difference between source and shadow partitions | > ❗ **IMPORTANT** > > Note the replication lag to estimate potential data loss during failover. The `Tasks` section shows the health of shadow link replication tasks. For details about what each task does, see [Shadow link tasks](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). ### [](#initiate-failover)Initiate failover A complete cluster failover is appropriate If you observe that the source cluster is no longer reachable: ```bash # Fail over all topics in the shadow link rpk shadow failover --all ``` For detailed command options, see [`rpk shadow failover`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-failover/). For selective topic failover (when only specific services are affected): ```bash # Fail over individual topics rpk shadow failover --topic rpk shadow failover --topic ``` ### [](#monitor-progress)Monitor failover progress Track the failover process: ```bash # Monitor status until all topics show FAILED_OVER watch -n 5 "rpk shadow status " # Check detailed topic status and lag during emergency rpk shadow status --print-topic ``` Example output during successful failover: shadow link: Overview: NAME UID STATE ACTIVE Tasks: Name Broker\_ID State Reason 1 ACTIVE 2 ACTIVE Topics: Name: , State: FAILED\_OVER Name: , State: FAILED\_OVER Name: , State: FAILING\_OVER **Wait for**: All critical topics to reach `FAILED_OVER` state before proceeding. ### [](#update-applications)Update application configuration Redirect your applications to the shadow cluster by updating connection strings in your applications to point to shadow cluster brokers. If using DNS-based service discovery, update DNS records accordingly. Restart applications to pick up new connection settings and verify connectivity from application hosts to shadow cluster. ### [](#verify-functionality)Verify application functionality Test critical application workflows: ```bash # Verify applications can produce messages rpk topic produce --brokers :9092 # Verify applications can consume messages rpk topic consume --brokers :9092 --num 1 ``` Test message production and consumption, consumer group functionality, and critical business workflows to ensure everything is working properly. ### [](#cleanup-stabilize)Clean up and stabilize After all applications are running normally: ```bash # Optional: Delete the shadow link (no longer needed) rpk shadow delete ``` For detailed command options, see [`rpk shadow delete`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-delete/). > 📝 **NOTE** > > This operation [force deletes](#force-delete-warning) the shadow link. Document the time of failover initiation and completion, applications affected and recovery times, data loss estimates based on replication lag, and issues encountered during failover. ## [](#troubleshoot-common-issues)Troubleshoot common issues ### [](#topics-stuck-in-failing_over-state)Topics stuck in FAILING_OVER state **Problem**: Topics remain in `FAILING_OVER` state for extended periods **Solution**: Check shadow cluster logs for specific error messages and ensure sufficient cluster resources (CPU, memory, disk space) are available on the shadow cluster. Verify network connectivity between shadow cluster nodes and confirm that all shadow topic partitions have elected leaders and the controller partition is properly replicated with an active leader. If topics remain stuck after addressing these cluster health issues and you need immediate failover, you can force delete the shadow link to failover all topics: ```bash # Force delete the shadow link to failover all topics rpk shadow delete --force ``` > ⚠️ **WARNING** > > Force deleting a shadow link immediately fails over all topics in the link. This action is irreversible and should only be used when topics are stuck and you need immediate access to all replicated data. ### [](#topics-in-faulted-state)Topics in FAULTED state **Problem**: Topics show `FAULTED` state and are not replicating **Solution**: Check for authentication issues, network connectivity problems, or source cluster unavailability. Verify that the shadow link service account still has the required permissions on the source cluster. Review shadow cluster logs for specific error messages about the faulted topics. ### [](#application-connection-failures)Application connection failures **Problem**: Applications cannot connect to shadow cluster after failover **Solution**: Verify shadow cluster broker endpoints are correct and check security group and firewall rules. Confirm authentication credentials are valid for the shadow cluster and test network connectivity from application hosts. ### [](#consumer-group-offset-issues)Consumer group offset issues **Problem**: Consumers start from beginning or wrong positions **Solution**: Verify consumer group offsets were replicated (check your filters) and use `rpk group describe ` to check offset positions. If necessary, manually reset offsets to appropriate positions. See [How to manage consumer group offsets in Redpanda](https://support.redpanda.com/hc/en-us/articles/23499121317399-How-to-manage-consumer-group-offsets-in-Redpanda) for detailed reset procedures. ## [](#next-steps)Next steps After successful failover, focus on recovery planning and process improvement. Begin by assessing the source cluster failure and determining whether to restore the original cluster or permanently promote the shadow cluster as your new primary. **Immediate recovery planning:** 1. **Assess source cluster**: Determine root cause of the outage 2. **Plan recovery**: Decide whether to restore source cluster or promote shadow cluster permanently 3. **Data synchronization**: Plan how to synchronize any data produced during failover 4. **Fail forward**: Create a new shadow link with the failed over shadow cluster as source to maintain a DR cluster **Process improvement:** 1. **Document the incident**: Record timeline, impact, and lessons learned 2. **Update runbooks**: Improve procedures based on what you learned 3. **Test regularly**: Schedule regular disaster recovery drills 4. **Review monitoring**: Ensure monitoring caught the issue appropriately --- # Page 140: Failover **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover.md --- # Failover > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Failover latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/failover page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/failover.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/failover.adoc description: Learn how failover can transform shadow topics into fully writable resources during disasters. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-12-16" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Failover is the process of modifying shadow topics or an entire shadow cluster from read-only replicas to fully writable resources, and ceasing replication from the source cluster. You can fail over individual topics for selective workload migration or fail over the entire cluster for comprehensive disaster recovery. This critical operation transforms your shadow resources into operational production assets, allowing you to redirect application traffic when the source cluster becomes unavailable. You can failover a shadow link using Redpanda Console, `rpk`, or the Admin API. > 📝 **NOTE** > > If you are using Kubernetes, you can also use the Redpanda Operator’s `ShadowLink` resource to manage failover. See [Kubernetes Shadow Link Failover](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-failover-runbook/) for details. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. ## [](#failover-behavior)Failover behavior When you initiate failover, Redpanda performs the following operations: 1. **Stops replication**: Halts all data fetching from the source cluster for the specified topics or entire shadow link 2. **Failover topics**: Converts read-only shadow topics into regular, writable topics 3. **Updates topic state**: Changes topic status from `ACTIVE` to `FAILING_OVER`, then `FAILED_OVER` Topic failover is irreversible. Once failed over, topics cannot return to shadow mode, and automatic fallback to the original source cluster is not supported. > 📝 **NOTE** > > To avoid a split-brain scenario after failover, ensure that all clients are reconfigured to point to the shadow cluster before resuming write activity. ## [](#failover-commands)Failover commands You can perform failover at different levels of granularity to match your disaster recovery needs: ### [](#individual-topic-failover)Individual topic failover To fail over a specific shadow topic while leaving other topics in the shadow link still replicating, run: ```bash rpk shadow failover --topic ``` For detailed command options, see [`rpk shadow failover`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-failover/). Use this approach when you need to selectively failover specific workloads or when testing failover procedures. ### [](#complete-shadow-link-failover-cluster-failover)Complete shadow link failover (cluster failover) To fail over all shadow topics associated with the shadow link simultaneously, run: ```bash rpk shadow failover --all ``` Use this approach during a complete regional disaster when you need to activate the entire shadow cluster as your new production environment. ### [](#force-delete-shadow-link-emergency-failover)Force delete shadow link (emergency failover) ```bash rpk shadow delete --force ``` > ⚠️ **WARNING** > > Force deleting a shadow link is irreversible and immediately fails over all topics in the link, bypassing the normal failover state transitions. This action should only be used as a last resort when topics are stuck in transitional states and you need immediate access to all replicated data. ## [](#failover-states)Failover states ### [](#shadow-link-states)Shadow link states The shadow link itself has a simple state model: - **`ACTIVE`**: Shadow link is operating normally, replicating data - **`PAUSED`**: Shadow link replication is temporarily halted by user action Shadow links do not have dedicated failover states. Instead, the link’s operational status is determined by the collective state of its shadow topics. ### [](#shadow-topic-states)Shadow topic states Individual shadow topics progress through specific states during failover: - **`ACTIVE`**: Normal replication state before failover - **`FAULTED`**: Shadow topic has encountered an error and is not replicating - **`FAILING_OVER`**: Failover initiated, replication stopping - **`FAILED_OVER`**: Failover completed successfully, topic fully writable - **`PAUSED`**: Replication temporarily halted by user action ## [](#monitor-failover-progress)Monitor failover progress To monitor failover progress using the status command, run: ```bash rpk shadow status ``` The output shows individual topic states and any issues encountered during the failover process. For detailed command options, see [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/). Task states during monitoring: - **`ACTIVE`**: Task is operating normally and replicating data - **`FAULTED`**: Task encountered an error and requires attention - **`NOT_RUNNING`**: Task is not currently executing - **`LINK_UNAVAILABLE`**: Task cannot communicate with the source cluster For detailed information about shadow link tasks and their roles, see [Shadow link tasks](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). ## [](#post-failover-cluster-behavior)Post-failover cluster behavior After successful failover, your shadow cluster exhibits the following characteristics: **Topic accessibility:** - Failed over topics become fully writable and readable. - Applications can produce and consume messages normally. - All Kafka APIs are available for failedover topics. - Original offsets and timestamps are preserved. **Shadow link status:** - The shadow link remains but stops replicating data. - Link status shows topics in `FAILED_OVER` state. - You can safely delete the shadow link after successful failover. **Operational limitations:** - No automatic fallback mechanism to the original source cluster. - Data transforms remain disabled until you manually re-enable them. - Audit log history from the source cluster is not available (new audit logs begin immediately). ## [](#failover-considerations-and-limitations)Failover considerations and limitations Before implementing failover procedures, understand these key considerations that affect your disaster recovery strategy and operational planning. **Data consistency:** - Some data loss may occur due to replication lag at the time of failover. - Consumer group offsets are preserved, allowing applications to resume from their last committed position. - In-flight transactions at the source cluster are not replicated and will be lost. **Recovery-point-objective (RPO):** The amount of potential data loss depends on replication lag when disaster occurs. Monitor lag metrics to understand your effective RPO. **Network partitions:** If the source cluster becomes accessible again after failover, do not attempt to write to both clusters simultaneously. This creates a scenario with potential data inconsistencies, since metadata starts to diverge. **Testing requirements:** Regularly test failover procedures in non-production environments to validate your disaster recovery processes and measure RTO. ## [](#next-steps)Next steps After completing failover: - Update your application connection strings to point to the shadow cluster - Verify that applications can produce and consume messages normally - Consider deleting the shadow link if failover was successful and permanent For emergency situations, see [Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/). --- # Page 141: Migrate Schemas from Confluent Schema Registry **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent.md --- # Migrate Schemas from Confluent Schema Registry > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Migrate Schemas from Confluent Schema Registry latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/migrate-schemas-confluent page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/migrate-schemas-confluent.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/migrate-schemas-confluent.adoc description: Replicate subjects, versions, and compatibility settings from a Confluent Schema Registry into a Redpanda shadow cluster. page-topic-type: how-to learning-objective-1: Configure a shadow link that continuously replicates schemas from a Confluent Schema Registry learning-objective-2: Filter replication by context or subject and map source contexts to destination contexts learning-objective-3: Monitor schema replication status and resolve validation errors page-git-created-date: "2026-07-20" page-git-modified-date: "2026-08-25" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. When you migrate to Redpanda from a deployment that uses a Confluent Schema Registry, your producers and consumers depend on the [schemas](https://docs.redpanda.com/streaming/current/reference/glossary/#schema) stored in that registry. Shadowing removes this migration obstacle: a [shadow link](https://docs.redpanda.com/streaming/current/reference/glossary/#shadow-link) continuously replicates schemas from the source Confluent Schema Registry into the [Schema Registry](https://docs.redpanda.com/streaming/current/reference/glossary/#schema-registry) built into the Redpanda [shadow cluster](https://docs.redpanda.com/streaming/current/reference/glossary/#shadow-cluster), preserving [subject](https://docs.redpanda.com/streaming/current/reference/glossary/#subject) names, versions, and compatibility settings. Because both registries stay synchronized until cutover, your applications keep working on Redpanda without a separate schema migration step. Use this approach when you migrate from Confluent to Redpanda, or when you maintain a Redpanda disaster recovery cluster for a system that keeps its schemas in a Confluent Schema Registry. After reading this page, you will be able to: - Configure a shadow link that continuously replicates schemas from a Confluent Schema Registry - Filter replication by context or subject and map source contexts to destination contexts - Monitor schema replication status and resolve validation errors ## [](#how-http-api-schema-replication-works)How HTTP API schema replication works When you configure a shadow link with the `shadow_schema_registry_api` option, the shadow cluster polls the source Schema Registry over HTTP and imports changes into its own Schema Registry. Two sync cycles keep the registries in step: - **Tail syncs** run frequently (default: every 10 seconds) to pick up incremental changes. - **Full syncs** scan all selected subjects (default: every 5 minutes) to catch anything a tail sync missed. Replicated schemas keep their original subject names and version IDs, so producers and consumers that reference schemas by ID continue to work after failover. Schemas that reference other schemas import in dependency order. Before importing a schema, Redpanda validates it against the Redpanda Schema Registry implementation. If a schema uses features that Redpanda does not support, the sync either reports an error and skips the schema, or removes the unsupported fields and imports the rest, depending on the [validation policy](#choose-a-validation-policy) you choose. While the link is active, the destination contexts that the link replicates into are read-only: the shadow cluster rejects client writes to those contexts so that replicated schemas remain identical to the source. Contexts outside the link’s filter remain writable. > 📝 **NOTE** > > This API-based mode is an alternative to the byte-for-byte `_schemas` topic replication described in [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#schema-registry-synchronization). A shadow link uses one Schema Registry sync mode or the other, not both: > > - Use **topic mode** (`shadow_schema_registry_topic`) when the source is another Redpanda cluster and you want an exact, complete replica of its Schema Registry. Topic mode shadows the `_schemas` topic byte for byte, so it does not filter, remap, or validate schemas. > > - Use **API mode** (`shadow_schema_registry_api`) when the source is a Confluent Schema Registry, or when you need to replicate only selected contexts or subjects, map source contexts to different destination contexts, or control how schemas that use unsupported features are handled with a validation policy. > > > Schema replication settings live in the shadow link configuration. Two cluster properties, `schema_registry_sync_memory_bytes` and `schema_registry_sync_parallelism`, tune how much memory and concurrency the shadow cluster uses while importing schemas. The defaults suit most deployments. ## [](#use-cases)Use cases - **Migrate from Confluent to Redpanda**: Replicate schemas continuously while [Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/) replicates your topic data, then cut applications over to Redpanda once both are in sync. No separate schema migration tooling is required. - **Phased migration**: Use context and subject filters to migrate one team, application, or environment at a time. - **Registry reorganization**: Map source contexts to different destination contexts to restructure your Schema Registry as part of the migration. ## [](#prerequisites)Prerequisites - A cluster running Redpanda version 26.2 or later. The schema replication feature activates after all brokers complete the upgrade. - To configure schema replication in [Redpanda Console](https://docs.redpanda.com/streaming/current/console/) rather than with `rpk`, Redpanda Console v3.9.0 or later. The Schema Registry fields appear only when the shadow cluster reports Redpanda 26.2 or later. On an earlier cluster, Redpanda Console falls back to a single toggle that enables `_schemas` topic replication, and API mode is unavailable. - Network connectivity from the shadow cluster to the source Schema Registry HTTP endpoint. - Credentials for the source registry with permission to read subjects, versions, and configuration. For Confluent Cloud, use a Schema Registry API key and secret. - Basic Shadow link settings. See [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/). - The destination contexts that the link replicates into, as determined by your `source_filter` and `destination` mapping, must be empty on the shadow cluster. The rest of the shadow cluster’s Schema Registry does not need to be empty: contexts outside the link’s mappings are unaffected. - To replicate contexts other than the default context, the [`schema_registry_enable_qualified_subjects`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#schema_registry_enable_qualified_subjects) cluster property must be enabled on the shadow cluster (the default). See [Schema Registry Contexts](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/#prerequisites). ## [](#limitations)Limitations - HTTP basic authentication and mTLS are the supported authentication methods for the source registry. You can also connect to a source registry that requires no authentication. - Replication is one way, from the source registry to the shadow cluster. Destination contexts owned by the link are read-only until failover. - Schemas that use Confluent features not supported by the Redpanda Schema Registry are not replicated as-is. Choose a [validation policy](#choose-a-validation-policy) to control whether these schemas are skipped or imported without the unsupported fields. - Topic data replication and schema replication are coordinated, but not immediately consistent: schema arrival on the shadow cluster can be delayed by the tail refresh interval (`tail_interval`), which defaults to 10 seconds. Records serialized in the Confluent SerDes wire format can therefore arrive before the schema IDs they contain. This does not cause replication errors, because schema IDs are not validated during topic data replication, but consumers that look up those schema IDs on the shadow cluster fail until the schemas arrive. See [Monitor replication status](#monitor-replication-status). - Role synchronization requires a Redpanda source. Leave `role_sync_options` unconfigured when the source is a Confluent cluster: a Roles Migrator task pointed at a non-Redpanda source reports `LINK_UNAVAILABLE` while the link itself stays `ACTIVE`. - Deleting and recreating a subject: A hard delete on the source replicates on the next sync, not instantly. Wait until the subject is removed from the shadow cluster before registering a new schema under the same name, as recreating it too soon prevents the sync job from detecting the subject/subject-version change. ## [](#configure-schema-replication)Configure schema replication Schema replication is one synchronization task on a shadow link. You configure it by adding the `shadow_schema_registry_api` option to the `schema_registry_sync_options` section of the same shadow link that replicates your topic data, consumer offsets, and ACLs. The sections below extend the shadow link configuration described in [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#create-a-shadow-link); they do not replace it. > ❗ **IMPORTANT** > > A shadow link whose configuration contains only `schema_registry_sync_options` replicates schemas and nothing else. For a full migration, keep your topic, consumer offset, and security synchronization options in the same configuration file. This workflow builds the configuration file that `rpk shadow create` consumes. The following collapsible sample shows where the schema replication settings sit in a complete shadow link configuration file. The highlighted lines are the schema replication settings, which the sections that follow explain. Explore a sample configuration file ```yaml # Sample shadow link configuration with API-mode Schema Registry replication name: confluent-migration # Unique name for this shadow link client_options: bootstrap_servers: # Source Kafka cluster brokers - : # Example: "pkc-xxxxx.us-east-1.aws.confluent.cloud:9092" - : # For the TLS and authentication settings that the shadow cluster uses to # connect to the source Kafka cluster, see the complete configuration file # reference in Configure Shadowing. # Keep the rest of your migration in the same file. Without these sections, # the link replicates schemas only. topic_metadata_sync_options: # ...your topic replication settings... consumer_offset_sync_options: # ...your consumer offset settings... security_sync_options: # ...your ACL replication settings... schema_registry_sync_options: shadow_schema_registry_api: # API mode: replicate from a Confluent Schema Registry source_url: https://psrc-xxxxx.us-east-1.aws.confluent.cloud # Source Schema Registry endpoint auth_options: basic: username: # Confluent Schema Registry API key password: # Confluent Schema Registry API secret tls_settings: enabled: true # Use TLS for the connection tail_interval: 10s # Optional: poll for incremental changes (default: 10s) full_sync_interval: 5m # Optional: full source scan interval (default: 5m) max_source_requests_per_second: 30 # Optional: rate limit for source requests (default: 30) source_filter: contexts: - "." # The default context subjects: [] # Empty: all subjects in the selected contexts destination: identity: {} # Keep source context names unsupported_schema_feature_policy: FAIL # FAIL (default) or REMOVE ``` For the complete configuration file, including consumer offset and security synchronization options, see [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#create-a-shadow-link). ### [](#generate-a-configuration-template)Generate a configuration template Generate a configuration file template that includes all available fields with comments: ```bash rpk shadow config generate --print-template -o shadow-config-template.yaml ``` For detailed command options, see [`rpk shadow config generate`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-config-generate/). ### [](#connect-to-the-source-registry)Connect to the source registry Configure the connection to the source Schema Registry: ```yaml schema_registry_sync_options: shadow_schema_registry_api: source_url: https://psrc-xxxxx.us-east-1.aws.confluent.cloud # Source Schema Registry endpoint auth_options: basic: username: # Confluent Schema Registry API key password: # Confluent Schema Registry API secret tls_settings: enabled: true # Use TLS for the connection tls_file_settings: ca_path: /path/to/ca.crt # Optional: CA certificate for custom trust tail_interval: 10s # How often to poll for incremental changes full_sync_interval: 5m # How often to run a full scan max_source_requests_per_second: 30 # Rate limit for requests to the source registry ``` The intervals and rate limit are optional. If you omit them, Redpanda uses the defaults shown above. To authenticate to the source registry with mTLS instead of HTTP basic authentication, omit `auth_options` and provide a client certificate and key in `tls_settings`. To connect to a source registry that requires no authentication, omit `auth_options` and do not provide a client certificate. ### [](#select-contexts-and-subjects)Select contexts and subjects By default, the link replicates the entire source registry. To replicate a subset, add a `source_filter` with the contexts or subjects to include: ```yaml schema_registry_sync_options: shadow_schema_registry_api: # ...connection settings... source_filter: contexts: - ".prod" # Replicate this entire context subjects: - orders-value # One subject from the default context ``` The two lists combine as a union: - `contexts` selects entire contexts: every subject in each listed context replicates. - `subjects` selects individual subjects, using qualified subject syntax: `orders-value` is the subject in the default context, and `:.staging:orders-value` is the subject of the same name in the `.staging` context. - When both lists are set, the link replicates everything selected by either list. A subject selected by both lists replicates once. For example, the preceding filter replicates every subject in the `.prod` context, plus the single `orders-value` subject from the default context. The `contexts` and `subjects` lists accept literal names only. Wildcard and prefix patterns are not supported. Schema Registry contexts provide independent namespaces for subjects within one registry. The default context is named `.`. For more information, see [Schema Registry Contexts](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/). ### [](#map-source-contexts-to-destination-contexts)Map source contexts to destination contexts Choose how replicated contexts are named on the shadow cluster: - `identity`: Keep the source context names (default behavior for migrations). - `exact`: Map each source context to a different destination context. ```yaml schema_registry_sync_options: shadow_schema_registry_api: # ...connection settings and filters... destination: identity: {} # Keep source context names ``` To rename contexts during replication: ```yaml schema_registry_sync_options: shadow_schema_registry_api: # ...connection settings and filters... destination: exact: mappings: - source: "." # Source context destination: ".shadow" # Destination context on the shadow cluster ``` > ❗ **IMPORTANT** > > With `exact` mapping, the mappings must cover every context that the link replicates. If the link encounters a source context that has no mapping, the schema replication task fails. If new contexts might be created on the source registry after you set up the link, scope the `source_filter` `contexts` list to the mapped contexts so that an unexpected context cannot stop replication. ### [](#choose-a-validation-policy)Choose a validation policy The `unsupported_schema_feature_policy` setting controls what happens when a source schema uses features that the Redpanda Schema Registry does not support. The unsupported Confluent Schema Registry features are: - In schema definitions: rule sets and metadata tags. - In subject configurations: override metadata, override rule sets, default metadata, default rule sets, and compatibility groups. Compatibility groups are not the same as compatibility levels, which do replicate. The policy determines how the sync handles a schema or configuration that uses these features: | Policy | Behavior | | --- | --- | | FAIL (default) | The schema is not replicated. The sync records an error, reports it in the link status, and continues with the remaining schemas. | | REMOVE | The unsupported fields are removed and the rest of the schema is imported. The sync records each modification in the link status. | ```yaml schema_registry_sync_options: shadow_schema_registry_api: # ...connection settings, filters, and destination... unsupported_schema_feature_policy: FAIL ``` ### [](#create-the-shadow-link)Create the shadow link Create the shadow link with your completed configuration file: ```bash rpk shadow create --config-file shadow-config.yaml ``` For detailed command options, see [`rpk shadow create`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-create/). To change the schema replication settings on an existing link, see [`rpk shadow update`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-update/). To create the link in Redpanda Console instead, see [Configure schema replication in Redpanda Console](#configure-schema-replication-in-redpanda-console). ## [](#configure-schema-replication-in-redpanda-console)Configure schema replication in Redpanda Console [Redpanda Console](https://docs.redpanda.com/streaming/current/console/) writes the same schema replication settings as a configuration file. The modes, filters, and validation policy behave identically; only the names of the fields differ. Use the [field mapping](#console-fields-and-configuration-keys) to move between the two. ### [](#create-a-shadow-link-with-api-mode-schema-replication)Create a shadow link with API-mode schema replication Every schema replication setting lives in the **Shadow Schema Registry** card. Selecting the **Other** mode tab reveals the source connection, scope, and sync behavior sections: ![The Shadow Schema Registry card in Redpanda Console, with the Other mode tab selected and the source connection, scope, and sync behavior sections visible](https://docs.redpanda.com/streaming/current/console/_images/shadow-link-schema-registry.png) 1. From the navigation menu, select **Shadow Links**, then click **Create shadow link**. 2. On the **Connection** step, complete the connection details for the source Kafka cluster. See [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#create-a-shadow-link). 3. On the **Configuration** step, in the **Shadow Schema Registry** card, select the **Other** mode tab. **Other** is API mode. **Redpanda** shadows the `_schemas` topic instead, and **None** leaves the shadow cluster’s Schema Registry independent. 4. Under **Source connection**, enter the **Source URL** of the source Schema Registry. 5. For **Authentication**, select the method that the source registry requires: - **HTTP Basic**: enter the **Username** and **Password**. For Confluent Cloud, these are the Schema Registry API key and secret. - **None**: Redpanda sends requests to the source registry without authentication. Also select **None** to authenticate with mTLS, then provide a client certificate and private key in the TLS settings in the next step. 6. Leave **Enable TLS** on to connect to the source registry over TLS. It is on by default. Turn it off only if the source registry does not use TLS. To trust a private certificate authority, upload a CA certificate. To authenticate with mTLS, provide a client certificate and private key. 7. Under **Scope**, choose what to replicate: - **Entire Schema Registry** replicates every context and subject. - **Specify contexts and subjects** limits replication to the **Contexts** and **Subjects** that you enter. Press Enter after each entry. The two lists combine as a union. For the selection rules and the qualified subject syntax, see [Select contexts and subjects](#select-contexts-and-subjects). 8. Under **Destination contexts**, select **Preserve source context names** to keep the source names, or **Map source contexts to explicit destination contexts** to rename them. With explicit mapping, every source context in scope must map to a distinct destination context. See [Map source contexts to destination contexts](#map-source-contexts-to-destination-contexts). 9. Optional: expand **Sync behavior** to change the **Tail interval**, **Full sync interval**, **Max source request rate**, or the **Unsupported schema features** policy. Leave these fields empty to use the cluster defaults. 10. Click **Create shadow link**. Redpanda Console does not test the source connection or list the matching subjects before you save. To confirm that the link works, [verify the configuration](#verify-the-configuration), then [monitor replication status](#monitor-replication-status). ### [](#console-fields-and-configuration-keys)Console fields and configuration keys Most fields carry the name of the configuration key they set, such as **Tail interval** for `tail_interval`. The following fields do not: | Redpanda Console | Configuration key | | --- | --- | | Shadow Schema Registry mode: Redpanda, Other, or None | shadow_schema_registry_topic, shadow_schema_registry_api, or neither | | Authentication: HTTP Basic or None | auth_options.basic, with username and password, or auth_options omitted | | Enable TLS, CA certificate, and client certificate and key | tls_settings | | Scope: Entire Schema Registry or Specify contexts and subjects, with Contexts and Subjects | source_filter.contexts and source_filter.subjects | | Destination contexts: Preserve source context names | destination.identity | | Destination contexts: Map source contexts to explicit destination contexts | destination.exact.mappings | | Max source request rate | max_source_requests_per_second | | Unsupported schema features: Fail the sync or Remove unsupported features | unsupported_schema_feature_policy: FAIL or REMOVE | ### [](#edit-an-existing-shadow-link)Edit an existing shadow link You can change the schema replication settings of an existing link in Redpanda Console, with the following constraints: - You cannot switch an existing link directly between **Redpanda** and **Other**. Redpanda Console locks the unavailable tab. To use a different Schema Registry replication mode, create a new shadow link. - Switching away from **Other** discards the stored Schema Registry connection settings, including credentials, scope, and sync behavior. Redpanda Console warns you before you save. - Turning off **Redpanda** mode does not remove a `_schemas` shadow topic that the link already added. To stop shadowing that topic, fail over the link, or delete the shadow topic after you save. Deleting the topic also discards the schemas that the link replicated into it, so fail over instead if you need to keep them. - When you edit a link that uses **HTTP Basic**, Redpanda Console requires you to re-enter the **Password**. It never populates the field with the stored value, and it does not accept an empty field. - Redpanda Console preserves settings that it does not expose, such as `paused`, when you save. ## [](#verify-the-configuration)Verify the configuration Confirm that the link is configured for API-based schema replication. ### [](#use-rpk)Use rpk ```bash rpk shadow describe --print-registry ``` The `--print-registry` flag prints the Schema Registry section, which includes the shadowing mode, source URL, sync intervals, validation policy, and your context and subject filters. Without it, `rpk shadow describe` prints only the overview and client sections. For detailed command options, see [`rpk shadow describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-describe/). ### [](#use-redpanda-console)Use Redpanda Console On the shadow link’s detail page, the **Schema Registry** section reports the stored configuration as read-only: | Field | Description | | --- | --- | | Connection | The Source URL of the source Schema Registry. | | Authentication | The authentication Type and Username. The Password shows as Set, Not set, or the date it last changed; Redpanda Console never displays the stored value. When the link does not use HTTP basic authentication, this section reports that no authentication is configured. | | TLS | Whether TLS is Enabled, whether the Trust store is a Custom CA or the System trust store, and whether Client auth uses an mTLS certificate. | | Scope | Either Entire Schema Registry, or the Contexts and Subjects that the link replicates. | | Destination contexts mapping | The source-to-destination context mappings, or the source names when the link preserves them. | | Sync behavior | The tail interval, full sync interval, max source request rate, and unsupported schema features policy. Shows Cluster defaults when the link does not override them. | ## [](#monitor-replication-status)Monitor replication status Check schema replication progress and errors for a link: ```bash rpk shadow status ``` The Schema Registry section of the output reports: | Field | Description | | --- | --- | | Inventory | The number of selected subjects and subject versions on the source, compared with the number of subjects and versions on the destination. The registries are synchronized when the destination counts match the selected source counts. A destination that is behind the source indicates that replication is still in progress. | | Current sync | The type of the sync in progress (FULL or TAIL) and the number of subject versions, compatibility configurations, and modes it has changed, including how many unsupported features were removed and how many errors occurred. | | Last full sync | Start time, finish time, and change counts for the most recent completed full sync. | | Totals since task start | Cumulative change and error counts since the schema replication task started. | | Last error | The most recent replication error. With the FAIL validation policy, schemas that fail validation appear here. | The schema replication task runs on the broker and shard that hosts the leader of the `_schemas` topic’s partition. The counters in the status output are not persisted: expect them to reset to zero when that broker restarts or when leadership of the `_schemas` partition moves to another broker. For detailed command options, see [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/). For general link monitoring, see [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/). ## [](#fail-over)Fail over For best results, shadow links should be failed over as a single unit, which prevents partial failover scenarios and unexpected results. Selective failover, such as topics only, does not stop schema replication. As part of failover, pause the schema replication task by setting `paused: true` in the `schema_registry_sync_options` section of the link configuration. Pausing the task stops further syncing from the source registry and makes the write-blocked destination contexts writable, so your applications can register new schemas on the promoted cluster. For the complete failover procedure, see [Failover](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover/). ## [](#next-steps)Next steps - [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/) to replicate topic data, consumer offsets, and ACLs alongside your schemas. - [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/) - [Schema Registry Contexts](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-contexts/) --- # Page 142: Monitor Shadowing **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor.md --- # Monitor Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Monitor Shadowing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/monitor page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/monitor.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/monitor.adoc description: Monitor Shadowing health with status commands, metrics, and best practices for tracking replication performance. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-08-25" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Monitor your [shadow links](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/) to ensure proper replication performance and understand your disaster recovery readiness. Use `rpk` commands, metrics, and status information to track shadow link health and troubleshoot issues. > 📝 **NOTE** > > If you’re running Redpanda in Kubernetes, see [Monitor Kubernetes Shadow Links](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-monitor-shadowing/). > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. ## [](#status-commands)Status commands To list existing shadow links: ```bash rpk shadow list ``` To view shadow link configuration details: ```bash rpk shadow describe ``` For detailed command options, see [`rpk shadow list`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-list/) and [`rpk shadow describe`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-describe/). This command shows the complete configuration of the shadow link, including connection settings, filters, and synchronization options. To check your shadow link status and ensure proper operation: ```bash rpk shadow status ``` - **Shadow link state**: Overall operational state (`ACTIVE`, `PAUSED`). - **Individual topic states**: Current state of each replicated topic (`ACTIVE`, `FAULTED`, `FAILING_OVER`, `FAILED_OVER`, `PAUSED`). - **Task status**: Health of replication tasks across brokers (`ACTIVE`, `FAULTED`, `NOT_RUNNING`, `LINK_UNAVAILABLE`). For details about shadow link tasks, see [Shadow link tasks](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). - **Failure reason**: When a shadow link or one of its tasks reports a failed state, the `Reason` field explains why. Check this field first when a link is not replicating as expected. - **Lag information**: Replication lag per partition showing source vs shadow high watermarks (HWM). - **Schema Registry sync status**: For links that replicate schemas through the Schema Registry API, inventory counts for source and destination subjects, sync progress, and the most recent error. See [Monitor replication status](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/#monitor-replication-status). - **Role sync counts**: For links that replicate RBAC roles, the `Reason` field of the [Roles Migrator task](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#roles-migrator-task) reports the counts from the most recent sync cycle (roles created, updated, deleted, and failures). See [Role filtering](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#role-filtering). ## [](#shadow-link-metrics)Metrics Shadowing provides comprehensive metrics to track replication performance and health with the [`public_metrics`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/) endpoint. | Metric | Type | Description | | --- | --- | --- | | redpanda_shadow_link_shadow_lag | Gauge | The lag of the shadow partition against the source partition, calculated as source partition LSO (Last Stable Offset) minus shadow partition HWM (High Watermark). Monitor by shadow_link_name, topic, and partition to understand replication lag for each partition. | | redpanda_shadow_link_total_bytes_fetched | Count | The total number of bytes fetched by a sharded replicator (bytes received by the client). Labeled by shadow_link_name and shard to track data transfer volume from the source cluster. | | redpanda_shadow_link_total_bytes_written | Count | The total number of bytes written by a sharded replicator (bytes written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor data written to the shadow cluster. | | redpanda_shadow_link_client_errors | Count | The number of errors seen by the client. Track by shadow_link_name and shard to identify connection or protocol issues between clusters. | | redpanda_shadow_link_shadow_topic_state | Gauge | Number of shadow topics in the respective states. Labeled by shadow_link_name and state to monitor topic state distribution across your shadow links. | | redpanda_shadow_link_total_records_fetched | Count | The total number of records fetched by the sharded replicator (records received by the client). Monitor by shadow_link_name and shard to track message throughput from the source. | | redpanda_shadow_link_total_records_written | Count | The total number of records written by a sharded replicator (records written to the write_at_offset_stm). Uses shadow_link_name and shard labels to monitor message throughput to the shadow cluster. | See also: [Public Metrics](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/) ## [](#monitoring-best-practices)Monitoring best practices ### [](#health-check-procedures)Health check procedures Establish regular monitoring workflows to ensure shadow link health: ```bash # Check all shadow links are active rpk shadow list | grep -v "ACTIVE" || echo "All shadow links healthy" # Monitor lag for critical topics rpk shadow status | grep -E "LAG|Lag" ``` ### [](#alert-conditions)Alert conditions Configure monitoring alerts for the following conditions, which indicate problems with Shadowing: - **High replication lag**: When `redpanda_shadow_link_shadow_lag` exceeds your RPO requirements - **Connection errors**: When `redpanda_shadow_link_client_errors` increases rapidly - **Topic state changes**: When topics move to `FAULTED` state - **Task failures**: When replication tasks enter `FAULTED` or `NOT_RUNNING` states - **Throughput drops**: When bytes/records fetched drops significantly - **Link unavailability**: When tasks show `LINK_UNAVAILABLE` indicating source cluster connectivity issues For more information about shadow link tasks and their states, see [Shadow link tasks](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). --- # Page 143: Shadowing Overview **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview.md --- # Shadowing Overview > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Shadowing Overview latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/overview page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/overview.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/overview.adoc description: Learn about disaster recovery using Shadowing for cross-region replication. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-08-25" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Shadowing is Redpanda’s enterprise-grade disaster recovery solution that establishes asynchronous, offset-preserving replication between two distinct Redpanda clusters. A cluster is able to create a dedicated client that continuously replicates source cluster data, including offsets, timestamps, and cluster metadata. This creates a read-only shadow cluster that you can quickly failover to handle production traffic during a disaster. Shadowing keeps data flowing, even during regional outages. > ❗ **IMPORTANT: Experiencing an active disaster?** > > See [Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/) for immediate step-by-step disaster procedures. Unlike traditional replication tools that re-produce messages, Shadowing copies data at the byte level, ensuring shadow topics contain identical copies of source topics with preserved offsets and timestamps. Shadowing replicates: - **Topic data**: All records with preserved offsets and timestamps - **Topic configurations**: Partition counts, retention policies, and other topic properties - **Consumer group offsets**: Enables seamless consumer resumption after failover - **Access control lists (ACLs)**: User permissions and security policies - **RBAC roles**: Role definitions and memberships, for the roles you select with filters - **Schema Registry data**: Schema definitions, versions, and compatibility settings, replicated from another Redpanda cluster or from a Confluent Schema Registry ## [](#how-shadowing-fits-into-disaster-recovery)How Shadowing fits into disaster recovery Shadowing addresses enterprise disaster recovery requirements driven by regulatory compliance and business continuity needs. Organizations typically want to minimize both recovery time objective (RTO) and recovery point objective (RPO), and Shadowing asynchronous replication helps you achieve both goals by reducing data loss during regional outages and enabling rapid application recovery. The architecture follows an active-passive pattern. The source cluster processes all production traffic while the shadow cluster remains in read-only mode, continuously receiving updates. If a disaster occurs, you can failover the shadow topics, making them fully writable. At that point, you can redirect your applications to the shadow cluster, which becomes the new production cluster. > 📝 **NOTE** > > To avoid a split-brain scenario after failover, ensure that all clients are reconfigured to point to the shadow cluster before resuming write activity. Shadowing complements Redpanda’s existing availability and recovery capabilities. [High availability](https://docs.redpanda.com/streaming/current/manage/high-availability/) actively protects your day-to-day operations, handling reads and writes seamlessly during node or availability zone failures within a region. Shadowing is your safety net for catastrophic regional disasters. While [Whole Cluster Restore](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/whole-cluster-restore/) provides point-in-time recovery from [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/), Shadowing delivers near real-time, cross-region replication for mission-critical applications that require rapid failover with minimal data loss. If you run Redpanda on Kubernetes and need multi-region availability without a separate standby cluster, you can instead deploy a single cluster across multiple Kubernetes clusters with synchronous replication. See [Deploy a Stretch Cluster on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). ## [](#limitations)Limitations Shadowing for disaster recovery currently has the following limitations: - Shadowing is designed for active-passive disaster recovery scenarios. Each shadow cluster can maintain only one shadow link. - Shadowing operates exclusively in asynchronous mode and doesn’t support active-active configurations. This means there will always be some replication lag. - [Data transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/) are not supported on shadow clusters while Shadowing is active. Writing to shadow topics is blocked. - During a disaster, [audit log](https://docs.redpanda.com/streaming/current/manage/audit-logging/) history from the source cluster is lost, though the shadow cluster begins generating new audit logs immediately after the failover. - After you failover shadow topics, automatic fallback to the original source cluster is not supported. ## [](#shadow-link-tasks)Shadow link tasks Shadow linking operates through specialized tasks that handle different aspects of replication. If you use a `shadow-config.yaml` configuration file to create the shadow link, each task corresponds to a section in the file. Tasks run continuously to maintain synchronization with the source cluster. #### Source Topic Sync The **Source Topic Sync task** manages topic discovery and metadata synchronization. This task periodically queries the source cluster to discover available topics, applies your configured topic filters to determine which topics should become shadow topics, and synchronizes topic properties between clusters. The task is controlled by the `topic_metadata_sync_options` section in the configuration file. It includes: - **Auto-creation filters**: Determines which source topics automatically become shadow topics - **Property synchronization**: Controls which topic properties replicate from source to shadow - **Starting offset**: Sets where new shadow topics begin replication (earliest, latest, or timestamp-based) - **Sync interval**: How frequently to check for new topics and property changes When this task discovers a new topic that matches your filters, it creates the corresponding shadow topic and begins replication from your configured starting offset. #### Consumer Group Shadowing The **Consumer Group Shadowing task** replicates consumer group offsets and membership information from the source cluster. This ensures that consumer applications can resume processing from the correct position after failover. The task is controlled by the `consumer_offset_sync_options` section in the configuration file. It includes: - **Group filters**: Determines which consumer groups have their offsets replicated - **Sync interval**: How frequently to synchronize consumer group offsets - **Offset clamping**: Automatically adjusts replicated offsets to valid ranges on the shadow cluster This task runs on brokers that host the `__consumer_offsets` topic and continuously tracks consumer group coordinators to optimize offset synchronization. #### Security Migrator The **Security Migrator task** replicates security policies, primarily ACLs (access control lists), from the source cluster to maintain consistent authorization across both environments. The task is controlled by the `security_sync_options` section in the configuration file. It includes: - **ACL filters**: Determines which security policies replicate - **Sync interval**: How frequently to synchronize security settings By default, all ACLs replicate to ensure your shadow cluster maintains the same security posture as your source cluster. #### Schema Registry Sync The **Schema Registry Sync task** replicates Schema Registry content so that applications that depend on schemas keep working after failover. The task is controlled by the `schema_registry_sync_options` section in the configuration file. It supports two modes: - **Topic mode** (`shadow_schema_registry_topic`): Shadows the `_schemas` system topic for byte-for-byte replication from another Redpanda cluster. - **API mode** (`shadow_schema_registry_api`): Polls the source Schema Registry over HTTP and imports selected contexts and subjects, with validation. Use this mode to replicate schemas from a Confluent Schema Registry. See [Migrate Schemas from Confluent Schema Registry](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/). A shadow link uses one mode or the other, not both. Only API mode runs as a separate task that appears in the shadow link status. Topic mode adds the `_schemas` topic to the set of shadowed topics, so it is monitored like any other shadow topic rather than as a separate task. #### Roles Migrator The **Roles Migrator task** replicates RBAC role definitions and role memberships from the source cluster, so that role-based permissions authorize the same operations on the shadow cluster after failover. The task is controlled by the `role_sync_options` section in the configuration file. It includes: - **Role name filters**: Determines which roles replicate. No roles replicate until you add at least one include filter. - **Sync interval**: How frequently to synchronize roles Within the configured filters, the task keeps the shadow cluster’s roles as a full mirror of the source, creating, updating, and deleting roles to match. It does not replicate user credentials or the ACLs bound to roles; the Security Migrator task replicates ACLs. See [Role filtering](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/#role-filtering). ### [](#task-status-and-monitoring)Task status and monitoring Each task reports its status through the shadow link status API. Task states include: - **`ACTIVE`**: Task is running normally and performing synchronization - **`PAUSED`**: Task has been manually paused through configuration - **`FAULTED`**: Task encountered an error and requires attention - **`NOT_RUNNING`**: Task is not currently executing - **`LINK_UNAVAILABLE`**: Task cannot communicate with the source cluster You can pause individual tasks by setting the `paused` field to `true` in the corresponding configuration section. This allows you to selectively disable parts of the replication process without affecting the entire shadow link. For monitoring task health and troubleshooting task issues, see [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/). ## [](#what-gets-replicated)What gets replicated Shadowing replicates your topic data with complete fidelity, preserving all message records with their original offsets, timestamps, headers, and metadata. The partition structure remains identical between source and shadow clusters, ensuring applications can resume processing from the exact same position after failover. Consumer group data flows according to your group filters, replicating offsets and membership information for matched groups. ACLs replicate based on your security filters. Schema Registry data synchronizes schema definitions, versions, and compatibility settings, either by shadowing the `_schemas` topic or through the [Schema Registry API](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/). Partition count is always replicated to ensure the shadow topic matches the source topic’s partition structure. ### [](#topic-properties-replication)Topic properties replication The [Source Topic Sync task](#shadow-link-tasks) handles topic property replication. For topic properties, Redpanda follows these replication rules: **Never replicated** - `redpanda.remote.readreplica` - `redpanda.remote.recovery` - `redpanda.remote.allowgaps` - `redpanda.virtual.cluster.id` - `redpanda.leaders.preference` - `redpanda.cloud_topic.enabled` **Always replicated** - `max.message.bytes` - `cleanup.policy` - `message.timestamp.type` **Always replicated (unless `exclude_default` is `true`)** - `compression.type` - `retention.bytes` - `retention.ms` - `delete.retention.ms` - `replication.factor` - `min.compaction.lag.ms` - `max.compaction.lag.ms` To replicate additional topic properties, explicitly list them in `synced_shadow_topic_properties`. The filtering system you configure determines the precise scope of replication across all components, allowing you to balance comprehensive disaster recovery with operational efficiency. ## [](#best-practices)Best practices To ensure reliable disaster recovery with Shadowing: - **Avoid write caching on source topics**: Do not shadow source topics that have [write caching](https://docs.redpanda.com/streaming/current/develop/manage-topics/config-topics/#configure-write-caching) enabled. Write caching can result in data loss on the source cluster during broker resets, causing cluster divergence if shadow links replicate data before it’s lost on the source. - **Do not modify shadow topic properties**: Avoid modifying synced topic properties on shadow topics, as these properties automatically revert to source topic values. ## [](#implementation-overview)Implementation overview Choose your implementation approach: - **[Setup and Configuration](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/)**: Initial shadow configuration, authentication, and topic selection - **[Monitoring and Operations](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/)**: Health checks, lag monitoring, and operational procedures - **[Planned Failover](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover/)**: Controlled disaster recovery testing and migrations - **[Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/)**: Rapid disaster response procedures > 💡 **TIP** > > You can create and manage shadow links with the [Redpanda Console](https://docs.redpanda.com/streaming/current/console/), the [Admin API v2](https://docs.redpanda.com/api/doc/admin/v2/), or [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk/), giving you flexibility in how you interact with your disaster recovery infrastructure. ## [](#next-steps)Next steps After setting up Shadowing for your Redpanda clusters, consider these additional steps: - **Test your disaster recovery procedures**: Regularly practice failover scenarios in a non-production environment. See [Failover Runbook](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover-runbook/) for step-by-step disaster procedures. - **Monitor shadow link health**: Set up alerting on the metrics described above to ensure early detection of replication issues. - **Implement automated failover**: Consider developing automation scripts that can detect outages and initiate failover based on predefined criteria. - **Review security policies**: Ensure your ACL filters replicate the appropriate security settings for your disaster recovery environment. - **Document your configuration**: Maintain up-to-date documentation of your shadow link configuration, including network settings, authentication details, and filter definitions. --- # Page 144: Configure Shadowing **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup.md --- # Configure Shadowing > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Configure Shadowing latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/setup page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/setup.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/setup.adoc description: Set up Shadowing for disaster recovery, including cross-region replication, data filters, networking, and authentication. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-08-25" support-status: supported --- You can create and manage shadow links with the [Redpanda Console](https://docs.redpanda.com/streaming/current/console/), the [Admin API v2](https://docs.redpanda.com/api/doc/admin/v2/), or [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk/), giving you flexibility in how you interact with your disaster recovery infrastructure. > 💡 **TIP** > > Deploy clusters in different geographic regions to protect against regional disasters. If you’re using Kubernetes, see [Configure Shadowing in Kubernetes](https://docs.redpanda.com/streaming/current/manage/kubernetes/shadowing/k-shadow-linking/) for Kubernetes-specific shadow link configuration. ## [](#prerequisites)Prerequisites > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. ### [](#license-and-cluster-requirements)License and cluster requirements - Both clusters must be running Redpanda v25.3 or later. - If you use Redpanda Console, ensure that it is running v3.3.0 or later. - You must have [Enterprise Edition](https://docs.redpanda.com/streaming/current/get-started/licensing/overview/) licenses on both clusters. > 💡 **TIP** > > When you later upgrade linked clusters, the upgrade order matters. See [Upgrade Clusters in a Shadowing Topology](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/upgrade/). ### [](#cluster-configuration)Cluster configuration The shadow cluster must have the [`enable_shadow_linking`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_shadow_linking) cluster property set to `true`. To enable this property, run: ```bash rpk cluster config set enable_shadow_linking true ``` > 📝 **NOTE** > > This cluster property must be configured using `rpk` or the Admin API v1 before you can create shadow links through any interface. To learn more about configuring cluster properties, see [Configure Cluster Properties](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-property-configuration/). ### [](#administrative-access)Administrative access Superuser access is required on both clusters through [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk/), the Admin API, or [Redpanda Console](https://docs.redpanda.com/streaming/current/console/) to create and manage shadow links. ### [](#replication-service-permissions)Replication service permissions You must configure a service account on the source cluster with the following [ACL](https://docs.redpanda.com/streaming/current/manage/security/authorization/acl/) permissions for shadow link replication: - **Topics**: `read` permission on all topics you want to replicate - **Topic configurations**: `describe_configs` permission on topics for configuration synchronization - **Consumer groups**: `describe` and `read` permission on consumer groups for offset replication - **ACLs**: `describe` permission on ACL resources to replicate security policies - **Cluster**: `describe` permission on the cluster resource to access ACLs This service account authenticates from the shadow cluster to the source cluster and performs the actual data replication. The credentials for this account are provided when you set up the shadow link. ### [](#network-and-authentication)Network and authentication You must configure network connectivity between clusters with appropriate firewall rules to allow the shadow cluster to connect to the source cluster for data replication. Shadowing uses a pull-based architecture where the shadow cluster fetches data from the source cluster. For detailed networking configuration, see [Networking](#networking). If using [authentication](https://docs.redpanda.com/streaming/current/manage/security/authentication/) for the shadow link connection, configure the source cluster with your chosen authentication method (SASL/SCRAM, SASL/PLAIN, TLS, mTLS) and ensure the shadow cluster has the proper credentials to authenticate to the source cluster. ## [](#set-up-shadowing)Set up Shadowing To set up Shadowing, you need to create a shadow link and configure filters to select which topics, consumer groups, ACLs, and Schema Registry data to replicate. ### [](#create-a-shadow-link)Create a shadow link Any cluster can create a shadow link to a source cluster. > 💡 **TIP** > > You can use `rpk` to generate a sample configuration file with common filter patterns: > > ```bash > # Generate a sample configuration file with placeholder values > rpk shadow config generate -o shadow-config.yaml > > # Or generate a template with detailed field documentation > rpk shadow config generate --print-template -o shadow-config-template.yaml > ``` > > This creates a complete YAML configuration file that you can customize for your environment. The template includes all available fields with comments explaining their purpose. For detailed command options, see [`rpk shadow config generate`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-config-generate/). Explore the configuration file ```yaml # Sample ShadowLinkConfig YAML with all fields name: # Unique name for this shadow link, example: "production-dr" client_options: bootstrap_servers: # Source cluster brokers to connect to - : # Example: "prod-kafka-1.example.com:9092" - : # Example: "prod-kafka-2.example.com:9092" - : # Example: "prod-kafka-3.example.com:9092" source_cluster_id: # Optional: UUID assigned by Redpanda # Example: a882bc98-7aca-40f6-a657-36a0b4daf1fd # To get source_cluster_id, run `rpk cluster config get cluster_id`. # TLS settings using file paths tls_settings: enabled: true # Enable TLS tls_file_settings: ca_path: # Path to CA certificate, example: "/etc/ssl/certs/ca.crt" key_path: # Optional: Path to client private key, example: "/etc/ssl/private/client.key" cert_path: # Optional: Path to client certificate, example: "/etc/ssl/certs/client.crt" do_not_set_sni_hostname: false # Optional: Skip SNI hostname when using TLS (default: false) # Create SASL credentials in the source cluster. # Then, with this configuration, ensure the shadow cluster uses the credentials # to authenticate to the source cluster. authentication_configuration: # SASL/SCRAM authentication scram_configuration: username: # SASL/SCRAM username, example: "shadow-replication-user" password: # SASL/SCRAM password, example: "secure-password-123" scram_mechanism: SCRAM_SHA_256 # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512" # SASL/PLAIN authentication plain_configuration: username: # SASL/PLAIN username, example: "shadow-replication-user" password: # SASL/PLAIN password, example: "secure-password-123" # Connection tuning - adjust based on network characteristics metadata_max_age_ms: 10000 # How often to refresh cluster metadata (default: 10000ms) connection_timeout_ms: 1000 # Connection timeout (default: 1000ms, increase for high latency) retry_backoff_ms: 100 # Backoff between retries (default: 100ms) fetch_wait_max_ms: 500 # Max time to wait for fetch requests (default: 500ms) fetch_min_bytes: 5242880 # Min bytes per fetch (default: 5MB) fetch_max_bytes: 20971520 # Max bytes per fetch (default: 20MB) fetch_partition_max_bytes: 5242880 # Max bytes per partition fetch (default: 5MB) topic_metadata_sync_options: interval: 30s # How often to sync topic metadata (examples: "30s", "1m", "5m") auto_create_shadow_topic_filters: # Filters for automatic topic creation - pattern_type: LITERAL # Include all topics (wildcard) filter_type: INCLUDE name: '*' - pattern_type: PREFIX # Exclude topics with specific prefix filter_type: EXCLUDE name: # Examples: "temp-", "test-", "debug-" synced_shadow_topic_properties: # Additional topic properties to sync (beyond defaults) - retention.ms # Topic retention time - segment.ms # Segment roll time exclude_default: false # Include default properties (compression, retention, etc.) start_at_earliest: {} # Start from the beginning of source topics (default) paused: false # Enable topic metadata synchronization consumer_offset_sync_options: interval: 30s # How often to sync consumer group offsets paused: false # Enable consumer offset synchronization group_filters: # Filters for consumer groups to sync - pattern_type: LITERAL filter_type: INCLUDE name: '*' # Include all consumer groups security_sync_options: interval: 30s # How often to sync security settings paused: false # Enable security settings synchronization acl_filters: # Filters for ACLs to sync - resource_filter: resource_type: TOPIC # Resource type: "TOPIC", "GROUP", "CLUSTER" pattern_type: PREFIXED # Pattern type: "LITERAL", "PREFIXED" name: # Examples: "prod-", "app-data-" access_filter: principal: User: # Principal name, example: "User:app-service" operation: ANY # Operation: "READ", "WRITE", "CREATE", "DELETE", "ALTER", "DESCRIBE", "ANY" permission_type: ALLOW # Permission: "ALLOW" or "DENY" host: '*' # Host pattern, examples: "*", "10.0.0.0/8", "app-server.example.com" schema_registry_sync_options: # Schema Registry synchronization options shadow_schema_registry_topic: {} # Byte-for-byte _schemas replication (Redpanda source) # For a Confluent source, replace the line above with a # shadow_schema_registry_api block (API mode). See Migrate Schemas # from Confluent Schema Registry. role_sync_options: # RBAC role synchronization options interval: 30s # How often to sync roles paused: false # Enable role synchronization role_name_filters: # Default empty: no roles sync until you add an INCLUDE filter - pattern_type: LITERAL # Pattern type: "LITERAL", "PREFIX" filter_type: INCLUDE # Filter type: "INCLUDE", "EXCLUDE" name: '*' # Include all roles ('*' must be the only character, LITERAL only) ``` To create a shadow link with the source cluster using `rpk`, run the following command from the shadow cluster: ```bash # Use the generated configuration file to create the shadow link rpk shadow create --config-file shadow-config.yaml ``` For detailed command options, see [`rpk shadow create`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-create/). > 💡 **TIP** > > Use [`rpk profile`](https://docs.redpanda.com/streaming/current/get-started/config-rpk-profile/) to save your cluster connection details and credentials for both source and shadow clusters. This allows you to easily switch between the two configurations. ### [](#set-filters)Set filters Filters determine which resources Shadowing automatically creates when establishing your shadow link. Topic filters select which topics Shadowing automatically creates as shadow topics when they appear on the source cluster. After Shadowing creates a shadow topic, it continues replicating until you failover the topic, delete it, or delete the entire shadow link. Consumer group and ACL filters control which groups and security policies replicate to maintain application functionality. #### [](#filter-types-and-patterns)Filter types and patterns Each filter uses two key settings: - **Pattern type**: Determines how names are matched - `LITERAL`: Matches names exactly (including the special wildcard `*` to match all items) - `PREFIX`: Matches names that start with the specified string - **Filter type**: Specifies whether to INCLUDE or EXCLUDE matching items - `INCLUDE`: Replicate items that match the pattern - `EXCLUDE`: Skip items that match the pattern #### [](#filter-processing-rules)Filter processing rules Redpanda processes filters in the order you define them with EXCLUDE filters taking precedence. Design your filter lists carefully: 1. **Exclude filters win**: If any EXCLUDE filter matches a resource, it is excluded regardless of INCLUDE filters. 2. **Order matters for INCLUDE filters**: Among INCLUDE filters, the first match determines the result. 3. **Default behavior**: Items that don’t match any filter are excluded from replication. #### [](#common-filtering-patterns)Common filtering patterns Replicate all topics except test topics: ```yaml topic_metadata_sync_options: auto_create_shadow_topic_filters: - pattern_type: PREFIX filter_type: EXCLUDE name: test- # Exclude all test topics - pattern_type: LITERAL filter_type: INCLUDE name: '*' # Include all other topics ``` Replicate only production topics: ```yaml topic_metadata_sync_options: auto_create_shadow_topic_filters: - pattern_type: PREFIX filter_type: INCLUDE name: prod- # Include production topics - pattern_type: PREFIX filter_type: INCLUDE name: production- # Alternative production prefix ``` Replicate specific consumer groups: ```yaml consumer_offset_sync_options: group_filters: - pattern_type: LITERAL filter_type: INCLUDE name: critical-app-consumers # Include specific consumer group - pattern_type: PREFIX filter_type: INCLUDE name: prod-consumer- # Include production consumers ``` #### [](#schema-registry-synchronization)Schema Registry synchronization Shadowing can replicate Schema Registry data in one of two modes: - **Topic mode** (`shadow_schema_registry_topic`): Shadows the `_schemas` system topic for byte-for-byte replication of schema definitions, versions, and compatibility settings from another Redpanda cluster. - **API mode** (`shadow_schema_registry_api`): Polls the source Schema Registry over HTTP and imports selected contexts and subjects, with validation. Use this mode to replicate schemas from a Confluent Schema Registry, or to replicate only part of the source registry. See [Migrate Schemas from Confluent Schema Registry](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/). A shadow link uses one mode or the other, not both. To enable topic mode, add the following to your shadow link configuration: ```yaml schema_registry_sync_options: shadow_schema_registry_topic: {} ``` To enable API mode instead, add a `shadow_schema_registry_api` block; its connection, filtering, and mapping options are described in [Migrate Schemas from Confluent Schema Registry](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/migrate-schemas-confluent/). Topic mode requirements: - The `_schemas` topic must exist on the source cluster - The `_schemas` topic must not exist on the shadow cluster, or must be empty - Once enabled, the `_schemas` topic will be replicated completely Important: After the `_schemas` topic becomes a shadow topic, it cannot be stopped without either failing over the topic or deleting it entirely. #### [](#system-topic-filtering-rules)System topic filtering rules Redpanda system topics have the following specific filtering restrictions: - Literal filters for `__consumer_offsets` and `_redpanda.audit_log` are rejected. - Prefix filters for topics starting with `_redpanda` or `__redpanda` are rejected. - Wildcard `*` filters will not match topics that start with `_redpanda` or `__redpanda`. - To shadow specific system topics, you must provide explicit literal filters for those individual topics. #### [](#acl-filtering)ACL filtering ACLs are replicated by the [Security Migrator task](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). This is recommended to ensure that your shadow cluster has the same permissions as your source cluster. To configure ACL filters: ```yaml security_sync_options: acl_filters: # Include read permissions for production topics - resource_filter: resource_type: TOPIC # Filter by topic resource pattern_type: PREFIXED # Match by prefix name: prod- # Production topic prefix access_filter: principal: User:app-user # Application service user operation: READ # Read operation permission_type: ALLOW # Allow permission host: '*' # Any host # Include consumer group permissions - resource_filter: resource_type: GROUP # Filter by consumer group pattern_type: LITERAL # Exact match name: '*' # All consumer groups access_filter: principal: User:app-user # Same application user operation: READ # Read operation permission_type: ALLOW # Allow permission host: '*' # Any host ``` #### [](#role-filtering)Role filtering After a failover, your users and applications need the same permissions on the shadow cluster that they had on the source. Role synchronization prepares the shadow cluster for that moment: the [Roles Migrator task](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#roles-migrator-task) periodically reads [RBAC](https://docs.redpanda.com/streaming/current/manage/security/authorization/rbac/) role definitions and role memberships from the source cluster and mirrors them to the shadow cluster. Use role synchronization together with [ACL filters](#acl-filtering) whenever your authorization model uses roles, so that role-based permissions keep working after failover. > ❗ **IMPORTANT** > > In the configuration file and `rpk`, no roles are replicated by default: the `role_name_filters` list defaults to empty, which synchronizes nothing. Add at least one `INCLUDE` filter to start replicating roles. When you create a shadow link in Redpanda Console on Redpanda 26.2 or later, the default is **All roles**. To replicate all roles: ```yaml role_sync_options: interval: 30s # How often to sync roles role_name_filters: - pattern_type: LITERAL filter_type: INCLUDE name: '*' # '*' must be the only character and requires LITERAL ``` Role filters use the same pattern and filter types as topic filters: `pattern_type` is `LITERAL` or `PREFIX`, and `filter_type` is `INCLUDE` or `EXCLUDE`. You can also configure role filters in Redpanda Console v3.11.0 or later, when you create a shadow link or when you edit an existing one on the **Shadowing** tab. The **Roles** card offers two modes: - **All roles**: Replicates every role. This is equivalent to a single `INCLUDE` filter with the literal name `*`, and it is the default when you create a shadow link in Redpanda Console. - **Specify roles**: Replicates only the roles that match the filters you add. Each filter takes a role name and a match mode: **Include specific roles**, **Include starting with**, **Exclude specific**, or **Exclude starting with**. The four modes correspond to the `filter_type` and `pattern_type` combinations above. When you edit a shadow link that has no role synchronization configured, the **Roles** card starts on **Specify roles** with no filters, and role synchronization remains off unless you add a filter or select **All roles**. Redpanda Console does not expose the `interval` or `paused` settings: a link created in Redpanda Console uses the default sync interval, and pausing role synchronization requires the configuration file or `rpk`. The shadow link details page shows the configured filters in the **Role replication** section. The section appears only when the link has role synchronization configured, and it shows **No role filters configured** when the filter list is empty. If the cluster does not support role synchronization (Redpanda versions earlier than 26.2), Redpanda Console hides the **Roles** card and sends no role sync options. Within the configured filters, the Roles Migrator task keeps the shadow cluster’s roles as a full mirror of the source: - The task creates roles that exist on the source but not on the shadow. - The task updates roles whose membership changed on the source. - The task deletes roles that were deleted on the source. The task does not modify roles on the shadow cluster that fall outside your filters. Role synchronization replicates role definitions and memberships only: - **Shadowing does not replicate user credentials.** Manage and distribute credentials outside of Shadowing, and provision users on the shadow cluster before failover. - **Role synchronization does not replicate the ACLs bound to roles.** Configure [ACL filters](#acl-filtering) as well, so that both the roles and the permissions granted to them exist on the shadow cluster. Role synchronization uses the same connection and service account as the rest of the shadow link. The cluster `describe` permission already required for ACL replication also covers role reads, and no additional network configuration is required. Both the source and shadow clusters must be running Redpanda 26.2 or later; the capability activates after the upgrade completes on all brokers. Role synchronization requires a Redpanda source cluster: if the source is an Apache Kafka cluster, the Roles Migrator task enters the `LINK_UNAVAILABLE` state, without affecting the other tasks on the link. Role synchronization continues to run after a [failover](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/failover/). Failover acts on topic data only: it stops topic replication and makes shadow topics writable, but the sync tasks, including the Roles Migrator, keep running on their intervals. To stop role synchronization, set `paused: true` in `role_sync_options`, or delete the shadow link. To monitor role synchronization, run [`rpk shadow status`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-status/). The Roles Migrator task’s `Reason` field reports the counts from the most recent sync cycle, for example: `Synced roles: 2 created, 0 updated, 0 deleted, 0 failures`. #### [](#consumer-group-filtering-and-behavior)Consumer group filtering and behavior Consumer group filters determine which consumer groups have their offsets replicated to the shadow cluster by the [Consumer Group Shadowing task](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks). Offset replication operates selectively within each consumer group. Only committed offsets for active shadow topics are synchronized, even if the consumer group has offsets for additional topics that aren’t being shadowed. For example, if consumer group "app-consumers" has committed offsets for "orders", "payments", and "inventory" topics, but only "orders" is an active shadow topic, then only the "orders" offsets will be replicated to the shadow cluster. ```yaml consumer_offset_sync_options: interval: 30s # How often to sync consumer group offsets paused: false # Enable consumer offset synchronization group_filters: - pattern_type: PREFIX filter_type: INCLUDE name: prod-consumer- # Include production consumer groups - pattern_type: LITERAL filter_type: EXCLUDE name: test-consumer-group # Exclude specific test groups ``` ##### [](#important-consumer-group-considerations)Important consumer group considerations **Avoid name conflicts:** If you plan to consume data from the shadow cluster, do not use the same consumer group names as those used on the source cluster. While this won’t break shadow linking, it can impact your RPO/RTO because conflicting group names may interfere with offset replication and consumer resumption during disaster recovery. **Offset clamping:** When Redpanda replicates consumer group offsets from the source cluster, offsets are automatically "clamped" during the commit process on the shadow cluster. If a committed offset from the source cluster is above the high watermark (HWM) of the corresponding shadow partition, Redpanda clamps the offset to the shadow partition’s HWM before committing it to the shadow cluster. This ensures offsets remain valid and prevents consumers from seeking beyond available data on the shadow cluster. #### [](#starting-offset-for-new-shadow-topics)Starting offset for new shadow topics When the [Source Topic Sync task](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/#shadow-link-tasks) creates a shadow topic for the first time, you can control where replication begins on the source topic. This setting only applies to empty shadow partitions and is crucial for disaster recovery planning. Changing this configuration only affects new shadow topics, existing shadow topics continue replicating from their current position. ```yaml topic_metadata_sync_options: start_at_earliest: {} ``` Alternatively, to start from the most recent offset: ```yaml topic_metadata_sync_options: start_at_latest: {} ``` Or to start from a specific timestamp: ```yaml topic_metadata_sync_options: start_at_timestamp: 2024-01-01T00:00:00Z ``` Starting offset options: - **`earliest`** (default): This replicates all existing data from the source topic. Use this for complete disaster recovery where you need full data history. - **`latest`**: This starts replication from the current end of the source topic, skipping existing data. Use this when you only need new data for disaster recovery and want to minimize initial replication time. - **`timestamp`**: This starts replication from the first record with a timestamp at or after the specified time. Use this for point-in-time disaster recovery scenarios. > ❗ **IMPORTANT** > > The starting offset only affects **new shadow topics**. After a shadow topic exists and has data, changing this setting has no effect on that topic’s replication. #### [](#networking)Networking Configure network connectivity between your source and shadow clusters to enable shadow link replication. The shadow cluster initiates connections to the source cluster using a pull-based architecture. For additional details about networking, see [Network and authentication](#network-and-authentication). ##### [](#connection-requirements)Connection requirements - **Direction**: Shadow cluster connects to source cluster (outbound from shadow, inbound to source) - **Protocol**: Kafka protocol over TCP (default port 9092, or your configured listener ports) - **Persistence**: Connections remain active for continuous replication ##### [](#firewall-configuration)Firewall configuration You must configure firewall rules to allow the shadow cluster to reach the source cluster. **On the source cluster network:** - Allow inbound TCP connections on Kafka listener ports (typically 9092). - Allow connections from the shadow cluster’s IP addresses or subnets. **On the shadow cluster network:** - Allow outbound TCP connections to the source cluster’s Kafka listener ports. - Ensure DNS resolution works for source cluster hostnames. ##### [](#bootstrap-servers)Bootstrap servers Specify multiple bootstrap servers in your shadow link configuration for high availability: ```yaml client_options: bootstrap_servers: # Source cluster brokers to connect to - : # Example: "prod-kafka-1.example.com:9092" - : # Example: "prod-kafka-2.example.com:9092" - : # Example: "prod-kafka-3.example.com:9092" ``` The shadow cluster uses these addresses to discover all brokers in the source cluster. If one bootstrap server is unavailable, the shadow cluster tries the next one in the list. ##### [](#network-security)Network security For production deployments, secure the network connection between clusters: TLS encryption: ```yaml client_options: tls_settings: enabled: true # Enable TLS tls_file_settings: ca_path: # Path to CA certificate, example: "/etc/ssl/certs/ca.crt" key_path: # Optional: Path to client private key, example: "/etc/ssl/private/client.key" cert_path: # Optional: Path to client certificate, example: "/etc/ssl/certs/client.crt" do_not_set_sni_hostname: false # Optional: Skip SNI hostname when using TLS (default: false) ``` Authentication: ```yaml client_options: authentication_configuration: # SASL/SCRAM authentication. # Create SASL credentials in the source cluster. # Then, with this configuration, ensure the shadow cluster uses the credentials # to authenticate to the source cluster. scram_configuration: username: # SASL/SCRAM username, example: "shadow-replication-user" password: # SASL/SCRAM password, example: "secure-password-123" scram_mechanism: SCRAM_SHA_256 # SCRAM mechanism: "SCRAM_SHA_256" or "SCRAM_SHA_512" # SASL/PLAIN authentication plain_configuration: username: # SASL/PLAIN username, example: "shadow-replication-user" password: # SASL/PLAIN password ``` ##### [](#connection-tuning)Connection tuning Adjust connection parameters based on your network characteristics. For example: ```yaml client_options: # Connection and metadata settings connection_timeout_ms: 1000 # Default 1000ms, increase for high-latency networks retry_backoff_ms: 100 # Default 100ms, backoff between connection retries metadata_max_age_ms: 10000 # Default 10000ms, how often to refresh cluster metadata # Fetch request settings fetch_wait_max_ms: 500 # Default 500ms, max time to wait for fetch requests fetch_min_bytes: 5242880 # Default 5MB, minimum bytes to fetch per request fetch_max_bytes: 20971520 # Default 20MB, maximum bytes to fetch per request fetch_partition_max_bytes: 5242880 # Default 5MB, maximum bytes to fetch per partition ``` ## [](#update-an-existing-shadow-link)Update an existing shadow link To modify a shadow link configuration after creation, run: ```bash rpk shadow update ``` For detailed command options, see [`rpk shadow update`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-shadow/rpk-shadow-update/). This opens your default editor to modify the shadow link configuration. Only changed fields are updated on the server. The shadow link name cannot be changed - you must delete and recreate the link to rename it. See also: [Admin API v2 reference](https://docs.redpanda.com/api/doc/admin/v2/) --- # Page 145: Upgrade Clusters in a Shadowing Topology **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/upgrade.md --- # Upgrade Clusters in a Shadowing Topology > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Upgrade Clusters in a Shadowing Topology latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/shadowing/upgrade page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/shadowing/upgrade.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/shadowing/upgrade.adoc description: "Upgrade clusters in a Shadowing topology safely: upgrade the shadow cluster before the source cluster, and use deferred finalization for bidirectional topologies." page-git-created-date: "2026-07-28" page-git-modified-date: "2026-07-28" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. When two clusters are connected by a [shadow link](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/overview/), an upgrade is no longer a single-cluster operation. The source and shadow clusters temporarily run different versions during the upgrade window, and the order in which you upgrade them determines whether topic metadata replication continues safely. This page gives you the upgrade order for shadow-linked clusters, and a coordinated procedure for bidirectional topologies, where each cluster shadows the other and no safe upgrade order exists. The bidirectional procedure uses [deferred upgrade finalization](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/) to keep new-version features inactive on both clusters until both are fully upgraded, so replication is never exposed to features that only one side supports. Use this guidance whenever you upgrade clusters that participate in a shadow link. Upgrading them like independent clusters risks breaking replication mid-upgrade. ## [](#upgrade-the-shadow-cluster-before-the-source-cluster)Upgrade the shadow cluster before the source cluster When you upgrade a source cluster and its shadow cluster, upgrade the shadow cluster first. Shadowing replicates topic configuration from the source to the shadow. If the source cluster is upgraded first, it can start using topic property values that the not-yet-upgraded shadow cluster does not support, which can break the topic metadata replication flow. Upgrading the shadow cluster first means the shadow always understands everything the source can send. For each cluster, follow the standard rolling upgrade procedure for your platform: [Linux](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/) or [Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/). After each cluster’s upgrade completes, verify shadow link health before proceeding: run `rpk shadow status ` on the shadow cluster and confirm that the link and its topics are in expected states. See [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/). ## [](#upgrade-a-bidirectional-topology)Upgrade a bidirectional topology In a bidirectional topology, two clusters each maintain a shadow link to the other, replicating non-overlapping sets of topics in opposite directions. Each cluster is an active source for its own topics and a shadow for the other cluster’s topics. The topic sets on the two links cannot overlap. Because each cluster is simultaneously a source and a shadow, the shadow-first upgrade order cannot be satisfied: whichever cluster you upgrade first is also a source for the other. The workaround is to defer upgrade finalization on both clusters. Both clusters roll to the new binaries while their version-gated features stay inactive, so neither side can start using functionality the other side doesn’t support yet. When both clusters are fully upgraded, you finalize both together. ### [](#prerequisites)Prerequisites - Both clusters must be running the same Redpanda version before you start the upgrade. - Both clusters must be running at least v26.1.9 or v25.3.15, and the upgrade target must be v26.2 or later. See [Defer Upgrade Finalization: Prerequisites](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/#prerequisites). - Both clusters need a valid Enterprise Edition license. Setting `features_auto_finalization` to `false` is rejected without one. - You need superuser access on both clusters. ### [](#procedure)Procedure 1. On both clusters, before beginning either cluster’s upgrade, disable automatic finalization: ```bash rpk cluster config set features_auto_finalization false ``` This is a runtime change; no restart is required. 2. Perform a rolling upgrade of the first cluster, following the standard procedure for your platform ([Linux](https://docs.redpanda.com/streaming/current/upgrade/rolling-upgrade/) or [Kubernetes](https://docs.redpanda.com/streaming/current/upgrade/k-rolling-upgrade/)). 3. Verify both replication directions: on each cluster, run `rpk shadow status ` and confirm the link, topic, and task states are healthy. See [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/). 4. Perform a rolling upgrade of the second cluster, then verify both replication directions again. 5. Confirm that both clusters are holding the upgrade open: ```bash rpk cluster upgrade status ``` Each cluster should report `ready to finalize`. Soak for as long as your validation requires. While either cluster is unfinalized, it can still [roll back to the previous release](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/#roll-back-to-the-previous-release). 6. Finalize both clusters promptly together. Do not leave one cluster finalized and the other unfinalized for longer than necessary. On each cluster: ```bash rpk cluster upgrade finalize ``` Finalization is asynchronous: confirm on each cluster that `rpk cluster upgrade status` reports `finalized` before moving on. See [Finalize the upgrade](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/#finalize-the-upgrade). 7. On both clusters, restore automatic finalization for future upgrades: ```bash rpk cluster config set features_auto_finalization true ``` 8. Start using new-version features only after both clusters are finalized. ### [](#what-to-expect-while-the-upgrade-is-unfinalized)What to expect while the upgrade is unfinalized - Existing shadow links keep replicating in both directions throughout the upgrade, the soak, and any rollback. - New shadow-link capabilities introduced by the new version cannot be configured yet. For example, on v26.2, configuring role sync or Schema Registry API-mode sync on a shadow link fails with a `FAILED_PRECONDITION` error stating that the feature cannot be configured until the upgrade is finalized. These errors are the feature gates working as designed. For the full list of gated features, see [What is unavailable while unfinalized](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/#what-is-unavailable-while-unfinalized). ## [](#limitations)Limitations - **Do not stay unfinalized indefinitely.** An unfinalized cluster blocks the next feature-release upgrade: Redpanda supports upgrades from exactly one feature release back, so both clusters must finalize before either can begin the next upgrade. - **New features wait for both clusters.** Adopt new-version features only after both clusters are upgraded and finalized. ## [](#suggested-reading)Suggested reading - [Defer Upgrade Finalization](https://docs.redpanda.com/streaming/current/upgrade/deferred-finalization/) - [Configure Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/setup/) - [Monitor Shadowing](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/shadowing/monitor/) --- # Page 146: Topic Recovery **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/topic-recovery.md --- # Topic Recovery > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Topic Recovery latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/topic-recovery page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/topic-recovery.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/topic-recovery.adoc description: Restore a single topic from object storage. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-11-19" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. When you create a topic, you can use remote recovery to download the topic data from object storage. This is useful when you need to restore a single topic in Tiered Storage that was accidentally deleted from a cluster. > ⚠️ **WARNING** > > While performing topic recovery, avoid adding additional load (such as produces, consumes, lists or additional recovery operations) to the target cluster. Doing so could destabilize the recovery process and result in either an unsuccessful or corrupted recovered topic. ## [](#prerequisites)Prerequisites You must have: - [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) enabled on your Redpanda cluster. - [Remote read](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#remote-read) (`redpanda.remote.read`) enabled on the topic you want to recover. ## [](#limitations)Limitations - Remote recovery is only safe when no other clusters are writing to the same bucket or container. - If you disable `redpanda.remote.read` after remote recovery, previously downloaded data will not be used to serve requests. ## [](#recover-a-topic)Recover a topic To create a new topic using remote recovery, in which the recovered topic can read and write in the cloud: ```bash rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=true -c redpanda.remote.read=true ``` To create a new topic using remote recovery, while also disabling the `redpanda.remote.write` property: ```bash rpk topic create -c redpanda.remote.recovery=true -c redpanda.remote.write=false -c redpanda.remote.read=true ``` --- # Page 147: Whole Cluster Restore **URL**: https://docs.redpanda.com/streaming/current/manage/disaster-recovery/whole-cluster-restore.md --- # Whole Cluster Restore > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Whole Cluster Restore latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: disaster-recovery/whole-cluster-restore page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: disaster-recovery/whole-cluster-restore.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/disaster-recovery/whole-cluster-restore.adoc description: Restore a failed cluster, including its metadata. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-11-19" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. With [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) enabled, you can use Whole Cluster Restore to restore data from a failed cluster (source cluster you are restoring from), including its metadata, onto a new cluster (target cluster you are restoring to). This is a simpler and cheaper alternative to active-active replication, for example with [MirrorMaker 2](https://docs.redpanda.com/streaming/current/migrate/data-migration/). Use this recovery method to restore your application to the latest functional state as quickly as possible. > ⚠️ **CAUTION** > > Whole Cluster Restore is not a fully-functional disaster recovery solution. It does not provide snapshot-style consistency. Some partitions in some topics will be more up-to-date than others. Committed transactions are not guaranteed to be atomic. > 💡 **TIP** > > If you need to restore only a subset of topic data, consider using [topic recovery](https://docs.redpanda.com/streaming/current/manage/disaster-recovery/topic-recovery/) instead of a Whole Cluster Restore. The following metadata is included in a Whole Cluster Restore: - Topic definitions. Topic data is restored only for topics that are archived to object storage. > 📝 **NOTE** > > Topics without archival to object storage are restored empty, with their definitions and configuration but no data. This does not apply to Cloud Topics (`redpanda.storage.mode=cloud`), whose data always resides in object storage and is fully restored. - Users and access control lists (ACLs). - [Schemas](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/). To ensure that your schemas are also archived and restored, you must also enable Tiered Storage for the `_schemas` topic. - The [consumer offsets topic](https://docs.redpanda.com/streaming/current/develop/consume-data/consumer-offsets/). Some restored committed consumer offsets may be truncated to a lower value than in the original cluster, to keep offsets at or below the highest restored offset in the partition. - Transaction metadata, up to the highest committed transaction. In-flight transactions are treated as aborted and will not be included in the restore. - [Cluster configurations](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/), including your Redpanda license key, with the exception of the following properties: - `cloud_storage_cache_size` - `cluster_id` - `cloud_storage_access_key` - `cloud_storage_secret_key` - `cloud_storage_region` - `cloud_storage_bucket` - `cloud_storage_api_endpoint` - `cloud_storage_credentials_source` - `cloud_storage_trust_file` - `cloud_storage_backend` - `cloud_storage_credentials_host` - `cloud_storage_azure_storage_account` - `cloud_storage_azure_container` - `cloud_storage_azure_shared_key` - `cloud_storage_azure_adls_endpoint` - `cloud_storage_azure_adls_port` ## [](#manage-source-metadata-uploads)Manage source metadata uploads By default, Redpanda uploads cluster metadata to object storage periodically. You can manage metadata uploads for your source cluster, or disable them entirely, with the following cluster configuration properties: - [`enable_cluster_metadata_upload_loop`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#enable_cluster_metadata_upload_loop): Enable metadata uploads. This property is enabled by default and is required for Whole Cluster Restore. - [`cloud_storage_cluster_metadata_upload_interval_ms`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_cluster_metadata_upload_interval_ms): Set the time interval to wait between metadata uploads. - [`controller_snapshot_max_age_sec`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#controller_snapshot_max_age_sec): Maximum amount of time that can pass before Redpanda attempts to take a controller snapshot after a new controller command appears. This property affects how current the uploaded metadata can be. - [`cloud_storage_cluster_name`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_cluster_name): This is an internal-only configuration and should be enabled only after consulting with Redpanda support. Specify a custom name for cluster’s metadata in object storage. For use when multiple clusters share the same storage bucket (for example, for Whole Cluster Restore). > 📝 **NOTE** > > You can monitor the [redpanda\_cluster\_latest\_cluster\_metadata\_manifest\_age](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_cluster_latest_cluster_metadata_manifest_age) metric to track the age of the most recent metadata upload. ## [](#restore-data-from-a-source-cluster)Restore data from a source cluster To restore data from a source cluster: 1. [Start a target cluster](#start-a-target-cluster) (new cluster). 2. [Restore data from a failed source cluster to the new cluster](#restore-to-target-cluster). ### [](#prerequisites)Prerequisites You must have the following: - Redpanda v23.3 or later on both source and target clusters. - [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) enabled on the source cluster. - Physical or virtual machines on which to deploy the target cluster. ### [](#limitations)Limitations - You cannot use Whole Cluster Restore if the target cluster is in [recovery mode](https://docs.redpanda.com/streaming/current/manage/recovery-mode/). - Whole Cluster Restore supports only one source cluster. It is not possible to consolidate multiple clusters onto the target cluster. - If a duplicate cluster configuration is found in the target cluster, it will be overwritten by the restore. - The target cluster should not contain user-managed or application-managed topic data, schemas, users, ACLs, or ongoing transactions. ### [](#start-a-target-cluster)Start a target cluster Follow the steps to [deploy a new cluster](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/). > 📝 **NOTE** > > Make sure to configure the target cluster with the same Tiered Storage settings as the source cluster. ### [](#restore-to-target-cluster)Restore to target cluster You can restore data from a source cluster to a target cluster using the [`rpk cluster storage restore`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-storage-restore/) command. 1. Restore data from the source cluster: ```bash rpk cluster storage restore start -w ``` The wait flag (`-w`) tells the command to poll the status of the restore process and then exit when completed. 2. Check if a rolling restart is required: ```bash rpk cluster config status ``` Example output when a restart is required: ```bash NODE CONFIG-VERSION NEEDS-RESTART INVALID UNKNOWN 1 4 true [] [] ``` 3. If a restart is required, perform a [rolling restart](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/rolling-restart/). When the cluster restore is successfully completed, you can redirect your application workload to the new cluster. Make sure to update your application code to use the new addresses of your brokers. ## [](#restore-data-from-multiple-clusters-sharing-the-same-bucket)Restore data from multiple clusters sharing the same bucket > ⚠️ **CAUTION** > > This is an advanced use case that should be performed only after consulting with Redpanda support. Typically, you will have a one-to-one mapping between a Redpanda cluster and its object storage bucket. However, it’s possible to run multiple clusters that share the same storage bucket. Sharing an object storage bucket allows you to move tenants between clusters without moving data. For example, you might wish to move topics (unmount on cluster A, mount on cluster B) to multiple clusters in the same bucket without having to move data. Running multiple clusters that share the same storage bucket presents unique challenges during Whole Cluster Restore operations. To manage these challenges, you must understand how Redpanda uses [UUIDs](#the-role-of-cluster-uuids-in-whole-cluster-restore) (universally unique identifiers) to identify clusters during a Whole Cluster Restore. This shared storage approach can create identification challenges during restore operations. ### [](#the-role-of-cluster-uuids-in-whole-cluster-restore)The role of cluster UUIDs in Whole Cluster Restore Each Redpanda cluster (single node or more) receives a unique UUID every time it starts. From that moment forward, all entities created by the cluster are identifiable using this cluster UUID. These entities include: - Topic data - Topic metadata - Whole Cluster Restore manifests - Controller log snapshots for Whole Cluster Restore - Consumer offsets for Whole Cluster Restore However, not all entities _managed_ by the cluster are identifiable using this cluster UUID. Each time a cluster uploads its metadata, the name of the object has two parts: the cluster UUID, which is unique each time you create a cluster (even after a restore it will have a new UUID), and a metadata (sequence) ID. When performing a restore, Redpanda scans the bucket to find the highest-sequenced ID uploaded by the cluster. It can be ambiguous what to restore when the highest sequential ID has been uploaded by another cluster, and result in a split-brain scenario, where you have two independent clusters that both believe they are the “rightful owner” of the same logical data. ### [](#configure-cluster-names-for-multiple-source-clusters)Configure cluster names for multiple source clusters To disambiguate cluster metadata from multiple clusters, use the [`cloud_storage_cluster_name`](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_cluster_name) property (off by default), which allows you to assign a unique name to each cluster sharing the same object storage bucket. Redpanda uses this name to organize the cluster metadata within the shared object storage bucket. This ensures that each cluster’s data remains distinct and prevents conflicts during recovery operations.The name must be unique within the bucket, 1-64 characters, and use only letters, numbers, underscores, and hyphens. Do not change this value once set. After setting, your object storage bucket organization may look like the following: ```bash / +- cluster_metadata/ | + /manifests/ | | +- 0/cluster_manifest.json | | +- 1/cluster_manifest.json | | +- 2/cluster_manifest.json | + /manifests/ | +- 0/cluster_manifest.json | +- 1/cluster_manifest.json # lost cluster +- cluster_name/ +- rp-foo/uuid/ +- rp-qux/uuid/ ``` During a Whole Cluster Restore, Redpanda looks for the cluster name specified in `cloud_storage_cluster_name` and only consider manifests associated with that name. Because the cluster name specified here is `rp-qux`, Redpanda only considers manifests for the clusters `` and `` (another new cluster sharing the bucket), ignoring cluster `` entirely. In this case, your object storage bucket may look like the following: ```bash +- cluster_metadata/ | + /manifests/ | | +- 0/cluster_manifest.json | | +- 1/cluster_manifest.json | | +- 2/cluster_manifest.json | + /manifests/ | | +- 0/cluster_manifest.json | | +- 1/cluster_manifest.json # lost cluster | + /manifests/ | +- 3/cluster_manifest.json # new cluster | # ^- next highest sequence number globally +- cluster_name/ +- rp-foo/uuid/ +- rp-qux/uuid/ +- +- # reference to new cluster ``` ### [](#resolve-repeated-recovery-failures)Resolve repeated recovery failures If you experience repeated failures when a cluster is lost and recreated, the automated recovery algorithm may have selected the manifest with the highest sequence number, which might be the most recent one with no data, instead of the original one containing the data. In such a scenario, your object storage bucket might be organized like the following: ```bash / +- cluster_metadata/ + /manifests/ | +- 0/cluster_manifest.json | +- 1/cluster_manifest.json #lost cluster + /manifests/ +- 3/cluster_manifest.json # lost again (not recovered) + /manifests/ +- 7/cluster_manifest.json # new attempt to recover uuid-b # it does not have the data ``` In such cases, you can explicitly specify the cluster UUID: #### rpk The `--cluster-uuid-override` option is available in v25.3.3 and later: ```bash rpk cluster storage restore start --cluster-uuid-override ``` #### Admin API ```bash curl -XPOST \ --data '{"cluster_uuid_override": ""}' \ http://localhost:9644/v1/cloud_storage/automated_recovery ``` For details, see the [Admin API reference](https://docs.redpanda.com/streaming/current/manage/use-admin-api/). --- # Page 148: Fast Commission and Decommission Brokers **URL**: https://docs.redpanda.com/streaming/current/manage/fast-commission-decommission.md --- # Fast Commission and Decommission Brokers > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Fast Commission and Decommission Brokers latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: fast-commission-decommission page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: fast-commission-decommission.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/fast-commission-decommission.adoc description: Configure fast partition movement during cluster resize for high availability. page-git-created-date: "2024-08-15" page-git-modified-date: "2025-07-31" support-status: supported --- Tiered Storage gives you the option to boost the speed and reduce the impact of broker operations, particularly when resizing a cluster. For instance, adding a new broker to increase capacity for an overburdened cluster could introduce additional stress as partition replicas are transferred to the new broker. For cloud deployments, commissioning or decommissioning a broker can be a slow and expensive process, especially across multiple availability zones (AZs). Tiered Storage can help make the cluster resize process faster and more cost-efficient by leveraging data that has already been uploaded to object storage. Instead of transferring all data from local storage to the reassigned replicas, the replicas can instead be initialized to rely more heavily on Tiered Storage for read requests. To accomplish this, Redpanda takes an on-demand snapshot at an offset that is already uploaded to object storage. This is a later offset compared to the default log start offset managed by [Raft](https://raft.github.io/). When an empty replica is initialized, the later offset uploaded to object storage can be used as the start offset source. Reads that access data before the uploaded offset can be executed as remote reads. Therefore, the only data that needs to be sent over the network to replicas is the data locally retained after the uploaded offset. This also results in less time taken overall to grow or shrink the cluster. ## [](#configure-fast-commission-and-decommission)Configure fast commission and decommission To activate fast commission and decommission when brokers enter and leave the cluster: 1. Make sure to [configure topics for Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage). 2. Configure at least one of the following cluster configuration properties. These properties limit the size of data replicated across brokers to local storage using Raft: - `initial_retention_local_target_bytes_default`: Initial local retention size target for partitions of topics with Tiered Storage enabled. The default is null. - Use the `initial.retention.local.target.bytes` topic configuration property to override on the topic level. - `initial_retention_local_target_ms_default`: Initial local retention time target for partitions of topics with Tiered Storage enabled. The default is null. - Use the `initial.retention.local.target.ms` topic configuration property to override on the topic level. If no values are set for the cluster configuration properties, all locally-retained data is delivered by default to the new broker (learner) when joining a partition replica set. > ❗ **IMPORTANT** > > Because topics become more reliant on object storage to serve data, you may experience higher latency reads for data beyond the range of the configured local [retention target](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#set-retention-limits). Carefully weigh the tradeoffs between faster broker commission/decommission and the increased read latency due to less data being available in local storage. ## [](#monitor-fast-commission-and-decommission)Monitor fast commission and decommission Use the following to monitor fast commission and decommission: - The `vectorized_cluster_partition_start_offset` metric on newly-joined brokers should be greater than on existing brokers. - Raft protocol logs taking an on-demand snapshot are logged. For example: ```bash INFO 2025-03-12 09:32:23,280 [shard 13:raft] raft - [follower: {id: 39, revision: 19481451}, term: 34] [group_id:116908, {kafka/foo/999}] - recovery_stm.cc:473 - creating on demand snapshot with last included offset: 316791, current leader start offset: 203732. Total partition size on leader 46.109MiB, expected to transfer to learner: 683.000 bytes ``` - Depending on retention settings, disk usage on a newly-joined broker should generally be much lower than on existing brokers. ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 149: High Availability **URL**: https://docs.redpanda.com/streaming/current/manage/high-availability.md --- # High Availability > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: High Availability latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: high-availability page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: high-availability.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/high-availability.adoc description: Learn about the trade-offs with different high availability configurations. page-git-created-date: "2025-11-19" page-git-modified-date: "2025-11-19" support-status: supported --- Redpanda is designed to ensure data integrity and high availability (HA), even at high-throughput levels. ## [](#deployment-strategies)Deployment strategies Consider the following Redpanda deployment strategies for the most common types of failures. | Failure | Impact | Mitigation strategy | | --- | --- | --- | | Broker failure | Loss of function for an individual broker or for any virtual machine (VM) that hosts the broker | Multi-broker deployment | | Rack or switch failure | Loss of brokers/VMs hosted within that rack, or loss of connectivity to them | Multi-broker deployment spread across multiple racks or network failure domains | | Data center failure | Loss of brokers/VMs hosted within that data center, or loss of connectivity to them | Multi-AZ or replicated deployment | | Region failure | Loss of brokers/VMs hosted within that region, or loss of connectivity to them | Geo-stretch (latency dependent), such as a Stretch Cluster on Kubernetes, or replicated deployment | | Global, systemic outage (DNS failures, routing failures) | Complete outage for all systems and services impacting customers and staff | Offline backups, replicas in 3rd-party domains | | Data loss or corruption (accidental or malicious) | Corrupt or unavailable data that also affects synchronous replicas | Offline backups | See also: [Deploy for Production](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment/) ## [](#ha-deployment-options)HA deployment options This section explains the trade-offs with different HA configurations. - [Multi-broker deployment](#multi-broker-deployment) - [Multi-AZ deployment](#multi-az-deployment) - [Multi-region deployment](#multi-region-deployment) - [Multi-cluster deployment](#multi-cluster-deployment) ### [](#multi-broker-deployment)Multi-broker deployment Redpanda is designed to be deployed in a cluster that consists of at least three brokers. Although clusters with a single broker are convenient for development and testing, they aren’t resilient to failure. Adding brokers to a cluster provides a way to handle individual broker failures. You can also use [\[rack awareness\]](<#rack awareness>) to assign brokers to different racks, which allows Redpanda to tolerate the loss of a rack or failure domain. ![Single-AZ deployment](https://docs.redpanda.com/streaming/current/shared/_images/single_az.png) See also: [Single-AZ deployments](#single-az-deployments) ### [](#multi-az-deployment)Multi-AZ deployment An availability zone (AZ) consists of one or more data centers served by high-bandwidth links with low latency (and typically within a close distance of one another). All AZs have discrete failure domains (power, cooling, fire, and network), but they also have common-cause failure domains, such as catastrophic events, that affect their geographical location. To safeguard against such possibilities, a cluster can be deployed across multiple AZs by configuring each AZ as a rack using rack awareness. Implementing Raft internally ensures that Redpanda can tolerate losing a minority of replicas for a given topic or for controller groups. For this to translate to a multi-AZ deployment, however, it’s necessary to deploy to at least three AZs (affording the loss of one zone). In a typical multi-AZ deployment, cluster performance is constrained by inter-AZ bandwidth and latency. See also: [Multi-AZ deployments](#multi-az-deployments) ![Multi-AZ deployment](https://docs.redpanda.com/streaming/current/shared/_images/multi_az.png) ### [](#multi-region-deployment)Multi-region deployment A multi-region deployment is similar to a multi-AZ deployment, in that it needs at least three regions to counter the loss of a single region. Note that this deployment strategy increases latency due to the physical distance between regions. In addition to higher produce and end-to-end latency and increased costs, multi-region deployments require careful tuning. Redpanda recommends that you work closely with Redpanda’s Customer Success team when implementing a multi-region deployment. Also consider the following strategies to mitigate these challenges: - Configure [Leader Pinning](https://docs.redpanda.com/streaming/current/develop/produce-data/leader-pinning/) to ensure that topic partition leaders are geographically closer to clients. This can help lower network costs and latency by routing produce requests to brokers located in specific AZs. - If your produce latency exceeds your requirements, you can configure producers to have `acks=1` instead of `acks=all`. This reduces latency by only waiting for the leader to acknowledge, rather than waiting for all brokers to respond. However, using this configuration can decrease message durability. If the partition leader goes offline, you may lose any messages that are acknowledged but not yet replicated. > 💡 **TIP** > > If you run Redpanda on Kubernetes, you can deploy one Redpanda cluster across multiple Kubernetes clusters in different regions. See [Deploy a Stretch Cluster on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). ### [](#multi-cluster-deployment)Multi-cluster deployment In a multi-cluster deployment, each cluster is configured using one of the other HA deployments, along with standby clusters or [Remote Read Replica](https://docs.redpanda.com/streaming/current/manage/remote-read-replicas/) clusters in one or more remote locations. A standby cluster is a fully functional cluster that can handle producers and consumers. A remote read replica is a read-only cluster that can act as a backup for topics. To replicate data across clusters in a multi-cluster deployment, use one of the following options: - [MirrorMaker2 replication](https://docs.redpanda.com/streaming/current/migrate/data-migration/) - [Remote Read Replicas](https://docs.redpanda.com/streaming/current/manage/remote-read-replicas/) - [Redpanda Edge Agent](https://github.com/redpanda-data/redpanda-edge-agent) Alternatively, you could dual-feed clusters in multiple regions. Dual feeding is the process of having producers connect to your cluster across multiple regions. However, this introduces additional complexity onto the producing application. It also requires consumers that have sufficient deduplication logic built in to handle offsets, since they won’t be the same across each cluster. > 📝 **NOTE** > > A multi-cluster deployment replicates data asynchronously between independent clusters, each with its own offsets. To run a single logical cluster whose brokers span multiple Kubernetes clusters, with synchronous replication and one set of offsets, see [Deploy a Stretch Cluster on Kubernetes](https://docs.redpanda.com/streaming/current/deploy/redpanda/kubernetes/k-stretch-clusters/). ## [](#ha-features-in-redpanda)HA features in Redpanda Redpanda includes the following high-availability features: - [Replica synchronization](#replica-synchronization) - [Rack awareness](#rack-awareness) - [Partition leadership](#partition-leadership) - [Producer acknowledgment](#producer-acknowledgment) - [Partition rebalancing](#partition-rebalancing) - [Tiered Storage and disaster recovery](#tiered-storage-and-disaster-recovery) ### [](#replica-synchronization)Replica synchronization A cluster’s availability is directly tied to replica synchronization. Brokers can be either leaders or replicas (followers) for a partition. A cluster’s replica brokers must be consistent with the leader to be available for consumers and producers. 1. The leader writes data to the disk. It then dispatches append entry requests to the followers in parallel with the disk flush. 2. The replicas receive messages written to the partition of the leader. They send acknowledgments to the leader after successfully replicating the message to their internal partition. 3. The leader sends an acknowledgment to the producer of the message, as determined by that producer’s `acks` value. Redpanda considers the group consistent after a majority has formed consensus; that is, a majority of participants acknowledged the write. While Apache Kafka® uses in-sync replicas, Redpanda uses a quorum-based majority with the Raft replication protocol. Kafka performance is negatively impacted when any "in-sync" replica is running slower than other replicas in the In-Sync Replica (ISR) set. Monitor the health of your cluster with the [`rpk cluster health`](https://docs.redpanda.com/streaming/current/reference/rpk/rpk-cluster/rpk-cluster-health/) command, which tells you if any brokers are down, and if you have any leaderless partitions. ### [](#rack-awareness)Rack awareness Rack awareness is one of the most important features for HA. It lets Redpanda spread partition replicas across available brokers in different failure zones. Rack awareness ensures that no more than a minority of replicas are placed on a single rack, even during cluster balancing. > 💡 **TIP** > > Make sure you assign separate rack IDs that actually correspond to a physical separation of brokers. See also: [Enable Rack Awareness](https://docs.redpanda.com/streaming/current/manage/rack-awareness/) ### [](#partition-leadership)Partition leadership Raft uses a heartbeat mechanism to maintain leadership authority and to trigger leader elections. The partition leader sends a periodic heartbeat to all followers to assert its leadership. If a follower does not receive a heartbeat over a period of time, then it triggers an election to choose a new partition leader. See also: [Partition leadership elections](https://docs.redpanda.com/streaming/current/get-started/architecture/#partition-leadership-elections) ### [](#producer-acknowledgment)Producer acknowledgment Producer acknowledgment defines how producer clients and broker leaders communicate their status while transferring data. The `acks` value determines producer and broker behavior when writing data to the event bus. See also: [Producer Acknowledgement Settings](https://docs.redpanda.com/streaming/current/develop/produce-data/configure-producers/) ### [](#partition-rebalancing)Partition rebalancing By default, Redpanda rebalances partition distribution when brokers are added or decommissioned. Continuous Data Balancing additionally rebalances partitions when brokers become unavailable or when disk space usage exceeds a threshold. See also: [Cluster Balancing](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/cluster-balancing/) ### [](#tiered-storage-and-disaster-recovery)Tiered Storage and disaster recovery In a disaster, your secondary cluster may still be available, but you need to quickly restore the original level of redundancy by bringing up a new primary cluster. In a containerized environment such as Kubernetes, all state is lost from pods that use only local storage. HA deployments with Tiered Storage address both these problems, since it offers long-term data retention and topic recovery. See also: [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) > ❗ **IMPORTANT** > > Tiered Storage operates as an asynchronous process and only applies to closed segments. Any open segments or segments existing only in local storage are not recoverable by your new primary cluster. ## [](#single-az-deployments)Single-AZ deployments When deploying a cluster for high availability into a single AZ or data center, you need to ensure that, within the AZ, single points of failure are minimized and that Redpanda is configured to be aware of any discrete failure domains within the AZ. This is achieved with Redpanda’s rack awareness, which deploys _n_ Redpanda brokers across three or more racks (or failure domains) within the AZ. Single-AZ deployments in the cloud have less network costs than multi-AZ deployments, and you can leverage resilient power supplies and networking infrastructure within the AZ to mitigate against all but total-AZ failure scenarios. You can balance the benefits of increased availability and fault tolerance against any increase in cost, performance, and complexity: - Cost: Redpanda operates the same Raft consensus algorithm whether it’s in HA mode or not. There may be infrastructure costs when deploying across multiple racks, but these are normally amortized across a wider datacenter operations program. - Performance: Spreading Redpanda replicas across racks and switches increases the number of network hops between Redpanda brokers; however, normal intra-data center network latency should be measured in microseconds rather than milliseconds. Ensure that there’s sufficient bandwidth between brokers to handle replication traffic. - Complexity: A benefit of Redpanda is the simplicity of deployment. Because Redpanda is deployed as a single binary with no external dependencies, it doesn’t need any infrastructure for ZooKeeper or for a Schema Registry. Redpanda also includes cluster balancing, so there’s no need to run Cruise Control. ### [](#single-az-infrastructure)Single-AZ infrastructure In a single-AZ deployment, ensure that brokers are spread across at least three failure domains. This generally means separate racks, under separate switches, ideally powered by separate electrical feeds or circuits. Also, ensure that there’s sufficient network bandwidth between brokers, particularly considering shared uplinks, which could be subject to high throughput intra-cluster replication traffic. In an on-premises network, this HA configuration refers to separate racks or data halls within a data center. Cloud providers support various HA configurations: - AWS [partition placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-partition) allow spreading hosts across multiple partitions (or failure domains) within an AZ. The default number of partitions is three, with a maximum of seven. This can be combined with Redpanda’s replication factor setting, so each topic partition replica is guaranteed to be isolated from the impact of hardware failure. - Microsoft Azure [flexible scale sets](https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-orchestration-modes#scale-sets-with-flexible-orchestration) let you assign VMs to specific fault domains. Each scale set can have up to five fault domains, depending on your region. Not all VM types support flexible orchestration; for example, Lsv2-series only supports uniform scale sets. - Google Cloud [instance placement policies](https://cloud.google.com/compute/docs/instances/define-instance-placement) let you specify how many availability domains you can have (up to eight) when using the Spread Instance Placement Policy. > 📝 **NOTE** > > Google Cloud doesn’t divulge which availability domain an instance has been placed into, so you must have an availability domain for each Redpanda broker. Essentially, this isn’t enabled with rack awareness, but it’s the only possibility for clusters with more than three brokers. You can automate this using Terraform or a similar infrastructure-as-code (IaC) tool. See [AWS](https://github.com/redpanda-data/deployment-automation/blob/main/aws/cluster.tf#L23-L24), [Azure](https://github.com/redpanda-data/deployment-automation/blob/main/azure/network.tf#L39-L50), and [GCP](https://github.com/redpanda-data/deployment-automation/blob/main/gcp/cluster.tf#L17-L19). ### [](#single-az-rack-awareness)Single-AZ rack awareness To make Redpanda aware of the topology it’s running on, configure the cluster to [enable rack awareness](https://docs.redpanda.com/streaming/current/manage/rack-awareness/), then configure each broker with the identifier of the rack. Set the `enable_rack_awareness custer` property either in `/etc/redpanda/.bootstrap.yaml` or with `rpk`: ```bash rpk cluster config set enable_rack_awareness true ``` For each broker, set the rack ID in `/etc/redpanda/redpanda.yaml` file or with `rpk`: ```bash rpk redpanda config set redpanda.rack ``` The modified Ansible playbooks take a per-instance rack variable from the Terraform output and use that to set the relevant cluster and broker configuration. Redpanda deployment automation can provision public cloud infrastructure with discrete failure domains (`-var=ha=true`) and use the resulting inventory to provision rack-aware clusters using Ansible. See also: [Automated Deployment](https://docs.redpanda.com/streaming/current/deploy/redpanda/manual/production/production-deployment-automation/) ### [](#single-az-example)Single-AZ example The following example deploys an HA cluster into AWS, Azure, or GCP using Terraform and Ansible. 1. Install all prerequisites, including all Ansible requirements: ```bash ansible-galaxy install -r ansible/requirements.yml ``` 2. Initialize a private key, if you haven’t done so already: ```bash ssh-keygen -f ~/.ssh/id_rsa ``` 3. Clone the deployment-automation repository: ```bash git clone https://github.com/redpanda-data/deployment-automation ``` 4. Initialize Terraform for your cloud provider: ```bash cd deployment-automation/aws (or cd deployment-automation/azure, or cd deployment-automation/gcp) terraform init ``` 5. Deploy the infrastructure (this assumes you have cloud credentials available): ```bash terraform apply -var=ha=true ``` 6. Verify that the racks have been correctly specified in the `host.ini` file: ```bash cd .. cat hosts.ini ``` ```ini [redpanda] 35.166.210.85 ansible_user=ubuntu ansible_become=True private_ip=172.31.7.173 rack=1 18.237.173.220 ansible_user=ubuntu ansible_become=True private_ip=172.31.2.138 rack=2 54.218.103.91 ansible_user=ubuntu ansible_become=True private_ip=172.31.2.93 rack=3 ``` 7. Provision the cluster with Ansible: ```bash ansible-playbook --private-key `cat ~/.ssh/id_rsa.pub | awk '{print $2}'` ansible/playbooks/provision-node.yml -i hosts.ini ``` 8. Verify that rack awareness is enabled: 1. Get connection details for the first Redpanda broker from the `hosts.ini` file: ```bash grep -A1 '\[redpanda]' hosts.ini ``` Example output: 35.166.210.85 ansible\_user=ubuntu ansible\_become=True private\_ip=172.31.7.173 rack=1 2. SSH into a cluster host with the username and hostname of that Redpanda broker: ```bash ssh -i ~/.ssh/id_rsa @ ``` 3. Verify that rack awareness is enabled: ```bash rpk cluster config get enable_rack_awareness ``` Example output: true 4. Check the rack assigned to this specific broker: ```bash rpk cluster status ``` Expected output: ```none CLUSTER = = = = redpanda.807d59af-e033-466a-98c3-bb0be15c255d BROKERS = = = = ID HOST PORT RACK 0* 10.0.1.7 9092 1 1 10.0.1.4 9092 2 2 10.0.1.8 9092 3 ``` ## [](#multi-az-deployments)Multi-AZ deployments In a multi-AZ (availability zone) deployment a single Redpanda cluster has brokers distributed over multiple availability zones. With rack awareness, Redpanda places replicas across brokers in different failure zones, resulting in a cluster that can survive a zone outage. > 📝 **NOTE** > > Adding a zone does not necessarily increase availability. The replication factor of a given partition is most important. If all of your partitions use a replication factor of three, then adding an additional broker in a fourth zone just means fewer partitions are affected by an outage (since the workload is more spread out). The primary reason to deploy across multiple availability zones is to achieve extremely high availability, even at the expense of other considerations. Before choosing this approach, carefully consider your system’s requirements. Some of the considerations of a multi-AZ approach include: - Cost: Maintaining presence across multiple availability zones may incur additional costs. You may require additional brokers to hit the minimum requirements for utilizing a multi-AZ deployment. Data sent between availability zones is often chargeable, resulting in additional cloud costs. - Performance: A multi-AZ approach introduces additional message latency. Your brokers are further apart in terms of network distance with additional routing hops in place. - Complexity: The Redpanda operational complexity is not appreciably increased, but the complexity of your overall cloud solution is. Maintaining presence across availability zones requires additional servers with corresponding maintenance, access control, and standard operational considerations. ### [](#multi-az-infrastructure-requirements)Multi-AZ infrastructure requirements Redpanda requires a minimum of three availability zones when using a multi-AZ approach. Deploying across only two availability zones is problematic. For example, given a cluster with three brokers spread across two availability zones, you either end up with all three brokers in one zone or a pair of brokers in one with a single broker in the other. Either way, it’s possible to lose a majority of your brokers with a single availability zone outage. You lose the ability to form consensus in affected partitions, negating the high availability state you desire. ### [](#multi-az-optimization)Multi-AZ optimization You can configure [follower fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/) to help ease the cross-AZ cost problems associated with a multi-AZ configuration. This is achieved by configuring consumers to advertise their preferred rack using the `client.rack` option within their consumer configuration. This allows consumers to read data from their closest replica rather than always reading from a (potentially non-local) partition leader. > 📝 **NOTE** > > With follower fetching enabled, a consumer chooses the closest replica rather than the leader. This reduces network transfer costs against the possibility of increased end-to-end latency. Make sure to monitor your system to determine if the cost savings are worth this latency risk. ### [](#multi-az-example)Multi-AZ example Redpanda provides an official [deployment automation](https://github.com/redpanda-data/deployment-automation) project using Ansible and Terraform to help self-managed users stand up multi-AZ deployments quickly and efficiently. #### [](#configure-terraform)Configure Terraform Configure the appropriate Terraform script for your cloud provider. Within the deployment-automation project, locate the file for your cloud provider and edit the `availability_zones` parameter. Include each availability zone you intend to use for your deployment. For example, under AWS, edit the `aws/main.tf` file: ```bash variable "availability_zone" { description = "The AWS AZ to deploy the infrastructure on" default = ["us-west-2a", "us-west-2b", "us-west-2c"] type = list(string) } ``` Alternatively, you can supply the configuration at the command line: ```bash $ terraform apply -var=availability_zone='["us-west-2a","us-west-2b","us-west-2c"]' ``` #### [](#deploy-using-terraform-and-ansible)Deploy using Terraform and Ansible After you configure Terraform for your cloud provider and choose availability zones, you can deploy your cluster. The following example deploys a multi-AZ cluster and validates the rack configuration. ```bash # Initialize a private key if you haven’t done so already ssh-keygen -f ~/.ssh/id_rsa # Clone the deployment-automation repository git clone https://github.com/redpanda-data/deployment-automation # Choose your cloud provider and initialize Terraform cd deployment-automation/aws # choose one: aws|azure|gcp terraform init # Deploy the infrastructure # (Note: This guidance is based on the assumption that you have cloud credentials available) terraform apply -var=availability_zone='["us-west-2a","us-west-2b","us-west-2c"]' # Verify you have correctly specified your racks in the host.ini file: cd .. export HOSTS=$(find . -name hosts.ini) head -4 $HOSTS [redpanda] 34.102.108.41 ansible_user=adminpanda ansible_become=True private_ip=10.168.0.41 rack=us-west2-a 35.236.32.47 ansible_user=adminpanda ansible_become=True private_ip=10.168.0.39 rack=us-west2-b 35.236.29.38 ansible_user=adminpanda ansible_become=True private_ip=10.168.0.40 rack=us-west2-c # Ensure the environment is ready export CLOUD_PROVIDER=aws # or azure or gcp accordingly export ANSIBLE_COLLECTIONS_PATH=${PWD}/artifacts/collections export ANSIBLE_ROLES_PATH=${PWD}/artifacts/roles export ANSIBLE_INVENTORY=${PWD}/${CLOUD_PROVIDER}/hosts.ini # Install Ansible Galaxy roles ansible-galaxy install -r ./requirements.yml # Provision the cluster with Ansible ansible-playbook ansible/provision-basic-cluster.yml -i $HOSTS ### Verify that rack awareness is enabled # SSH into a cluster node substituting the username and hostname from the values above ssh -i ~/.ssh/id_rsa @ # Check to confirm that rack awareness is enabled rpk cluster config get enable_rack_awareness true # Check to confirm that the brokers are assigned to distinct racks rpk cluster status | grep RACK -A3 ID HOST PORT RACK 0* 34.102.108.41 9092 us-west2-a 1 35.236.32.47 9092 us-west2-b 2 35.236.29.38 9092 us-west2-c ``` #### [](#use-follower-fetching)Use follower fetching Use [follower fetching](https://docs.redpanda.com/streaming/current/develop/consume-data/follower-fetching/) to reduce the latency and potential costs involved in a multi-AZ deployment. ```bash # SSH into a node using appropriate credentials ssh -i ~/.ssh/id_rsa @ # Create a topic with 1 partition and 3 replicas rpk topic create foo -p1 -r3 TOPIC STATUS foo OK # Determine which broker is the leader rpk topic describe foo -a | grep HIGH-WATERMARK -A1 PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK 0 0 1 [0 1 2] 0 3 # Produce 1000 records using rpk for i in {1..1000}; do echo $(cat /dev/urandom | head -c50 | base64); done | rpk topic produce foo Produced to partition 0 at offset 0 with timestamp 1687508554559. Produced to partition 0 at offset 1 with timestamp 1687508554574. Produced to partition 0 at offset 2 with timestamp 1687508554593. ... 997 more lines ... # Consume for three seconds, writing debug logs and ignoring regular output timeout 3 rpk topic consume foo -v --rack us-west2-c 1>/dev/null 2>debug.log # Filter the debug log to only show lines of interest cat debug.log | grep -v ApiVersions | egrep 'opening|read' 08:25:14.974 DEBUG opening connection to broker {"addr": "10.168.0.41:9092", "broker": "seed 0"} 08:25:14.976 DEBUG read Metadata v7 {"broker": "seed 0", "bytes_read": 236, "read_wait": "36.312µs", "time_to_read": "534.898µs", "err": null} 08:25:14.977 DEBUG opening connection to broker {"addr": "34.102.108.41:9092", "broker": "0"} 08:25:14.980 DEBUG read ListOffsets v4 {"broker": "0", "bytes_read": 51, "read_wait": "16.19µs", "time_to_read": "1.090468ms", "err": null} 08:25:14.981 DEBUG opening connection to broker {"addr": "34.102.108.41:9092", "broker": "0"} 08:25:14.982 DEBUG read Fetch v11 {"broker": "0", "bytes_read": 73, "read_wait": "17.705µs", "time_to_read": "858.613µs", "err": null} 08:25:14.982 DEBUG opening connection to broker {"addr": "35.236.29.38:9092", "broker": "2"} 08:25:14.989 DEBUG read Fetch v11 {"broker": "2", "bytes_read": 130337, "read_wait": "54.712µs", "time_to_read": "4.466249ms", "err": null} 08:25:17.946 DEBUG read Fetch v11 {"broker": "2", "bytes_read": 0, "read_wait": "41.144µs", "time_to_read": "2.955927224s", "err": "context canceled"} 08:25:17.947 DEBUG read Fetch v11 {"broker": "0", "bytes_read": 22, "read_wait": "175.952µs", "time_to_read": "500.832µs", "err": null} ``` ## [](#suggested-reading)Suggested reading - [Redpanda’s official Jepsen report](https://redpanda.com/blog/redpanda-official-jepsen-report-and-analysis?utm_assettype=report&utm_assetname=roi_report&utm_source=gated_content&utm_medium=content&utm_campaign=jepsen_blog) - [Simplifying Redpanda Raft implementation](https://redpanda.com/blog/simplifying-raft-replication-in-redpanda) - [An availability footprint of the Redpanda and Apache Kafka replication protocols](https://redpanda.com/blog/kafka-redpanda-availability) - [How we built Tiered Storage to supercharge storage and data streaming](https://www.redpanda.com/blog/tiered-storage-architecture-deep-dive) ## Suggested labs - [Migrate from Confluent with Shadowing](https://docs.redpanda.com/labs/docker-compose/confluent-schema-registry-shadowing/) - [Disaster Recovery with Envoy and Shadowing](https://docs.redpanda.com/labs/docker-compose/envoy-shadowing/) - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Stream Jira Issues to Redpanda for Real-Time Metrics](https://docs.redpanda.com/labs/docker-compose/jira-metrics-pipeline/) - [Migrate Data with Redpanda Migrator](https://docs.redpanda.com/labs/docker-compose/redpanda-migrator/) - [Start a Single Redpanda Broker with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/single-broker/) - [Start a Cluster of Redpanda Brokers with Redpanda Console in Docker](https://docs.redpanda.com/labs/docker-compose/three-brokers/) - [Set Up GitOps for the Redpanda Helm Chart](https://docs.redpanda.com/labs/kubernetes/gitops-helm/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) - [Set Up MySQL CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-mysql-json/) - [Set Up Postgres CDC with Debezium and Redpanda](https://docs.redpanda.com/labs/docker-compose/cdc-postgres-json/) See more [Search all labs](https://docs.redpanda.com/labs) --- # Page 150: Integrate Redpanda with Iceberg **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg.md --- # Integrate Redpanda with Iceberg > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Integrate Redpanda with Iceberg latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/index.adoc description: Generate Iceberg tables for your Redpanda topics for data lakehouse access. page-git-created-date: "2025-02-07" page-git-modified-date: "2025-02-07" support-status: supported --- - [About Iceberg Topics](about-iceberg-topics/) Learn how Redpanda can integrate topics with Apache Iceberg. - [Migrate to Iceberg Topics](migrate-to-iceberg-topics/) Migrate existing Iceberg integrations to Redpanda Iceberg topics. - [Specify Iceberg Schema](specify-iceberg-schema/) Learn about supported Iceberg modes and how you can integrate schemas with Iceberg topics. - [Use Iceberg Catalogs](use-iceberg-catalogs/) Learn how to access Redpanda topic data stored in Iceberg tables, using table metadata or a catalog integration. - [Integrate with REST Catalogs](rest-catalog/) Integrate Redpanda topics with managed Iceberg REST Catalogs. - [Query Iceberg Topics](query-iceberg-topics/) Query Redpanda topic data stored in Iceberg tables, based on the topic Iceberg mode and schema. - [Migrate Iceberg Catalogs](migrate-iceberg-catalog/) Switch the Iceberg catalog backend for an existing Redpanda cluster without losing untranslated topic data. - [Tune Performance for Iceberg Topics](iceberg-performance-tuning/) Optimize query performance and translation throughput for Iceberg topics with partitioning, compaction, flush threshold tuning, and cluster sizing guidance. - [Troubleshoot Iceberg Topics](iceberg-troubleshooting/) Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. --- # Page 151: About Iceberg Topics **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics.md --- # About Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: About Iceberg Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/about-iceberg-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/about-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/about-iceberg-topics.adoc description: Learn how Redpanda can integrate topics with Apache Iceberg. page-git-created-date: "2025-04-08" page-git-modified-date: "2026-07-28" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. The Apache Iceberg integration for Redpanda allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems, including data warehouses like Snowflake, Databricks, ClickHouse, and Redshift, without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, such as Apache Spark and Flink, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines. Redpanda supports [version 2](https://iceberg.apache.org/spec/#format-versioning) of the Iceberg table format. ## [](#iceberg-concepts)Iceberg concepts [Apache Iceberg](https://iceberg.apache.org) is an open source format specification for defining structured tables in a data lake. The table format lets you quickly and easily manage, query, and process huge amounts of structured and unstructured data. This is similar to the way you would manage and run SQL queries against relational data in a database or data warehouse. The open format lets you use many different languages, tools, and applications to process the same data in a consistent way, so you can avoid vendor lock-in. This data management system is also known as a _data lakehouse_. In the Iceberg specification, tables consist of the following layers: - **Data layer**: Stores the data in data files. The Iceberg integration currently supports the Parquet file format. Parquet files are column-based and suitable for analytical workloads at scale. They come with compression capabilities that optimize files for object storage. - **Metadata layer**: Stores table metadata separately from data files. The metadata layer allows multiple writers to stage metadata changes and apply updates atomically. It also supports database snapshots, and time travel queries that query the database at a previous point in time. - Manifest files: Track data files and contain metadata about these files, such as record count, partition membership, and file paths. - Manifest list: Tracks all the manifest files belonging to a table, including file paths and upper and lower bounds for partition fields. - Metadata file: Stores metadata about the table, including its schema, partition information, and snapshots. Whenever a change is made to the table, a new metadata file is created and becomes the latest version of the metadata in the catalog. For Iceberg-enabled topics, the manifest files are in JSON format. - **Catalog**: Contains the current metadata pointer for the table. Clients reading and writing data to the table see the same version of the current state of the table. The Iceberg integration supports two [catalog integration](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/) types. You can configure Redpanda to catalog files stored in the same object storage bucket or container where the Iceberg data files are located, or you can configure Redpanda to use an [Iceberg REST catalog](https://iceberg.apache.org/terms/#decoupling-using-the-rest-catalog) endpoint to update an externally-managed catalog when there are changes to the Iceberg data and metadata. ![Redpanda’s Iceberg integration](https://docs.redpanda.com/streaming/current/shared/_images/iceberg-integration-optimized.png) When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers store streaming data in the Iceberg-compatible format in Parquet files in object storage, in addition to the log segments uploaded using Tiered Storage. Storing the streaming data in Iceberg tables in the cloud allows you to derive real-time insights through many compatible data lakehouse, data engineering, and business intelligence [tools](https://iceberg.apache.org/vendors/). ## [](#prerequisites)Prerequisites To enable Iceberg for Redpanda topics, you must have the following: - **rpk**: See [Install or Update rpk](https://docs.redpanda.com/streaming/current/get-started/rpk-install/). - **Enterprise license**: To check if you already have a license key applied to your cluster: ```bash rpk cluster license info ``` - **Tiered Storage**: Enable [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#set-up-tiered-storage) for the topics for which you want to generate Iceberg tables. ## [](#limitations)Limitations - It is not possible to append topic data to an existing Iceberg table that is not created by Redpanda. - If you enable the Iceberg integration on an existing Redpanda topic, Redpanda does not backfill the generated Iceberg table with topic data. - JSON schemas are supported starting with Redpanda version 25.2. ## [](#enable-iceberg-integration)Enable Iceberg integration To create an Iceberg table for a Redpanda topic, you must set the cluster configuration property `[iceberg_enabled](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_enabled)` to `true`, and also configure the topic property [`redpanda.iceberg.mode`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-mode). You can choose to provide a schema if you need the Iceberg table to be structured with defined columns. 1. Set the `iceberg_enabled` configuration option on your cluster to `true`. When multiple clusters write to the same catalog, each cluster must use a distinct namespace to avoid table name collisions. This is especially critical for REST catalog providers that offer a single global catalog per account (such as AWS Glue), where there is no other isolation mechanism. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace for your cluster’s REST catalog integration, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. You cannot change this property after you enable Iceberg topics on the cluster. ```bash rpk cluster config set iceberg_enabled true # Optional: set a custom namespace (default is "redpanda") # rpk cluster config set iceberg_default_catalog_namespace '[""]' ``` ```bash Successfully updated configuration. New configuration version is 2. ``` You must restart your cluster if you change this configuration for a running cluster. 2. (Optional) Create a new topic. ```bash rpk topic create ``` ```bash TOPIC STATUS OK ``` 3. Configure `redpanda.iceberg.mode` for the topic. You can choose one of the following [Iceberg modes](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/): - `key_value`: Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. - `value_schema_id_prefix`: Creates an Iceberg table whose structure matches the Redpanda schema for this topic, with columns corresponding to each field. You must register a schema in the Schema Registry (see next step), and producers must write to the topic using the Schema Registry wire format. - `value_schema_latest`: Creates an Iceberg table whose structure matches the latest schema registered for the subject in the Schema Registry. - `disabled` (default): Disables writing to an Iceberg table for this topic. ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` ```bash TOPIC STATUS OK ``` 4. Register a schema for the topic. This step is required for the `value_schema_id_prefix` and `value_schema_latest` modes. ```bash rpk registry schema create --schema --type ``` ```bash SUBJECT VERSION ID TYPE 1 1 PROTOBUF ``` As you produce records to the topic, the data also becomes available in object storage for Iceberg-compatible clients to consume. You can use the same analytical tools to [read the Iceberg topic data](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) in a data lake as you would for a relational database. See also: [Schema types translation](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#schema-types-translation). ### [](#iceberg-data-retention)Iceberg data retention Data in an Iceberg-enabled topic is consumable from Kafka based on the configured [topic retention policy](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/disk-utilization/). Conversely, data written to Iceberg remains queryable as Iceberg tables indefinitely. The Iceberg table persists unless you: - Delete the Redpanda topic associated with the Iceberg table. This is the default behavior set by the `[iceberg_delete](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_delete)` cluster property and the `redpanda.iceberg.delete` topic property. If you set this property to `false`, the Iceberg table remains even after you delete the topic. - Explicitly delete data from the Iceberg table using a query engine. - Disable the Iceberg integration for the topic and delete the Parquet files in object storage. The DLQ table (`~dlq`) follows the same persistence rules as the main Iceberg table. ## [](#schema-evolution)Schema evolution Redpanda supports schema evolution in accordance with the [Iceberg specification](https://iceberg.apache.org/spec/#schema-evolution). Permitted schema evolutions include reordering fields and promoting field types. When you update the schema in Schema Registry, Redpanda automatically updates the Iceberg table schema to match the new schema. For example, if you produce records to a topic `demo-topic` with the following Avro schema: schema\_1.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" } ] } ``` ```bash rpk registry schema create demo-topic-value --schema schema_1.avsc echo '{"user_id":23, "event_type":"BUTTON_CLICK"}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Then, you update the schema to add a new field `ts`, and produce records with the updated schema: schema\_2.avsc ```avro { "type": "record", "name": "ClickEvent", "fields": [ { "name": "user_id", "type": "int" }, { "name": "event_type", "type": "string" }, { "name": "ts", "type": [ "null", { "type": "long", "logicalType": "timestamp-millis" } ], "default": null # Default value for the new field } ] } ``` The `ts` field can be either null or a long representing epoch milliseconds. The default value is null. ```bash rpk registry schema create demo-topic-value --schema schema_2.avsc echo '{"user_id":858, "event_type":"BUTTON_CLICK", "ts":1737998723230}' | rpk topic produce demo-topic --format='%v\n' --schema-id=topic ``` Querying the Iceberg table for `demo-topic` includes the new column `ts`: ```bash +---------+--------------+--------------------------+ | user_id | event_type | ts | +---------+--------------+--------------------------+ | 858 | BUTTON_CLICK | 2025-02-26T20:05:23.230Z | | 23 | BUTTON_CLICK | NULL | +---------+--------------+--------------------------+ ``` ## [](#monitor-iceberg-health)Monitor Iceberg health To test connectivity with your external Iceberg REST catalog and check the per-partition commit lag for each Iceberg topic, use the read-only `GetIcebergStatus` Admin API endpoint (Redpanda version 26.2 and later). Commit lag is the number of records that Redpanda has written to Parquet but not yet committed to the catalog. This endpoint requires superuser privileges if authentication is enabled on your cluster. See the [Admin API reference](https://docs.redpanda.com/api/doc/admin/v2/operation/operation-redpanda-core-admin-v2-icebergservice-geticebergstatus) for the full request and response schema. ## [](#next-steps)Next steps - [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/) - [Tune Performance for Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-performance-tuning/) - [Troubleshoot Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/) ## [](#suggested-reading)Suggested reading - [Server-Side Schema ID Validation](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-id-validation/) - [Understanding Apache Kafka Schema Registry](https://www.redpanda.com/blog/schema-registry-kafka-streaming#how-does-serialization-work-with-schema-registry-in-kafka) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 152: Tune Performance for Iceberg Topics **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-performance-tuning.md --- # Tune Performance for Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Tune Performance for Iceberg Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/iceberg-performance-tuning page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-performance-tuning.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-performance-tuning.adoc description: Optimize query performance and translation throughput for Iceberg topics with partitioning, compaction, flush threshold tuning, and cluster sizing guidance. page-topic-type: best-practices personas: ops_admin, streaming_developer page-git-created-date: "2026-05-06" page-git-modified-date: "2026-05-06" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide covers strategies for optimizing the performance of Iceberg topics in Redpanda, including improving downstream query performance, tuning the Iceberg translation pipeline, and monitoring translation throughput. After reading this page, you will be able to: - Apply partitioning and compaction strategies to improve query performance - Choose appropriate flush threshold and lag target values for your workload - Identify translation performance signals using Iceberg metrics ## [](#prerequisites)Prerequisites You must be familiar with how Iceberg topics work in Redpanda. See [About Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/). ## [](#optimize-query-performance)Optimize query performance Query engines read Parquet files from object storage to process Iceberg table data. Partitioning, compaction, and schema design affect how efficiently those reads perform. ### [](#use-custom-partitioning)Use custom partitioning To improve query performance, consider implementing custom [partitioning](https://iceberg.apache.org/docs/nightly/partitioning/) for the Iceberg topic. Use the [`redpanda.iceberg.partition.spec`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-partition-spec) topic property to define the partitioning scheme: ```bash # Create new topic with five topic partitions, replication factor 3, and custom table partitioning for Iceberg rpk topic create -p5 -r3 -c redpanda.iceberg.mode=value_schema_id_prefix -c "redpanda.iceberg.partition.spec=(, , ...)" ``` Valid `` values include a source column name or a transformation of a column. The columns referenced can be Redpanda-defined (such as `redpanda.timestamp`) or user-defined based on a schema that you register for the topic. The Iceberg table stores records that share different partition key values in separate files based on this specification. For example: - To partition the table by a single key, such as a column `col1`, use: `redpanda.iceberg.partition.spec=(col1)`. - To partition by multiple columns, use a comma-separated list: `redpanda.iceberg.partition.spec=(col1, col2)`. - To partition by the year of a timestamp column `ts1`, and a string column `col1`, use: `redpanda.iceberg.partition.spec=(year(ts1), col1)`. To learn more about how partitioning schemes can affect query performance, and for details on the partitioning specification such as allowed transforms, see the [Apache Iceberg documentation](https://iceberg.apache.org/spec/#partitioning). > 💡 **TIP** > > - Partition by columns that you frequently use in queries. Columns with relatively few unique values (low cardinality) are good candidates for partitioning. > > - If you must partition based on columns with high cardinality, for example timestamps, use Iceberg’s available transforms such as extracting the year, month, or day to avoid creating too many partitions. Too many partitions can be detrimental to performance because more files need to be scanned and managed. ### [](#compact-iceberg-tables)Compact Iceberg tables Over time, Iceberg translation can produce many small Parquet files, especially with low-throughput topics or short lag targets. Compaction merges small files into larger ones, reducing the number of metadata operations query engines must perform and improving read performance. - Automatic compaction: Some catalog and data platform services, such as AWS Glue and Databricks, automatically compact Iceberg tables. - Manual or scheduled compaction: Tools like [Apache Spark](https://spark.apache.org/) can run compaction jobs on a schedule. This is useful if your catalog or platform does not compact automatically. If you observe degraded read performance or a high number of small files, investigate whether your catalog or platform supports automatic compaction or schedule periodic compaction jobs. ### [](#avoid-high-column-count)Avoid high column count A high column count or schema field count results in more overhead when translating topics to the Iceberg table format. Small message sizes can also increase CPU utilization. To minimize the performance impact on your cluster, keep to a low column count and large message size for Iceberg topics. ## [](#tune-translation-performance)Tune translation performance Translation is the process in which Redpanda converts topic data into Parquet files for the Iceberg table. Each round of translation processes one topic partition at a time. Under typical conditions, Iceberg translation has the following performance characteristics: - Throughput: Approximately 5 MiB/s per core. - Flush threshold: Controlled by [`datalake_translator_flush_bytes`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#datalake_translator_flush_bytes) (default: 32 MiB). Each translation process uploads its on-disk data when accumulated data reaches this threshold. This is the primary control for Parquet file size. - Lag target: Controlled by [`iceberg_target_lag_ms`](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_target_lag_ms) (default: 1 minute). Redpanda tries to commit all data produced to an Iceberg-enabled topic within this window. The flush threshold and lag target together determine the size of the Parquet files written to object storage. Larger Parquet files generally improve downstream query performance by reducing the number of metadata operations query engines must perform. ### [](#tune-flush-threshold-and-lag-target)Tune flush threshold and lag target Increase both the flush threshold and the lag target together to produce larger Parquet files with more records per file. This is the primary way to tune Iceberg translation performance. 1. Increase `datalake_translator_flush_bytes` to control the size of Parquet files. A good starting value depends on your workload: ```bash rpk cluster config set datalake_translator_flush_bytes ``` For example, to set a 64 MiB flush threshold: ```bash rpk cluster config set datalake_translator_flush_bytes 67108864 ``` 2. Increase `iceberg_target_lag_ms` to give translators more time to accumulate data before committing: ```bash rpk cluster config set iceberg_target_lag_ms 300000 ``` You can also set the lag target per topic using the [`redpanda.iceberg.target.lag.ms`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-target-lag-ms) topic property. > 📝 **NOTE** > > Increasing the lag target means Iceberg tables receive new data less frequently. Choose a lag value that balances file efficiency against how current your downstream data must be. > 💡 **TIP** > > `datalake_translator_flush_bytes` and `iceberg_target_lag_ms` work best when tuned together. A high flush threshold combined with a short lag window may not improve file sizes if the lag window expires before enough data has accumulated. To check the current values of key translation properties: ```bash rpk cluster config get datalake_translator_flush_bytes rpk cluster config get iceberg_target_lag_ms ``` To check topic-level overrides: ```bash rpk topic describe -c ``` ### [](#optimize-message-size)Optimize message size Redpanda has validated 32 MiB as the maximum recommended message size for Iceberg-enabled topics. With large messages, each Parquet file contains fewer records because the flush threshold is reached sooner. This can reduce the efficiency of analytical queries that need to scan many records. If query latency is a concern and your workload produces large messages, consider: - Reducing individual message sizes if your data model allows it. - Increasing `datalake_translator_flush_bytes` and `iceberg_target_lag_ms` to produce Parquet files with more records per file. See [Tune flush threshold and lag target](#tune-flush-threshold-and-lag-target). ### [](#size-clusters-for-iceberg-workloads)Size clusters for Iceberg workloads When you enable Iceberg for any substantial workload and start translating topic data to the Iceberg format, you may see most of your cluster’s CPU utilization increase. If this additional workload overwhelms the brokers and causes the Iceberg table lag to exceed the configured target lag, Redpanda automatically increases the scheduling priority of Iceberg translation to help it catch up with incoming data. However, this does not substitute for adequate cluster resources. You may need to increase the size of your Redpanda cluster to accommodate the additional workload. To ensure that your cluster is sized appropriately, contact the Redpanda Customer Success team. ### [](#monitor-translation-performance)Monitor translation performance Use the following [Iceberg metrics](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#iceberg-metrics) to understand whether translation is keeping pace with incoming data: - [`redpanda_iceberg_translation_raw_bytes_processed`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_raw_bytes_processed): Total raw bytes consumed for translation input. Use this to monitor input throughput and compare against the expected 5 MiB/s per core baseline. - [`redpanda_iceberg_translation_parquet_bytes_added`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_parquet_bytes_added): Total bytes written to Parquet files. Divide by `redpanda_iceberg_translation_files_created` to estimate the average file size produced by your workload. - [`redpanda_iceberg_translation_files_created`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_files_created): Number of Parquet files created. A high file creation rate relative to bytes added indicates many small files. Consider increasing `datalake_translator_flush_bytes` and `iceberg_target_lag_ms`. - [`redpanda_iceberg_translation_parquet_rows_added`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_parquet_rows_added): Total rows written to Parquet files. Useful for understanding record-level throughput. - [`redpanda_iceberg_translation_translations_finished`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_translations_finished): Number of completed translator executions. A stalling or zero rate indicates translation has stopped. For metrics related to DLQ files, invalid records, and catalog commit failures, see [Troubleshooting metrics](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#troubleshooting-metrics). > 💡 **TIP** > > If translation consistently lags despite available CPU headroom, the workload may be partition-bound. Each core translates its assigned partitions independently, so distributing data across more partitions allows more cores to contribute to translation and can improve total throughput. ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 153: Query Iceberg Topics using AWS Glue **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-aws-glue.md --- # Query Iceberg Topics using AWS Glue > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Query Iceberg Topics using AWS Glue page-beta-text: This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/iceberg-topics-aws-glue page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-aws-glue.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc description: Add Redpanda topics as Iceberg tables that you can query from AWS Glue Data Catalog. # Beta release status page-beta: "true" page-git-created-date: "2025-07-30" page-git-modified-date: "2026-07-17" support-status: supported release-status: beta - This is a beta feature. Beta features are available for testing and feedback. They are not supported by Redpanda and should not be used in production environments. --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide walks you through querying Redpanda topics as Iceberg tables stored in AWS S3, using a catalog integration with [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/components-overview.html#data-catalog-intro). For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). ## [](#prerequisites)Prerequisites - An AWS account with access to [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html). - Redpanda version 25.1.7 or later. - [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) installed or updated to the latest version. - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You also use the S3 bucket URI to set the base location for AWS Glue Data Catalog. - Admin permissions to create IAM policies and roles in AWS. ## [](#limitations)Limitations ### [](#lowercase-field-names-required)Lowercase field names required Use only lowercase field names. AWS Glue converts all table column names to lowercase, and Redpanda requires exact column name matches to manage schemas. Using uppercase letters prevents Redpanda from finding matching columns, which breaks schema management. ### [](#nested-partition-spec-support)Nested partition spec support AWS Glue does not support partitioning on nested fields. If Redpanda detects that the default partitioning `(hour(redpanda.timestamp))` based on the record metadata is in use, it will instead apply an empty partition spec `()`, which means the table will not be partitioned. To use partitioning, you must implement custom partitioning using your own partition columns (that is, columns that are not nested). > 📝 **NOTE** > > In Redpanda versions 25.2.1 and earlier, an empty partition spec `()` can cause a known issue that prevents certain engines like Amazon Redshift from successfully querying the table. To resolve this issue, specify custom partitioning, or upgrade Redpanda to versions 25.2.2 or later. ### [](#manual-deletion-of-iceberg-tables)Manual deletion of Iceberg tables The AWS Glue catalog integration does not support automatic deletion of Iceberg tables from Redpanda. To manually delete Iceberg tables in AWS Glue, you must either: - Set the cluster property `[iceberg_delete](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_delete)` to `false` when you configure the catalog integration. - Override the cluster property `iceberg_delete` by setting the topic property [`redpanda.iceberg.delete`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-delete) to `false` for the topic you want to delete. When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic. ## [](#authorize-access-to-aws-glue)Authorize access to AWS Glue You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: - Root catalog (`catalog`) - All databases (`database/*`) - All tables (`table/*/*`) Your IAM policy should include a statement similar to the following: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:*" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::database/*", "arn:aws:glue:::table/*/*" ] } ] } ``` For more information on configuring IAM permissions, see the [AWS Glue documentation](https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html). ## [](#configure-authentication-and-credentials)Configure authentication and credentials You must configure credentials for the AWS Glue Data Catalog integration in either of the following ways: - Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach. - If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: - `[iceberg_rest_catalog_credentials_source](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_credentials_source)` overrides `[cloud_storage_credentials_source](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_credentials_source)` - `[iceberg_rest_catalog_aws_access_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_access_key)` overrides `[cloud_storage_access_key](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_access_key)` - `[iceberg_rest_catalog_aws_secret_key](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_secret_key)` overrides `[cloud_storage_secret_key](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_secret_key)` - `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` overrides `[cloud_storage_region](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_region)` ## [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with the AWS Glue Data Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. Because AWS Glue provides a single catalog per account, each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. To set a unique namespace, also set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` when you set `iceberg_enabled`. This property cannot be changed after Iceberg is enabled. Run `rpk cluster config edit` to update these properties: ```bash iceberg_enabled: true # Set a custom namespace instead of the default "redpanda" iceberg_default_catalog_namespace: [""] # Glue requires Redpanda Iceberg tables to be manually deleted iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 # Because Redpanda does not support the use of distinct buckets for Iceberg, # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to # use separate AWS credentials for the catalog, or omit these lines to reuse S3 # (cloud_storage_*) credentials. # For access using access keys only, use iceberg_rest_catalog_aws_access_key # and iceberg_rest_catalog_aws_secret_key. For access with an IAM role, use # iceberg_rest_catalog_credentials_source only. # iceberg_rest_catalog_aws_region: # iceberg_rest_catalog_aws_access_key: # iceberg_rest_catalog_aws_secret_key: # iceberg_rest_catalog_credentials_source: ``` Use your own values for the following placeholders: - ``: A unique namespace for this cluster’s Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used. - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your `[cloud_storage_region](https://docs.redpanda.com/streaming/current/reference/properties/object-storage-properties/#cloud_storage_region)` or `[iceberg_rest_catalog_aws_region](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_rest_catalog_aws_region)` property. - `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. `` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster. As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with your cluster’s data in object storage. ```bash Successfully updated configuration. New configuration version is 2. ``` 2. If you change the configuration for a running cluster, you must restart that cluster now. 3. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 4. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` You should see the topic as a table with data in AWS Glue Data Catalog. The data may take some time to become visible, depending on your `[iceberg_target_lag_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_target_lag_ms)` setting. 1. In AWS Glue Studio, go to Databases. 2. Select the `redpanda` database. The `redpanda` database and the table within are automatically added for you. The table name is the same as the topic name. ## [](#query-iceberg-table)Query Iceberg table You can query the Iceberg table using different engines, such as Amazon Athena, PyIceberg, or Apache Spark. To query the table or view the table data in AWS Glue, ensure that your account has the necessary permissions to access the catalog, database, and table. To query the table in Amazon Athena: 1. On the list of tables in AWS Glue Studio, click "Table data" under the **View data** column. 2. Click "Proceed" to be redirected to the Athena query editor. 3. In the query editor, select AwsDataCatalog as the data source, and select the `redpanda` database. If you set a custom namespace for your cluster, select that database instead of `redpanda`. 4. The SQL query editor should be pre-populated with a query that selects 10 rows from the Iceberg table. Run the query to see a preview of the table data. ```sql SELECT * FROM "AwsDataCatalog"."redpanda"."" limit 10; ``` Your query results should look like the following: ```sql +-----------------------------------------------------+----------------+ | redpanda | value | +-----------------------------------------------------+----------------+ | {partition=0, offset=0, timestamp=2025-07-21 | 77 6f 72 6c 64 | | 18:11:25.070000, headers=null, key=[B@1900af31} | | +-----------------------------------------------------+----------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and the AWS Glue Data Catalog. To grant your end users and query engines (such as Amazon Athena or Apache Spark) read access to the Iceberg tables, use [AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) to assign table-level and column-level permissions. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 154: Query Iceberg Topics using Databricks and Unity Catalog **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-databricks-unity.md --- # Query Iceberg Topics using Databricks and Unity Catalog > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Query Iceberg Topics using Databricks and Unity Catalog latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/iceberg-topics-databricks-unity page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-databricks-unity.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc description: Add Redpanda topics as Iceberg tables that you can query in Databricks managed by Unity Catalog. page-git-created-date: "2025-06-12" page-git-modified-date: "2026-08-13" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide walks you through querying Redpanda topics as managed Iceberg tables in Databricks, with AWS S3 as object storage and a catalog integration using [Unity Catalog](https://docs.databricks.com/aws/en/data-governance/unity-catalog). For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). After reading this page, you will be able to: - Configure a Unity Catalog integration for Redpanda Iceberg topics with AWS S3 - Query Redpanda topic data as Iceberg tables in Databricks SQL ## [](#prerequisites)Prerequisites - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You need the AWS S3 bucket URI, so you can configure it as an external location in Unity Catalog. - A Databricks workspace in the same region as your S3 bucket. See the [list of supported AWS regions](https://docs.databricks.com/aws/en/resources/supported-regions#supported-regions-list). - Unity Catalog enabled in your Databricks workspace. See the [Databricks documentation](https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started) to set up Unity Catalog for your workspace. - [Predictive optimization](https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-predictive-optimization) enabled for Unity Catalog. > 📝 **NOTE** > > When you enable predictive optimization, you must also set the following configurations in your Databricks workspace. These configurations allow predictive optimization to automatically generate column statistics and carry out background compaction for Iceberg tables: > > ```sql > SET spark.databricks.delta.liquid.lazyClustering.backfillStats=true; > SET spark.databricks.delta.computeStats.autoConflictResolution=true; > > /* > After setting these configurations, you can optionally run OPTIMIZE to > immediately trigger compaction and liquid clustering, or let predictive > optimization handle it automatically later. > */ > OPTIMIZE ``.redpanda.``; > ``` - [External data access](https://docs.databricks.com/aws/en/external-access/admin) enabled in your metastore. - Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster’s Tiered Storage bucket to Databricks. ## [](#limitations)Limitations The following data types are not currently supported for managed Iceberg tables: | Iceberg type | Equivalent Avro type | | --- | --- | | uuid | uuid | | fixed(L) | fixed | | time | time-millis, time-micros | There are no limitations for Protobuf types. ## [](#create-a-unity-catalog-storage-credential)Create a Unity Catalog storage credential A storage credential is a Databricks object that controls access to external object storage, in this case S3. You associate a storage credential with an AWS IAM role that defines what actions Unity Catalog can perform in the S3 bucket. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials) to create an AWS IAM role that has the required permissions for the bucket. When you have completed these steps, you should have the following configured in AWS and Databricks: - A self-assuming IAM role, meaning you’ve defined the role trust policy so the role trusts itself. - Two IAM policies attached to the IAM role. The first policy grants Unity Catalog read and write access to the bucket. The second policy allows Unity Catalog to configure file events. - A storage credential in Databricks associated with the IAM role, using the role’s ARN. You also use the storage credential’s external ID in the role’s trust relationship policy to make the role self-assuming. ## [](#create-a-unity-catalog-external-location)Create a Unity Catalog external location The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for Tiered Storage for your Redpanda cluster. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations) to **manually** create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing Tiered Storage bucket URL, `s3://`. ## [](#choose-a-catalog-setup)Choose a catalog setup You can either create a new catalog dedicated to Redpanda topics or use an existing catalog. If you create a new catalog, Redpanda automatically creates the required schema for you. If you need to integrate with an existing catalog, you must manually create the schema in that catalog before Redpanda creates any Iceberg tables. After you set up your catalog, the authorization and Redpanda configuration steps are the same for both options. ### [](#option-1-create-a-new-catalog-recommended)Option 1: Create a new catalog (recommended) Follow the steps in the Databricks documentation to [create a standard catalog](https://docs.databricks.com/aws/en/catalogs/create-catalog). When you create the catalog, specify the external location you created in the previous step as the storage location. In this setup, Redpanda creates the default `redpanda` schema for you. You use the catalog name when you set the Iceberg cluster configuration properties in Redpanda in a later step. ### [](#option-2-use-an-existing-catalog-with-a-pre-created-schema)Option 2: Use an existing catalog with a pre-created schema If you need to integrate Redpanda with an existing Unity Catalog catalog object, follow the steps to [create a schema](https://docs.databricks.com/aws/en/schemas/create-schema) in the catalog. - By default, Redpanda creates tables in a schema named `redpanda`. If you want to use a different schema, set `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)` before enabling Iceberg, then manually create that schema in the catalog. - Set the schema’s managed storage location to the same S3 bucket used for Tiered Storage, using the external location you created in the previous step. Unity Catalog resolves managed storage locations through a hierarchy of metastore > catalog > schema. If you assign the schema its own managed storage location, Redpanda can use the existing catalog while the schema stores its managed Iceberg data in the schema-specific location. For example: - Your existing Unity Catalog catalog stores managed data in `s3://`. - You manually create a `redpanda` schema in that catalog and override its managed storage location, through the external location, to `s3://`, which matches the S3 bucket that Redpanda uses for Tiered Storage. For more information, see the [Unity Catalog managed storage location hierarchy](https://docs.databricks.com/aws/en/data-governance/unity-catalog/#managed-storage-location-hierarchy) in the Databricks documentation. ## [](#authorize-access-to-unity-catalog)Authorize access to Unity Catalog Redpanda recommends using OAuth for service principals to grant Redpanda access to Unity Catalog. 1. Follow the steps in the [Databricks documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) to create a service principal, and then generate an OAuth secret. You use the client ID and secret to set Iceberg cluster configuration properties in Redpanda in the next step. 2. Open your catalog in the Catalog Explorer, then click **Permissions**. 3. Click **Grant** to grant the service principal the following permissions on the catalog: - `ALL PRIVILEGES` - `EXTERNAL USE SCHEMA` The Iceberg integration for Redpanda also supports using bearer tokens. ## [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Unity Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Run `rpk cluster config edit` to update these properties: ```bash iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https:///api/2.1/unity-catalog/iceberg-rest iceberg_rest_catalog_authentication_mode: oauth2 iceberg_rest_catalog_oauth2_server_uri: https:///oidc/v1/token iceberg_rest_catalog_oauth2_scope: all-apis iceberg_rest_catalog_client_id: iceberg_rest_catalog_client_secret: iceberg_rest_catalog_warehouse: iceberg_disable_snapshot_tagging: true # Optional. Set a custom namespace only if you want to use a schema other than the default `redpanda` # iceberg_default_catalog_namespace: [""] ``` Use your own values for the following placeholders: - ``: The URL of your [Databricks workspace instance](https://docs.databricks.com/aws/en/workspace/workspace-details#workspace-instance-names-urls-and-ids); for example, `cust-success.cloud.databricks.com`. - ``: The client ID of the service principal you created in an earlier step. - ``: The client secret of the service principal you created in an earlier step. - ``: The name of your catalog in Unity Catalog. ```bash Successfully updated configuration. New configuration version is 2. ``` 2. You must restart your cluster if you change the configuration for a running cluster. 3. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates an Iceberg table for the topic consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 4. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` You should see the topic as a table with data in Unity Catalog. The data may take some time to become visible, depending on your `[iceberg_target_lag_ms](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_target_lag_ms)` setting. 1. In Catalog Explorer, open your catalog. You should see a `redpanda` schema (or the namespace you configured with `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`), in addition to `default` and `information_schema`. 2. The schema and the table residing within it are automatically added for you. The table name is the same as the topic name. ## [](#query-iceberg-table-using-databricks-sql)Query Iceberg table using Databricks SQL You can query the Iceberg table using different engines, such as Databricks SQL, PyIceberg, or Apache Spark. To query the table or view the table data in Catalog Explorer, ensure that your account has the necessary permissions to read the table. The following example shows how to query the Iceberg table using SQL in Databricks SQL. 1. In the Databricks console, open **SQL Editor**. 2. In the query editor, run: ```sql /* Ensure that the catalog and table name are correctly parsed in case they contain special characters. If you set iceberg_default_catalog_namespace to a custom namespace, replace `redpanda` with that namespace in the query below. */ SELECT * FROM ``.redpanda.`` LIMIT 10; ``` Your query results should look like the following: ```sql -- Example for redpanda.iceberg.mode=key_value with 1 record produced to topic +----------------------------------------------------------------------+------------+ | redpanda | value | +----------------------------------------------------------------------+------------+ | {"partition":0,"offset":"0","timestamp":"2025-04-02T18:57:11.127Z", | 776f726c64 | | "headers":null,"key":"68656c6c6f"} | | +----------------------------------------------------------------------+------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and Unity Catalog. To grant your end users or query engines read access to the Iceberg tables, use Unity Catalog to assign the appropriate privileges. Review the Databricks documentation on [granting permissions to objects](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/?language=SQL#grant-permissions-on-objects-in-a-unity-catalog-metastore) and [Unity Catalog privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges) for details. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 155: Use Iceberg Topics with GCP Lakehouse **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-gcp-biglake.md --- # Use Iceberg Topics with GCP Lakehouse > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Use Iceberg Topics with GCP Lakehouse latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/iceberg-topics-gcp-biglake page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-topics-gcp-biglake.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-topics-gcp-biglake.adoc description: Add Redpanda topics as Iceberg tables to Google Lakehouse for Apache Iceberg that you can query from Google BigQuery. page-git-created-date: "2025-11-19" page-git-modified-date: "2026-07-17" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. > 💡 **TIP** > > This guide is for integrating Iceberg topics with a managed REST catalog. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. > > The blog post uses a Redpanda Cloud cluster, but you follow the same steps for a Self-Managed cluster. This guide walks you through querying Redpanda topics as Iceberg tables stored in Google Cloud Storage, using a REST catalog integration with [Google Lakehouse for Apache Iceberg](https://docs.cloud.google.com/lakehouse/docs/introduction) (formerly BigLake). After completing this guide, you will be able to: - Create a catalog in GCP Lakehouse for Iceberg topic data. - Configure a Redpanda cluster to use GCP Lakehouse as an Iceberg REST catalog. - Query Iceberg topic data from Google BigQuery. This guide also includes optional steps to deploy a Redpanda quickstart cluster on a GCP VM instance using Docker Compose, which you can use to quickly test the Lakehouse Iceberg integration. For general information about Iceberg catalog integrations in Redpanda, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). > 📝 **NOTE** > > Check the [Lakehouse product page](https://docs.cloud.google.com/lakehouse/docs) for the latest status and availability of the REST Catalog API. ## [](#prerequisites)Prerequisites - A Google Cloud Platform (GCP) project. If you do not have permissions to manage GCP resources such as VMs, storage buckets, and service accounts in your project, ask your project owner to create or update them for you. - The [`gcloud` CLI](https://docs.cloud.google.com/sdk/docs/install) installed and configured for your GCP project. - [Lakehouse (BigLake) API](https://cloud.google.com/biglake/docs/enable-biglake-api) enabled for your GCP project. - Redpanda version 25.3 or later. Your Redpanda cluster must be deployed on GCP VMs. - `rpk` [installed or updated](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to the latest version. - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You also use the GCS bucket URI to set the warehouse location for the Lakehouse catalog. ## [](#limitations)Limitations ### [](#multi-region-bucket-support)Multi-region bucket support The Lakehouse runtime catalog does not support multi-region buckets. Use single-region buckets to store your Iceberg topics. ### [](#catalog-deletion)Catalog deletion Currently, it is not possible to delete non-empty Lakehouse Iceberg catalogs through the Lakehouse interface. If you need to reconfigure your setup, create a new bucket or use the REST API to remove the existing catalog. ### [](#topic-names)Topic names Lakehouse does not support Iceberg table names that contain dots (`.`). When creating Iceberg topics in Redpanda that you plan to access through Lakehouse, either: - Use the `iceberg_topic_name_dot_replacement` cluster property to set a replacement string for dots in topic names. Ensure that the replacement value does not cause table name collisions. For example, `current.orders` and `current_orders` would both map to the same table name if you set the replacement to an underscore (`_`). - Ensure that the new topic names do not include dots. You must also set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). See [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) for the list of cluster properties to set when enabling the Lakehouse REST catalog integration. ## [](#set-up-google-cloud-resources)Set up Google Cloud resources ### [](#create-a-service-account-for-redpanda)Create a service account for Redpanda If you don’t already have a Google Cloud service account to use, create a new service account that will be used by the VMs running Redpanda. Redpanda uses this account for writing data to Tiered Storage, Iceberg data and metadata, and for interacting with the Lakehouse catalog: ```bash gcloud iam service-accounts create --display-name "" ``` Replace the placeholder values: - ``: You can use a [name](https://docs.cloud.google.com/iam/docs/service-accounts-create) that contains lowercase alphanumeric characters and dashes. - ``: Enter a display name for the service account. ### [](#grant-required-permissions)Grant required permissions Grant the necessary permissions to your service account. To run the following commands, replace the placeholder values: - ``: The name of your service account. - ``: The name of your storage bucket. 1. Grant the service account the [Storage Object Admin role](https://docs.cloud.google.com/storage/docs/access-control/iam-roles) to access the bucket: ```bash gcloud storage buckets add-iam-policy-binding gs:// \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/storage.objectAdmin" ``` 2. Grant [Service Usage Consumer](https://docs.cloud.google.com/iam/docs/roles-permissions/serviceusage) and [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake#biglake.editor) roles for using the Iceberg REST catalog: ```bash gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/serviceusage.serviceUsageConsumer" gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:@$(gcloud config get-value project).iam.gserviceaccount.com" \ --role="roles/biglake.editor" ``` ### [](#create-a-lakehouse-catalog)Create a Lakehouse catalog Create a Lakehouse Iceberg REST catalog using the [`gcloud biglake`](https://docs.cloud.google.com/sdk/gcloud/reference/biglake/iceberg/catalogs/create) command: ```bash gcloud biglake iceberg catalogs create --catalog-type=gcs-bucket --project= ``` Replace the placeholder values: - ``: Use the name of your storage bucket as the catalog ID. - ``: Your GCP project ID. ## [](#optional-deploy-redpanda-quickstart-on-gcp)Optional: Deploy Redpanda quickstart on GCP If you want to quickly test Iceberg topics in Lakehouse, you can deploy a test environment using the Redpanda Streaming quickstart. You create a new storage bucket for Tiered Storage and Iceberg data, configure a Redpanda cluster for the Lakehouse catalog integration, and deploy the cluster on a GCP Linux VM instance using Docker Compose. > 📝 **NOTE** > > If you already have a Redpanda cluster deployed on GCP, skip to [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg). ### [](#create-a-storage-bucket)Create a storage bucket Create a new Google Cloud Storage bucket to store Iceberg data: ```bash gcloud storage buckets create gs:// --location= ``` Replace the placeholder values: - ``: A globally unique name for your bucket. - ``: The region where you want to create the bucket, for example, `europe-west2`. > 📝 **NOTE** > > Ensure that the service account you created earlier has the [required permissions](#grant-required-permissions) to access this bucket. ### [](#create-vm-instances)Create VM instances Create a VM instance to run Redpanda: ```bash gcloud compute instances create \ --zone= \ --machine-type=e2-medium \ --service-account=@$(gcloud config get-value project).iam.gserviceaccount.com \ --scopes=https://www.googleapis.com/auth/cloud-platform \ --create-disk=auto-delete=yes,boot=yes,device-name=,image=projects/debian-cloud/global/images/debian-12-bookworm-v20251014,mode=rw,size=20,type=pd-standard ``` Replace the placeholder values: - ``: A name for your VM instance. - ``: The name of the service account you created earlier. - ``: The fully-qualified zone name, for example, `europe-west2-a`. ### [](#install-and-configure-redpanda)Install and configure Redpanda 1. Connect to your VM instance. It may take a few moments for the instance to be ready to accept SSH connections: ```bash gcloud compute ssh --zone ``` 2. Install Docker and Docker Compose following the [Docker installation guide](https://docs.docker.com/engine/install/debian/) for Debian. ```bash # Add Docker's official GPG key: sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc # Add the repository to Apt sources: echo \ "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update # Install Docker Engine, CLI, and Compose sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin ``` 3. Download the Redpanda Streaming quickstart files: ```bash mkdir redpanda-quickstart && cd redpanda-quickstart && \ (1) curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \ (2) cd docker-compose (3) ``` | 1 | Create and navigate to the redpanda-quickstart directory. | | --- | --- | | 2 | Download and extract the archive. | | 3 | Navigate to the Docker Compose configuration directory. | 4. Edit the `bootstrap.yaml` file to enable Tiered Storage and Iceberg features. Add or modify these sections: ```yaml # Enable Tiered Storage cloud_storage_enabled: true cloud_storage_region: n/a # GCP does not require region to be set cloud_storage_api_endpoint: storage.googleapis.com cloud_storage_api_endpoint_port: 443 cloud_storage_disable_tls: false cloud_storage_bucket: cloud_storage_credentials_source: gcp_instance_metadata # Configure Iceberg REST catalog integration with Lakehouse iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). > 📝 **NOTE** > > If you edit `bootstrap.yml`, you can skip the cluster configuration step in [Configure Redpanda for Iceberg](#configure-redpanda-for-iceberg) and proceed to the next step in that section to enable Iceberg for a topic. 5. Start Redpanda: ```bash docker compose up -d ``` 6. Install and configure `rpk`: ```bash sudo apt-get install unzip curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ rpk profile create quickstart --from-profile rpk-profile.yaml ``` ## [](#configure-redpanda-for-iceberg)Configure Redpanda for Iceberg 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. Run `rpk cluster config edit` to update these properties: ```yaml iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://biglake.googleapis.com/iceberg/v1/restcatalog iceberg_rest_catalog_oauth2_server_uri: https://oauth2.googleapis.com/token iceberg_rest_catalog_authentication_mode: gcp iceberg_rest_catalog_warehouse: gs:/// iceberg_rest_catalog_gcp_user_project: iceberg_dlq_table_suffix: _dlq ``` - Replace `` with your bucket name and `` with your Google Cloud project ID. - You must set the `iceberg_dlq_table_suffix` property to a value that does not include dots or tildes (`~`). The example above uses `_dlq` as the suffix for the [dead-letter queue (DLQ) table](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). 2. If you change the configuration for a running cluster, you must restart that cluster now. 3. Enable the REST catalog integration for a topic by configuring the topic property `redpanda.iceberg.mode`. The following examples show how to use [`rpk`](https://docs.redpanda.com/streaming/current/get-started/rpk-install/) to either create a new topic or alter the configuration for an existing topic and set the Iceberg mode to `key_value`. The `key_value` mode creates a two-column Iceberg table for the topic, with one column for the record metadata including the key, and another binary column for the record’s value. See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) for more details on Iceberg modes. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` > 📝 **NOTE** > > If you’re using the Self-managed quickstart for testing, your Redpanda cluster includes a `transactions` topic with data in it, and a sample schema in the Schema Registry. To enable Iceberg for the `transactions` topic, run: > > ```bash > rpk topic alter-config transactions --set redpanda.iceberg.mode=value_schema_latest:subject=transactions > ``` Iceberg data can take a few moments to become available in Lakehouse. ## [](#query-iceberg-topics-in-bigquery)Query Iceberg topics in BigQuery 1. Navigate to the [BigQuery console](https://console.cloud.google.com/bigquery). 2. Query your Iceberg topic using SQL. For example, to query the `transactions` topic in the quickstart cluster: ```sql SELECT * FROM `>redpanda`.transactions ORDER BY redpanda.timestamp DESC LIMIT 10 ``` Replace `` with your bucket name. Your Redpanda topic is now available as Iceberg tables in Lakehouse, allowing you to run analytics queries directly on your streaming data. ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and the BigLake catalog. To grant your end users and query engines read access to the Iceberg tables in BigQuery, see [Grant permissions for BigLake tables](https://cloud.google.com/bigquery/docs/manage-open-source-metadata#grant_permissions) in the Google Cloud documentation. ## [](#optional-clean-up-resources)Optional: Clean up resources When you’re finished with the quickstart example, you can clean up the resources you created: ```bash # Delete VM instances gcloud compute instances delete --zone= # Delete the storage bucket gcloud storage buckets delete gs:// # Delete the service account gcloud iam service-accounts delete @$(gcloud config get-value project).iam.gserviceaccount.com ``` > 📝 **NOTE** > > Manually delete the Lakehouse catalog using the [REST API](https://docs.cloud.google.com/bigquery/docs/reference/biglake/rest/v1/projects.locations.catalogs/delete). ## [](#suggested-reading)Suggested reading - [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/) - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) - [Google Lakehouse for Apache Iceberg documentation](https://docs.cloud.google.com/lakehouse/docs/introduction) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 156: Troubleshoot Iceberg Topics **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting.md --- # Troubleshoot Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Troubleshoot Iceberg Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/iceberg-troubleshooting page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/iceberg-troubleshooting.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/iceberg-troubleshooting.adoc description: Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. page-git-created-date: "2026-05-06" page-git-modified-date: "2026-07-28" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. Diagnose and resolve errors in Redpanda Iceberg translation, including dead-letter queue (DLQ) inspection and record reprocessing. Use this page to: - Diagnose Iceberg translation errors using DLQ tables and metrics - Reprocess or drop invalid records from the DLQ table ## [](#dead-letter-queue)Dead-letter queue If Redpanda encounters an error while writing a record to the Iceberg table, Redpanda by default writes the record to a separate DLQ Iceberg table named `~dlq`. The following can cause errors to occur when translating records in the `value_schema_id_prefix` and `value_schema_latest` modes to the Iceberg table format: - Redpanda cannot find the embedded schema ID in the Schema Registry. - Redpanda cannot resolve the record’s schema within the topic’s configured Schema Registry context. For example, the schema exists only in a different context. See [Resolve schemas within a Schema Registry context](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#resolve-schemas-within-a-context). - Redpanda fails to translate one or more schema data types to an Iceberg type. - In `value_schema_id_prefix` mode, you do not use the Schema Registry wire format with the magic byte. The DLQ table itself uses the `key_value` schema, consisting of two columns: the record metadata including the key, and a binary column for the record’s value. > 📝 **NOTE** > > Topic property misconfiguration, such as [overriding the default behavior of `value_schema_latest` mode](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#override-value-schema-latest-default) but not specifying the fully qualified Protobuf message name, does not cause records to be written to the DLQ table. Instead, Redpanda pauses the topic data translation to the Iceberg table until you fix the misconfiguration. ### [](#inspect-dlq-table)Inspect DLQ table You can inspect the DLQ table for records that failed to write to the Iceberg table, and you can take further action on these records, such as transforming and reprocessing them, or debugging issues that occurred upstream. The following example produces a record to a topic named `ClickEvent` and does not use the Schema Registry wire format that includes the magic byte and schema ID: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent --format='%k %v\n' ``` Querying the DLQ table returns the record that was not translated: ```sql SELECT value FROM ."ClickEvent~dlq"; -- Fully qualified table name ``` ```bash +-------------------------------------------------+ | value | +-------------------------------------------------+ | 7b 22 75 73 65 72 5f 69 64 22 3a 32 33 32 34 2c | | 22 65 76 65 6e 74 5f 74 79 70 65 22 3a 22 42 55 | | 54 54 4f 4e 5f 43 4c 49 43 4b 22 2c 22 74 73 22 | | 3a 22 32 30 32 34 2d 31 31 2d 32 35 54 32 30 3a | | 32 33 3a 35 39 2e 33 38 30 5a 22 7d | +-------------------------------------------------+ ``` The data is in binary format, and the first byte is not `0x00`, indicating that it was not produced with a schema. ### [](#reprocess-dlq-records)Reprocess DLQ records You can apply a transformation and reprocess the record in your data lakehouse to the original Iceberg table. In this case, you have a JSON value represented as a UTF-8 binary. Depending on your query engine, you might need to decode the binary value first before extracting the JSON fields. Some query engines decode the binary value automatically: ClickHouse SQL example to reprocess DLQ record ```sql SELECT CAST(jsonExtractString(json, 'user_id') AS Int32) AS user_id, jsonExtractString(json, 'event_type') AS event_type, jsonExtractString(json, 'ts') AS ts FROM ( SELECT CAST(value AS String) AS json FROM .`ClickEvent~dlq` -- Ensure that the table name is properly parsed ); ``` ```bash +---------+--------------+--------------------------+ | user_id | event_type | ts | +---------+--------------+--------------------------+ | 2324 | BUTTON_CLICK | 2024-11-25T20:23:59.380Z | +---------+--------------+--------------------------+ ``` You can now insert the transformed record back into the main Iceberg table. Redpanda recommends using an exactly-once processing strategy to avoid duplicates when reprocessing records. ### [](#drop-invalid-records)Drop invalid records To disable the default behavior and drop an invalid record, set the [`redpanda.iceberg.invalid.record.action`](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-invalid-record-action) topic property to `drop`. You can also configure the default cluster-wide behavior for invalid records by setting the `iceberg_invalid_record_action` property. ## [](#troubleshooting-metrics)Troubleshooting metrics The following [Iceberg metrics](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#iceberg-metrics) help identify translation errors, invalid records, and catalog connectivity issues: - [`redpanda_iceberg_translation_dlq_files_created`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_dlq_files_created): Number of DLQ Parquet files created. A non-zero and increasing value indicates records are failing to translate. See [Inspect DLQ table](#inspect-dlq-table) to examine the failed records. - [`redpanda_iceberg_translation_invalid_records`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_translation_invalid_records): Number of invalid records encountered during translation, labeled by cause. See [Drop invalid records](#drop-invalid-records) to configure how Redpanda handles these records. - [`redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#redpanda_iceberg_rest_client_num_commit_table_update_requests_failed): Failed table commit requests to the REST catalog. Applies only when using a REST catalog (`iceberg_catalog_type: rest`). Persistent failures indicate catalog connectivity or permission issues. For an on-demand check of external catalog connectivity and per-partition commit lag, use the `GetIcebergStatus` Admin API endpoint. Commit lag is the number of records written to Parquet but not yet committed to the catalog; the endpoint does not report translation lag (how far Iceberg translation trails the Kafka log). See [Monitor Iceberg health](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/#monitor-iceberg-health). --- # Page 157: Migrate Iceberg Catalogs **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/migrate-iceberg-catalog.md --- # Migrate Iceberg Catalogs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Migrate Iceberg Catalogs latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/migrate-iceberg-catalog page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/migrate-iceberg-catalog.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/migrate-iceberg-catalog.adoc description: Switch the Iceberg catalog backend for an existing Redpanda cluster without losing untranslated topic data. page-git-created-date: "2026-05-30" page-git-modified-date: "2026-05-30" support-status: supported --- Use the procedures in this topic when moving from the filesystem-based `object_storage` catalog to a managed REST catalog, or when changing between REST catalogs. By doing so, you can switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data. This procedure guides you through pausing Iceberg translation per topic, draining pending commits to the old catalog, applying the new catalog configuration, and restarting the cluster. After completing these steps, you will be able to: - Verify that a target Iceberg catalog supports your existing schemas and partition specs - Pause Iceberg translation and drain pending commits without losing untranslated data - Apply new catalog configuration and resume translation safely > ❗ **IMPORTANT** > > Do not change `[iceberg_catalog_type](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_catalog_type)` or any other catalog cluster property in place without following this procedure. In-flight commits and untranslated data can be lost or stuck if the catalog changes mid-translation. ## [](#prerequisites)Prerequisites - Iceberg topics enabled and running on your Redpanda cluster. - Network connectivity from all brokers to the new catalog endpoint. - Credentials configured for the new catalog (REST endpoint, authentication mode, secret or token). For configuration guidance for each catalog type, see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/). - The new catalog must support the current schema and partition spec of every Iceberg topic. See [Verify catalog compatibility](#verify-catalog-compatibility). ## [](#verify-catalog-compatibility)Verify catalog compatibility Before starting the migration, verify that the new catalog can host every Iceberg topic’s table with its existing schema and partition spec. If the new catalog is incompatible with your existing schemas or partition specs, already-translated Parquet files will fail to commit, and translation will stall in a state that is difficult to recover from. The simplest validation is to manually create a test table in the new catalog with the same schema and partition spec as one of your Iceberg topics. If the create call fails, fix the partition spec or schema before migrating. Delete the test table after validation. > ⚠️ **CAUTION** > > AWS Glue does not support partitioning on a nested field, which is Redpanda’s default partition spec for Iceberg topics. If you migrate to AWS Glue, you must change the partition spec to a Glue-compatible form before starting the migration procedure. ## [](#run-the-migration)Run the migration 1. Save the current `retention.ms` and `retention.bytes` values for every Iceberg topic, then set both to `-1` (infinite retention): ```bash rpk topic alter-config --set retention.ms=-1 --set retention.bytes=-1 ``` While Iceberg translation is paused in the next step, the topic’s retention anchor on the log is released. Without infinite retention, the cluster could delete untranslated data before the migration completes. 2. Pause Iceberg translation on every Iceberg topic by setting `redpanda.iceberg.mode` to `disabled`. Save each topic’s previous mode value so you can restore it later. ```bash rpk topic alter-config --set redpanda.iceberg.mode=disabled ``` Setting the mode to `disabled` stops new translation while letting already-translated data finish committing to the old catalog. For more about Iceberg modes, see [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/). > 📝 **NOTE** > > Do not change `[iceberg_enabled](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_enabled)` at the cluster level. The Iceberg integration must remain enabled at the cluster level so that pending commits can drain to the old catalog. 3. Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` metric until it reaches `0` for every Iceberg topic-partition. This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. A non-zero value here is expected while translation is paused, and reflects new records the cluster has not yet translated. > 💡 **TIP** > > If you scrape Prometheus, the following expression returns `0` only when every Iceberg-topic partition has fully drained: > > ```promql > sum(redpanda_iceberg_pending_commit_lag) > ``` 4. Apply the new catalog configuration. For example, to switch from `object_storage` to a REST catalog, update the catalog cluster properties: ```bash rpk cluster config set iceberg_catalog_type rest rpk cluster config set iceberg_rest_catalog_endpoint rpk cluster config set iceberg_rest_catalog_authentication_mode oauth2 # Set additional credential properties for your chosen authentication mode. ``` For full guidance on setting catalog cluster properties, see [Connect to a REST catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/#rest) and the individual [REST catalog integration pages](https://docs.redpanda.com/streaming/current/manage/iceberg/rest-catalog/). 5. Restart Redpanda. The catalog cluster properties require a restart to take effect. For instructions, see [Perform a Rolling Restart](https://docs.redpanda.com/streaming/current/manage/cluster-maintenance/rolling-restart/). 6. After the cluster comes up, check broker logs for successful catalog requests and the absence of authentication errors to verify the new catalog connection. 7. Resume Iceberg translation by restoring `redpanda.iceberg.mode` on every Iceberg topic to its previous value: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` 8. Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved before starting the migration. ## [](#verify-the-migration)Verify the migration After the migration completes, confirm that new data is reaching the new catalog: - Query an Iceberg table in your query engine using the new catalog and confirm that row counts continue to increase as your topic produces new records. - Check broker logs for any commit failures referencing the new catalog. Repeated failures often indicate a schema or partition spec mismatch. See [Troubleshooting](#troubleshooting) for details. ## [](#troubleshooting)Troubleshooting - Pending commits stuck after restart: A schema or partition spec mismatch between the original tables and the new catalog is the most common cause. See [Verify catalog compatibility](#verify-catalog-compatibility). If you cannot resolve the mismatch, contact [Redpanda Support](https://support.redpanda.com/hc/en-us/requests/new). - Authentication errors against the new REST catalog: Verify that the credential cluster properties (for example, `iceberg_rest_catalog_client_id`, `iceberg_rest_catalog_client_secret`, `iceberg_rest_catalog_token`) match what the new catalog expects. For OAuth, also check `iceberg_rest_catalog_oauth2_server_uri`. - Translation does not resume after restoring `redpanda.iceberg.mode`: Check that `redpanda_iceberg_pending_translation_lag` is increasing as new records are produced. If it remains `0`, the cluster is not translating new records. Verify that your producer is still writing to the topic and that the topic’s mode value is one of `key_value`, `value_schema_id_prefix`, or `value_schema_latest`. --- # Page 158: Migrate to Iceberg Topics **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/migrate-to-iceberg-topics.md --- # Migrate to Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Migrate to Iceberg Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/migrate-to-iceberg-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/migrate-to-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/migrate-to-iceberg-topics.adoc description: Migrate existing Iceberg integrations to Redpanda Iceberg topics. page-topic-type: how-to learning-objective-1: Compare external Iceberg integrations with Iceberg Topics architectures learning-objective-2: Implement data merge strategies using SQL patterns learning-objective-3: Execute validation checks and perform cutover procedures page-git-created-date: "2026-02-28" page-git-modified-date: "2026-05-26" support-status: supported --- Migrate existing Iceberg pipelines to Redpanda Iceberg topics to simplify your architecture and reduce operational overhead. After reading this page, you will be able to: - Compare external Iceberg integrations with Iceberg Topics architectures - Implement data merge strategies using SQL patterns - Execute validation checks and perform cutover procedures ## [](#why-migrate-to-iceberg-topics)Why migrate to Iceberg Topics Redpanda’s built-in Iceberg-enabled topics offer a simpler alternative to external Iceberg integrations for writing streaming data to Iceberg tables. > 📝 **NOTE** > > This page focuses on migrating from Kafka Connect Iceberg Sink. The migration patterns and SQL examples can be adapted for other Iceberg sources such as Apache Flink or Spark. ### [](#kafka-connect-iceberg-sink-comparison)Kafka Connect Iceberg Sink comparison The following table compares Kafka Connect Iceberg Sink with Redpanda Iceberg Topics: | Aspect | Kafka Connect Iceberg Sink | Iceberg Topics | | --- | --- | --- | | Infrastructure | Requires external Kafka Connect cluster | Built into Redpanda brokers | | Dependencies | Separate service to manage | No external dependencies | | Setup time | Medium (deploy connector) | Fast (enable topic property and post schema) | ## [](#prerequisites)Prerequisites To migrate from an existing Iceberg integration to Iceberg Topics, you must have: - [Tiered Storage](https://docs.redpanda.com/streaming/current/manage/tiered-storage/) enabled. - [Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/) enabled on your Redpanda cluster. - Understanding of your current schema format (Avro, Protobuf, or JSON Schema). - For Kafka Connect migrations, knowledge of your Kafka Connect configuration, especially if using `iceberg.tables.route-field` for multi-table routing. - If migrating multi-table fan-out patterns, [data transforms](https://docs.redpanda.com/streaming/current/develop/data-transforms/how-transforms-work/) enabled on your cluster. - Access to both source and target (Iceberg Topics) tables in your query engine. - Query engine access (Snowflake, Databricks, ClickHouse, or Spark) for data merging. ## [](#migration-steps)Migration steps Redpanda recommends following a phased approach to ensure data consistency and minimize risk: 1. Enable Iceberg on target topics and verify new data flows. 2. Run both systems concurrently during transition. 3. Choose a strategy to combine historical and new data. 4. Verify data completeness and accuracy. 5. Disable the external Iceberg integration. > ❗ **IMPORTANT** > > Iceberg Topics cannot append to existing Iceberg tables that are not created by Redpanda. You must create new Iceberg tables and merge historical data separately. ### [](#enable-iceberg-topics)Enable Iceberg Topics For simple migrations (one topic mapping to one Iceberg table), enable the Iceberg integration for your Redpanda topics. 1. Set the `iceberg_enabled` configuration property on your cluster to `true`: ```bash rpk cluster config set iceberg_enabled true ``` If you change this configuration for a running cluster, restart the cluster. 2. Configure the `redpanda.iceberg.mode` property for the topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` Choose the mode based on your message format and schema configuration. For Kafka Connect migrations, use this mapping: | Kafka Connect Converter | Recommended Iceberg Mode | | --- | --- | | io.confluent.connect.avro.AvroConverter | value_schema_id_prefix (messages already use Schema Registry wire format) | | io.confluent.connect.protobuf.ProtobufConverter | value_schema_id_prefix (messages already use Schema Registry wire format) | | org.apache.kafka.connect.json.JsonConverter with schemas | value_schema_latest (Schema Registry resolves schema automatically) | | org.apache.kafka.connect.json.JsonConverter with embedded schemas | key_value (schema included with each message) | See [Specify Iceberg Schema](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) to learn more about the different Iceberg modes. 3. If using `value_schema_id_prefix` or `value_schema_latest` modes, register a schema for the topic: ```bash rpk registry schema create -value --schema --type ``` > ❗ **IMPORTANT** > > If using the `value_schema_id_prefix` mode, schema subjects must use the `-value` [naming convention](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-id-validation/#set-subject-name-strategy-per-topic) (TopicNameStrategy). Note the schema ID returned, in case you need it for troubleshooting. 4. Verify that new records are being written to the Iceberg table: - Check that data appears in your query engine. - Validate that the schema translation is correct. - Confirm record counts are increasing. #### [](#multi-table-fan-out-pattern)Multi-table fan-out pattern If your existing integration routes records to multiple Iceberg tables based on a field value (for example, Kafka Connect’s `iceberg.tables.route-field` property), you need to implement equivalent routing logic. You create separate Iceberg-enabled topics for each target table, and Redpanda automatically creates corresponding Iceberg tables. Use either of the following approaches to route records to the correct topic: ##### [](#option-1-data-transforms-with-separate-topics-recommended)Option 1: Data transforms with separate topics (recommended) Use a data transform to read the routing field from each message and write records to separate Iceberg-enabled topics. This approach keeps routing logic within Redpanda and avoids external dependencies. When using Iceberg modes that require schema validation, the transform can register schemas dynamically and encode messages with the appropriate format. 1. Enable data transforms on your cluster: ```bash rpk cluster config set data_transforms_enabled true ``` 2. Create output topics and enable Iceberg with Schema Registry validation: ```bash rpk topic create rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix rpk topic alter-config --set redpanda.iceberg.mode=value_schema_id_prefix ``` 3. Implement a transform function that: 1. Reads the routing field from each input message. 2. If using Schema Registry validation, registers schemas dynamically and encodes messages with the appropriate format. 3. Writes to a specific output topic based on the routing field. 4. Deploy the transform, specifying multiple output topics: ```bash rpk transform deploy \ --file transform.wasm \ --name \ --input-topic \ --output-topic \ --output-topic \ --output-topic ``` 5. Validate the fanout by checking that each output topic receives the correct records. For a complete implementation example with dynamic schema registration, see [Multi-topic fan-out with Schema Registry](https://docs.redpanda.com/streaming/current/develop/data-transforms/build/#multi-topic-fanout). The example demonstrates Schema Registry wire format encoding for use with `value_schema_id_prefix` mode. ##### [](#option-2-external-stream-processor)Option 2: External stream processor Use an external stream processor for complex routing logic: 1. Use a stream processor ([Redpanda Connect](https://docs.redpanda.com/connect/home/) or Flink) to split records. 2. Write to separate Iceberg-enabled topics. This approach is more complex but offers more flexibility for advanced routing requirements not supported by data transforms. ### [](#validate-schema-registry-integration)Validate Schema Registry integration If using [`value_schema_id_prefix`](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/#value_schema_id_prefix) mode, verify that messages use the Schema Registry [wire format](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/#wire-format). ```bash rpk topic consume --num=1 --format='%v\n' | xxd | head -n 1 ``` If the first byte is not `00` (magic byte), you must configure your producer to use the wire format. The `value_schema_id_prefix` mode also requires that schema subjects follow the TopicNameStrategy: `-value`. Verify your schemas use the correct naming: ```bash rpk registry schema list ``` #### [](#verify-no-records-in-dlq)Verify no records in DLQ Check that no records failed validation and were written to the dead-letter queue. If records are present, see [Records in DLQ table](#records-in-dlq-table) for resolution steps. ```sql SELECT COUNT(*) FROM ."~dlq"; ``` ### [](#run-systems-in-parallel)Run systems in parallel Keep your existing Iceberg integration running while Iceberg Topics is enabled. This provides a safety net during the transition period: - New data flows to both the source tables and new Iceberg Topics tables. - You can validate data consistency between both systems. - You have a fallback option if issues arise. Run a query to compare record counts between systems: ```sql -- Source table SELECT COUNT(*) AS source_count FROM .; -- Iceberg Topics table SELECT COUNT(*) AS iceberg_topics_count FROM .; ``` Record counts should increase at similar rates, accounting for the time Iceberg Topics was enabled. Check for DLQ records (see [Records in DLQ table](#records-in-dlq-table)). Monitor Iceberg topic metrics to validate that data is flowing at expected rates: - `redpanda_iceberg_translation_parquet_rows_added`: Track rows written to Iceberg tables (compare with source write rate) - `redpanda_iceberg_translation_translations_finished`: Number of completed translation executions - `redpanda_iceberg_translation_invalid_records`: Records that failed validation - `redpanda_iceberg_translation_dlq_files_created`: Dead-letter queue activity - `redpanda_iceberg_rest_client_num_commit_table_update_requests_failed`: Failed table commits to catalog If using data transforms for multi-table fanout, also monitor: - `redpanda_transform_processor_lag`: Records pending processing in transform input topic For a complete list of Iceberg metrics, see the [Iceberg metrics reference](https://docs.redpanda.com/streaming/current/reference/public-metrics-reference/#iceberg-metrics). > 💡 **TIP** > > Run both systems for at least 24-48 hours to ensure stability before proceeding with data merge. ### [](#merge-historical-data)Merge historical data Choose a strategy to combine your historical data with new Iceberg Topics data. #### [](#option-1-insert-into-pattern-recommended)Option 1: INSERT INTO pattern (recommended) Use this approach to create a unified table with all data, taking into consideration the following: - You want a single table for queries. - You can afford the one-time data copy cost. - You need optimal query performance. This SQL pattern uses partition and offset metadata to identify and copy only records not yet in the target table: ```sql -- Step 1: Find the latest offset per partition in the target (Iceberg Topics) table WITH latest_offsets AS ( SELECT partition, MAX(offset) AS max_offset FROM target_iceberg_topics_table GROUP BY partition ) -- Step 2: Insert records from source table that don't exist in target INSERT INTO target_iceberg_topics_table SELECT s.* FROM source_table AS s LEFT JOIN latest_offsets AS t ON s.partition = t.partition WHERE t.max_offset IS NULL -- Partition not seen before in target OR s.offset > t.max_offset; -- Record is newer than target's latest offset ``` - The `latest_offsets` CTE finds the highest offset in the target table for each partition. - The `LEFT JOIN` ensures you include partitions never seen before in the target (`t.max_offset IS NULL`). - The `WHERE` clause filters to only records with offsets greater than the target’s latest. - This avoids duplicates by using Kafka partition and offset as the deduplication key. This approach may take significant time for large datasets. Consider executing this process during low-query periods. You can also execute on an incremental basis to ease the load on your query engine, for example, by date or partition ranges. #### [](#option-2-view-based-query-federation)Option 2: View-based query federation Use this approach to query both tables without copying data if: - You cannot afford data copy time or cost. - You need immediate access to a unified view. - Query complexity and performance are acceptable with federated queries. - You may consolidate data later. Create a view that queries both tables and deduplicates on the fly: ```sql CREATE VIEW unified_iceberg_view AS WITH latest_offsets AS ( SELECT partition, MAX(offset) AS max_offset FROM target_iceberg_topics_table GROUP BY partition ), historical_data AS ( SELECT s.* FROM source_table AS s LEFT JOIN latest_offsets AS t ON s.partition = t.partition WHERE t.max_offset IS NULL OR s.offset <= t.max_offset -- Only historical records not in target ), new_data AS ( SELECT * FROM target_iceberg_topics_table ) SELECT * FROM historical_data UNION ALL SELECT * FROM new_data; ``` Most Iceberg-compatible query engines support views, including Snowflake, Databricks, ClickHouse, and Spark. ### [](#validate-the-migration)Validate the migration After completing the data merge, verify the migration before cutting over: - Record counts match between source and target: ```sql -- Compare record counts SELECT 'Source' AS table_name, COUNT(*) AS record_count FROM . UNION ALL SELECT 'Target', COUNT(*) FROM .; ``` - All partitions are represented in the target: ```sql -- Check for missing partitions SELECT DISTINCT partition FROM . EXCEPT SELECT DISTINCT partition FROM .; -- Should return no rows ``` - Date ranges cover the full historical period. Compare `MIN(timestamp)` and `MAX(timestamp)` between source and target tables to ensure the target covers the same time range. - No gaps in offset sequences: ```sql -- Check for offset gaps (may indicate missing data) WITH offset_check AS ( SELECT partition, offset, LAG(offset) OVER (PARTITION BY partition ORDER BY offset) AS prev_offset FROM . ) SELECT * FROM offset_check WHERE offset - prev_offset > 1; -- Should return no rows ``` - Sample queries return expected results. Spot check specific records by ID to verify data accuracy. - Schema translation is correct. Run `DESCRIBE` on both tables and verify all fields are present with correct data types. - New records are flowing to Iceberg Topics. Check record count for a recent time window (for example, the last hour). - Query performance is acceptable. - Monitoring and alerts are configured. - No records in DLQ (see [Records in DLQ table](#records-in-dlq-table)). ### [](#troubleshoot-common-migration-issues)Troubleshoot common migration issues #### [](#records-in-dlq-table)Records in DLQ table Iceberg Topics write records that fail validation to a dead-letter queue (DLQ) table. Records may appear in the DLQ due to: - Schema Registry issues. For example, using the wrong schema subject name, or Redpanda cannot find the embedded schema ID in Schema Registry. - When using `value_schema_id_prefix` mode: messages not encoded with Schema Registry wire format. - Incompatible schema changes. For example, changing field types or removing required fields. - Data type translation failures. To check for DLQ records during migration: ```sql SELECT COUNT(*) FROM ."~dlq"; ``` If the count is greater than zero, inspect the failed records. See [Troubleshoot Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/) for steps to inspect and reprocess DLQ records. #### [](#multi-table-fan-out-transform-issues)Multi-table fan-out transform issues If the transform does not process messages, check if: - The specified output topics don’t exist or aren’t enabled with Iceberg. - The routing logic in the transform is incorrect, or the routing field is missing from input messages. - (When using Schema Registry validation) The schema registration failed during initialization, preventing the transform from starting. To check the transform status: ```bash rpk transform list ``` To view logs and check for errors: ```bash rpk transform logs ``` To check for routing errors: ```bash rpk transform logs | grep -i "unknown\|error" ``` If using Schema Registry validation, verify schema registration: ```bash # Check transform logs for schema registration messages rpk transform logs | grep -i "schema" # List registered schemas rpk registry schema list ``` ### [](#plan-for-rollback)Plan for rollback Before cutting over, ensure you have a rollback strategy. See the [Pre-cutover checklist](#pre-cutover-checklist) in the cutover section to verify you’re ready. #### [](#rollback-during-parallel-operation)Rollback during parallel operation If you discover issues while both systems are running: 1. Keep producing to both systems. 2. Point consumers back to source tables. 3. Investigate Iceberg Topics issues using troubleshooting section. 4. Fix issues and re-validate. 5. Attempt cutover again when ready. #### [](#rollback-after-external-integration-disabled)Rollback after external integration disabled > ⚠️ **WARNING** > > Rollback after stopping your external Iceberg integration may result in data loss or gaps. If you must rollback after disabling the external integration: 1. Restart your external Iceberg integration immediately. 2. Identify data written only to Iceberg Topics during the gap. 3. Export that data from Iceberg Topics tables: ```sql SELECT * FROM iceberg_topics_table WHERE timestamp > ''; ``` 4. Write exported data back to the source system (for example, Kafka Connect input topics or directly to source tables). 5. Verify data completeness across both systems. 6. Resume operations on the external integration. Redpanda recommends maintaining the ability to rollback for at least seven days after cutover to allow for issue discovery. ### [](#cut-over-to-iceberg-topics)Cut over to Iceberg Topics #### [](#pre-cutover-checklist)Pre-cutover checklist Before disabling your external Iceberg integration, ensure you have completed all validation steps: - All historical data is successfully merged (see [Merge historical data](#merge-historical-data)). - Parallel operation is complete and stable for at least 24-48 hours. - All validation queries pass (see [Validate the migration](#validate-the-migration)). - No records in DLQ tables, or all DLQ records are investigated and resolved. - Query performance meets requirements. - Downstream consumers are successfully tested with Iceberg Topics tables. - Monitoring and alerts are configured. - Rollback plan is verified and documented. #### [](#cutover-procedure)Cutover procedure 1. Set an appropriate maintenance window, ideally during low-traffic periods. 2. Stop your external Iceberg integration. 3. Monitor Iceberg Topics to ensure data continues flowing. 4. Verify that no new records are being written to source tables: ```sql SELECT MAX(timestamp) FROM .; -- Should not change after integration is stopped ``` 5. Run validation queries from [Validate the migration](#validate-the-migration) after 1-2 hours of operation. 6. Wait for a short period, such as 24-48 hours, to monitor and validate stability. 7. If migrating to a unified table of historical plus new data, optionally delete old source tables after an extended validation period (for example, at least seven days): > 📝 **NOTE** > > Ensure you have backups before deleting historical data. Some organizations keep old tables for compliance or audit purposes. ```sql DROP TABLE .; ``` 8. Decommission external Iceberg infrastructure after an extended safety period (30+ days, for example). If any issues arise during cutover, see [Plan for rollback](#plan-for-rollback). ## [](#next-steps)Next steps - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) - [About Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/) --- # Page 159: Query Iceberg Topics **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics.md --- # Query Iceberg Topics > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Query Iceberg Topics latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/query-iceberg-topics page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/query-iceberg-topics.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/query-iceberg-topics.adoc description: Query Redpanda topic data stored in Iceberg tables, based on the topic Iceberg mode and schema. page-git-created-date: "2025-04-07" page-git-modified-date: "2026-07-17" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. When you access Iceberg topics from a data lakehouse or other Iceberg-compatible tools, how you consume the data depends on the topic [Iceberg mode](https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema/) and whether you’ve registered a schema for the topic in the [Redpanda Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/). You do not need to rely on complex ETL jobs or pipelines to access real-time data from Redpanda. After reading this page, you will be able to: - Query Redpanda topic data from an Iceberg-compatible engine - Grant end-user and query engine access to Iceberg data ## [](#access-iceberg-tables)Access Iceberg tables > ❗ **IMPORTANT** > > Redpanda v25.3 introduces breaking schema changes for Iceberg topics. If you are using Iceberg topics and want to retain the data in the corresponding Iceberg tables, review [Schema Changes and Migration Guide for Iceberg Topics in Redpanda v25.3](https://docs.redpanda.com/streaming/current/upgrade/iceberg-schema-changes-and-migration-guide/) before upgrading your cluster, and follow the required migration steps to avoid sending new records to a dead-letter queue table. Redpanda generates an Iceberg table with the same name as the topic. Depending on the processing engine and your Iceberg [catalog implementation](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/), you may also need to define the table (for example using `CREATE TABLE`) to point the data lakehouse to its location in the catalog. For an example, see [Query Iceberg Topics using Snowflake and Open Catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog/). ### [](#grant-access-to-query-engine-users)Grant access to query engine users Redpanda manages the service-to-service permissions between Redpanda and the catalog (see [Use Iceberg Catalogs](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/)). However, you are responsible for granting your end users and query engines (such as Amazon Athena, Apache Spark, Trino, or Snowflake) read access to the Iceberg data. Use either or both of the following approaches to control access: #### [](#cloud-storage-prefix-level-access)Cloud storage prefix-level access Grant query engine roles or users read access to the Iceberg data prefix in the cluster’s storage bucket. This controls who can read the underlying data and metadata files. Scope permissions to specific prefixes to restrict access to individual tables. - AWS (S3): Use IAM policies to grant `s3:GetObject` and `s3:ListBucket` on the Iceberg prefix (for example, `/redpanda-iceberg-catalog/*`). See [Using IAM policies with Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-iam-policies.html). - GCP (GCS): Use IAM conditions or bucket-level policies to grant `storage.objects.get` and `storage.objects.list` on the Iceberg prefix. See [GCS IAM permissions](https://cloud.google.com/storage/docs/access-control/iam). - Azure (Blob Storage): Use Azure RBAC roles such as Storage Blob Data Reader scoped to the container or prefix. See [Authorize access to blob data](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory). #### [](#catalog-level-table-access)Catalog-level table access If you use a REST catalog, you can control access at the table level through the catalog’s own access control layer. Use this approach when query engines access tables through the catalog rather than reading files directly. - AWS Glue: Use [AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) to grant table-level and column-level permissions. - Databricks Unity Catalog: See the [Unity Catalog privileges documentation](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/index.html). - Snowflake Open Catalog: See [Open Catalog access control](https://docs.snowflake.com/en/user-guide/opencatalog/access-control). - GCP BigLake: See [BigLake table permissions](https://cloud.google.com/bigquery/docs/manage-open-source-metadata#grant_permissions). ### [](#refresh-table-data)Refresh table data Some query engines may require you to manually refresh the Iceberg table snapshot (for example, by running a command like `ALTER TABLE REFRESH;`) to see the latest data. If your engine needs the full JSON metadata path, use the following: ```none redpanda-iceberg-catalog/redpanda//metadata/v.metadata.json ``` This provides read access to all snapshots written as of the specified table version (denoted by `version-number`). > 📝 **NOTE** > > Redpanda automatically removes expired snapshots on a periodic basis. Snapshot expiry helps maintain a smaller metadata size and reduces the window available for [time travel](#time-travel-queries). ## [](#query-examples)Query examples To follow along with the examples on this page, suppose you produce the same stream of events to a topic `ClickEvent`, which uses a schema, and another topic `ClickEvent_key_value`, which uses the key-value mode. The topics have [Tiered Storage](https://docs.redpanda.com/streaming/current/reference/glossary/#tiered-storage) configured to an AWS S3 bucket. A sample record contains the following data: ```bash {"user_id": 2324, "event_type": "BUTTON_CLICK", "ts": "2024-11-25T20:23:59.380Z"} ``` > 📝 **NOTE** > > The query examples on this page use `redpanda` as the Iceberg namespace, which is the default. If you configured a different namespace using `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`, replace `redpanda` with your configured namespace. ### [](#topic-with-schema-value_schema_id_prefix-mode)Topic with schema (`value_schema_id_prefix` mode) > 📝 **NOTE** > > The steps in this section also apply to the `value_schema_latest` mode, except the produce step. The `value_schema_latest` mode is not compatible with the Schema Registry wire format. The [`rpk topic produce`](#reference:rpk/rpk-topic/rpk-topic-produce) command embeds the wire format header, so you must use your own producer code with `value_schema_latest`. Assume that you have created the `ClickEvent` topic, set `redpanda.iceberg.mode` to `value_schema_id_prefix`, and are connecting to a REST-based Iceberg catalog. The following is an Avro schema for `ClickEvent`: `schema.avsc` ```avro { "type" : "record", "namespace" : "com.redpanda.examples.avro", "name" : "ClickEvent", "fields" : [ { "name": "user_id", "type" : "int" }, { "name": "event_type", "type" : "string" }, { "name": "ts", "type": "string" } ] } ``` 1. Register the schema under the `ClickEvent-value` subject: ```bash rpk registry schema create ClickEvent-value --schema path/to/schema.avsc --type avro ``` 2. Produce to the `ClickEvent` topic using the following format: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent --format='%k %v\n' --schema-id=topic ``` The `value_schema_id_prefix` mode requires that you produce to a topic using the [Schema Registry wire format](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/#wire-format), which includes the magic byte and schema ID in the prefix of the message payload. This allows Redpanda to identify the correct schema version in the Schema Registry for a record. 3. The following Spark SQL query returns values from columns in the `ClickEvent` table, with the table structure derived from the schema, and column names matching the schema fields. If you’ve integrated a catalog, query engines such as Spark SQL provide Iceberg integrations that allow easy discovery and access to existing Iceberg tables in object storage. ```sql SELECT * FROM ``.redpanda.ClickEvent; ``` ```bash +-----------------------------------+---------+--------------+--------------------------+ | redpanda | user_id | event_type | ts | +-----------------------------------+---------+--------------+--------------------------+ | {"partition":0,"offset":0,"timestamp":2025-03-05 15:09:20.436,"headers":null,"key":null} | 2324 | BUTTON_CLICK | 2024-11-25T20:23:59.380Z | +-----------------------------------+---------+--------------+--------------------------+ ``` ### [](#topic-in-key-value-mode)Topic in key-value mode In `key_value` mode, you do not associate the topic with a schema in the Schema Registry, which means using semi-structured data in Iceberg. The record keys and values can have an arbitrary structure, so Redpanda stores them in [binary format](https://apache.github.io/iceberg/spec/?h=spec#primitive-types) in Iceberg. In this example, assume that you have created the `ClickEvent_key_value` topic, and set `redpanda.iceberg.mode` to `key_value`. 1. Produce to the `ClickEvent_key_value` topic using the following format: ```bash echo '"key1" {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"}' | rpk topic produce ClickEvent_key_value --format='%k %v\n' ``` 2. The following Spark SQL query returns the semi-structured data in the `ClickEvent_key_value` table. The table consists of two columns: one named `redpanda`, containing the record key and other metadata, and another binary column named `value` for the record’s value: ```sql SELECT * FROM ``.redpanda.ClickEvent_key_value; ``` ```bash +-----------------------------------+------------------------------------------------------------------------------+ | redpanda | value | +-----------------------------------+------------------------------------------------------------------------------+ | {"partition":0,"offset":0,"timestamp":2025-03-05 15:14:30.931,"headers":null,"key":key1} | {"user_id":2324,"event_type":"BUTTON_CLICK","ts":"2024-11-25T20:23:59.380Z"} | +-----------------------------------+------------------------------------------------------------------------------+ ``` Depending on your query engine, you might need to first decode the binary value to display the record key and value using a SQL helper function. For example, see the [`decode` and `unhex`](https://spark.apache.org/docs/latest/api/sql/index.html#unhex) Spark SQL functions, or the [HEX\_DECODE\_STRING](https://docs.snowflake.com/en/sql-reference/functions/hex_decode_string) Snowflake function. Some engines may also automatically decode the binary value for you. ### [](#time-travel-queries)Time travel queries Some query engines, such as Spark, support time travel with Iceberg, allowing you to query the table as it existed at a specific point in the past. You can run a time travel query by specifying a timestamp or version number. Redpanda automatically removes expired snapshots on a periodic basis, which also reduces the window available for time travel queries. By default, Redpanda retains snapshots for five days, so you can query Iceberg tables as of up to five days ago. The following example queries a `ClickEvent` table at a specific timestamp in Spark: ```sql SELECT * FROM ``.redpanda.ClickEvent TIMESTAMP AS OF '2025-03-02 10:00:00'; ``` ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 160: Query Iceberg Topics using Snowflake and Open Catalog **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog.md --- # Query Iceberg Topics using Snowflake and Open Catalog > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Query Iceberg Topics using Snowflake and Open Catalog latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/redpanda-topics-iceberg-snowflake-catalog page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc description: Add Redpanda topics as Iceberg tables that you can query in Snowflake using an Open Catalog integration. page-git-created-date: "2025-02-07" page-git-modified-date: "2026-08-13" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. This guide walks you through querying Redpanda topics as Iceberg tables in [Snowflake](https://docs.snowflake.com/en/user-guide/tables-iceberg), with Amazon S3 as object storage and a catalog integration using [Open Catalog](https://docs.snowflake.com/en/user-guide/opencatalog/overview). After reading this page, you will be able to: - Configure AWS IAM credentials granting Open Catalog access to your S3 bucket - Integrate Redpanda Iceberg topics with Snowflake using Open Catalog ## [](#prerequisites)Prerequisites - [Object storage configured](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#configure-object-storage) for your cluster and [Tiered Storage enabled](https://docs.redpanda.com/streaming/current/manage/tiered-storage/#enable-tiered-storage) for the topics for which you want to generate Iceberg tables. You need the S3 bucket URI to configure it as external storage for Open Catalog. - A Snowflake account. - An Open Catalog account. To [create an Open Catalog account](https://other-docs.snowflake.com/en/opencatalog/create-open-catalog-account), you require ORGADMIN access in Snowflake. ## [](#authorize-access-to-open-catalog)Authorize access to Open Catalog You must create an AWS IAM policy and role that Open Catalog and Snowflake use to access the S3 bucket where your Iceberg data is stored. Redpanda writes Iceberg data and metadata files to the bucket using your cluster’s existing object storage credentials, so Redpanda does not need additional IAM configuration for its own S3 access. You finish configuring this role’s trust policy when you create the catalog and the external volume, using values that Open Catalog and Snowflake generate for you. ### [](#create-an-iam-policy)Create an IAM policy Create an IAM policy with the following S3 permissions, scoped to your cluster’s storage bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion" ], "Resource": "arn:aws:s3:::/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::" } ] } ``` Replace `` with the name of your cluster’s object storage bucket. You use this same IAM policy for both the Open Catalog role and the Snowflake external volume. ### [](#create-an-iam-role)Create an IAM role Create an IAM role and attach the IAM policy you created. Open Catalog and Snowflake each need this role’s ARN before they can generate the IAM user and external ID values you use to finish configuring the trust policy, so create the role with a temporary trust relationship for now: 1. In the AWS IAM console, create a new role. 2. For the trusted entity type, select **AWS account**. Under **An AWS account**, select **This account**. 3. Attach the IAM policy you created. 4. Note the role’s ARN (``). You update this role’s trust policy after Open Catalog and Snowflake generate the values you need, when you create the catalog and the external volume. ## [](#create-a-catalog-in-open-catalog)Create a catalog in Open Catalog Create the catalog that Redpanda’s Iceberg data and metadata are registered to, with your Tiered Storage S3 bucket configured as external storage: 1. In Open Catalog, create a new catalog. 2. For **S3 role ARN**, enter ``, the ARN of the IAM role you created. 3. Configure the catalog’s external storage location to point to your Tiered Storage S3 bucket. > 📝 **NOTE** > > Your Open Catalog account must be in the same AWS region as your S3 bucket. 4. On the Open Catalog home page, in the **Catalogs** pane, select the catalog you created. Under **Storage Details**, copy the **IAM user arn** (``). 5. If you didn’t specify an external ID when you created the IAM role, Open Catalog generates one for you (``). Record this value. For complete steps, see the [Open Catalog documentation](https://docs.snowflake.com/en/user-guide/opencatalog/create-catalog). ### [](#update-the-iam-role-trust-policy-for-open-catalog)Update the IAM role trust policy for Open Catalog Update the trust policy for the IAM role you created, using the IAM user ARN and external ID that Open Catalog generated when you created the catalog: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` After you update the trust policy, Open Catalog can assume the role to read and write Iceberg data and metadata in your S3 bucket. ## [](#create-a-snowflake-external-volume)Create a Snowflake external volume Create a Snowflake external volume using the same IAM role you created for Open Catalog. Snowflake provisions its own IAM user to assume the role, so you add a second statement to the role’s trust policy rather than replacing the statement that trusts Open Catalog: 1. In Snowflake, run `CREATE EXTERNAL VOLUME`, pointing `STORAGE_AWS_ROLE_ARN` to the IAM role you created: ```sql CREATE OR REPLACE EXTERNAL VOLUME STORAGE_LOCATIONS = ( ( NAME = '' STORAGE_PROVIDER = 'S3' STORAGE_BASE_URL = 's3:///' STORAGE_AWS_ROLE_ARN = '' STORAGE_AWS_EXTERNAL_ID = '' ) ) ALLOW_WRITES = TRUE; ``` Use your own values for the following placeholders: - ``: Provide a name for your external volume in Snowflake. - ``: Provide a name for the storage location. - ``: Choose an external ID for the volume’s trust relationship, distinct from the external ID Open Catalog uses. If you don’t set `STORAGE_AWS_EXTERNAL_ID`, Snowflake generates one for you. 2. Run `DESC EXTERNAL VOLUME` to retrieve the IAM user ARN that Snowflake generated for the volume: ```sql DESC EXTERNAL VOLUME ; ``` Record the `STORAGE_AWS_IAM_USER_ARN` value from the output (``). 3. Add a new statement to the IAM role’s trust policy for the volume’s IAM user ARN and external ID, alongside the existing statement that trusts Open Catalog: > ❗ **IMPORTANT** > > Add this as a new statement in the trust policy’s `Statement` array. If you replace the existing trust policy instead, Open Catalog loses access to the bucket. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` 4. Run `SYSTEM$VERIFY_EXTERNAL_VOLUME` to confirm Snowflake can access the bucket: ```sql SELECT SYSTEM$VERIFY_EXTERNAL_VOLUME(''); ``` For complete steps, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-external-volume-s3). ## [](#set-up-catalog-integration-using-open-catalog)Set up catalog integration using Open Catalog To integrate Iceberg-enabled topics with Open Catalog, create a service connection and configure catalog roles. ### [](#create-a-new-open-catalog-service-connection-for-redpanda)Create a new Open Catalog service connection for Redpanda To create a new service connection to integrate the Iceberg-enabled topics into Open Catalog: 1. In Open Catalog, select **Connections**, then **\+ Connection**. 2. In **Configure Service Connection**, provide a name. Open Catalog creates a new principal with this name. 3. Make sure **Create new principal role** is selected. 4. Enter a name for the principal role. Then, click **Create**. After you create the connection, get the client ID and client secret. Save these credentials to add to your cluster configuration in a later step. ### [](#create-a-catalog-role)Create a catalog role Grant privileges to the principal created in the previous step: 1. In Open Catalog, select **Catalogs**, and select your catalog. 2. On the **Roles** tab of your catalog, click **\+ Catalog Role**. 3. Give the catalog role a name. 4. Under **Privileges**, select `CATALOG_MANAGE_CONTENT`. This provides full management [privileges](https://docs.snowflake.com/en/user-guide/opencatalog/access-control#catalog-privileges) for the catalog. Then, click **Create**. 5. On the **Roles** tab of the catalog, click **Grant to Principal Role**. 6. Select the catalog role you just created. 7. Select the principal role you created earlier. Click **Grant**. ### [](#update-cluster-configuration)Update cluster configuration To configure your Redpanda cluster to enable Iceberg on a topic and integrate with Open Catalog: 1. Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below. You must restart your cluster if you change this configuration for a running cluster. You can run `rpk cluster config edit` to update these properties: ```bash iceberg_enabled: true iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://-.snowflakecomputing.com/polaris/api/catalog iceberg_rest_catalog_authentication_mode: oauth2 iceberg_rest_catalog_client_id: iceberg_rest_catalog_client_secret: iceberg_rest_catalog_warehouse: # Optional iceberg_translation_interval_ms_default: 1000 iceberg_catalog_commit_interval_ms: 1000 ``` Use your own values for the following placeholders: - `` and ``: Your [Open Catalog account URI](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters) is composed of these values. > 💡 **TIP** > > In Snowflake, navigate to **Admin**, then **Accounts**. Click the ellipsis near your Open Catalog account name, and select **Manage URLs**. The **Current URL** contains `` and ``. - ``: The client ID of the service connection you created in an earlier step. - ``: The client secret of the service connection you created in an earlier step. - ``: The name of your catalog in Open Catalog. ```bash Successfully updated configuration. New configuration version is 2. ``` 2. You must restart your cluster so that the configuration changes take effect. 3. Enable the integration for a topic by configuring the topic property `redpanda.iceberg.mode`. This mode creates an Iceberg table for the topic consisting of two columns: one for the record metadata including the key, and another binary column for the record’s value. See [Enable Iceberg integration](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration) for more details on Iceberg modes. The following examples show how to use `rpk` to create a new topic or alter the configuration for an existing topic, setting the Iceberg mode to `key_value`. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode=key_value ``` Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode=key_value ``` 4. Produce to the topic. For example, ```bash echo "hello world\nfoo bar\nbaz qux" | rpk topic produce --format='%k %v\n' ``` The topic appears as a table in Open Catalog. 1. In Open Catalog, select **Catalogs**, then open your catalog. 2. Under your catalog, you will see the `redpanda` namespace (or the namespace you configured with `[iceberg_default_catalog_namespace](https://docs.redpanda.com/streaming/current/reference/properties/cluster-properties/#iceberg_default_catalog_namespace)`), and a table with the name of your topic. The namespace and the table are automatically added for you. ## [](#query-iceberg-table-in-snowflake)Query Iceberg table in Snowflake To query the topic in Snowflake, you must create a [catalog integration](https://docs.snowflake.com/en/user-guide/tables-iceberg#catalog-integration) so that Snowflake has access to the table data and metadata. ### [](#configure-catalog-integration-with-snowflake)Configure catalog integration with Snowflake 1. Run the [`CREATE CATALOG INTEGRATION`](https://docs.snowflake.com/sql-reference/sql/create-catalog-integration-open-catalog) command in Snowflake: ```sql CREATE CATALOG INTEGRATION CATALOG_SOURCE = POLARIS TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'redpanda' REST_CONFIG = ( CATALOG_URI = '' WAREHOUSE = '' ) REST_AUTHENTICATION = ( TYPE = OAUTH OAUTH_CLIENT_ID = '' OAUTH_CLIENT_SECRET = '' OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL') ) REFRESH_INTERVAL_SECONDS = 30 ENABLED = TRUE; ``` Use your own values for the following placeholders: - ``: Provide a name for your Iceberg catalog integration in Snowflake. - ``: Your [Open Catalog account URI](https://docs.snowflake.com/en/sql-reference/sql/create-catalog-integration-open-catalog#required-parameters) (`[https://-.snowflakecomputing.com/polaris/api/catalog](https://\-\.snowflakecomputing.com/polaris/api/catalog)`). - ``: The name of your catalog in Open Catalog. - ``: The client ID of the service connection you created in an earlier step. - ``: The client secret of the service connection you created in an earlier step. 2. Run the following command to verify that the catalog is integrated correctly: ```sql SELECT SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG(''); ``` ```bash # Example result for redpanda.iceberg.mode=key_value +-----------------------------------------------------------------------+ | SYSTEM$LIST_ICEBERG_TABLES_FROM_CATALOG('') | +-----------------------------------------------------------------------+ | [{"namespace":"redpanda","name":""}] | +-----------------------------------------------------------------------+ ``` ### [](#create-iceberg-table-in-snowflake)Create Iceberg table in Snowflake After creating the catalog integration, you must create an externally-managed table in Snowflake. You must run your Snowflake queries against this table. In your Snowflake database, run the [CREATE ICEBERG TABLE](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-rest) command. The following example also specifies that the table should automatically refresh metadata: ```sql CREATE ICEBERG TABLE CATALOG = '' EXTERNAL_VOLUME = '' CATALOG_TABLE_NAME = '' AUTO_REFRESH = TRUE ``` Use your own values for the following placeholders: - ``: Provide a name for your table in Snowflake. - ``: The name of the catalog integration you configured in an earlier step. - ``: The name of the external volume you configured using the Tiered Storage bucket. - ``: The name of the table in your catalog, which is the same as your Redpanda topic name. ### [](#query-the-iceberg-table)Query the Iceberg table To verify that Snowflake has successfully created the table containing the topic data, run the following: ```sql SELECT * FROM ; ``` Your query results should look like the following: ```bash # Example for redpanda.iceberg.mode=key_value with 3 records produced to topic +--------------------------------------------------------------------------------------------------------------+------------+ | REDPANDA | VALUE | +--------------------------------------------------------------------------------------------------------------+------------+ | { "partition": 0, "offset": 0, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "68656C6C6F"} | 776F726C64 | | { "partition": 0, "offset": 1, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "666F6F"} | 626172 | | { "partition": 0, "offset": 2, "timestamp": "2025-02-07 16:29:50.122", "headers": null, "key": "62617A" } | 717578 | +--------------------------------------------------------------------------------------------------------------+------------+ ``` ### [](#manage-access-for-query-engine-users)Manage access for query engine users Redpanda manages the permissions between Redpanda and Open Catalog. To grant your Snowflake users or other query engines read access to the Iceberg tables, use [Open Catalog access control](https://docs.snowflake.com/en/user-guide/opencatalog/access-control) to assign catalog privileges. For example, you can grant `TABLE_READ_DATA` to a read-only role rather than the `CATALOG_MANAGE_CONTENT` privilege used by the Redpanda service principal. ## [](#suggested-reading)Suggested reading - [Query Iceberg Topics](https://docs.redpanda.com/streaming/current/manage/iceberg/query-iceberg-topics/) ## Suggested labs - [Redpanda Iceberg Docker Compose Example](https://docs.redpanda.com/labs/docker-compose/iceberg/) - [Iceberg Streaming on Kubernetes with Redpanda, MinIO, and Spark](https://docs.redpanda.com/labs/kubernetes/iceberg/) [Search all labs](https://docs.redpanda.com/labs) --- # Page 161: Integrate with REST Catalogs **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/rest-catalog.md --- # Integrate with REST Catalogs > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Integrate with REST Catalogs latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/rest-catalog/index page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/rest-catalog/index.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/rest-catalog/index.adoc description: Integrate Redpanda topics with managed Iceberg REST Catalogs. page-git-created-date: "2025-07-30" page-git-modified-date: "2025-11-27" support-status: supported --- > 💡 **TIP** > > These guides are for integrating Iceberg topics with managed REST catalogs. Integrating with a REST catalog is recommended for production deployments. If it is not possible to use a REST catalog, you can use the [filesystem-based catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/use-iceberg-catalogs/#object-storage). For an example of using the filesystem-based catalog to access Iceberg topics, see the [Getting Started with Iceberg Topics on Redpanda BYOC](https://www.redpanda.com/blog/iceberg-topics-redpanda-cloud-byoc-setup) blog post. The blog post uses a Redpanda Cloud cluster, but you follow the same steps for a Self-Managed cluster. - [Query Iceberg Topics using AWS Glue](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-aws-glue/) Add Redpanda topics as Iceberg tables that you can query from AWS Glue Data Catalog. - [Query Iceberg Topics using Databricks and Unity Catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-databricks-unity/) Add Redpanda topics as Iceberg tables that you can query in Databricks managed by Unity Catalog. - [Use Iceberg Topics with GCP Lakehouse](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-topics-gcp-biglake/) Add Redpanda topics as Iceberg tables to Google Lakehouse for Apache Iceberg that you can query from Google BigQuery. - [Query Iceberg Topics using Snowflake and Open Catalog](https://docs.redpanda.com/streaming/current/manage/iceberg/redpanda-topics-iceberg-snowflake-catalog/) Add Redpanda topics as Iceberg tables that you can query in Snowflake using an Open Catalog integration. --- # Page 162: Specify Iceberg Schema **URL**: https://docs.redpanda.com/streaming/current/manage/iceberg/specify-iceberg-schema.md --- # Specify Iceberg Schema > For the complete documentation index, see [llms.txt](https://docs.redpanda.com/llms.txt). Component-specific: [streaming-full.txt](https://docs.redpanda.com/streaming-full.txt) --- title: Specify Iceberg Schema latest-redpanda-tag: v26.2.2 latest-console-tag: v3.11.0 latest-operator-version: v26.2.2 # EOL = End-of-Life (support lifecycle status) page-is-nearing-eol: "false" page-is-past-eol: "false" page-eol-date: July 28, 2027 latest-connect-version: 4.107.0 docname: iceberg/specify-iceberg-schema page-component-name: streaming page-version: "26.2" page-component-version: "26.2" page-component-title: Streaming page-relative-src-path: iceberg/specify-iceberg-schema.adoc page-edit-url: https://github.com/redpanda-data/docs/edit/main/modules/manage/pages/iceberg/specify-iceberg-schema.adoc description: Learn about supported Iceberg modes and how you can integrate schemas with Iceberg topics. page-git-created-date: "2025-07-30" page-git-modified-date: "2026-08-04" support-status: supported --- > 📝 **NOTE** > > This feature requires an [enterprise license](https://docs.redpanda.com/streaming/current/get-started/licensing/). To get a trial license key or extend your trial period, [generate a new trial license key](https://redpanda.com/try-enterprise). To purchase a license, contact [Redpanda Sales](https://redpanda.com/upgrade). > > If Redpanda has enterprise features enabled and it cannot find a valid license, [restrictions](https://docs.redpanda.com/streaming/current/get-started/licensing/#self-managed) apply. In [Iceberg-enabled clusters](https://docs.redpanda.com/streaming/current/manage/iceberg/about-iceberg-topics/#enable-iceberg-integration), the `redpanda.iceberg.mode` topic property determines how Redpanda maps topic data to the Iceberg table structure. You can have the generated Iceberg table match the structure of a schema in Schema Registry, or you can use the `key_value` mode where Redpanda stores the record values as-is in the table. After reading this page, you will be able to: - Configure the redpanda.iceberg.mode property when you create or update a topic - Choose the Iceberg mode that produces the table structure your data consumers need - Apply independent translation for record keys, values, and headers ## [](#supported-iceberg-modes)Supported Iceberg modes Redpanda supports the following [modes](https://docs.redpanda.com/streaming/current/reference/properties/topic-properties/#redpanda-iceberg-mode) for Iceberg topics: ### [](#key_value)key_value Creates an Iceberg table using a simple schema, consisting of two columns, one for the record metadata including the key, and another binary column for the record’s value. ### [](#value_schema_id_prefix)value_schema_id_prefix Creates an Iceberg table whose structure matches the Redpanda schema for the topic, with columns corresponding to each field. You must register a schema in [Schema Registry](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/) and producers must write to the topic using the Schema Registry wire format. In the [Schema Registry wire format](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-reg-overview/#wire-format), a "magic byte" and schema ID are embedded in the message payload header. Producers to the topic must use the wire format in the serialization process so Redpanda can determine the schema used for each record, use the schema to define the Iceberg table, and store the topic values in the corresponding table columns. ### [](#value_schema_latest)value_schema_latest Creates an Iceberg table whose structure matches the latest schema registered for the subject in Schema Registry. You must register a schema in Schema Registry. Producers cannot use the wire format in `value_schema_latest` mode. Redpanda expects the serialized message as-is without the magic byte or schema ID prefix in the record value. > 📝 **NOTE** > > The `value_schema_latest` mode is not compatible with the [`rpk topic produce`](#reference:rpk/rpk-topic/rpk-topic-produce) command which embeds the wire format header. You must use your own producer code to produce to topics in `value_schema_latest` mode. The latest schema is cached periodically. The cache period is defined by the cluster property `iceberg_latest_schema_cache_ttl_ms` (default: 5 minutes). ### [](#disabled)disabled Default for `redpanda.iceberg.mode`. Disables writing to an Iceberg table for the topic. > 📝 **NOTE** > > The following modes are compatible with producing to an Iceberg topic using Redpanda Console: > > - `key_value` > > - Starting in version 25.2, `value_schema_latest` with a JSON schema > > > Otherwise, records may fail to write to the Iceberg table and instead write to the [dead-letter queue](https://docs.redpanda.com/streaming/current/manage/iceberg/iceberg-troubleshooting/#dead-letter-queue). ## [](#configure-iceberg-mode-for-a-topic)Configure Iceberg mode for a topic You can set the Iceberg mode for a topic when you create the topic, or you can update the mode for an existing topic. Option 1. Create a new topic and set `redpanda.iceberg.mode`: ```bash rpk topic create --topic-config=redpanda.iceberg.mode= ``` Option 2. Set `redpanda.iceberg.mode` for an existing topic: ```bash rpk topic alter-config --set redpanda.iceberg.mode= ``` ### [](#override-value-schema-latest-default)Override `value_schema_latest` default In `value_schema_latest` mode, you only need to set the property value to the string `value_schema_latest`. This enables the default behavior of `value_schema_latest` mode, which determines the subject for the topic using the [TopicNameStrategy](https://docs.redpanda.com/streaming/current/manage/schema-reg/schema-id-validation/#set-subject-name-strategy-per-topic). For example, if your topic is named `sensor` the schema is looked up in the `sensor-value` subject. For Protobuf data, the default behavior also deserializes records using the first message defined in the corresponding Protobuf schema stored in Schema Registry. If you use a different strategy other than the topic name to derive the subject name, you can override the default behavior of `value_schema_latest` mode and explicitly set the subject name. To override the default behavior, use the following optional syntax: ```bash value_schema_latest:subject=,protobuf_name= ``` - For both Avro and Protobuf, specify a different subject name by using the key-value pair `subject=`, for example `value_schema_latest:subject=sensor-data`. - For Protobuf only: - Specify a different message definition by using a key-value pair `protobuf_name=`. You must use the fully qualified name, which includes the package name, for example, `value_schema_latest:protobuf_name=com.example.manufacturing.SensorData`. - To specify both a different subject and message definition, separate the key-value pairs with a comma, for example: `value_schema_latest:subject=my_protobuf_schema,protobuf_name=com.example.manufacturing.SensorData`. > 📝 **NOTE** > > If you don’t specify the fully qualified Protobuf message name, Redpanda pauses the data translation to the Iceberg table until you fix the topic misconfiguration. ## [](#configure-key-value-and-header-translation)Configure key, value, and header translation For Redpanda clusters version 26.2 and later, in addition to the [supported modes](#supported-iceberg-modes), `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections (see [Iceberg mode shorthands](#iceberg-mode-shorthands)). The `key` and `headers` sections change fields inside the `redpanda` system struct column (`redpanda.key` and the `value` field of each entry in `redpanda.headers`), while the `value` section changes the columns outside that struct. See [How Iceberg modes translate to table format](#how-iceberg-modes-translate-to-table-format) for the base row structure that every generated table includes. Use the following syntax to configure one or more sections: ```bash
: