Connect

Tune the Iceberg output for throughput

Tune the iceberg output for high write throughput to Apache Iceberg tables. Use these four tested recipes to choose the right trade-off between preserving message ordering, maximizing throughput, and applying keyed change-data-capture (CDC) operations.

After reading this page, you will be able to:

  • Configure an order-preserving Iceberg sink that sustains high throughput

  • Configure an unordered Iceberg sink for maximum throughput

  • Choose the recipe that matches your ordering and throughput requirements

  • Tune a keyed change-data-capture sink that uses upsert or delete operations

The two append recipes target roughly 10,000 rows or 10 seconds per commit, whichever comes first. This keeps commit sizes large enough to amortize the cost of each Iceberg snapshot while bounding commit latency.

Prerequisites

These recipes tune append-only sinks, where every message becomes a new row. They use a high max_in_flight to commit batches concurrently. Do not apply them to keyed change-data-capture workloads that use row_operation with upsert or delete. Keyed workloads require max_in_flight: 1 to preserve per-key order, and their tuning levers are different: see Recipe C.

Recipe A: preserve message ordering

Use this recipe when you want to process records as a single ordered stream, rather than in parallel per partition.

A single merged input stream preserves the processing order of records, because Redpanda Connect consumes the stream in order instead of parallelizing consumption per partition. To keep that ordered stream fast, decouple the input from the commit-bound output and batch at the output level:

  • A memory buffer absorbs short input bursts and commit stalls, decoupling the input from commit latency. The buffer is bounded (500 MiB in this recipe): if the input sustains a higher rate than the sink can commit, the buffer fills and applies backpressure to the input. Size the buffer to your throughput multiplied by the commit interval.

  • Output-level batching feeds the Iceberg commit coalescer, so max_in_flight can dispatch concurrent writers. Without output batching, a single buffered stream starves the committer. In smoke tests at 1 vCPU, adding output batching raised throughput from 0.8 to 12.7 MB/s.

# Configure catalog, storage, namespace, and table as usual.
buffer:
  memory:
    limit: 524288000        # 500 MiB: size to throughput x commit interval
    batch_policy:
      count: 10000
      period: 10s
output:
  iceberg:
    max_in_flight: 16
    batching:
      count: 10000
      period: 10s
    commit:
      max_snapshot_age: 24h

max_in_flight: 16 lets the output commit batches concurrently for throughput. Because concurrent batches can commit in any order, the order in which rows become visible in the Iceberg table is not guaranteed to match the input order, even though the input stream itself is processed in order. If you need commits applied in strict order, set max_in_flight: 1, at the cost of throughput.

This recipe’s values are validated optima for this pipeline shape, so there is no need to raise them further. In benchmarks against an AWS Glue REST catalog, larger max_in_flight values and larger batch sizes produced the same mean throughput with a burstier delivery pattern, because Iceberg commits to a single table are applied sequentially. The recipe’s values give the smoothest commit cadence at full throughput. Batch cadences of 5 and 10 seconds performed identically.

Recipe B: maximize throughput

Use this recipe when preserving processing order is not required, which is typical for append-only Iceberg sinks. This recipe reaches the highest throughput ceiling.

Recipe B removes the buffer and parallelizes at the input instead. The redpanda input's unordered_processing processes each partition’s records in parallel and batches them in the read-ahead fetch buffer, feeding many concurrent commits to the coalescer. A higher max_in_flight of 32 lets those commits overlap. This configuration trades ordered processing for higher throughput.

input:
  redpanda:
    # Configure seed_brokers, topics, and consumer_group as usual.
    unordered_processing:
      enabled: true
      checkpoint_limit: 20000
      batching:
        count: 10000
        period: 10s
output:
  iceberg:
    max_in_flight: 32

checkpoint_limit must exceed batching.count, because it caps how many unacknowledged records a partition can hold in flight. If the limit is lower than the batch size (for example, the default checkpoint_limit of 1024 combined with count: 10000), batches can never fill to their target count. Every batch then flushes on the period timer at a fraction of the intended size, and each undersized batch becomes its own Iceberg commit. Set checkpoint_limit to at least twice batching.count.

Compare recipe throughput

The following table shows mean write throughput in MB/s for each append recipe as vCPUs scale. Keyed workloads follow a different cost model: see Recipe C and Recipe D. Results were measured on an Amazon Web Services (AWS) c8g.4xlarge instance against a roughly 178 GB dataset, using an AWS Glue REST catalog with Amazon S3 storage.

vCPU Recipe A (ordered) Recipe B (unordered)

1

15.9

38.3

2

69.1

64.1

4

114.2

98.6

8

109.2

122.4

When CPU is constrained, Recipe B’s per-partition parallelism gives a clear advantage. Recipe A scales strongly up to 4 vCPUs and then plateaus, while Recipe B continues to climb as more vCPUs become available.

Your results depend on your catalog, storage backend, message size, and partition count. Use these numbers as a starting point and benchmark against your own workload.

Choose a recipe

Requirement Recipe

Ordered processing is not required (typical for append-only Iceberg sinks)

Recipe B: maximize throughput

You want the stream processed in order

Recipe A: preserve message ordering

The workload is keyed CDC (row_operation with upsert or delete)

Recipe C: keyed CDC with merge-on-read

The workload is keyed CDC and the table must be readable by engines that cannot read equality deletes (for example, Snowflake or Databricks Unity Catalog)

Recipe D: keyed CDC with clustered-key copy-on-write

Recipe C: keyed CDC with merge-on-read

Keyed workloads, where row_operation resolves to upsert or delete, follow a different cost model than the append recipes above, so they are tuned with different levers:

  • max_in_flight must be 1. The output validates this at startup to guarantee last-writer-wins correctness: concurrent batches could otherwise commit out of order. See Row-level operations.

  • Each batch that contains an upsert or delete becomes its own Iceberg snapshot. This one-to-one mapping is what guarantees correct results, so batch boundaries directly define snapshot boundaries.

With one snapshot per batch and sequential commits, batch size is the primary throughput lever: throughput is the rows carried per batch divided by the time each commit takes. In benchmarks against an AWS Glue REST catalog, raising the batch size from 10,000 to 50,000 rows more than doubled measured upsert throughput. Size the batch against your latency budget and memory, and run regular snapshot expiry and compaction, because a keyed sink produces one snapshot per batch.

# Configure catalog, storage, namespace, and table as usual.
buffer:
  memory:
    limit: 524288000        # 500 MiB: size to throughput x commit interval
    batch_policy:
      count: 50000
      period: 10s
output:
  iceberg:
    max_in_flight: 1        # required for keyed workloads
    row_operation: upsert   # or an interpolated expression driven by the data
    identifier_fields: [ id ]
    batching:
      count: 50000
      period: 10s
    commit:
      max_snapshot_age: 24h

Choose a merge strategy

The merge_strategy field controls how mutations are written, and it dominates keyed throughput:

  • merge-on-read (the default) writes small equality-delete files and sustains streaming rates: the numbers above were measured in this mode. Only engines that can read equality deletes can query the result.

  • copy-on-write rewrites every data file that contains an updated key, which makes the table readable by every engine. Its throughput is set by how the updated keys land across data files: keys arriving in contiguous runs sustained a rate comparable to merge-on-read in the same benchmark (see Recipe D), while keys spread across the whole table sustained a small fraction of that, because every commit rewrites every file containing a touched key, and the scattered-key cost grows with table size.

Use merge-on-read for continuous CDC streams whenever your query engines can read it. Use copy-on-write when they cannot, following Recipe D and its key-clustering prerequisite. For the full decision guide and support matrix, see the merge_strategy field in the iceberg output reference.

Recipe D: keyed CDC with clustered-key copy-on-write

Use this recipe when the table must be readable by engines that cannot read equality deletes, such as Snowflake or the Databricks Unity Catalog, so merge-on-read is not an option.

Copy-on-write rewrites every data file that contains an updated key, so the write cost per batch is set by how many files the batch’s keys touch. That makes key arrival order the decisive factor:

  • Clustered keys (arriving in contiguous runs) concentrate each batch’s updates into the few files covering that key range. Measured with the Recipe C configuration plus merge_strategy: copy-on-write against an AWS Glue REST catalog, this sustained a steady rate in the same range as merge-on-read on the identical workload, with no degradation over time.

  • Scattered keys spread each batch’s updates across many files, so each commit rewrites a large share of the table. This sustains a small fraction of the clustered rate, and that share grows with table size. This is an inherent property of the copy-on-write format, not a tunable. Address scattered keys upstream, in the data layout, as described below.

The configuration is Recipe C with one change:

# Configure catalog, storage, namespace, and table as usual.
buffer:
  memory:
    limit: 524288000
    batch_policy:
      count: 50000
      period: 10s
output:
  iceberg:
    max_in_flight: 1        # required for keyed workloads
    row_operation: upsert
    identifier_fields: [ id ]
    merge_strategy: copy-on-write
    batching:
      count: 50000
      period: 10s
    commit:
      max_snapshot_age: 24h

To satisfy the clustering prerequisite when your source does not already produce keys in runs, arrange it upstream: order the capture by primary key where the source supports it, or partition the source topic by key range so each partition carries a contiguous slice of the key space. If the keys cannot be clustered, expect scattered-key performance and treat the sink as batch or low-churn only.

Plan for regular table maintenance with this recipe: every mutating batch rewrites files and adds a snapshot, so schedule compaction, snapshot expiry, and orphan-file removal as part of normal operations. See the merge_strategy field in the iceberg output reference for the maintenance guidance and support matrix.

Next steps