Iceberg Streaming on Kubernetes with Redpanda, RustFS, and Spark
This lab provides a local Kubernetes environment to help you quickly get started with Redpanda and its integration with Apache Iceberg. It showcases how Redpanda, when paired with a Tiered Storage solution like RustFS, can write data in the Iceberg format, enabling seamless analytics workflows. The lab also includes a Spark environment configured for querying the Iceberg tables using SQL within a Jupyter Notebook interface.
In this setup, you will:
-
Produce data to Redpanda topics that are Iceberg-enabled.
-
Observe how Redpanda writes this data in Iceberg format to RustFS as the Tiered Storage backend.
-
Use Spark to query the Iceberg tables, demonstrating a complete pipeline from data production to querying.
This environment is ideal for experimenting with Redpanda’s Iceberg and Tiered Storage capabilities, enabling you to test end-to-end workflows for analytics and data lake architectures.
Prerequisites
Run the lab
Follow these steps to set up the lab environment:
Clone the repository
git clone https://github.com/redpanda-data/redpanda-labs.git
cd redpanda-labs/kubernetes/iceberg
Create the Kubernetes cluster and namespace
kind create cluster --config kind.yaml
kubectl create namespace iceberg-lab
Configure Redpanda for Iceberg
Configuration for Redpanda is provided in configmap.yaml and secret.yaml. This sets up the necessary configurations for Iceberg topics and Tiered Storage.
kubectl apply -f configmap.yaml --namespace iceberg-lab
kubectl apply -f secret.yaml --namespace iceberg-lab
Deploy RustFS
This lab uses RustFS, an S3-compatible object store, for Tiered Storage and the Iceberg warehouse. RustFS is for development and evaluation only and is not supported for production deployments of Redpanda.
-
Deploy RustFS:
kubectl apply -f rustfs.yaml --namespace iceberg-lab -
Wait for RustFS to be ready:
kubectl rollout status deployment/rustfs --namespace iceberg-lab --timeout=300s
Set up the RustFS bucket and permissions
This step creates a job that creates the redpanda bucket and makes it readable without credentials. Redpanda, the Iceberg REST catalog, and Spark use the lab’s credentials.
kubectl apply -f rustfs-setup-job.yaml --namespace iceberg-lab
kubectl wait --for=condition=complete job/rustfs-setup --namespace iceberg-lab --timeout=60s
If the job times out, check the logs with kubectl logs job/rustfs-setup --namespace iceberg-lab.
Deploy the Iceberg REST catalog
This step deploys the Iceberg REST catalog, which allows you to interact with Iceberg tables over HTTP. It uses an init container approach to automatically resolve DNS issues with bucket-style S3 URLs.
kubectl apply -f iceberg-rest.yaml
kubectl wait --for=condition=available deployment/iceberg-rest --namespace iceberg-lab --timeout=120s
Prepare Spark
-
Label and taint your dedicated Spark worker node:
kubectl label node kind-worker node-role.kubernetes.io/spark-node=true kubectl taint nodes kind-worker dedicated=spark:NoSchedule -
Create a ConfigMap with the Spark configuration and the notebook:
kubectl create configmap spark-files --namespace iceberg-lab \ --from-file=spark-defaults.conf=spark/spark-defaults.conf \ --from-file=query-redpanda-table.ipynb="spark/notebooks/Iceberg - Query Redpanda Table.ipynb"Spark runs from the prebuilt
quay.io/jupyter/pyspark-notebookimage, so there is nothing to build. The first Spark session you start downloads the Iceberg libraries, so your first query can take a minute. -
Deploy Spark:
kubectl apply -f spark.yaml kubectl wait --for=condition=available deployment/spark-iceberg --namespace iceberg-lab --timeout=120s
Deploy Redpanda
-
Deploy the Redpanda Operator and its custom resource definitions (CRDs):
helm repo add jetstack https://charts.jetstack.io helm repo add redpanda https://charts.redpanda.com helm repo update helm install cert-manager jetstack/cert-manager \ --set crds.enabled=true \ --namespace cert-manager \ --create-namespace \ --version 1.17.4 helm upgrade --install redpanda-controller redpanda/operator \ --namespace iceberg-lab \ --create-namespace \ --version 26.2.6 \ --set crds.enabled=true -
Ensure that the Deployment is successfully rolled out:
kubectl --namespace iceberg-lab rollout status --watch deployment/redpanda-controller-operatorWhen you see
deployment "redpanda-controller-operator" successfully rolled out, the operator is ready. -
Create the Redpanda cluster:
kubectl apply -f redpanda.yaml --namespace iceberg-lab -
Wait for the Redpanda cluster to be ready:
kubectl get redpanda --namespace iceberg-lab --watchWhen the Redpanda cluster is ready, the output should look similar to the following:
NAME READY STATUS redpanda True Redpanda reconciliation succeeded
Expose services
In this step, you set up access to the RustFS console, Spark Jupyter Notebook, and Redpanda Console.
Set up RustFS console access
For reliable access to the RustFS console, create a NodePort service:
kubectl apply -f rustfs-nodeport.yaml --namespace iceberg-lab
The NodePort service exposes the RustFS console on port 32090 of all cluster nodes. In a kind cluster, you can access it directly at: http://localhost:32090/rustfs/console/
NodePort gives the console a stable address that does not depend on a kubectl port-forward tunnel staying up.
|
Set up port forwarding for other services
For Spark Jupyter Notebook and Redpanda Console, use port forwarding:
kubectl port-forward deploy/spark-iceberg 8888:8888 --namespace iceberg-lab &
kubectl port-forward svc/redpanda-console 8080:8080 --namespace iceberg-lab &
You can run these commands in separate terminals, or run them in the background by appending & as shown above.
This way, all port-forward processes will run in the background in the same terminal. You can bring them to the foreground with fg or stop them with kill if needed.
Create and validate Iceberg topics
You can validate your setup by performing the following steps:
-
Alias the Redpanda CLI:
alias internal-rpk="kubectl --namespace iceberg-lab exec -i -t redpanda-0 -c redpanda -- rpk"This command allows you to run
rpkcommands directly against the Redpanda broker in theiceberg-labnamespace using theinternal-rpkalias. You can also usekubectl exec -i -t redpanda-0 -c redpanda — rpkdirectly if you prefer not to set an alias. -
Create Iceberg topics:
internal-rpk topic create key_value --topic-config=redpanda.iceberg.mode=key_value -
Produce sample data:
echo "hello world" | internal-rpk topic produce key_value --format='%k %v\n' -
Open Redpanda Console at http://localhost:8080/topics to see that the topics exist in Redpanda.
-
Open the RustFS console at http://localhost:32090/rustfs/console/ to view your data stored in the S3-compatible object store.
Login credentials:
-
Account: rustfs
-
Key: rustfs123
-
-
Open the Jupyter Notebook server at http://localhost:8888. The notebook guides you through querying the Iceberg table created from your Redpanda topic.
Clean up
When you’re finished with the lab, you can clean up the resources:
-
Stop all port forwarding processes:
pkill -f "kubectl port-forward"You can also use Ctrl+C if the port forwarding is running in the foreground.
-
Delete the RustFS NodePort service:
kubectl delete service rustfs-nodeport -n iceberg-lab -
Delete the kind cluster (this removes everything):
kind delete cluster
Or, if you want to keep the cluster but remove just the lab resources:
# Delete the namespace (removes all lab resources)
kubectl delete namespace iceberg-lab
# Delete cert-manager
helm uninstall cert-manager --namespace cert-manager
kubectl delete namespace cert-manager
Troubleshoot
Redpanda bucket access errors
If Redpanda logs show bucket not found errors after setup:
-
Verify the bucket exists:
kubectl run s3-ls --rm -i --restart=Never -n iceberg-lab --image=amazon/aws-cli:2.37.1 \ --env AWS_ACCESS_KEY_ID=rustfs --env AWS_SECRET_ACCESS_KEY=rustfs123 --env AWS_REGION=local \ -- --endpoint-url http://rustfs.iceberg-lab.svc.cluster.local:9000 s3 ls -
Check that Redpanda can reach RustFS:
kubectl exec -n iceberg-lab redpanda-0 -c redpanda -- curl -I http://rustfs.iceberg-lab.svc.cluster.local:9000/redpanda
Iceberg REST catalog 500 errors
If the Iceberg REST catalog shows UnknownHostException errors in the logs:
-
Check the catalog logs for DNS resolution errors:
kubectl logs -n iceberg-lab deployment/iceberg-rest | grep -i "unknownhost\|resolve" -
If you see errors, check the init container logs to see if DNS resolution failed:
kubectl logs -n iceberg-lab deployment/iceberg-rest -c dns-resolver -
The init container maps the bucket-style hostnames to the ClusterIP of the
rustfsService, which stays the same when the RustFS pod restarts. If you delete and re-create therustfsService, restart the Iceberg REST catalog so that the init container picks up the new IP:kubectl rollout restart deployment/iceberg-rest -n iceberg-lab kubectl rollout status deployment/iceberg-rest -n iceberg-lab