Open your Kafka cluster's AWS bill and look closely. Most of the cost isn't in EC2. It lies in two lines people often overlook: cross-AZ data transfer and EBS. A three-broker cluster spread across three availability zones with a replication factor of 3 means every byte a producer writes is copied across AZ boundaries at least twice — and the cloud provider charges for both directions. With a workload of several hundred MB/s, this cost can easily be many times higher than compute costs.
That is why KIP-1150 "Diskless Topics" — accepted by the Kafka community in March 2026 — is the biggest architectural change since KRaft. Not because it adds features, but because it shifts where the money is spent.
Why Kafka on the cloud is unexpectedly expensive
Kafka was designed in 2011 for self-managed datacenters: cheap disks, free internal bandwidth. Its replication architecture — leader receives writes, followers pull, ISR acknowledges — was perfectly logical in that context.
Bringing this exact architecture to the cloud causes three assumptions to collapse:
- Cross-AZ bandwidth is not free. A producer in AZ-a writes to a leader in AZ-b: you get charged. The leader replicates to a follower in AZ-c: you get charged again. A consumer in a different AZ reads: you get charged again.
- Disks are attached to brokers. To increase throughput, you must add brokers; adding brokers requires rebalancing partitions — in large clusters, a single rebalance can last many hours, during which the cluster runs in a degraded state.
- Hot partitions are a constant drama. A hot partition gets pinned to a specific broker because the data resides on that broker's disk. You cannot "load balance" without moving data.
Meanwhile, S3 and equivalent object storage already provide what Kafka builds itself via replication: 11 nines of durability, cross-AZ replication, and — the key point — no cross-AZ charges for read/write within the same region.
what does KIP-1150 do
Blunt idea: for topics marked as diskless, brokers no longer write logs to local disk. Batches from producers are aggregated and written directly to object storage. Replication disappears — object storage handles it. Broker disks only have two tasks: read cache and KRaft metadata.
graph LR
P[Producer] --> B1[Broker bất kỳ<br/>stateless với topic diskless]
B1 -->|batch + commit| S3[(Object Storage<br/>S3 / GCS / Azure Blob)]
S3 --> B2[Broker khác<br/>cache đọc]
B2 --> C[Consumer]
K[KRaft metadata<br/>+ batch coordinator] -.-> B1
K -.-> B2
Architectural implications are more important than cost savings:
- Brokers become almost stateless with diskless topics. Leaders no longer "own" data — any broker can accept writes for any partition. Producers write to the broker in the same AZ: cross-AZ traffic drops to near zero.
- Scale in seconds, not hours. Adding a broker does not trigger data migration. Autoscale Kafka clusters based on traffic — previously nearly impossible — becomes standard.
- Hot partitions become a single-machine issue. Data resides in object storage; cache can be deployed across multiple brokers.
The ~80% cloud cost savings cited by the community come from exactly this: eliminating cross-AZ replication and replacing expensive EBS with cheap S3.
The trade-off is clear: latency. Writing to object storage means waiting to batch and waiting for S3 PUT confirmation. End-to-end latency jumps from a few ms to hundreds of ms or even seconds, depending on batch configuration. This is not a bug; it is a deliberate trade-off. And because diskless is a per-topic attribute, the same cluster can simultaneously run low-latency classic topics and low-cost diskless topics.
The vendor has been running for three years.
KIP-1150 did not invent this model — it standardizes what the market has already proven:
| WarpStream | AutoMQ | Aiven (Inkless) | |
|---|---|---|---|
| Approach | Rewrite from scratch, protocol-compatible, agent stateless | Fork Kafka, replace storage layer with S3 + WAL | Main contribution to KIP-1150, runs on Kafka upstream |
| Compatibility | Kafka protocol, not Kafka codebase | Keep Kafka codebase, change storage engine | Real Kafka, following upstream |
| p99 latency | Highest (pure object storage, cost-optimized) | Lower thanks to WAL buffering before pushing to S3 | Depends on batch configuration, accepts second-level latency |
| Main risk | Lock-in to Confluent (already acquired) | Fork — how well can it keep up with upstream? | Slowest in terms of features, but safest in the long term |
The 2026 streaming landscape summarized by Kai Waehner revolves around this axis: the competition is no longer "who has Kafka" but "who runs Kafka on object storage best". The acceptance of KIP-1150 changes the game — vendor proprietary advantage shrinks into a time advantage.
The remaining piece: diskless topic meets lakehouse
Once Kafka data is already on object storage, the natural question arises: why not read it as a table?
This is where Confluent Tableflow (now GA for Iceberg and Delta) and the streaming-first lakehouse trend converge. Tableflow automatically materializes topics into Iceberg tables — schema, CDC, publishing to the catalog — without needing custom Spark pipelines to "dump Kafka into the lake." Combined with Iceberg V3's new deletion vectors and row lineage, the CDC-from-Kafka-to-queryable-table chain is streamlined from three self-patched systems into a single path.
For data platform teams, this is more valuable than infrastructure savings: an entire Kafka-to-lake ETL pipeline layer — which still breaks at 2 AM — becomes mere configuration.
Decision framework: wait for upstream or go with a vendor.
The practical question isn't "is diskless good" but "which workload, at what time."
Step 1 — classify topics by latency sensitivity:
- Can tolerate second-level latency: log aggregation, CDC ingest to lakehouse, clickstream, analytics feed, ML feature pipeline. Usually accounts for most throughput (and most of the bill).
- Must stick to classic Kafka: matching engine, payments, real-time fraud detection, anything with a p99 SLA under ~100ms.
Step 2 — audit the bill. If cross-AZ transfer + EBS is under 30% of total Kafka costs, or your cluster is small, do nothing. The complexity isn't worth it.
Step 3 — choose a path:
- Pain now, six-figure bill: Go with a vendor. AutoMQ if you want to maintain maximum Kafka codebase compatibility; WarpStream if you are already in the Confluent ecosystem and can accept higher latency.
- Painful but bearable for 12–18 months: Wait for upstream. KIP-1150 was only accepted in 3/2026; the journey from accepted to a production-ready release for critical topics is a long road. While waiting, the most useful thing is to label latency classifications for each topic right now — so that migration day is just a config change.
- Already using managed Kafka (MSK, Confluent Cloud, Aiven): Stay put. Competitive pressure will drive diskless pricing down to you; no need to migrate yourself.
Kafka is not dying, nor is it being replaced. It is doing what long-lived infrastructure systems always do: acknowledging that the hardware assumptions of 2011 have expired, and sacrificing what it once took most pride in — the hard disk — to preserve something more important: its default position in every data architecture. From now on, paying for cross-AZ replication for log and CDC topics is no longer an operational cost. It is a choice — and will soon be a hard choice to justify.