Skip to content
Manifester Data Streams (MDS) is live. Kafka-compatible, open source under Apache 2.0.Read the launch post →
mmanifester.io
Blog

Manifester Data Streams: Kafka on S3, 88% cheaper than MSK

MDS stores data in S3 behind stateless brokers, and it is open source under Apache 2.0. It sustained 1 million messages/s on the OpenMessaging Benchmark.

Published
Reading
14 min
Tags
#kafka#object-storage#s3

Today we are launching Manifester Data Streams (MDS), a Kafka-compatible streaming engine written in Go. It is open source under the Apache 2.0 license.

MDS sustained 1 million messages/s (1.02 GB/s of writes, with the same cluster serving the reads) on the OpenMessaging Benchmark. At that load the estimated monthly bill is $9,400 and for a similar workload Amazon MSK will cost $81,200

A classic 3-AZ Kafka bill is mostly disks and network: every byte is stored 3 times and crosses zones. MDS stores each byte once in Object store - S3 and its brokers are stateless.

Every byte is paid for 3 times

In a classic 3-AZ Kafka cluster, every byte lands on 1 disk and is copied to 2 more disks in other zones which pays network fees at each zone crossing. AWS charges $0.01/GB in each direction, so a crossing costs $0.02/GB.

HopDefaultWith follower fetchingWhy
Producer to leader0.670.672 of 3 producers sit in another zone than the leader
Leader to followers2.002.00Both followers sit in other zones
Leader to consumer0.670Consumer reads the replica in its own zone
Total3.332.67Per consumer group
Legend: Each cell is zone crossings per produced byte, for replication factor 3 with 1 replica per zone, producers and consumers spread evenly over 3 zones, and 1 consumer group. 0.67 is the 2 of 3 clients that sit outside the leader’s zone. At $0.02/GB per crossing, 3.33 times is $0.067 for every GB produced. Follower fetching is KIP-392.

Where one produced byte crosses a zone in 3-AZ Kafka

Ingesting 100 MB/s generates 259 TB of data a month. Because producers, leaders, replicas, and consumers sit across different availability zones, that traffic multiplies rapidly:

  • Default setup: Moves 864 TB across zones, costing $17,300 a month in network fees alone.
  • With follower fetching: Drops to 691 TB ($13,800 a month) by letting consumers read from local replicas, though cross-AZ replication hops still remain.
  • Additional consumer groups: Without follower fetching, each extra consumer group adds another 0.67x multiplier to your cross-AZ bill.

Managed services shift these costs around without eliminating them. Amazon MSK absorbs the leader-to-follower replication fees into its instance pricing, leaving you to pay directly for the producer-to-broker hop and for storing all three copies on disk.

Disks now cost more than the network. In their 2023 VLDB paper on exploiting cloud object storage for high-performance analytics, Dominik Durner, Viktor Leis, and Thomas Neumann lay out the stark economics of cloud storage tiers. They price EBS gp2 at $102.40/TiB-month and local instance NVMe at $158.40/TB-month - nearly seven times the $23.20/TiB-month cost of S3. Distributed systems like Kafka compound this gap by replicating data three ways. Consider a workload ingesting 100 MB/s with a 7-day retention window:

  • Raw data: 60 TB
  • On-disk footprint (3 times replication): 181 TB Storing those 181 TB on EBS gp3 ($0.08/GB-month) runs $14,500 a month and that assumes your disks are packed to 100% capacity. Storing that same 60 TB just once in S3 Standard costs $1,400.

Storage is also tightly coupled to compute. Running out of disk space forces you to provision an entire broker, paying for extra CPU and RAM you never needed. Worse, scaling out or replacing a failed node triggers massive partition rebalances, copying terabytes across availability zones while the cluster is trying to serve production traffic. Tiered storage (KIP-405) softens the blow by offloading cold log segments to S3, yet the active segment of every partition still sits on replicated local disks. It trims the storage bill without fixing the architectural root cause: durable state remains pinned to a broker-attached disk.

Traditional 3-AZ Kafka vs. MDS on object storage

Why S3 can be the log now

S3 is 4.4 times cheaper per TiB than EBS gp2 and delivers 11 nines of durability, making it the clear winner on both cost and reliability (Finding 1 of the Durner paper). So why wasn’t Kafka built on object storage from day one?

The blocker was never cost or durability, it was network bandwidth. As Durner et al. says Until recently, local storage was unavoidable to process large tables efficiently due to the bandwidth limitations of the network infrastructure in public clouds. However, the gap between remote network and local NVMe bandwidth is closing

That gap effectively closed in 2018 when AWS introduced 100 Gbit/s (12.5 GB/s) networking, a 4 times jump in per-instance bandwidth:

  • Local NVMe vs. Network: Even on an i3en.24xlarge (AWS’s fastest local-NVMe instance), disk reads hit 16 GB/s while network throughput reaches 12.5 GB/s.
  • Modest hardware premium: Unlocking that 100 Gbit/s tier costs just 22% more per instance : c5n.18xlarge vs. c5.18xlarge (adjusted for DRAM).
  • Real-world S3 throughput: In eu-central-1, a single c5n.18xlarge reading from S3 sustained a median throughput of at least 75 Gbit/s with most runs hitting 80 to 90 Gbit/s, right against the instance ceiling. Object retrieval can now saturate the network pipe (Finding 2).

Still, S3 is not a drop-in replacement for a local filesystem. Three physical realities dictate how you have to engineer around it:

  • Bandwidth comes only from concurrency. Single S3 requests are slow: median base latency is 30 ms plus 20 ms per MiB, capping a single stream at ~95 MiB/s (55-60 MiB/s median). That is spinning-disk speed, meaning an 80 Gbit/s read is hitting roughly 100 spinning HDDs in parallel behind the scenes. Filling a 100 Gbit/s pipe requires keeping 200 to 250 requests in flight at once (Finding 4).
  • You pay per request, not per byte. A 1 KiB GET costs the exact same as a 1 TiB GET, making 8 to 16 MiB chunks the economic sweet spot for bulk reads (Finding 3).
  • Tail latency is long. While most 16 MiB GETs finish in under 600 ms, occasional requests stall. You work around this via request hedging firing a duplicate request if a GET returns no first byte within 200 ms (hedging is on the MDS roadmap).

Two practical caveats apply when sizing brokers:

  1. TLS isn’t free: HTTPS consumes more than 2x the CPU of plain HTTP at identical throughput, making encryption overhead a real line item.
  2. Reads aren’t writes: The Durner benchmarks measure large analytical reads, not the small, latency-sensitive PUTs that drive a streaming write path.

On the flip side, S3 hands you multi-AZ resilience out of the box: data is automatically replicated across at least three availability zones, and traffic between S3 and EC2 instances in the same region costs $0.

In short: S3 is a massive pool of slow disks behind an HTTP API and if you want it to act as your log, you have to build like it.

How MDS writes and reads

Write path and ordering

In MDS, S3 is the log and every produce request moves through following steps:

  1. Same-zone ingress: A producer sends a batch to a broker in its own availability zone, crossing zero zone boundaries.
  2. Shared buffering: The broker appends the batch to a single in-memory buffer shared across all its partitions.
  3. Threshold flush: The buffer flushes as a single S3 object once it hits either a size limit or a time limit and defaulting to 16 MiB or 200 ms after the segment’s first batch, whichever comes first
  4. Durable PUT: The broker writes one object containing the data for every buffered partition alongside an index footer mapping each partition’s byte range.
  5. Metadata commit: Once S3 confirms durability, the broker commits the object’s metadata to the control plane, which assigns offsets in commit order.
  6. Client ack: Only after the metadata commit succeeds does the broker acknowledge the producer.

MDS write path

Two guarantees follow from this sequence:

  1. Decoupled ordering: Because the control plane defines offset order at commit time, objects can land in S3 in any order and offsets simply do not exist until the commit succeeds.
  2. Garbage, never corruption: If a broker crashes between the S3 PUT and the metadata commit, no ack is ever sent. The producer safely retries, leaving behind an unreferenced object for a background sweeper to clean up.

Commit order stays linearizable because the control plane stores its metadata log in PostgreSQL (or a cloud-native equivalent like DynamoDB). As Martin Kleppmann defines it in Designing Data-Intensive Applications, linearizability makes a system behave “as if there were only one copy of the data.” Before an ack, no client sees the data and after a commit, every client sees the exact same order.

If you already run Postgres, the MDS control plane runs right on it. If the control plane becomes unreachable, produce requests simply stall, never acking out of order while everything already committed remains safe in S3. What disappears with this architecture is the

  • Replication machinery: Broker-to-broker replication, in-sync replica (ISR) sets, partition leader elections, under-replicated partitions, and cross-AZ replication traffic all vanish.
  • Data ownership & copy storms: Because no broker owns data, any broker can serve any partition. Scaling out means starting a stateless node; scaling in means stopping one; a broker crash just means clients reconnect elsewhere. Even a full zone outage triggers zero data re-replication.
  • Resource coupling & spot penalties: Compute scales independently with CPU and network while storage scales with retention. Without local disks to fill or rebalance, stateless brokers can run on spot instances at up to 90% discounts, something stateful brokers cannot do without severe performance cliffs.

3 nodes with CPU and disk, against 3 nodes with CPU and S3

Zone-aligned routing eliminates cross-AZ data fees. Traditional Kafka clients query metadata to locate partition leaders across zones. In MDS, clients embed their availability zone in their client ID, and brokers reply to every metadata request with a broker in that exact zone. Producers and consumers talk strictly to same-zone brokers, while brokers talk only to S3 (free within the same region) and the control plane. The 2.67x - 3.33x cross-AZ data multiplier drops to 0.

Batching and S3 request economics

Aggressive batching is what makes S3 affordable as a primary log. AWS charges $5.00 per million PUTs and $0.40 per million GETs:

  • Unbatched writes: Issuing one PUT per message at 100,000 messages/s generates 259 billion PUTs a month - a $1,296,000 API bill.
  • Time-batched writes: Six brokers flushing every 250 ms make just 62 million PUTs a month, costing $311.

With shared buffering, your S3 PUT bill depends on broker count and flush interval rather than message rate. The message volume only shows up in storage cost, which is the cheap part.

The trade-off is latency: every millisecond spent filling a buffer adds directly to produce latency. MDS solves this by flushing on whichever threshold hits first (size or time) and then merging small objects into larger ones in the background. That background compaction shifts reads toward the 8-16 MiB sweet spot, purely as a physical storage layout optimization, completely distinct from Kafka’s log-compacted topics.

Read path and caching

Catch-up reads are fundamentally a concurrency problem. The Durner paper models required concurrency as:

Requests=Throughput×Base Latency+(Size×Data Latency)Size\text{Requests} = \text{Throughput} \times \frac{\text{Base Latency} + (\text{Size} \times \text{Data Latency})}{\text{Size}}

Plugging in S3’s median latencies (30 ms base + 20 ms/MiB) for a consumer replaying history at 1 GiB/s:

  • 1 MiB reads (50 ms each): 51 requests in flight
  • 4 MiB reads (110 ms each): 28 requests in flight
  • 16 MiB reads (350 ms each): 22 requests in flight

Larger byte ranges require less concurrency and fewer billable GETs. Today, each Fetch reads contiguous runs of adjacent batches using a single ranged GET. Every PUT, metadata commit, and read retries up to 6 times with exponential backoff from 20 ms to 640 ms. Dynamic range sizing via this formula and hedged requests are both on the roadmap.

Why Go fits the read path: Keeping hundreds of S3 requests in flight simultaneously is where Go earns its keep. The Durner paper’s AnyBlob engine uses io_uring to multiplex hundreds of requests per thread because spawning one OS thread per request oversubscribes the CPU. Go achieves the same effect natively multiplexing lightweight goroutines onto a handful of OS threads and parking network I/O on the runtime’s netpoller. Hundreds of concurrent S3 requests cost a few kilobytes of goroutine stack rather than hundreds of OS threads. (Note: The broker currently has no read cache; every Fetch in our benchmarks hit S3 directly, with a cache planned next in the roadmap.)

1 million messages/s on 6 brokers

We benchmarked MDS using the public OpenMessaging Benchmark, completely unmodified, with the stock Kafka client driver. A 6-broker cluster sustained 1 million messages/s (1.02 GB/s) of 1 KiB writes while simultaneously serving all consumer reads with consumers keeping pace in real time and zero errors. Testbed & Workload Configuration

  • Cluster: 6 stateless m6in.4xlarge brokers (16 vCPU, 64 GiB RAM each) backed by S3, paired with 1 active + 1 standby control plane (m6in.xlarge) on DynamoDB on-demand.
  • Load Generation: 2 m6in.8xlarge instances running 64 producers and 64 consumers across 3 availability zones in ap-south-2.
  • Workload: 1 topic, 288 partitions, 1 KiB messages (50% random payload), fixed-rate ingest, acks=all, linger.ms=25, batch.size=100000, and lz4 compression.
Target RatePublishedWrite ThroughputPublish p99End-to-End p99
750k msg/s751,369769 MB/s702 ms1.07 s
1M msg/s1,001,3161,025 MB/s743 ms1.19 s
Legend: At peak throughput (1,001,316 messages/s), median publish latency holds at 400 ms p50 (743 ms p99), and full end-to-end producer-to-consumer latency clocks in at 620 ms p50 (1.19 s p99) all without a single local disk or broker-side read cache.

How physical layout unlocked 1 GB/s reads

Early runs proved that the write path, control plane, and DynamoDB had ample headroom, while consumer throughput hit a serialization bottleneck on the read path. Because batches arrived interleaved across partitions, a single Fetch had to issue sequential S3 GET requests for scattered byte ranges capping consumer drain rates at 750k messages/s while compute and S3 bandwidth sat underutilized.

By sorting batches by partition at segment seal time, contiguous partition data now sits side by side inside every S3 object. A Fetch reads an entire run of adjacent batches in a single ranged GET, collapsing S3 request overhead and slashing end-to-end p99 latency from 6.8 s down to 1.07 s at 750k msg/s and unlocking full 1 million messages/s throughput with headroom to spare.

Cost

At 1 million messages/s (1.02 GB/s) with 1-day retention across 3 availability zones (ap-south-2 list pricing), running MDS costs $9,400/month compared to $81,200/month for an equivalent provisioned Amazon MSK cluster an 88% reduction. Infrastructure & Assumptions

  • MDS: 6 stateless m6in.4xlarge brokers, 2 m6in.xlarge control plane nodes, single-copy S3 storage (80.6 TiB), S3 + DynamoDB API requests, and zone-local client traffic.
  • Amazon MSK: 18 m7g.4xlarge brokers (sized to AWS throughput and per-broker storage limits), 3x replicated EBS storage (242 TiB), and cross-AZ producer ingress across 2,658 TB/month.
Monthly Cost BreakdownMDSAmazon MSK (Provisioned)
Brokers$5,100 (6 x m6in.4xlarge)$15,300 (18 x m7g.4xlarge)
Control plane$340 (2 x m6in.xlarge)$0 (included)
Storage (1-day retention)$2,200 (S3, 80.6 TiB, 1x copy)$30,300 (EBS, 242 TiB, 3x copies)
Requests (S3 + DynamoDB)$1,750$0 (included)
Cross-AZ client traffic$0 (zone-local)$35,600
Total$9,400$81,200

Two architectural differences drive nearly the entire $71,800/month savings: storing data once in S3 instead of replicating it three ways on EBS ($28,100 saved), and eliminating cross-AZ client traffic ($35,600 saved). Together, those two line items alone account for 78% of the MSK bill.

Roadmap

  • Latency: Built for high-throughput pipelines (400 ms p50 / 743 ms p99 publish), not sub-10 ms workloads. S3 Express One Zone based quorum design is on the roadmap for close to single-digit millisecond latency.
  • Delivery & Topics: At-least-once delivery today (set enable.idempotence=false on Kafka 3.0+ clients) - idempotent producers, transactions, and compacted topics are in active development.
  • Read Path: Benchmarks ran uncached directly against S3 and same zone broker read caching and hedged GETs are shipping next.
  • Egress: Zero inter-AZ fees inside a region, though standard AWS cross-region ($0.02/GB) and internet egress ($0.09/GB) still apply.

Both planes are open

Many object-storage-native systems leave the data plane in your account while keeping the control plane and your ordering metadata is locked inside a vendor’s hosted service. MDS ships both planes under Apache 2.0 with zero license keys, hosted dependencies, or telemetry, speaking the Kafka wire protocol out of the box so your existing clients connect unchanged. Kafka is just the first system built on this foundation, and at Manifester Cloud we are building the rest in the open.

Check out the code and quickstart at github.com/manifesterio/mds, spin it up against your own traffic, and join us on Discord to share feedback or open issues.

References