Mosaic Streaming Router: Kafka POC

Modeled ingress

0 MB/s

Hot path

0%

Bulk routed

0%

Replay retained

0 d

Projected savings

$0

Failover ready

0

Policy decisions

0

Routing p95

0 ms

Control plane

Kafka traffic classification

Mosaic evaluates topic class, payload sensitivity, consumer lag, retention need, and incident state before selecting a streaming destination.

Simulated live stream

Producers

Orders, payments, clickstream, audit

Kafka protocol clients emit business events with topic and tenant hints.

→

Mosaic router

Policy + schema + lag guardrails

Premium hot streams, cheaper bulk streams, and replay archives are split without app rewrites.

→

Destinations

Confluent, WarpStream, AutoMQ, Redpanda, Aiven

Kafka-compatible endpoints are treated as policy targets, not as a single forever choice.

Routing evidence

Recent topic decisions

Retail event storm

Time	Class	Topic	Tenant	Size	Destination	Reason

Destination fit

Provider roles

Kafka-compatible

Premium hot path

Confluent Cloud

42%

Best fit for Schema Registry, Connect, Flink, governance, enterprise support, and low-risk migrations.

Cheapest bulk

WarpStream

26%

Strong candidate when object-storage economics and Kafka protocol compatibility matter more than ultra-low latency or full broker ownership.

S3-backed Kafka

AutoMQ

16%

Strong candidate when the buyer wants cloud-native Kafka with storage decoupled to S3-style object storage.

Operational simplicity

Redpanda Cloud

18%

Kafka API-compatible path with simpler broker operations and strong performance profile.

Managed OSS

Aiven for Kafka

9%

Useful when the buyer wants managed open-source Kafka across several clouds with straightforward service boundaries.

Policy knobs

Business guardrails

Editable assumptions

Confluent target share35%

Replay retention30 d

Ingress rate85 MB/s

All-Confluent baseline

$0

Mosaic routed model

$0

Savings estimate

$0

Cost model

Streaming spend analogy

This is a directional model, not a vendor quote. It uses editable assumptions so the demo can show the shape of a conversation before live SKU pricing and discounts are plugged in.

Assumption driven

Monthly ingress

0 TB

Premium hot data

0 TB

Cheaper bulk data

0 TB

Retention multiplier

1.0x

Why this maps to observability

Hot path vs context path

Where it differs

Kafka has harder correctness constraints

Bulk analytics

Clickstream, product telemetry, menu/catalog sync

Best first wedge. Route high-volume, replayable topics to WarpStream or AutoMQ while keeping compact control topics in Confluent.

Audit and compliance

Login audit, policy change, billing ledger copies

Durable retention, long replay windows, and lower query frequency fit object-storage-backed Kafka economics.

Migration and dual write

Confluent-to-cheaper target assessment

Mirror traffic, compare schema compatibility, lag, ordering drift, and consumer behavior before cutover.

Incident expansion

Payments, fraud, order criticality

Temporarily widen Confluent hot-path coverage during customer-impacting incidents, then contract after lag clears.

Dead-letter replay

DLQ fan-out and recovery windows

Keep cheap replay capacity for poison-message recovery without paying premium rates for every retained payload.

Tenant-aware routing

Enterprise customers, regulated data, regional rules

Route by tenant tier, data class, region, and contractual posture instead of treating all Kafka bytes equally.

Provider fit matrix

How Mosaic should choose a target

Decision evidence

Target	Best-fit workload	Why it belongs	Watch-outs
Confluent Cloud	Critical operational streams, governance-heavy teams, Flink/Connect/SR-heavy estates	Lowest migration risk and strongest enterprise platform gravity.	Premium economics make it expensive as the default sink for every retained byte.
WarpStream	Bulk analytics, replay, event lake, high-retention topics	Kafka protocol compatibility with object-storage economics makes it the cleanest cheap-capacity story.	Latency-sensitive and complex transactional workloads need careful validation.
AutoMQ	S3-backed cloud-native Kafka, elastic storage-heavy topics, BYOC-style control	Decoupled storage maps tightly to the OpenObserve/R2 analogy and can preserve Kafka semantics better than a pure event lake.	Buyer confidence depends on maturity, ecosystem compatibility, and operational proof.
Redpanda Cloud	Kafka API-compatible performance workloads and broker simplification	Good alternative when operational simplicity and performance are the buyer's pain.	Less purely a cheap-retention story than WarpStream/AutoMQ.
Aiven for Kafka	Managed open-source Kafka across clouds, private/network-sensitive estates	Useful where open-source posture and cloud choice matter.	May not create the same dramatic storage-cost wedge by itself.

Reference architecture

Mosaic Kafka control plane

Protocol-aware edge

Accept producer metadata through sidecars, Kafka Connect, MirrorMaker 2, topic policies, or SDK hints. Do not require every producer to change on day one.

Policy engine

Score topic class, schema compatibility, key ordering, PII flags, tenant tier, region, lag budget, retention window, and incident mode.

Dual-write and mirror evidence

Before moving traffic, compare offsets, key distribution, compaction behavior, schema evolution, and consumer lag across targets.

Destination adapters

Treat Confluent, WarpStream, AutoMQ, Redpanda, and Aiven as policy targets with health, cost, SLA, and replay evidence.

Failback controller

If lag, ordering drift, schema rejection, or target health violates policy, promote the topic back to Confluent and retain a replay trail.

Correctness risks

What makes Kafka harder than logs

Ordering

Key-level ordering and partition behavior must be preserved or explicitly scoped.

Exactly-once

Transactional producers and idempotent writes need conservative migration treatment.

Schema evolution

Compatibility rules must be checked across registries and consumers.

Consumer lag

Routing is only successful if downstream consumers keep their recovery budgets.

Retention

Cold replay topics are usually easiest to optimize.

Compaction

Log-compacted topics require target-specific validation.

PII and region

Tenant and data residency rules can override pure cost routing.

Observability

Lag, route, and replay evidence are highly demoable and buyer-friendly.

90-day wedge

Realistic build path

Practical MVP

Week 1-2: Topic inventory

Ingest topic configs, producer/consumer groups, schemas, retention, compaction, throughput, lag, and tenant metadata. Output a route-readiness score.

Week 3-4: Recommendation-only mode

Rank topics by savings potential and migration risk. Start with analytics, audit, replay, and DLQ topics.

Week 5-7: Shadow mirror

Mirror selected topics to WarpStream or AutoMQ and compare offsets, lag, schema rejections, and replay completeness.

Week 8-10: Policy-controlled cutover

Move low-risk producers or consumers behind a controlled route with automatic failback to Confluent.

Week 11-12: Buyer proof pack

Generate savings evidence, incident safety proof, replay tests, schema compatibility reports, and an executive migration plan.

Day-one command

Find waste

Which topics have high retention, low SLA, low consumer criticality, and no strict ordering constraints?

Day-one proof

Mirror safely

Can the cheaper target keep up for 72 hours with no schema rejects, no unexplained offset gaps, and acceptable replay speed?

Day-one buyer

Platform team

The initial buyer is the Kafka/platform owner under cost pressure, not every application team at once.

Morning brief

Is there a Kafka version of the Datadog/OpenObserve wedge?

Yes, but narrower

The clean analogy

Confluent = premium default.

It is the closest Kafka equivalent to Datadog in your existing story: enterprise feature depth, integrations, governance, support, and lower adoption risk.

WarpStream and AutoMQ are the closest cheaper-cloud analogies.

Both lean into object-storage economics, which makes them feel most like the OpenObserve/R2 side of the observability POC.

Redpanda and Aiven are also viable targets.

They are less purely OpenObserve-style cheap storage, but credible Kafka-compatible choices depending on performance, cloud, and open-source posture.

The product wedge

Do not start as a universal Kafka broker.

The first shippable version should be a policy router and migration copilot for topics, consumers, replay, and retention placement.

Start with duplicate-write and replay-safe topics.

Analytics, clickstream, audit, and cold replay streams are easier than exactly-once payment events or tightly ordered command topics.

Sell control-plane evidence.

The buyer wants proof that lag, schema compatibility, ordering risk, retention, and failback are being managed before moving production traffic.

Reference links

Vendor pages to refresh before a customer-facing price claim

Live verification recommended

Confluent Cloud pricing WarpStream pricing AutoMQ Redpanda Cloud Aiven for Apache Kafka