When one source broke at billion-row scale, the fix was not to rebuild everything

At a Glance

• Database changes flow in through standard change data capture across multiple production databases
• Finance data syncs in from Stripe, untouched by the rebuild
• One product analytics source, generating event volume in the billions of rows, gets its own custom pipeline
• Clean, aggregated data flows back out to the go-to-market and marketing tools the team already works in

‍

Client Background

A voice AI SaaS platform that turns speech into formatted, application-ready text. The product runs on a pipeline pulling from multiple production databases, Stripe for billing, and a dedicated product analytics tool for usage events.

Most of that pipeline was unremarkable, in a good way. Database changes flowed in through standard change data capture. Finance data synced in from Stripe. Go-to-market and marketing data came in through their own paths. One source didn't play along. A product analytics tool was generating event volume in the billions of rows, more than the standard managed ingestion service was built to absorb at that shape and scale.

‍

The Infrastructure Problem

The pipeline lagged. It failed silently. And on retry, it sometimes reloaded a file it had already processed, creating duplicate data downstream that was hard to trace back to its source.

This wasn't a company-wide infrastructure failure. Every other source, production databases, Stripe, go-to-market tools, kept working exactly as it should. But every hour that one source stayed broken was an hour of engineering time spent firefighting instead of building product, and every retry that silently duplicated data made the numbers less trustworthy the longer it went unfixed.

‍

Why the Existing Setup Could Not Keep Up

The instinctive response when one part of a pipeline breaks at scale is to rebuild the whole thing. That instinct is usually wrong, and it would have been wrong here. Replacing the entire ingestion setup to fix one broken path means introducing risk into every part of the system that was already working, the CDC pipelines across production databases, the Stripe sync, the go-to-market ingestion, none of which had ever failed.

What the setup actually needed wasn't a full rebuild. It needed the one failing piece isolated, diagnosed, and fixed on its own terms, without disturbing anything else.

‍

What Datum Labs Built

Rather than replacing the ingestion setup that was working everywhere else, we isolated the one path that wasn't and rebuilt just that piece. A custom pipeline, built in Dagster and Python, now handles this specific source directly, with real monitoring so lag and failures are visible immediately instead of surfacing days later as a support ticket.

Every other source kept running exactly as before. Database changes still flow through standard change data capture. Finance data still syncs in from Stripe. Only the one broken path changed.

Downstream, every record lands in a raw zone before anything touches it, and a transformation layer removes duplicates using a unique identifier before the data reaches a report. Everything still flows into the same central warehouse, ClickHouse Cloud running on 2 or more replicas, as every other source, refreshed roughly every hour, so there's one consistent set of tables, not a separate system carved out for the one troublesome pipeline.

We also closed a loop that hadn't existed before. Clean, aggregated data now flows back out of the warehouse into the go-to-market and marketing tools the team already uses day to day, on an automated schedule through reverse ETL, so insight reaches the tools where decisions get made instead of sitting in a dashboard nobody opens.

What Changed

• Standard sources stayed exactly as they were, database changes via CDC, finance data via Stripe, no disruption to anything already working
• The problem source now runs on its own custom pipeline, sized for its actual scale, instead of straining a general-purpose ingestion tool
• Duplicate records get caught and removed automatically through dbt's unique identifier deduplication, before any report sees them
• Pipeline health is visible to engineers directly, lag and failures surface immediately instead of arriving as a support ticket days later
• Clean, aggregated data flows back into the go-to-market and marketing tools the team already works in, closing a loop that didn't exist before

Nobody had to choose between fixing the broken source and touching everything else. The fix was scoped to exactly the piece that needed it.

‍

Built to Extend as the Business Grows

The pipeline is structured so a new source, or a future source that outgrows standard ingestion the same way this one did, slots in without requiring the same investigation from scratch. The pattern is established: standard sources run on managed CDC ingestion, and anything that outgrows that gets its own isolated, monitored pipeline into the same central warehouse.

For a platform generating event volume at this pace, that pattern is not optional. It's what keeps one difficult source from ever again becoming a full afternoon of firefighting.