Systems we rebuilt while they stayed in production
4 representative engagements. Clients are described by sector rather than named; we publish outcomes, not logos, unless a reference is explicitly cleared.
Cutting warehouse spend 61% without touching the SLA
Challenge
Warehouse costs had grown 4x in eighteen months while query latency got worse. Nobody could attribute spend to a team, so nobody could cut it.
Approach
We instrumented query-level attribution, rebuilt the worst offenders as incremental models, and introduced tiered storage with automatic suspension on idle warehouses.
What changed
Before
dbt models 340 models
Full refresh every run
Always-on warehouse never suspended
Untagged queries
One line item no owner
After
dbt models incremental
Changed rows only
Tiered + auto-suspend idle in 60s
Tagged queries attributed
Spend per team query-level
The SLA was the constraint, not the target. Nothing below was allowed to slow down while this was happening.
Fleet telemetry landed in a nightly batch, so operations were always reacting to yesterday. Peak-season backfills routinely ran past the next morning.
Approach
We introduced a Kafka backbone with Flink stateful processing, kept the legacy batch path running in parallel for six weeks, and cut over per-region behind flags.
What changed
Before
Fleet telemetry
Nightly batch 6h 40m
Warehouse stale by a day
Next morning
Operations reacting to yesterday
After
Fleet telemetry
Append
Kafka 12M events/s
Windowed joins
Flink stateful
End to end <800ms
Operations live
The batch path above kept running beside this one for six weeks. Cutover was per-region, behind a flag, and reversible until the last region was done.
SOC2 evidence generated by the pipeline, not by hand
Challenge
Audit prep consumed a quarter of the platform team every cycle. PII masking was applied inconsistently and lineage was reconstructed manually from tribal memory.
Approach
We pushed policy into the transformation layer, column-level classification, automated masking, and lineage emitted as a build artefact that satisfies the auditor directly.
What changed
Before
Transformations policy applied by hand
Inconsistent masking
Warehouse
Lineage from memory each cycle
Audit evidence a quarter of the team
After
Transformations policy as code
Masking at build column-level
Warehouse 100% classified
Lineage artefact every build
Auditor reads the artefact
Evidence is a build output here, so it cannot drift from the pipeline that produced it. If the build passes, the evidence is current.
Training and serving used two separate feature paths. Offline metrics never matched production, and every model rollback became a forensic investigation.
Approach
One declarative feature definition compiled to both offline and online paths, with automated skew detection wired into CI so drift fails the build.
What changed
Before
Two definitions training and serving
Written twice
Two feature paths drift unmeasured
Offline never matched online
Production rollback = investigation
After
One definition declarative
Compiled to both
Offline + online compiled from it
Skew check in CI drift fails the build
Production skew <0.5%
The check runs in CI rather than on a dashboard, because a drift number nobody is paged for is a number nobody reads.