Case studies / Data foundations
Data foundations · Case study
From inbox alerts to one view of platform health
A DataGrokr DataOps team took over the day-to-day running of a national grocer's enterprise data platform, and built the monitoring around it — as the number of jobs it watched grew from 100+ to 1,600+.
The challenge
The retailer's enterprise data platform fed finance and sales reporting, marketing, and data science, and it ran across several technologies: Snowflake, Kafka, Azure Data Factory, scheduled Python loads, and later Google Cloud. Keeping it healthy depended on people reading email.
The operations team faced upwards of 10,000 emails and notices a week. There were no standard keywords for success or failure, no links from an alert to the relevant log or runbook, and each engineer had developed a personal way of working. Onboarding a new team member was slow, and a failed load could sit unnoticed in an inbox.
How we approached it
Replace the inbox with one dashboard
Twenty separate email-based checks became a single Power BI dashboard that opens a ServiceNow incident automatically when a job fails. On top of it we added targeted monitors: a point-of-sale ingestion monitor that replaced manual SQL checks every 15 to 20 minutes and opens an incident if loading runs past 40 minutes, coverage for 35+ Azure Data Factory pipelines that had been running unmonitored, and a view of 350+ daily customer-data jobs.
Remove the noise and the defects
We deactivated more than 3,000 false data-quality alerts, rewrote 122 production views that broke whenever a base table changed, and reviewed 544 Snowpipes, fixing 180 that could load files from the wrong folder. The team spent less time chasing alerts and more time on real failures.
Make the monitoring see real state
As pipelines began running on Google Cloud, we rebuilt the monitoring there. We designed a three-layer audit that reads Composer's own state, so paused, retried and mid-run-failed DAGs show up instead of hiding behind false successes; the client approved the design within five days. We also consolidated five separate Kafka monitoring systems into one, and added an offset monitor that raises an incident when messages go missing.
Under the hood
The monitoring stack and the platforms it watches:
The result
One dashboard now shows the state of every production DAG, with each DAG's lifecycle, SLA breaches and pause status visible in one place. The slowest dashboard went from 10 to 15 minutes to open to one to two seconds, after we replaced a chain of 22 stacked views with a date-partitioned table refreshed every 15 minutes. Refresh cost in BigQuery fell from about four hours of slot time to under 30 seconds.
What we learned
A platform's monitoring has to be designed for the platform, not carried over from the one before it. Status logic that looked reliable showed false successes once we compared it with the scheduler's real state. We now read state from the source of truth and keep each kind of status separate, so a finished job can never hide a failed process.
Running a data platform on email alerts and tribal knowledge?
Facing something similar?
Scope a diagnostic →