Case studies / Data foundations

Data foundations · Case study

From inbox alerts to one view of platform health

A DataGrokr DataOps team took over the day-to-day running of a national grocer's enterprise data platform, and built the monitoring around it — as the number of jobs it watched grew from 100+ to 1,600+.

10–15 min → 1–2 sec
Dashboard load time, after a redesign
1,600+
Airflow DAGs and tasks monitored daily (from 100+ pipelines in 2023)
20 → 1
Email-based monitoring processes replaced by one dashboard
Sector
Grocery retail
Engagement
Multi-year DataOps and monitoring
Problem area
Data foundations
Client
A large US grocery retailer
Our role
Ran production operations; designed and built the monitoring
Platform
Snowflake, Kafka, Azure and Google Cloud

The challenge

The retailer's enterprise data platform fed finance and sales reporting, marketing, and data science, and it ran across several technologies: Snowflake, Kafka, Azure Data Factory, scheduled Python loads, and later Google Cloud. Keeping it healthy depended on people reading email.

The operations team faced upwards of 10,000 emails and notices a week. There were no standard keywords for success or failure, no links from an alert to the relevant log or runbook, and each engineer had developed a personal way of working. Onboarding a new team member was slow, and a failed load could sit unnoticed in an inbox.

How we approached it

01

Replace the inbox with one dashboard

Twenty separate email-based checks became a single Power BI dashboard that opens a ServiceNow incident automatically when a job fails. On top of it we added targeted monitors: a point-of-sale ingestion monitor that replaced manual SQL checks every 15 to 20 minutes and opens an incident if loading runs past 40 minutes, coverage for 35+ Azure Data Factory pipelines that had been running unmonitored, and a view of 350+ daily customer-data jobs.

02

Remove the noise and the defects

We deactivated more than 3,000 false data-quality alerts, rewrote 122 production views that broke whenever a base table changed, and reviewed 544 Snowpipes, fixing 180 that could load files from the wrong folder. The team spent less time chasing alerts and more time on real failures.

03

Make the monitoring see real state

As pipelines began running on Google Cloud, we rebuilt the monitoring there. We designed a three-layer audit that reads Composer's own state, so paused, retried and mid-run-failed DAGs show up instead of hiding behind false successes; the client approved the design within five days. We also consolidated five separate Kafka monitoring systems into one, and added an offset monitor that raises an incident when messages go missing.

Under the hood

The monitoring stack and the platforms it watches:

Power BIServiceNowSnowflakeBigQueryCloud Composer (Airflow)Apache KafkaDataflowAzure Data FactoryStonebranchPython & SQL

The result

One dashboard now shows the state of every production DAG, with each DAG's lifecycle, SLA breaches and pause status visible in one place. The slowest dashboard went from 10 to 15 minutes to open to one to two seconds, after we replaced a chain of 22 stacked views with a date-partitioned table refreshed every 15 minutes. Refresh cost in BigQuery fell from about four hours of slot time to under 30 seconds.

What we learned

A platform's monitoring has to be designed for the platform, not carried over from the one before it. Status logic that looked reliable showed false successes once we compared it with the scheduler's real state. We now read state from the source of truth and keep each kind of status separate, so a finished job can never hide a failed process.

Running a data platform on email alerts and tribal knowledge?

Facing something similar?

Scope a diagnostic →