Case studies / Cloud

Cloud · Case study

A petabyte-scale SQL Server platform, rebuilt for AWS

A multi-year re-architecture — not a lift-and-shift — for a financial data and analytics business running 50+ on-premises SQL Server instances and growing by roughly 10TB a month.

40%
Lower annual spend ($10M → $6M)
300+
ETL pipelines redesigned
12 hrs
Down from weeks, slowest workloads
Sector
Financial data & analytics
Engagement
Multi-year platform modernization
Problem area
Cloud
Client
Argus Information & Advisory Services (now Verisk Financial Solutions)
Team at peak
~20 engineers
Data under management
~1PB (+10TB/month)

The challenge

In 2017, the client's infrastructure, data, and applications were hosted entirely in an on-premises data center. Microsoft SQL Server was the primary data platform — more than 50 instances holding close to a petabyte of data, growing by roughly 10TB every month, all flowing through multiple complex pipelines.

From the outset, this wasn't going to be a simple lift-and-shift from on-premises to the cloud. The goal was to take full advantage of what AWS offered, which meant fundamentally re-architecting the platform around AWS-native services — many of which the client's own team was only beginning to build expertise in.

How we approached it

01

Prove the approach

DataGrokr joined early, running quick-cycle proofs of concept across candidate AWS services before settling on Amazon Redshift, Amazon EMR, and an S3-based data lake as the target architecture — using transient EMR clusters sized to each workload, with Spot and fleet instances, carefully chosen instance types, and scaling rules tied to data volume.

02

Redesign and validate at scale

More than 300 complex ETL pipelines were redesigned, rebuilt, and thoroughly tested — validating that output from the new pipelines matched the legacy environment before any cutover. Pipelines ran several times faster on the new platform; some processes that had taken weeks in the legacy environment finished in about 12 hours.

03

Operate, secure, and advise

DataGrokr helped establish the AWS cloud architecture, networking strategy, system operations, security framework, and DevOps pipelines, growing to a team of roughly 20 engineers who worked directly with internal users — moving beyond execution into an ongoing advisory role on architecture and planning.

Under the hood

The rebuilt platform, and the legacy environment it replaced:

AWS RedshiftAmazon EMR (transient, Spot/fleet clusters)Amazon S3 data lakeAWS networking & system operationsCloud security frameworkDevOps / CI-CD pipelines300+ ETL pipelinesLegacy: Microsoft SQL Server (50+ instances)

The result

The modernization cut annual platform spend by 40% — from $10M to $6M — and reduced day-to-day (BAU) time-to-delivery by several weeks.

What stood out most was that DataGrokr was never just an execution partner. They consistently advised us on design, planning, and problem-solving, and brought the expertise, flexibility, and collaborative mindset a modernization of this scale requires.

Gautam ValasaeDirector of Engineering at the time of the engagement, Verisk Financial Solutions

Modernizing a financial data platform without losing what already works?

Facing something similar?

Scope a diagnostic →