Case studies / Data foundations
Data foundations · Marketing analytics
A modern, PCI-compliant data lake on AWS — built from scratch
No scalable platform existed to ingest from dozens of public and proprietary sources, and no environment was built to PCI security standards. We built the whole thing from scratch.
Built from scratch on AWS, ingesting billions of transactions.
A fraction of the cost of the previous approach.
Full PCI-compliant environment from day one.
The challenge
No platform built for scale, or for compliance
The client needed to ingest from many public and proprietary sources at growing volume, with no scalable, cost-effective platform to do it on — and no cloud environment built to PCI security standards from the outset.
Before
- No unified ingestion platform across sources
- Compliance bolted on late, if at all
- Rising cost per unit of data processed
- Manual, ad-hoc data handling
After
- 45TB data lake, built from scratch on AWS
- PCI-compliant environment from day one
- Operating cost under $10K/month
- Supported and maintained ongoing
How we approached it
Build the cloud, design for cost, embed the practice
Build the environment from scratch
We stood up the entire AWS environment to PCI-compliant security standards, then built a comprehensive ingestion and processing platform on it using a range of AWS data services — not a layer on top of existing infrastructure, but the foundation itself.
Design storage for cost and change
S3 was the storage layer for data that rarely changed, keeping cost low; Postgres on RDS held data that did change, exported to S3 once stable. Transformations ran on Glue and Athena, and the pipelines were tuned for both cost and performance.
Embed MLOps, then stay
A single scrum team delivered the platform in six months. We implemented MLOps with SageMaker and GitLab, embedded our cloud engineers alongside the client's data scientists to transfer the practice, and continued to support the platform after launch.
Under the hood
Next step
Carrying data across many sources, with compliance on the line?
We design ingestion and compliance together from day one — not as a retrofit once something's already live.