25PB cross-cloud data platform & self-service feed onboarding

01The problem

Two problems, one platform. First: enterprise data platforms tend to fuse themselves to one cloud vendor — every pipeline hard-codes provider APIs until migration is fiction. The mandate was a 25PB+ pipeline platform with genuine GCP/AWS portability, so workloads could move for cost, resilience, or regulatory reasons.

Second: onboarding a new data feed was a 5-hour manual process — tickets, hand-edited configuration, environment-specific setup, waiting. At enterprise feed volume, that's an engineer's week burned every week, plus every manual step is an incident waiting to happen.

02Approach

• Portability via abstraction boundaries: pipelines target a platform interface layer, not provider APIs directly. Provider-specific adapters live at the edges — microservice and distributed-system patterns (idempotent processing, event-driven handoffs) do the rest.
• Feed onboarding as software, not process: I rebuilt the manual flow into a self-service system (Java, Spring Boot, Kubernetes) — declarative feed definitions validated and provisioned automatically. A ticket queue became an API call.
• Reliability as a design input: at petabyte scale, retry storms and partial failures are the norm; the platform assumes them — checkpointed processing, dead-lettering, and observability wired in from day one.

03Key decisions

• Buy portability with an interface, pay for it once. The abstraction layer costs some indirection up front, but it converts a multi-quarter cloud migration into an adapter swap — and it kept every pipeline testable against a local fake.
• Declarative feed definitions. Making onboarding data-driven (validated config in, provisioned feed out) is what allowed 5 hours of expert steps to compress into 30 automated seconds — and made every feed reproducible.
• Design for the operator, not the demo. Dead-letter queues, checkpoints, and metrics aren't features anyone asks for until 2 a.m. — building them in first is why the platform holds at 25PB.

04Impact

• 25PB+ platform running with genuine cross-cloud portability and enterprise reliability.
• Feed onboarding: 5 hours → 30 seconds (−99%), saving $120K annually and removing a full class of manual-error incidents.
• Delivered with a global cross-functional team across India–US time zones; related work took 1st place at Ford's Data Platform Engineering hackathon.

Architecture shown at the pattern level; internal implementation details are generalized to respect Ford confidentiality. Figures are from my resume.