Agentic BigQuery Optimizer

01The problem

A data platform at 25PB+ scale accumulates cost pathologies faster than humans can review them: over-provisioned slot reservations, queries scanning far more data than they return, tables partitioned for access patterns that changed a year ago. Each individual issue is findable by a skilled engineer with time; the problem is that there are thousands of them, they regenerate constantly, and expert time is the scarcest resource on the team.

The insight behind this project: warehouse optimization is fundamentally a reasoning task over structured telemetry — exactly the shape of problem LLM agents are good at, if you ground them in real data and constrain what they're allowed to do.

02Approach

I built an agentic system where LLM agents consume workload metadata — query logs, slot utilization, table/partition statistics — and produce concrete, reviewable optimization actions:

• Analysis agents scan workload telemetry for waste patterns: inefficient query shapes, mis-sized reservations, cold data on hot storage, missing partitioning or clustering.
• A recommendation layer converts findings into ranked, costed actions — each with a projected savings estimate, so impact is measurable before anything changes.
• Guardrails first: agents propose; changes flow through review and safe application paths. An agent that can silently rewrite production workloads is a liability, not an optimizer.
• Measurement built in: every applied optimization is tracked against its projection — the $2.4M figure is a bottom-up sum of measured and projected per-action savings, not a vibe.

03Key decisions

• Ground agents in telemetry, not prompts about telemetry. The agents work from actual workload metadata pulled from the warehouse itself — hallucinating an optimization is structurally difficult when every claim must reference a real query or reservation.
• Costed recommendations over autonomous action. Attaching a dollar projection to every action made the system legible to leadership and safe to adopt incrementally — trust was earned action by action.
• Optimize the platform, not one query. Query-level tuning is a local maximum; the agents also reason about reservations, storage tiering, and partitioning strategy — where the structural savings live.

04Impact

• $2.4M projected annual savings through automated resource allocation and query optimization.
• Cost review transformed from a periodic expert-driven exercise into a continuous automated loop.
• Recognized as Top AI Innovator at Ford (2025); related work took 1st place at Ford's Data Platform Engineering hackathon.

Architecture shown at the pattern level; internal implementation details are generalized to respect Ford confidentiality. Figures are from my resume and internal projections.