Zillow’s Toby Roberts says AI ROI only holds when you measure first, not after

Zillow's SVP of Engineering Toby Roberts made a blunt point at VB Transform 2026: AI ROI numbers only hold up if you measure before you build, not after. In other words, if you wait until the AI rollout is already underway to set your benchmarks, you cannot credibly separate “AI caused this” from “we just happened to change a bunch of other things at the same time.” Roberts tied Zillow’s attribution of a 40% increase in shipped code to AI adoption to a DORA metrics baseline that the team put in place years earlier.

Why open with this? Because too many enterprise AI stories start with results and work backward. Zillow’s framing is the opposite. Start with the measurement backbone first, then run the AI program in a way you can defend to finance, the board, and auditors. Roberts’ point is practical, not theoretical: the ability to show cause and effect becomes part of the AI product itself. If your measurement plan is an afterthought, your AI ROI story becomes a vibes-based deck.

The deeper challenge Zillow described is that real estate is not one conversation. Customers move through multiple steps: from a phone screen to a loan officer to a real estate agent. Sometimes this happens over months or years. In that reality, a single chatbot cannot carry the thread, because the “thread” is not just text. It is the evolving state of where a customer is in a journey, what’s been discussed, what’s been decided, and what needs to happen next when a different professional picks up the case.

Roberts said Zillow identified early that it needed a persistent context layer that could meet customers and professionals wherever they were. Data foundation sounded like the obvious place to start, and Zillow did begin with it. Roberts described a large push to make sure the data had the right foundation: a data mesh approach, clear data lineage, and a governance structure with permissions and identity attached to the data itself. None of that, he argued, was the hardest part.

The hard part was building the system that remembered. Not “remembered” like a marketing slogan, but like an engineering requirement: the context layer had to live so it could support customers at any point in their journey, regardless of which surface they showed up on next. Roberts also said Zillow chose to own that layer itself rather than relying on a single external chat interface. The team’s decision, in his telling, came quickly once they looked at the shape of a real transaction rather than a single conversation.

To make it work, Zillow built its own harness rather than routing work through a single model API. Roberts said the approach drew on 20 years of machine learning history behind products like Zestimate, and leaned into smaller, task-specific fine-tuned models instead of one general-purpose model. That harness runs alongside Glean internally. Jain added that Zillow now has thousands of Glean agents in production, handling repetitive tasks with tens of thousands of executions across the company.

Here’s where Glean’s role becomes more than a vendor pitch. Jain argued that enterprises pay a hidden tax when integration work is duplicated across teams like finance, legal, and marketing. Instead of letting each group rebuild its own connections, Glean’s pitch is centralizing that integration once through the Glean MCP gateway. Jain also described two mechanisms meant to control cost: model routing, which sends most tasks to smaller, cheaper models instead of defaulting to “frontier” models, and precomputed context, which avoids an agent burning tokens to assemble its own context from scratch.

Jain’s comments on token burn were especially direct. He said Claude is “also very slow” because the first part of assembling context “actually takes forever,” and that routing that request through Glean can cut token consumption by as much as half. The larger enterprise lesson is that context is not just a capability, it is a cost driver. In the session’s framing, “models by themselves are not enough to bring automation with AI inside your enterprise.” You still have to connect automation to enterprise context.

Zillow’s compliance stance added another layer of realism. Jain and Roberts did not present permission-aware architecture as a single magic switch. Even with permissions-aware context in place, Zillow layered hard rules and a standing compliance check for its most sensitive categories, rather than trusting the architecture to handle it automatically. That matters because “permission inheritance” sounds neat in a slide, but regulated data tends to punish neatness.

Taken together, the message for builders and board members is clear. If you want agentic AI that can move across real workflows, context is the harder engineering problem than raw data. And if you want defensible AI ROI, your baseline has to exist before the AI rollout begins. Enterprises that skip either part will still ship pilots, but they will struggle to prove they are shipping value.