The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO
Neutral Harnesses Prevent Model Lock-In
Chase argues that developers need model-neutral agent harnesses because running agents accumulates valuable state, context, and memory outside the model weights. Tying that context to a single model provider's proprietary harness locks a company in, preventing them from switching to newer frontier models or defending against price hikes.
Domain Experts Capture Value via Vertical Agents
When asked who wins if agent value comes from output verification, Chase contends that domain experts capture the value because they alone can define and quantify what good performance looks like in their field. He advises vertical agent builders to start with high-ROI context and harness engineering before attempting model post-training once they have substantial usage.
Smart Model Routing Belongs in the Harness
Chase distinguishes between API gateways like OpenRouter, which he estimates are used primarily for unified model access, and true dynamic model routing. While gateway-level routing allows drop-in model swapping for closed tools like Claude Code or horizontal chatbots, he argues that vertical agents require harness-level routing because the harness holds full state awareness, prompt cache status, and model-specific prompt strategies.
Why Agent Memory Remains Unsolved
Examining the data and memory layer, Chase argues that traditional databases like ClickHouse are not enough because agent memory requires an intelligent system around the database to distill raw user interactions into useful context. He notes that LLMs currently struggle with meta-reflection, though he expects labs to improve this over the next year.
How LangChain Cut Its 30% Coding Agent Spend
Chase reports that while annual developer surveys still rank accuracy as the top priority—nearly double latency, cost, and security—enterprise concern over token cost is rising rapidly due to multi-agent loops. At LangChain, coding agent expenditure reached roughly 30% of engineering payroll, including an incident where a frontend DOM-snapshot plugin accidentally burned $40,000 in a single week.
Massive Inference Demand and AI Investing Rules
Chase admits the two markets he underestimated most since founding LangChain are inference and coding agents, arguing that overall inference demand is so vast that providers like Fireworks, Crusoe, and AWS can all succeed simultaneously. Stepping into a hypothetical venture role, he says he would invest in inference, low-level infrastructure primitives like Modal and Temporal, and unglamorous vertical agents with deep domain expertise.
