Executive Summary
AI agents are moving from demos into real operations, and three new papers show where the hard problems now sit: cost, control and security. None of them is about a smarter model. All of them are about whether organizations can deploy agents responsibly at scale.
Cost: Capability Is Not the Same as Efficiency
Running a frontier model on millions of similar items, such as classifying product search results, is expensive. The "Agent in a Bottle" research asks whether an agent can build its own cheaper solution, such as a small trained model or a reusable program. The answer is mixed. Most agent runs fell short of the model's own direct performance, and strong direct results did not predict good "bottling." Yet the best run kept about 82% of the quality at roughly 657 times lower cost. For leaders, the lesson is that the economics of repetitive AI workloads can change dramatically, but you must measure the outcome rather than assume it.
Control: You Cannot Review Everything
The healthcare platform case study describes a production system built by coding agents and governed by a non-engineer. Reviewing every line of code was not realistic, so the operator layered agents that wrote, supervised and reviewed the work. Those safeguards failed in quiet ways: monitors tracked proxies instead of outcomes, audits failed silently, and missing checks vanished from reports. The takeaway is that oversight must stay anchored to the real business objective, with clear permissions and a human making the final call.
Security: Agents Read Untrusted Pages
Web agents act on content written by third parties, which makes them vulnerable to prompt injection, where hidden instructions on a page hijack the agent. AdvSim2Real trains a small agent inside a simulated web, against attackers that keep adapting. The result is an agent that is both more capable and more robust, with a 33.6% relative improvement in task completion against an attacker it had never seen. Security improves when you train against adversaries that evolve, not against a fixed list of known attacks.
What This Means for Your Organization
Together, the papers suggest a practical playbook. First, treat inference cost as a design problem and test whether repetitive workloads can be converted into cheaper artifacts. Second, define outcome-based measures before delegating work to coding agents, and never trust a single layer of automated checks. Third, ask vendors how their web agents were tested against adaptive attackers, not only static examples. Organizations that build these habits early will capture the savings of agentic AI while avoiding the failures that quietly erode trust.
Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
What they did. The researchers define "bottling": an agent's ability to take a general capability and turn it into a cheap, task-specific solution for a large, repetitive workload. They built a benchmark called BOTTLED, in which an agent receives an entire unlabelled workload and must finish it within fixed time, compute and API budgets. The agent chooses its own method, such as training a small model or writing a reusable program. Ten models were tested across three tasks.
Why it matters. Strong direct performance did not reliably predict strong bottling. Of 60 runs, 48 scored below the lower bound of the confidence interval for their model's direct performance, and 31 underperformed a simple baseline that distills into a small model with the same token budget. Models with similar direct scores could end up far apart. Still, the upside is real: on query-product relevance classification, one model kept about 82% of its direct macro-F1 at roughly 657 times lower reported cost, and recovered about 94% of the quality of a specialized cheap-inference model at a quarter of its projected cost.
Business implications. If your organization runs the same kind of AI task thousands or millions of times, such as tagging, routing, moderation or classification, you should not default to calling a frontier model for every item. Ask whether an agent can build a reusable, cheaper component, and benchmark the result against simple alternatives. Do not pick agents for this job on general leaderboard scores alone; test them on your own workload and cost targets.
A Case Study in Assuring AI-Written Software
What they did. The authors document a production healthcare platform built through coding agents and run by an operator with no formal software-engineering training. Over time the workflow grew into a human-led system of meta-agents: one agent wrote code, others supervised and reviewed it, and project rules carried lessons forward from one cycle to the next.
Why it matters. The paper's central point is that exhaustive code review cannot be the sole basis for human control, either because the operator lacks the expertise or because the volume of code exceeds what even experts can inspect. The supervision tools were themselves fallible. Some monitors measured proxies rather than real outcomes, some audits failed silently, missing checks simply dropped out of reported results, and one automated repair caused an operational disruption. Control held up best when the intended outcome, the evidence used to judge it, the agents' permissions and the final human decision were all tied to the same underlying objective.
Business implications. Many organizations will soon have non-engineers building internal tools with coding agents, and this is a candid preview of the risks. Governance should begin with clear statements of the outcome you care about, then work backward to the evidence that would show it. Review whether each monitor measures the outcome or a convenient stand-in, make sure failed or missing checks are loudly visible, limit what each agent is permitted to do, and keep a named human accountable for the final decision. This matters most in regulated settings such as healthcare, where a silent failure carries real consequences.
AdvSim2Real: Training Web Agents Against Adaptive Prompt Injection in a Web World Model
What they did. Web agents must read pages written by third parties, so a planted instruction can redirect them away from the user's goal. Existing defenses train on injections fixed in advance, which attackers can bypass by adapting. AdvSim2Real instead co-evolves three components inside a frozen simulated web: a task curriculum rewarded for tasks the agent solves about half the time, an adversary rewarded only when its injection turns a judged success into a failure, and the agent itself.
Why it matters. The approach produced a 4B-parameter agent that became both more capable and more robust. Its task completion rose with and without attacks, held up against a frontier-model adversary it never trained against, and the capability gains carried over to a real browser. On 150 web tasks, completion under the unseen adversary improved by 33.6% relative to the base agent. The design also solves a training problem: keeping tasks at the edge of the agent's ability means they continue to teach.
Business implications. Prompt injection is one of the most pressing security concerns for any agent that browses, reads email or processes outside documents. A defense tested only against known attack strings offers limited comfort. When evaluating vendors or building in-house, ask whether agents were trained and tested against adversaries that adapt, whether results hold against attackers not seen in training, and whether gains transfer to real environments. Smaller, well-trained agents may also offer a cost-effective path for secure automation.
Key Takeaways
• Test whether your highest-volume AI workloads can be converted into cheaper task-specific solutions, since the best case in one study kept about 82% of quality at roughly 657 times lower cost.
• Do not assume a model that scores well on direct tasks will be good at building its own cheap solution, because the two abilities diverged in most runs tested.
• Compare any agent-built shortcut against simple baselines, such as distilling into a small model, before adopting it, since many agent runs failed to beat them.
• When coding agents produce more code than anyone can review, anchor oversight to the business outcome, the evidence, the permissions and the final human decision.
• Audit your monitors and automated checks for proxy metrics and silent failures, because a missing check can disappear from reports without anyone noticing.
• Ask vendors how their web agents were tested against attackers that adapt, not only against fixed lists of known prompt injections.
• Simulated environments can make agents both more capable and more secure, so consider simulation based training and testing before exposing agents to the live web.