This Week: The Hidden Costs of Long-Running AI Systems
Three papers this week converge on a single executive concern: what happens to AI systems over time, and how do you keep them trustworthy as they age? The findings are practical, uncomfortable, and directly relevant to any organization that has moved past AI pilots into sustained deployment.
The Prompt Bloat Problem Is Real and Measurable
If your engineering teams use coding agents, they almost certainly maintain a configuration file — often called CLAUDE.md, AGENTS.md, or something similar — that tells the agent how to behave in your codebase. A new study of 1,867 real repositories and nearly 250,000 instruction lifetimes shows these files grow without bound, tripling in size over their lifetime and gaining almost five net instructions per commit. The mechanism is asymmetric: adding a rule is cheap, but once the rationale for a rule is forgotten, safely deleting it requires exhaustive testing. The authors call this catastrophic remembering — the mirror image of the catastrophic forgetting problem researchers already know about. The fix they propose is deceptively simple: add comments that explain why each instruction exists. In controlled tests, this removed 99.3% of unnecessary bloat and improved real-world instruction-following by up to 23.1%.
Verifiable Honesty Moves From Aspiration to Theory
Regulators, boards, and enterprise customers increasingly demand that AI systems be honest — meaning their stated confidence in predictions actually corresponds to reality. A new result co-authored by Turing laureate Yoshua Bengio establishes the complexity-theoretic foundations for verifying, in polynomial time, whether a probabilistic AI predictor is internally self-consistent. This is a foundational step, not a shipping product, but it opens a path toward audit tools that can catch models contradicting themselves across related questions — a failure mode that current evaluation frameworks largely miss.
AI as Genuine Research Collaborator
A team of mathematicians and computer scientists used an AI research system to tighten the best-known bounds on the Grothendieck constant, a well-studied problem at the boundary of combinatorial and continuous optimization. The improvements were judged genuinely novel by domain experts. The paper is candid about what worked and what didn't, and the meta-lesson is important for business leaders: extracting breakthrough work from AI requires deliberately engineered conditions — good problem framing, sustained iteration, and expert review — not just larger models. This is a template for how R&D functions should be organizing their AI investments.
Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding
What they did: Chakrabarti analyzed 247,694 instruction lifetimes across 1,867 real repositories that use agentic coding configuration files. The data shows these files grow by 226% over their lifetime, gain 4.9 net instructions per commit, and rarely shrink — the older an instruction gets, the less likely anyone is to remove it. The author formalizes the underlying dynamic as catastrophic remembering: appending an instruction is O(1), but deleting one safely requires reasoning about interactions with every other instruction, which is exponential. The proposed remedy is to add structured comments that encode the reasoning behind each instruction, so future maintainers can safely prune. In inverted IFEval benchmarks, comments removed 99.3% of excess instructions; in real-world WildIFEval settings, they improved instruction-following by up to 23.1%.
Why it matters for business: Prompt configuration files are load-bearing infrastructure for every organization running coding agents, customer service agents, or any long-lived AI system. This paper is the first to quantify the drift problem and offer a mechanically simple fix. Practically: mandate that every instruction added to a system prompt carries a comment explaining its rationale, and schedule quarterly prompt reviews the same way you schedule dependency audits. Without this discipline, your agent's behavior becomes governed by a document nobody fully understands, which is a governance and reliability risk that compounds silently.
How to Verify Consistency of Probabilistic Claims
What they did: Paradise, Richardson, Bengio, and Goldwasser tackle a foundational question in AI safety: if an AI model makes many probabilistic predictions, can we verify in reasonable time that those predictions don't contradict each other? They construct an interactive proof system in which a polynomial-time verifier, given a model and a proof oracle, can check the approximate consistency of exponentially many implied probabilistic claims by evaluating the model at only a few points. The result places approximate probabilistic consistency in a well-understood complexity class and provides the theoretical foundation for building practical consistency-checking tools.
Why it matters for business: Current AI evaluation frameworks measure output quality — did the model give a good answer? — but rarely measure internal coherence, which is where subtle safety failures hide. A model that says an outcome is 90% likely in one context and 10% likely in a semantically identical context is unreliable in ways users won't notice until it matters. As the EU AI Act and comparable regimes push honesty and transparency requirements into procurement, boards will need audit primitives beyond user-satisfaction metrics. This paper is upstream of the tooling that will make those audits tractable. Watch for practical implementations to appear over the next 12–24 months.
Long-Horizon AI Research for the Grothendieck Constant
What they did: The team used an AI research system to improve both the upper and lower bounds on the Grothendieck constant, a mathematical quantity that captures the hardness of relating combinatorial problems to their continuous relaxations. The improvements were validated by domain experts as genuinely novel — not paraphrases of known techniques. Just as valuable as the mathematical result, the paper is a candid retrospective on what worked: careful problem framing, sustained multi-turn collaboration, and human review at critical junctions. It is equally clear about weaknesses, including the AI's tendency to produce plausible-but-wrong intermediate steps that required expert filtering.
Why it matters for business: Most enterprise AI conversations are still stuck on autocomplete-style use cases — faster code, faster emails. This paper is a proof point for a fundamentally different mode: AI as a collaborator on genuinely hard, long-horizon problems. The lesson for R&D leaders is that this capability is real but conditional. You don't get breakthrough insights by handing a model a vague brief; you get them by engineering the collaboration — structuring the problem, iterating with discipline, and keeping domain experts in the loop as filters. Organizations that build this operating model early will compound an advantage that pure-tools access cannot replicate.
Key Takeaways
• Agentic coding prompts grow by 226% over their lifetime because adding instructions is cheap but deleting them safely is exponentially expensive — treat prompt bloat as a first-class operational risk, not a housekeeping chore.
• The older an instruction gets, the less likely anyone will ever delete it (log-hazard -0.032 per commit) — schedule mandatory prompt rewrites the same way you schedule dependency upgrades.
• Adding structured comments that encode the *why* behind each instruction eliminates 99.3% of prompt bloat in controlled settings and improves real-world instruction-following by up to 23.1% — English needs comments, just like code does.
• New complexity-theoretic results show it is possible to verify in polynomial time whether an AI's probabilistic predictions are internally consistent — a foundation for the honesty guarantees regulators are starting to require.
• Self-consistency checks give boards a concrete audit primitive that goes beyond output-quality evals: you can now test whether a model contradicts itself across related predictions, not just whether it sounds confident.
• A team of humans and AI agents tightened the best-known bounds on the Grothendieck constant — a decades-old open problem — proving AI can produce insights domain experts recognize as novel, not just accelerate known workflows.
• The Grothendieck case study shows breakthrough AI collaboration requires deliberately engineered conditions — problem framing, iteration cadence, and expert-in-the-loop review — not just better models.