← Back to Insights

Insight

Your Fork Has a Half-Life

Ariel Agor
•
A custom model starts decaying against the frontier once you freeze it. It lost because it was designed to be finished.

Listen · Read by Ariel · click any word to jump

0:00 / n/a
· loading…

On October 29, 2026, Mistral published the open weights for Mistral Large 4, a trillion-parameter model the company had put into public preview earlier in the month. The model is good. The detail I care about sits further down the announcement. Between the preview and the open release, Mistral kept running reinforcement learning for about three weeks, and its score on Terminal-Bench 4.0 rose 13 points to 41%, with 30% fewer output tokens. Mistral's own phrase for the trend is "Gains continue to accelerate, with no signs of saturation."

Picture a team that downloaded the preview on the first day, built a fine-tuning pipeline around it and shipped a custom model to its claims department. That team has spent the month building on a base that the vendor has already left behind. Nobody made a mistake. The ground moved.

Most companies frame custom AI versus off-the-shelf tools as a one-time purchase decision: build the thing, or buy the thing. That framing made sense when software sat still between releases. It fails for models, because a model you customize starts decaying against the frontier the moment you freeze it. Every fork has a half-life. In my view the real build-versus-buy question is who pays to re-fork, how often, and whether your organization can do it without starting from zero each time.

Two trillion-class releases in fourteen days

Look at the calendar. On October 15, 2026, Meta released Muse Spark Open. It is an 816-billion-parameter mixture-of-experts model with 42 billion active parameters, a one-million-token context window and native multimodal input. Meta called it "the most capable US open-weight model to date." The weights are on Hugging Face, Amazon SageMaker AI, Google Cloud Model Garden and Microsoft Foundry Models. Meta's developer post says you can customize it with PyTorch and TorchTitan. Meta also says it runs on two Apple M5 Ultra Mac Studios using MLX, which it describes as "the first model in its class to run on your desktop."

Fourteen days later, Mistral's weights landed. You can self-host Large 4 with vLLM or SGLang, run it through NVIDIA's NIM container, deploy it privately on Azure through Microsoft Foundry, or call it on Mistral's API at $0.70 per million input tokens and $2.10 per million output tokens. The licence is a modified MIT licence. Mistral says you can use, modify, fine-tune and commercialize the model freely. The one condition is that a business selling the model itself as a hosted service needs a separate commercial licence from Mistral. Pierre Stock, Mistral's VP of Science, told TechCrunch that open weights mean access "can't be cut off by definition."

That quote is the open-weight pitch in one line, and it works. If a closed provider changes its terms, raises prices or retires a model, the weights on your own servers keep running. For a bank, a defence contractor or a hospital network, that resilience is worth a lot.

Running and staying current are separate problems, though. Weights that can't be cut off can still be overtaken. Mistral's own release notes show that: the open weights beat the preview weights by a wide margin after three more weeks of training. Whatever you build on Large 4 will meet a Large 4.1, a Muse Spark Open successor, or a new Kimi or DeepSeek checkpoint well before your finance team has finished amortizing the project.

What three weeks of training buys

The numbers matter here, so I'll stay with Mistral's figures. Reinforcement learning took the model's BrowseComp score from 75% at the supervised fine-tuning checkpoint to 84.5%. On Mistral's B3 agent security benchmark, attack resistance went from 93.3% in the preview to 97.1% in the open release. On the Artificial Analysis Coding Agent Index, Mistral reports 48%, roughly level with GLM-5.3, while using 47% fewer tokens per task.

Those are vendor numbers, and you should read them as vendor numbers. Even so, the direction holds across the industry, and the size of the move is what matters for strategy. The base model gained a double-digit improvement on a hard agentic benchmark in the time a typical enterprise needs to approve a statement of work.

Now think about what a custom fine-tune contains. It carries your data, your labels, your tone and your edge cases, layered onto a particular set of base weights. When the base improves, your layer does not move with it. In most real deployments you cannot lift a fine-tune off one base and drop it onto the next. LoRA adapters are tied to the architecture and weights they were trained against. A distilled small model inherits the ceiling of the teacher it learned from. A carefully tuned prompt chain was tuned against one model's habits and quirks.

So the custom model gets a little worse every week. Its absolute quality stays the same, but the thing next to it keeps improving. Suppose your fine-tuned claims model beat the off-the-shelf base by eight points on your internal evaluation the day you shipped it. If the next base checkpoint gains ten points on the general skills your task depends on, your custom model now trails a model you could have had for nothing. That arithmetic is invented for illustration, but Mistral's 13-point move in three weeks makes it a plausible case.

The buy side already voted

Enterprises have noticed, even if most haven't put it this way. Menlo Ventures published its State of Generative AI in the Enterprise report on December 9, 2025. Based on a survey of 495 US enterprise AI decision-makers, it found that 76% of AI use cases were purchased rather than built internally. In 2024 the split had been 47% built and 53% purchased. The same report put enterprise spending on generative AI at $37 billion in 2025, up from $11.5 billion in 2024.

The usual reading of that swing is that companies found building hard and went shopping. I read it differently. Buying an off-the-shelf tool means renting someone else's re-fork schedule. When the vendor swaps the model underneath, the customer gets the improvement without retraining anything. The 76% are paying, partly, for freshness they never have to manage.

That's a sensible trade, but it has a known price. The vendor's upgrade cadence serves the vendor's average customer. Your edge cases, your regulatory constraints and your vocabulary get whatever attention the average customer's needs leave over. When the vendor upgrades, your behaviour changes on their timetable, and sometimes it gets worse for you while it improves for everyone else. You find out from your support queue.

The October open-weight releases put pressure on that trade from the other side. With an 816-billion-parameter model that runs on two desktop machines, and a trillion-parameter model whose quantized version Mistral says fits on four Nvidia B200 or B300 GPUs, owning the model is no longer a fantasy for mid-sized companies. Owning it is cheap. Keeping it current is the expensive part.

Custom AI versus off-the-shelf tools, measured in half-lives

Here is the frame I now use with clients. Every AI capability in your company has a half-life: the time until the gap between your deployed version and the best available alternative grows large enough to cost you money. Off-the-shelf tools have short half-lives that someone else pays to reset. Custom models have half-lives you pay to reset yourself. The strategic question is what it costs you to reset the clock.

The snapshot trap

Most custom AI projects I see are built as snapshots. A team picks a base model, collects a dataset, fine-tunes, evaluates once, ships and celebrates. The project closes. The budget line ends. The people move on to the next initiative.

Six months later the base model is two generations old, and nobody remembers which data cleaning decisions went into the training set. The evaluation suite was a spreadsheet one analyst built for the launch review. When someone proposes upgrading to the new base, the honest answer is that the company would have to start over. So it doesn't upgrade. The custom model hardens into legacy software with a shelf life measured in quarters, and the off-the-shelf competitor that renews itself every few weeks passes it.

That is the hidden reason so many custom builds lose to bought tools. The custom version usually started out better. It lost because it was designed to be finished.

The rebase cost

Software engineers have a word for moving your changes onto a newer version of the code underneath: rebasing. A good engineering culture keeps rebases cheap. Changes stay small and isolated, tests run automatically, and nobody is surprised by what breaks.

Custom AI needs the same discipline, and almost nobody budgets for it. Say it takes your team eight weeks to rebuild a fine-tune from scratch on a new base: two weeks hunting for the training data, three weeks rebuilding evaluation, two weeks tuning, one week of review. If the frontier moves meaningfully every month, you will never catch up. Suppose instead the same rebase takes four days because the data is versioned, the evaluation runs in a pipeline and the tuning recipe is a script. Then a new base model is a Thursday task, and every gain the labs make arrives at your company within a week.

The difference between those two companies has nothing to do with model choice. It comes down to whether they treated customization as a product or as a process.

What you actually own

If the weights you fine-tune will be stale within a quarter, what is the durable asset in a custom AI program? I count four things, and each outlasts any particular model.

The evaluation suite

Your evaluation suite is the most valuable AI asset most companies don't have. I mean a versioned, automated set of test cases built from real work, scored the way your business scores outcomes. With it, every new open-weight release and every vendor upgrade becomes a measurement: run the suite, compare, decide. Without it, every model decision becomes an argument between whoever saw the most impressive demo and whoever is most afraid of change.

Mistral reported 13 points of improvement on Terminal-Bench. You should be able to say, within a day of any release, how many points the new model gains or loses on your work. Public benchmarks tell you where to look. Your own suite tells you what to buy.

The data pipeline

Your training and retrieval data should be reproducible from source. The same goes for labels, filters and exclusions. When a new base arrives, the data step should be a rerun. Teams that hand-assembled their first dataset in a notebook pay the full assembly cost again on every rebase, which is usually the moment they stop rebasing.

The tuning recipe

The specific weights you produce are disposable. The recipe that produced them is the asset: hyperparameters, data mixtures, reward definitions, the sequence of supervised and reinforcement steps. Meta points developers to TorchTitan for customizing Muse Spark Open. Mistral offers Forge for tailoring Large 4 to proprietary data and guardrails, and NeMo AutoModel through NVIDIA. Whichever tooling you pick, write the recipe so a new base model is a parameter rather than a rewrite.

The behavioural contract

This is the least technical piece and the one most often missing. It is a written statement of how your AI must behave: what it refuses, what it escalates, what tone it takes with an angry customer, which regulatory lines it never crosses. Mistral reports a 96.6% average refusal rate on malicious cyber requests for Large 4, and Meta says Muse Spark Open got extra safety post-training because it ships as open weights. Those are the vendors' policies. Your policy has to survive every swap of the model underneath, which means it has to live outside the model, as tests in the evaluation suite and rules in the orchestration layer.

The licence clause that tells you where the money is

Mistral's licence is free for almost everyone, with one exception: companies that offer the model as a hosted service need a commercial licence. Meta's Muse Model License comes "with limited commercial terms." Both companies are giving away the weights and holding on to the business of running them for others.

That tells you where they think the value sits. The weights are a loss leader. The money is in serving, updating and supporting the model over time. Mistral says it can send Applied AI experts to work with customer teams, and it gave Orange Cyberdefense, a design partner, preview access to a specialized cybersecurity variant under vetted, contractually restricted terms. The labs know the hard part comes after the download.

For an operator, that suggests a split. Treat the base weights as a commodity that you expect to replace. Treat the evaluation suite, data pipeline, recipe and behavioural contract as owned infrastructure that you expect to keep for years. Then the build-or-buy decision becomes a series of smaller decisions about which layer to rent and which to own.

A rule set for forks that survive

I'll be concrete, because frameworks without rules turn into slide decks.

First, never approve a custom AI project that lacks a rebase plan. The project charter should state how the team will move to the next base model and how long that should take. If the answer is "we'll figure it out," the project is a snapshot, and you should price it as one.

Second, build the evaluation suite before the model. A team that cannot measure an off-the-shelf tool against your work cannot tell you whether its custom version beats one. Make the suite the first deliverable. It pays for itself the first time it stops you from buying the wrong vendor.

Third, keep customization as thin as the job allows. Retrieval and well-structured context move to a new base with almost no work. Prompt and tool scaffolding move with modest work. Full fine-tunes move with the most. Reach for heavy customization only where a measured gap justifies the rebase cost you are signing up for.

Fourth, set a re-fork cadence and fund it as an operating cost. Tie it to your own evaluation results rather than release headlines. When a candidate base beats your deployed model on your suite by a margin you defined in advance, the rebase starts. Funding it as ongoing operations, rather than as a fresh project each time, keeps it from dying in budget review.

Fifth, keep an off-the-shelf benchmark running permanently. Even if you own your stack, score the best bought tool against your suite every month. It is your control group. The day it beats your custom model and stays ahead, retire the custom model without sentiment.

What the October releases change for operators

Before this month, I could tell many mid-sized companies that serious custom AI meant taking on infrastructure they couldn't support. October 15 and October 29 weakened that argument. A capable model running on two desktop machines, and a trillion-parameter model with a quantized build for four GPUs and a licence that permits commercial fine-tuning, puts owning a frontier-adjacent model within reach of a regional insurer or a logistics firm with a decent engineering team.

The barrier has moved. Getting the model is easy. Keeping it current costs real time and attention. Companies that read the October news as a reason to download and fine-tune will produce a crop of snapshots that look impressive at the board meeting in December and fall behind by spring. Companies that read it as a reason to build rebase machinery will gain an option the 76% who buy everything lack: they can absorb every gain the labs produce, on their own schedule, shaped to their own work.

The vendors understand this, which is why they give the weights away and charge for continuity. When the people selling you something make the object free and charge for the upkeep, the upkeep is where the value is.

Architect the re-fork, or rent someone else's

Buying a tool is easy and downloading weights is easy. Neither one is a strategy. The strategic work is designing the company's ability to change models quickly, cheaply and safely, again and again, as the frontier keeps moving. That means an evaluation suite grounded in your actual work, a data pipeline you can rerun, a tuning recipe that treats the base model as a variable, and a behavioural contract that outlasts any single set of weights. It also means deciding, layer by layer, where your company should own the cadence and where it should rent someone else's.

This is architecture work. It crosses engineering, operations, legal, risk and finance, and no vendor will do it for you, because every vendor's incentive is to be the base you never leave. Agor AI Advisory does this work with leadership teams: we map your AI capabilities by half-life, decide which layers to own, build the evaluation and rebase machinery, and set the re-fork cadence that keeps a custom investment ahead of the off-the-shelf alternative. The next base model is probably already in training somewhere, and when it ships you will either be ready to adopt it quickly or stuck on the version you have now.

Sources

Want this kind of automation working for your business?

Agor AI designs and ships the systems these posts describe, scoped in weeks, not quarters.

Book a Free Strategy Call