Part IV specified what an AI system may do. This piece asks whether it should be there at all. The mathematics of establishing that a strategy has skill is settled, old, and almost universally ignored — and in April 2026 the framework every firm cited for model governance was rescinded and replaced by one that excludes generative AI by name. Validation just stopped being a compliance exercise. It is now purely a question of whether you want to know.
For fifteen years, any conversation about model governance in a US financial institution ended in the same place: SR 11-7. The 2011 interagency guidance on model risk management was the document everyone cited, the framework every AI governance policy was written against, and the reason "independent validation" meant something specific rather than whatever a firm wanted it to mean.
On 17 April 2026 the OCC, Federal Reserve and FDIC rescinded it. The replacement makes four moves, and every one of them reduces the constraint on an AI deployment.
"Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."OCC Bulletin 2026-13, "Model Risk Management: Revised Guidance," 17 April 2026
That is the first move, and it is dispositive. The second is a scoping statement: the guidance is "expected to be most relevant to banking organizations with over $30 billion in total assets" — language the 2011 OCC and Federal Reserve guidance did not contain at all. Be precise here, because it is widely misreported: this is a relevance statement rather than an applicability threshold, smaller institutions with significant model risk may still be expected to apply it, and the only prior asset threshold anywhere in this lineage was the FDIC's $1bn, adopted in 2017. The third move: the guidance states plainly that it "does not set forth enforceable standards or prescriptive requirements; accordingly, non-compliance with this guidance will not result in supervisory criticism against a banking organization." The fourth: validation independence was relaxed from SR 11-7's detailed separation-of-duties requirements to a principle that "the quality of validation process depends on the rigor and effectiveness of the review rather than on organizational structure."
The rescission is also broader than the headline suggests. Alongside SR 11-7 and OCC 2011-12 it withdraws the 1997 credit-scoring guidance, the 2021 BSA/AML model guidance, and the Model Risk Management booklet of the Comptroller's Handbook. And the agencies have said they plan to issue a request for information covering banks' use of AI — announced in April, not issued as of this writing.
Any firm whose AI governance policy currently says it applies SR 11-7 principles to its AI models is citing a rescinded document for a use case its successor explicitly disclaims. That is worth checking this week; it is a sentence that appears in a great many policies.
This does not create an unregulated zone. It fragments oversight — across consumer protection law, adverse-action requirements, NYDFS Part 500, third-party risk management, SEC anti-fraud authority, and a growing patchwork of state regimes. What it removes is the single consolidating framework that firms were using to organise the problem, and it removes it precisely for the technology that most needs organising.
So the question this piece answers is no longer "what does the regulator require." Nobody requires this. The question is narrower and more uncomfortable: if you wanted to know whether your model actually works, what would you have to do?
The answer divides cleanly. For anything that produces a track record — a strategy, a signal, an allocation model — the mathematics has been settled since 2014 and is almost universally ignored. For anything built on a language model, the methods are immature, the tooling is worse than it looks, and the honest answer is that nobody has solved it. We take them in that order.
Below, a specified number of strategy variants are backtested against synthetic return data. You control how many variants are tried, how much history they are tried on, and — critically — whether the underlying process contains any genuine skill at all. The default is zero skill: the returns are pure noise, and every strategy is worthless by construction.
The engine selects the best performer in sample, exactly as a researcher would, and then reports how that selected strategy does out of sample. It also computes what the best Sharpe ratio should be under the null hypothesis of no skill, which is the number that makes the whole exercise tractable.
Set true skill to zero and drag the number of variants from ten to a thousand. The best in-sample Sharpe climbs steadily — past 1.0, past 1.5 — while out of sample it wanders around zero. Nothing has been discovered. The search itself manufactured the number.
Then set true skill to something real, say 0.6, and run it again. The in-sample Sharpe is inflated well above 0.6 by the same selection effect, but out of sample the strategy delivers roughly what the process actually contains. Selection bias does not destroy real skill; it obscures it, by making skill and noise produce identical-looking backtests. That is precisely why the deflated Sharpe ratio exists — it is the only number on that panel that distinguishes the two cases.
The correlation control is the one worth pushing deliberately, because it is the honest defence against "we tried ten thousand configurations." Ten thousand re-parameterisations of one idea is not ten thousand independent chances, and the deflation ought to reflect that. Push it above 0.5 and watch what happens to the deflated Sharpe: as the effective trial count collapses, the no-skill benchmark falls and the deflation becomes less protective — precisely when the research process was least diverse. That is not a bug in the simulator; it is the framework's weak point, and it is why the out-of-sample column stays on screen.
One result this simulator deliberately does not claim to show. The literature reports that overfitting on a process with memory — an autoregressive or mean-reverting series — produces not merely zero expected out-of-sample performance but persistent losses, because the fitted strategy has learned the sample's specific phase. That is a real and important finding. Demonstrating it requires strategies whose parameters are actually fitted to the series, which these are not: here each variant is an independent return stream, so there is no phase for it to learn. We mention the result because it matters and attribute it to the source rather than staging a demonstration that would not be one.
Three formulas do the work. They are twelve years old, they are not controversial, and implementing them takes an afternoon.
If you run N independent trials on a process with no skill, the maximum Sharpe ratio you observe is not zero. It is a predictable positive number, because the maximum of N draws from a zero-mean distribution rises with N. The False Strategy Theorem gives it:
The growth is logarithmic, bounded above by √(2·ln N), which is the single most useful intuition in this area. Going from 10 trials to 10,000 — three orders of magnitude more searching — only moves the expected maximum from about 1.57 to about 3.86. Overfitting is cheap at the start and gets expensive fast.
| Independent trials | Expected max Sharpe | Years of data needed for a credible SR of 1 |
|---|---|---|
| 7 | 1.39 | 1.9 |
| 10 | 1.57 | 2.5 |
| 45 | 2.24 | 5.0 |
| 100 | 2.53 | 6.4 |
| 1,000 | 3.26 | 10.6 |
| 10,000 | 3.86 | 14.9 |
Read the last column as a constraint rather than a curiosity. The published claim is stark: "if only five years of data are available, no more than forty-five independent model configurations should be tried." Forty-five configurations is not a research programme; it is a morning. And the authors are blunt about the consequence: "Any perseverant researcher will always be able to find a backtest with a desired Sharpe ratio regardless of the sample length requested."
One nuance the popular version of this result loses. Manufacturing a spurious Sharpe of 1.0 on five years is trivially cheap at 45 trials. Manufacturing a spurious 2.0 on the same data takes on the order of a hundred thousand genuinely independent trials. The cost scales brutally, so a very high reported Sharpe on a short sample is not automatically evidence of overfitting — it may be evidence of something else entirely, including a data error.
The deflated Sharpe ratio asks the only question that matters: given that this strategy was selected as the best of N attempts, what is the probability its true skill exceeds zero?
Two implementation details determine whether your answer is right, and both are easy to get wrong silently.
Everything must be in the raw sampling frequency. If your returns are daily, the Sharpe in that formula is the daily Sharpe. Feeding it an annualised Sharpe does not produce a slightly wrong answer — in the paper's own worked example it turns a DSR of 0.89 into exactly 1.00, converting a marginal result into apparent certainty. It is the single most destructive bug in this area.
γ₄ is raw kurtosis, not excess kurtosis. A Gaussian has γ₄ = 3, not 0. The most widely used statistics library in Python returns excess kurtosis by default. The error is small at low Sharpe and grows with the square of it, so a passing test on a modest strategy will not protect you on a high-frequency one.
The Probability of Backtest Overfitting takes a different angle: instead of deflating one number, it asks whether your selection process carries any information at all. Split the sample into an even number of blocks — sixteen is the recommended default, giving 12,870 combinations — and for every way of splitting those blocks into two halves, pick the best strategy in one half and record where it ranks in the other. PBO is simply the fraction of splits where the in-sample winner landed below the out-of-sample median.
PBO near 0.5 means your selection is a coin flip. Above 0.5 means it is actively anti-predictive: choosing the in-sample best is worse than choosing at random. It is worth stating what PBO does not tell you — it measures selection quality, not profitability, and a family of uniformly terrible strategies can post an excellent PBO because the ranking is stable.
Combinatorial purged cross-validation extends the idea into a full backtesting method, and its real contribution is that it produces many backtest paths instead of one. Walk-forward gives you a single Sharpe with no sampling distribution, which means you cannot compute a variance, cannot compute PBO, and cannot parameterise a DSR. CPCV gives you a distribution.
It also adds two operations that walk-forward and ordinary cross-validation both lack, and which matter more than the combinatorics:
Purging removes from the training set any observation whose label was formed over a time interval overlapping a test observation's label interval. Without it, a training label and a test label can be functions of the same future returns — the model is literally trained on the answer. This must be applied on both sides of the test fold, because training data after the test period leaks backwards through its own lookback window.
Embargoing drops an additional block of observations immediately after each test fold, typically around 1% of the sample. Purging handles deterministic label overlap; embargoing handles serial correlation, the residual dependence that survives purging because markets react with a lag. It is one-sided where purging is two-sided.
The series would be doing what it criticises if it presented this literature as settled consensus, so: the mathematics above is not in dispute — the False Strategy Theorem is a straightforward extreme-value result — but the empirical claim built on it very much is.
On one side, Harvey, Liu and Zhu catalogued 316 published factors and argued a new one needs a t-statistic above 3.0 rather than the conventional 2.0, concluding that "most claimed research findings in financial economics are likely false." Hou, Xue and Zhang rebuilt 452 anomalies using NYSE breakpoints and value-weighted returns to mitigate the influence of microcaps, and found that 65% could not clear a t-statistic of 1.96 — rising to 82% against a stricter multiple-testing hurdle of 2.78 — with the failure heavily concentrated in the trading-frictions category at 96%.
On the other, Jensen, Kelly and Pedersen rebuilt the analysis as a hierarchical Bayesian model across 153 factors and 93 countries and found 82% replicate. Their deepest point inverts the whole logic: under a hierarchical model, a large number of observed factors is evidence for an underlying signal, not against it, because it lets you estimate the prior. And Andrew Chen's thought experiment is hard to dismiss — if the published factors were all spurious, "if 10,000 researchers generate 8 factors every day, it takes hundreds of years" to produce the observed distribution. Dozens of published t-statistics exceed 6.0. P-hacking cannot manufacture tails that fat.
The honest position: run the deflation because the arithmetic is free and the failure mode is real, and hold the "most research is false" conclusion loosely, because it depends on a methodological choice — frequentist family-wise error control versus Bayesian shrinkage — on which serious people disagree in top journals.
Everything above assumes a return series. A language model summarising a client meeting, answering a research question, or drafting portfolio commentary has no Sharpe ratio to deflate. The validation problem is different, the methods are younger, and the honest summary is that the field has not solved it.
The published benchmarks between them define a usable taxonomy. Seven dimensions, and a firm that measures fewer than five of them does not know what it has deployed.
| Dimension | What it asks | Where the method comes from |
|---|---|---|
| Groundedness | Is every claim in the output supported by the retrieved evidence? Measured as the share of extracted claims that can be verified against the provided context. | RAGAS faithfulness; the standard production metric |
| Factual accuracy | Is the answer right, against a known ground truth? Kept separate from groundedness — an answer can be faithful to bad context. | FinanceBench, and its three-way scoring of correct / incorrect / refused |
| Retrieval quality | Was the right evidence found at all? Precision and recall of the retrieved context, measured independently of the generation step. | FinanceBench's condition decomposition — 85% correct with perfect evidence, 19% with a shared document store |
| Abstention | Does it decline when it should? Appropriate refusal must score positively, not as a failure to answer, or you optimise directly for confident fabrication. | FinanceBench; FailSafeQA |
| Robustness | What happens under degraded input — a truncated document, an OCR error, a missing page, an irrelevant context? Production inputs are not clean. | FailSafeQA |
| Consistency | Does it give the same answer to the same question on Tuesday? Measured on extracted decision variables across repeated runs, not on raw text. | Part II's protocol; the non-determinism literature |
| Invariance | Does the answer change when an irrelevant attribute of the client changes — name, gender, an implied demographic? Pass/fail, never graded. | Part II's persona battery |
The retrieval row deserves emphasis because it reorders priorities. If accuracy moves thirty-one points on retrieval scope alone, then measuring the model without separately measuring retrieval tells you almost nothing actionable — you cannot distinguish a model that reasons badly from a model that was handed the wrong document. Instrument them separately or you will tune the wrong component.
At any realistic scale, evaluation is done by another language model. The justification is a widely-cited result: GPT-4 agreed with human evaluators 85% of the time, "which is even higher than the agreement among humans (81%)." That is a real finding and it is the reason the practice exists. It is also a finding about general chat quality, not financial groundedness, and the same paper documents why the practice needs controls.
Position bias. When two candidate answers were swapped, the best judge in the study held its verdict only 65% of the time. Claude-v1 held 23.8%; GPT-3.5, 46.2%. Few-shot prompting improved GPT-4 to 77.5% — better, not fixed.
State the setup honestly, because it matters: those pairs were deliberately similar, being two generations from the same model at temperature 0.7, so these are close to worst-case numbers rather than general consistency rates. But that is exactly why they belong here. Position bias is strongest when the two candidates are close in quality — and a champion-versus-challenger comparison is, by construction, two similar answers to the same question. The worst case for the judge is the normal case for your promotion pipeline.
Self-preference, and what it actually is. Models rate their own outputs higher — GPT-4 by about 10 points of win rate, Claude-v1 by about 25. The obvious inference is vanity, and the obvious fix is to use a different vendor's model as judge. Both are wrong. The mechanism is perplexity: models "assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated." The bias is toward familiar-sounding text in general.
That has a consequence worth sitting with. An LLM judge systematically rewards outputs that sound like an LLM. If what you actually want is house-style analyst prose that reads as though a person wrote it, your judge is biased against your objective — and switching vendors does not help, because the bias is not about self.
Evaluation produces a score. Promotion is the decision that score feeds. Five gates, each producing an artifact, each with a defined way to fail.
Validation is a claim about a moment. Three things erode it, and only two are on anyone's dashboard.
Data drift is a change in the input distribution. The workhorse detector is the population stability index — bin the reference and current distributions, then sum (actual% − expected%) × ln(actual%/expected%) across bins. The conventional reading is below 0.10 stable, 0.10 to 0.25 moderate, above 0.25 significant. It is cheap, interpretable, and sensitive to binning choices; the Kolmogorov–Smirnov statistic and Jensen–Shannon divergence are the usual companions.
Concept drift is a change in the relationship between inputs and outcomes — the inputs look the same and the mapping has moved. This is the one that matters in markets and the one you cannot detect from inputs alone; it requires outcomes, which arrive late.
Model drift is the one specific to this technology, and it has no established detection practice at all: the model changes without you changing anything.
Every major provider publishes a deprecation policy. They vary by more than an order of magnitude — roughly two weeks' notice for a Google rolling alias, sixty days from Anthropic, six months for OpenAI's generally-available models, twelve months on Bedrock. A firm running a multi-provider stack inherits the shortest of these as its effective change-management window, which is a useful thing to know before you discover it.
But retirement notice is the wrong thing to be watching. Those policies govern when a model goes away. The operational risk is behaviour changing while the name stays the same, and there the picture is thinner than most firms assume:
| What you might assume | What is actually committed |
|---|---|
| A pinned snapshot is behaviourally immutable | Committed by nobody — and the clearest provider says so explicitly. Anthropic goes furthest of the four, committing that it "does not update the weights or configuration of an existing model ID." The same page then disclaims the thing you actually care about: serving infrastructure — request router, safety classifiers, sampling logic — "can change over time," and such updates "produce minor differences in observable behavior even when the model ID and weights have not changed." OpenAI and Bedrock make no published commitment in either direction. Weight immutability is available. Behavioural immutability is not. |
| You get notice before behaviour changes | One provider, one case. Google commits to about two weeks' email notice before the model behind a latest alias is swapped. We found no commitment from any provider to notify of behavioural change to a pinned version. |
| Serving infrastructure is out of scope | It is in scope, and it is the gap. One provider names it explicitly as a source of behavioural change; the others do not address it. Quantisation, routing, sampling and safety-filter updates all move behaviour with the weights untouched — which is why a stable model ID is not a stable system. |
| Version identifiers stay auditable | Getting worse. The dated-snapshot convention that made earlier pinning auditable is not consistently visible across current model lines. For a firm needing to evidence "we ran version X on date Y," that is a regression. |
FINRA's 2026 report lists "tracking which model version was used and when" as an effective practice. It is the right instruction and it is not sufficient, because it tracks the label rather than the behaviour. The practice that actually protects you is a frozen regression suite that runs on a schedule, not only on upgrade — so that a silent change shows up as a score movement rather than as a client complaint. Log the model identifier the API returns on every call, not the one you requested. Treat the shortest provider notice in your stack as the binding constraint on how often that suite runs.
Part II established the underlying problem: a thousand identical prompts at temperature zero produced eighty unique completions. Classical validation assumes a function — same input, same output, verifiable. That assumption is simply false here, and pretending otherwise produces documentation rather than assurance.
The workable reformulation is to validate the distribution rather than the output. Acceptance criteria become statistical: not "the model answers this correctly" but "across n runs, the model answers correctly at a rate above a floor, with a worst-case no worse than a bound." Report the floor, not the mean, for anything client-facing — a system that is right nine times in ten is not a 90% system to the tenth client. Then monitor that rate over time with the same statistical process control you would apply to any manufacturing output, because a drop in the rate is the earliest signal you will get that something upstream has changed.
None of this is written down in any supervisory guidance. It is the reasonable extension of methods that exist, applied to a problem the guidance explicitly declined to cover.
Given §01, a firm that wants a defensible governance position has to assemble one. Three components are available and none is a regulation.
NIST's AI Risk Management Framework is voluntary, sector-neutral, and organised around four functions — govern, map, measure, manage — with a generative AI profile added in July 2024 that supplies a twelve-category risk taxonomy. It prescribes nothing testable, which is both its weakness and the reason it maps onto anything. Use it for structure, not for standards.
ISO/IEC 42001 certifies an AI management system — the processes around AI rather than any model's performance. It is the only certifiable standard in this space, which makes it the natural procurement filter. As Part III found, no advisor-AI vendor we could locate publishes it, so it does not yet work as a screen; that is likely to change, and asking for it now is how it changes.
Three lines of defence still works as an organising principle even with the independence requirement relaxed: the business owns the model, a risk function challenges it, and internal audit tests whether the challenge is real. The revised guidance's position — that validation quality depends on rigour rather than organisational structure — is defensible in principle and, in practice, an invitation to let the builder mark their own homework. A firm that keeps genuine independence now does it because it wants the answer, not because anyone requires it.
Which is the theme of this piece and, increasingly, of this series. The controls in Part IV are not required. The evaluation protocol in Part II is not required. The deflation in §03 has never been required. Every serious control we have described across five pieces is voluntary, and the firms that adopt them will do so because the alternative is not knowing whether the thing they have deployed works.
| Item | Status |
|---|---|
| Result tables of the 2024 evaluation comparing CPCV against walk-forward | Only the abstract was retrievable; the publisher blocks automated access. Its headline finding favours CPCV, and we searched specifically for contrary findings and located none. We quote no figures from it. Note also that it tests in a synthetic controlled environment, where CPCV's assumptions hold by construction. |
| The exact trial variance in the deflated Sharpe paper's worked example | Our reconstruction implies roughly 0.60 annualised; one rendering of the paper gives 0.67, which does not reproduce the paper's own stated answer. The two figures that matter — SR₀ of 1.93 and DSR of 0.90 — are firm and our implementation reproduces both. |
| Per-judge numerical position-consistency figures from the position-bias study | The results tables did not render and a follow-up fetch was rate-limited. The magnitudes quoted in §04 are from the MT-Bench paper and are attributed there; the position-bias study is cited for its framework and its qualitative conclusions only. |
| Whether any provider guarantees behavioural immutability of a pinned snapshot | None does. One provider commits explicitly that weights and configuration do not change for a given model ID, and in the same document warns that serving-infrastructure changes produce observable behavioural differences anyway. Two others make no published commitment either way, so for them this remains an absence-of-evidence claim about documentation rather than a claim that their snapshots change. |
| Whether the announced request for information on banks' AI use has been issued | Announced in the April 2026 release as forthcoming. We found no evidence it had been published as of 2 August 2026, and the piece is written on that basis. |
| Which version of the anomaly-replication study to cite | An earlier draft used the 2017 working-paper figures (447 anomalies, 64%, 93%). We have moved to the published 2020 journal version (452 anomalies, 65%, 96%, and 82% against a 2.78 hurdle). The two are not interchangeable and should not be mixed. |
| A published dedicated critique of PBO or CSCV | None located. The limitations listed in §03 are our own analysis, labelled as such. An alternative Bayesian approach exists but positions itself as an alternative rather than a rebuttal. |
| Whether automated groundedness metrics correlate reliably with human judgment | Partially, and less strongly than the tooling implies. We could not find a clean, finance-specific validation study of these metrics against expert human scoring — which is itself the finding, since these metrics are widely deployed as production gates. |
On the simulator: the expected-maximum approximation is an asymptotic result and runs roughly 1–2.5% high for moderate trial counts, and about 8% low at two trials. We guard the small-N case. The demonstration generates independent Gaussian returns because that is exactly the null the theorem describes; the correlation and autocorrelation controls deliberately depart from it and are labelled where they do.
Part VI is the living document the rest of the series has been pointing at: a governance and regulatory tracker, maintained rather than published once. The 2026–2027 cycle is unusually active — a rescinded model risk framework with a promised request for information behind it, a Rule 2210 rewrite in its comment window, congressional pressure on agentic trading meeting a regulatory agenda that contains no AI item, and an EU regime whose high-risk deadlines keep moving. A static piece on any of that is wrong within a quarter.
Part VII closes the series by turning the argument around: instead of auditing what exists, specifying what a platform designed around these controls from the beginning would look like — and asking which of the incumbents is closest to building it.