Open Source Is Not an Audit Trail

There is exactly one question that separates a benchmark from a number on a screen. Take the published value. Take the constituents published alongside it. Apply the stated methodology. Do you get the published value back?

That is what IOSCO Principle 15 and the record-keeping requirements of the EU and UK Benchmarks Regulation exist to make possible. It is what a clearing house’s risk committee asks before a series becomes a settlement source. It is not a high bar. It is the floor, and it is the one property that cannot be argued as a modelling choice: either the arithmetic closes or it does not.

FSKU is unusually well set up to pass it. The methodology is not described in a PDF — it is executable. PricingEngine.calculate_sku_index_summaries groups observations by SKU and takes the median. The shipped tape is twenty-six observations across six providers. The shipped snapshots are four quarterly records, each carrying a label, an observation count and a checksum, displayed in the dashboard under a subheading reading “Point-in-time immutable audit archives stored in FSKUDb.”

So: run the code on the data. What follows is that exercise and nothing more exotic. Every number in this piece comes from the repository itself; section 09 sets out, claim by claim, exactly where each one was checked.

The most recent snapshot, labelled “August 2026 Current Production Tape,” publishes thirteen SKU index values. Recomputing each one as a median of the matching rows in the tape shipped in the same repository gives the picture below.

Seven of thirteen fail. But the six that reconcile are the more interesting half, because of how they reconcile.

Ask a different question. Not “is the published value the median of its bucket?” but “is the published value equal to one specific row in the tape?” The structure appears immediately.

Ten of the thirteen published index values equal a RunPod on-demand list price exactly. The eleventh — A100 80GB at $2.70 — equals CoreWeave’s on-demand rate exactly. The remaining two, MI300X at $2.40 and GH200 at $2.99, match nothing in the tape.

That includes the ones that “reconcile.” H100 SXM has four observations: RunPod’s single-GPU pod at $2.69, Lambda’s at $2.49, CoreWeave’s eight-GPU HGX node at $6.155, and CoreWeave’s spot at $2.464. The median of those four is $2.590. The published index is $2.69 — the RunPod row. The A100 80GB bucket holds CoreWeave’s on-demand $2.700 and its spot $1.206; the published index is $2.700, the on-demand row, and the +38.2% gap in Figure 1 is precisely the spot observation being dropped.

Five of the six SKUs that reconcile perfectly are the five with a single RunPod row in the tape. They agree because there is nothing to disagree with. The sixth, B200, has three rows and reconciles only because the RunPod quote happens to fall between CoreWeave’s on-demand and spot prints. The published index is one vendor’s price list wearing an index’s clothes, and the stated aggregation methodology is not what produced it.

Then run the same test on the three earlier tapes — the Q1, late-Q1 and mid-year 2026 snapshots.

Each of the four snapshots carries a twelve-character value in a field the dashboard labels “SHA-256 Checksum,” under a subheading reading “Point-in-time immutable audit archives stored in FSKUDb.” The README describes the same machinery as a “point-in-time immutable snapshot engine with SHA-256 integrity checksums,” and calls the seed directory “out-of-the-box verified seed checkpoints.”

The observations array inside all four is empty.

Read the checksum column downward. a1, b2, c3, d4. The pattern is visible without a computer.

The truncated SHA-256 of an empty JSON array is 4f53cda18c2b. None of the four shipped values is that. We also hashed each record’s observations, its index values, its label, its timestamp and the whole document: no match on any of the four. These strings are not hashes of anything present.

Be precise about where the fault lies, because it matters. The code is fine. FSKUDb.create_snapshot computes a genuine truncated SHA-256 over the observation JSON and stores the observations alongside it, so a snapshot a user generates is internally verifiable — as far as it goes. The hash covers only the observations; the median, the dispersion figure and the published index values sit outside it, so even a real checksum attests that the constituents are unaltered, not that the number derived from them is right.

The four snapshots that ship are a different object. They carry a field, sku_indices, that create_snapshot never writes. Which means a snapshot produced by the working code path contributes nothing to the history chart at all: the history endpoint reads sku_indices first and only falls back to recomputing from observations — and a real snapshot has no sku_indices.

So the demonstration history and the engine output are two different artefacts that look identical in the interface.

Nothing in the product distinguishes a number that was computed from a number that was typed. The checksum column is what makes this more than a seeding shortcut: it is an integrity signal attached to records that carry no integrity.

That is the whole argument in one screen. Every line of this system is public. You still cannot tell which numbers are real.

The reconciliation failure is the finding that cannot be argued as a modelling choice. The unit error is the one that would survive even if every number reconciled.

FSKU buckets observations on the free-text gpu string its adapters emit. Its headline dispersion metric is looser still — the H100 test is a regular expression — and on the shipped tape it reports nine things that are not one market as one market.

A 3.3× spread between the two on-demand SXM units alone. A capacity block you pay for whether or not you use it, and a spot pod that can be reclaimed mid-job, are not the same hour. These are not outliers to be trimmed; they are different products.

The six aggregation methodologies FSKU offers as robustness — median, two trimmed means, provider-balanced, GPU-weighted, simple mean — all execute after this. Every one of them is a robust estimator of a quantity that does not exist. This is the denominator problem one level further up: before you can argue about what an hour delivers, you have to settle what an hour is.

There is a governance consequence, and it is worse than a naming quibble. The adapters emit "H100 SXM (1x)" and "H100 SXM (HGX 8x)"; the seed tape says "H100 SXM". Run the six adapters and you get 32 observations across 16 SKU strings, of which three overlap the thirteen strings in the shipped tape. The sync engine then matches on the tuple of provider, SKU string, instance, basis and GPU count — and deletes every non-matching prior row for a polled provider.

A single sync does not merely re-bucket the index. It replaces almost the entire tape, silently: no error, no version bump, no continuity break flagged. For a benchmark, an unannounced rebucketing event is a governance failure independent of whether the new number is better.

FSKU extrapolates a forward as F(T) = S₀ × [(1 + c)(1 − d)]^T, with c a user-supplied carry or scarcity rate defaulting to 5%, and d a technological-decay rate inferred from matched-provider generational price pairs.

Start with the anchor, because the unit error from section 04 arrives here first. On the shipped tape, S₀ for the H100 curve is $3.9405 — the median of the eight firm-basis rows the family regex catches, spanning $1.99 to $12.29. FSKU’s own published H100 SXM index on the same tape is $2.69. The curve does not start from the number the platform publishes; it starts 46% above it.

Then run the engine on its own defaults. It returns an annual factor of 0.8901 — a decline of 11.0% a year. Now fit the platform’s own published H100 SXM history: $4.10, $3.65, $3.10, $2.69 across 0.59 years, a log-linear rate of −51.2% a year. (Endpoint to endpoint gives −51.1%, so it is not an artifact of the fit.)

State the caveat plainly. The historical series has no constituents, as section 02 established. So this is a finding about internal inconsistency, not a claim about the realized H100 rental market. That is what makes it strong rather than weak: it requires no external data to establish, and no reader has to trust our tape against theirs. A platform that publishes a history and publishes a forward, and never checks one against the other, has not built the instrument it thinks it has built.

Two problems inside d

The decay term cannot be negative. The estimator retains a generational pair only when the newer generation prices above the older. On the shipped tape that silently discards exactly one pair — Google Cloud, H100 at $11.061 against H200 at $10.601, implying −2.2% — which moves the all-pairs median from 0.2527 to 0.2802 and the published H100 value from 0.1506 to 0.1523. The result is then clamped to a floor of 0.01. The estimator cannot report zero or negative technological decay regardless of what the tape contains. The curve can still slope upward, but only because a user dials up the carry rate — never because the data said prices held. And the censoring bias grows precisely as generational pricing converges, which is exactly when curve shape starts to matter.

It is identified off the wrong variation. d comes from a cross-section of contemporaneous prices — what an H100 costs today versus what an H200 costs today — divided by an assumed 24-month release cadence to become a rate. A level effect and a rate of change are different objects. The cross-sectional ratio also embeds fleet vintage, product mix and each provider’s amortisation posture, none of which is the passage of time. It is the same error as inferring a term structure from a cross-section of credits with different maturities and calling the result a forward curve.

And the unit error propagates straight in. The single largest pair the estimator retains anywhere on the tape is Azure’s A100 at $1.469 against Azure’s H100 at $12.290 — a 65.4% implied annual decay that is really a single A100 retail-API rate measured against an eight-GPU HGX node’s retail-API rate. It is mostly topology. It sets the published A100 curve outright, at a median d of 0.316.

There is also a fallback table. When no pair exists in which the target family is the older member, the estimator returns a hardcoded constant — H100 0.19, A100 0.30, B300 0.07 — and it checks that table before it will use pairs where the family appears on the newer side. So the hardcoded number wins even when usable data is present. The output is described throughout as data-derived.

The forward’s confidence band is the cash quartile pair multiplied by the same deterministic factor as the base case. The arithmetic consequence is that the band is a fixed ratio of the forward at every horizon.

Constant at 122% at every tenor out to five years — and because the base case is declining, the absolute width falls 44%. Uncertainty about a five-year-forward rental rate is asserted to be narrower, in dollars, than uncertainty about today’s.

This is not a term structure of uncertainty. It is today’s cross-sectional dispersion — itself an artifact of the unit error, since it is the spread between a PCIe card and an HGX node — painted forward along a deterministic decay path.

Reasonable people can argue about shape. A pure diffusion band widens monotonically. A floor-anchored model, in which price decays toward cash operating cost rather than toward zero, widens and then compresses as an uncertain scarcity rent decays toward a known floor. Both are defensible. FSKU has not chosen a shape at all — which is why nothing downstream of it can be margined, and why it is not a candidate settlement source for anything cleared.

The critique above is severe and the code does not deserve to be dismissed. FSKU identifies the right problems and builds real instruments for four of them — instruments most administrators do not publish at all.

• The source-ablation engine. Exclude one contributor, observe the index move. Nothing in the CFTC’s cash-settlement guidance requires this test by name, but it is the most direct evidence a venue could offer against what that guidance does require — that a settlement series be reliable, representative of the cash market, and not distortable by a single participant.
• The methodology-sensitivity matrix. Six aggregation methods, same constituents, displayed side by side. Every administrator should have to show this.
• The provenance ledger. Every observation carries a timestamp and a source id resolving to a named, URL-bearing entry.
• The point-in-time snapshot pattern, as designed in the snapshot constructor: constituents stored with the summary and hashed. The design is right. The shipped examples do not follow it.

All four are Apache-2.0 with an express patent grant — free to fork, extend and commercialise. The instrument set is genuinely reusable. It is the answers that are not.

And the sharpest way to see that is to point the instruments back at the platform. Run FSKU’s own ablation on FSKU’s own tape and it already reports four contributors whose removal moves an index by more than 20%, the worst at 43.4% — drop CoreWeave from H200. Run the same test on resolved contract units rather than text-string buckets and it gets worse: remove one provider from the H100 SXM HGX-8x on-demand unit and the index moves 28.5%. Score all twenty-two resolved contract units on observation count, contributor count, contributor concentration and ablation fragility, and none passes. Eighteen have a single contributor.

FSKU ships the right instrument, publishes results that fail its own implied thresholds, and lets nothing downstream depend on the answer.

That is a data-volume problem rather than a code problem, and it is fixable. Twenty-six static observations, with one of six adapters making an actual network request, is a demonstration corpus. Point the same harness at a real feed and every diagnostic in it starts earning its keep.

Read the release as strategy and it stops being surprising.

NATIVX is doing two things at once. It has open-sourced the posted-rate normalization layer under a permissive licence. And since the July 1 announcement with ICE, it is taking an energy-normalized index — COIL — toward a regulated venue, with contracts described as “expected to be launched later this year, subject to completion of relevant regulatory processes.” The normalization layer is being priced at zero on purpose. The value being protected sits in settlement and in the energy-normalized methodology that was not open-sourced. Textbook commoditize-the-complement.

Note what is not on that list. ICE announced a GPU compute futures partnership with Ornn, on the OCPI index, on May 19, 2026 — six weeks before the NATIVX announcement, also subject to regulatory approval, also unlisted as of writing. The venue is not exclusive. So venue distribution is not the moat either, and the defensible ground narrows rather than widens.

Now put that beside the other August development. Compute Desk’s benchmarks moved under GX Benchmarks Ltd, a wholly-owned FCA-regulated subsidiary of General Index, which calculates and publishes them to IOSCO and EU/UK Benchmarks Regulation standards — a firm with no commercial position in compute at all.

Two moves in opposite directions, one conclusion.

Which is why this release is more useful to the people building the layer than to the people who published it. Any pitch whose differentiation is “we normalize posted rates transparently” is now competing against free, permissively licensed, and already written. And sections 02 and 03 are the argument for the administrator, made by the opposition: FSKU is maximally transparent and its published numbers still cannot be reproduced. Transparency is necessary and nowhere near sufficient. What is actually for sale is governed reproducibility — and that lands harder coming from a live, timestamped, re-runnable example than from an IOSCO citation.

There is a specific product requirement in here too. A benchmark whose uncertainty band is a flat 122% of the forward at every tenor cannot support a margin model, let alone a capacity-backed insurance product written against it. A benchmark that decomposes the forward into a cost floor and a decaying scarcity rent, each with its own uncertainty term structure, can. “The index is not fit to underwrite against” is usually an abstract objection. Figure 5 is a concrete instance of it.

One more number, because it is the cleanest read on how much of a posted-rate index is list price. Within a single provider, matched on family and topology, the on-demand rate against the spot rate for the identical physical box:

A posted-rate index and a transaction-anchored index of the same box appear to be measuring numbers roughly two to two and a half times apart. That spread is the cross-index basis, it is computable from public data today, and it is the quantitative answer to the question of how much of a posted-rate benchmark is list-price fiction. The premium also declines across the Hopper-to-Blackwell sequence — 2.50×, 2.41×, 2.02× — while the oldest generation breaks the pattern at 2.24×, which is what you would expect if the premium tracks current supply tightness rather than hardware age.

The test FSKU fails is the test everyone should have to sit, and it costs an afternoon. It applies to a paywalled assessment exactly as it applies to a public repository — the only difference is who is allowed to run it.

1. Reproduce it. Take a published value, take the constituents published with it, apply the stated methodology. Does the number come back?
2. Show the constituents. Not a count, not a checksum — the rows. A hash over an empty array is a hash, not an audit trail.
3. Define the unit. Family, form factor, topology, contract basis, region. If the unit is a text string, the index is a text search.
4. Ablate a contributor. Drop the largest one and publish the move. Above 20% and it is a quote, not an index.
5. Backtest the curve against your own history. If the published forward and the published history disagree by 4.7×, one of them is wrong and the administrator should say which.

Nothing on that list requires an index to be open source. FSKU is the proof: it answers question 1 with executable code, and still fails it.

The repository was cloned fresh on August 29, 2026 at commit 28fd6e1 — a single commit dated August 25, 2026, 3,756 lines of Python, version 0.9.0, Apache-2.0 with an express patent grant. Every figure in this piece was recomputed from the data shipped in that clone, with no external inputs. The engine was also run directly, unmodified, on its own seed tape with its own defaults, so that the forward, decay and dispersion figures are FSKU’s own output rather than our reimplementation of it. The test suite was run: twenty-six tests, twenty-six passing.

Each claim below was then re-checked independently, against the code and data rather than against our own working notes. That pass corrected five substantive errors in our first draft — including our own arithmetic on the headline count, and one regulatory citation that turned out not to exist and was removed. The ledger records what survived.

What this piece does not claim

• Not that the rental market fell 51% a year. The historical series has no constituents. The −51.2% figure characterises FSKU’s own published history, and the comparison against its −11.0% model is a statement about internal consistency, nothing more.
• Not that the tape is representative. Twenty-six static observations with one live adapter is a demonstration corpus, and a live sync would produce different magnitudes. The construction defects in sections 03 through 06 are properties of the code and survive any tape.
• Not that this is bad engineering. It is clean, tested, documented, well-abstracted code aimed at the right problems, released at v0.9.0 with no maintenance or support claims. The gap is between what a research release is and what the surrounding language — “immutable audit archives,” “settlement prices,” “verified seed checkpoints” — asserts it to be.
• Not a statement about COIL. COIL is energy-normalized and we have located no published methodology for it. Nothing here evaluates it. FSKU is the posted-rate layer that was given away, not the index going to a venue.
• Not a claim of misconduct. Nothing here alleges manipulation, bad faith or impropriety by anyone. A seeded demonstration dataset in a pre-1.0 research release is an ordinary engineering decision. What this piece argues is that the surrounding claims of immutability and integrity are load-bearing in a way the artefacts do not support — and that the general lesson is about benchmark governance, not about one repository.

On 31 August, six days after the release this piece examined and one day after it was published, NATIVX Engineering wrote to us. They did not dispute a finding. They shipped commit 6e2459f and asked us to re-run the checks.

We did, against a fresh clone, using the same method as the original audit. What follows separates what is now closed from what is not, and one finding that has become larger as a direct result of the fixes.

What genuinely closed

Six of the findings are simply gone, and the fix is the right one in each case rather than a patch over the symptom. Constituents are stored with the summary and hashed together; the checksum is computed over a canonical serialization, so it is reproducible by a third party rather than merely present; and the history endpoint now derives every point from those constituents instead of reading a pre-baked field. We verified the four shipped hashes ourselves, from outside the codebase, and all four match.

The taxonomy fix is the one we would have argued hardest for. The pooled “H100” bucket is gone, and an eight-GPU HGX node at $8.61 is no longer averaged against a single sliced pod at $2.69, a PCIe card at $1.99 and an NVL module at $2.59. That is four instruments where there was one.

What did not close

Contract basis is still pooled. Form factor and topology are now resolved; the contract basis is not. Six of the sixteen buckets still mix incompatible commercial terms, and the headline one is the worst: H100 SXM (HGX 8x) holds a CoreWeave spot print at $2.464, on-demand rates from $2.49 to $11.06, and an Azure retail-API rate at $12.29 — a 4.99× range inside a single published index. A spot instance that can be reclaimed mid-job and a retail API call are not the same hour, whatever the silicon. The pooling is narrower than it was. It has not gone away.

The decay estimator is untouched. The sign censor still discards any generational pair where the newer part prices below the older, the result is still clamped to a floor of 0.01, the hardcoded per-family table is still consulted before usable newer-side data, and the rate is still inferred from a cross-section of contemporaneous prices divided by an assumed release cadence. None of that appeared in NATIVX’s summary, and none of it has changed.

And there is still no backtest. Which brings us to the finding that moved in the wrong direction.

The contradiction got bigger, and that is the interesting part

The forward engine now returns an annual factor of 0.9251 — a decline of 7.5% a year. The platform’s own published history, across the four rebuilt snapshots, falls at 47.1% a year. The model understates its own history by a factor of 6.3, where before it was 4.7.

Read that carefully, because it is the whole argument of this piece arriving in miniature.

Before the fix, that comparison was weak: the history had no constituents, so it was a contradiction between two unsourced numbers. Now the history is real, checksummed, and recomputable from its own rows — which means the forward model is being falsified by an auditable series. Reproducibility did not repair the model. It made the model’s error provable.

That is what a reproducible benchmark is for. It does not make a number right. It makes a number checkable, which is the precondition for finding out that it is wrong. The gap between the curve and the history is now a fact about the platform rather than an argument about it, and closing it requires a decision about which of the two is the product.

What this does to the thesis

Honesty requires conceding something here. This piece argued that open source is not an audit trail, and that transparency is necessary but nowhere near sufficient. Both statements survive: FSKU was fully inspectable on 25 August and its numbers still could not be reproduced, which was the point.

But the six days that followed demonstrated a property of open publication that a closed administrator does not have. An outsider could run the test at all. The finding could be stated precisely enough to be actionable, because the constituents, the code and the commit hash were public. And the response is itself verifiable — we did not have to take the email’s word for any of it, and readers do not have to take ours.

No closed benchmark in this market could have been audited this way by anyone outside it, and none could have demonstrated a fix this way either. Open source is still not an audit trail. It turned out to be a fast route to one.

The governance point is unchanged and, if anything, sharper. What made this work was not the licence; it was that someone chose to answer. A benchmark that carries real exposure cannot depend on that choice — which is the case for an administrator with a control framework, contributors under an enforceable agreement, and a restatement policy that operates whether or not anyone is feeling responsive that week. FSKU has now shown it can produce an audit trail. What it does not yet have is an obligation to.

Everything above is re-runnable. Clone the repository at 6e2459f, recompute each published index from the observations stored in the same snapshot, and hash the canonical serialization of those observations. We will keep doing the same on any subsequent commit.

• The subject. The nativ-x/fsku repository, v0.9.0, Apache-2.0, single commit 28fd6e1 dated August 25, 2026; cloned and analysed August 29–30, 2026, and re-verified at commit 6e2459f (31 August 2026) on 1 September 2026. All quantitative claims derive from fsku/data/seeds/, fsku/core/, fsku/sync/, fsku/api/, fsku/web/index.html and the repository README at that commit.
• Benchmark standards. IOSCO Principles for Financial Benchmarks (Principle 15, audit trail and record keeping). Appendix C to 17 CFR part 38, paragraph (c)(3), on third-party settlement price sources — the reliability and representativeness requirements referenced in section 07.
• Venue and administration. ICE and NATIVX, July 1, 2026. ICE and Ornn, May 19, 2026. General Index on the Compute Desk arrangement, announced August 13 and posted August 17, 2026; GX Benchmarks Ltd, FCA registration 933348 as stated by the issuer.
• Kinetic Alpha. Five Indices, One Price — what a benchmark has to do before a contract can reference it, and why the hardest deficiency is independence rather than data. How the Indices Calculate Price — the component-level catalogue. Silicon Data’s Product Suite — the eleven-product audit and the utilization blind spot. The Contract Has a Date. It Needs a Dealer. — who actually holds a floating compute exposure.