Research•25 min read

ASIF: An Origin Report on a Self-Measuring Multi-Agent Control Plane at Portfolio Scale

July 6, 2026 by Asif Waliuddin, NXTG.AI

multi-agent systemsagent harnessobservabilityevaluationcontrol planedeterministic scoringLLM agents

One-line thesis: of 570 recorded agent-pulse outcomes on our own harness, just 5.4% were productive (31/570) and 93.3% were overhead — one-shot AI is not enough; real AI work needs a control plane: memory, evidence, governance, coordination, escalation, and correction across many projects. This is an honest origin account of that control plane running in production, and it includes the first registered result to come off it in full — a routing experiment that returned a clean tie.

Why we built it

A single model answering a single prompt is a demo, not an operation. The moment the work is portfolio-scale — many projects, many agents, months of continuity, decisions that must not be silently lost — the hard problem stops being the quality of any one answer and becomes the plane the answers run on: what the fleet remembers, what it can prove, what it is allowed to do, how it coordinates, when it escalates, and whether it catches and corrects itself before a mistake becomes durable.

ASIF is that plane. It is a production portfolio-governance layer built as a team of LLM agents surrounded by a deterministic control harness — shared typed state, hard pre-action checks, pre-committed evaluation gates, and a "one seat produces, a different seat grades" structure. The research object of this report is the control plane at portfolio scale — a global multi-agent, cross-project orchestration plane that measures itself, catches failures, and prunes its own waste. Its registered value claims are about orchestration: interception (catching errors mid-stream before they reach anything durable), coordination, typed governance, and continuity — evidenced by the operational corpus the plane emits as it runs.

We report this as an industrial experience report on purpose. In that genre, a single organization operating a real system in production (n = 1 organization, n = 1 fleet) is a valued datapoint, not a disqualifying weakness: it is the only vantage from which the in-situ behavior of a persistent control plane can be described at all. We are the operators, the subjects, and the authors at once. We treat that as the central constraint on interpretation — stated up front, returned to in the weaknesses section — not as a footnote.

We are not reporting a marketing win. We are reporting a plane, a dataset of its own graded outputs, one clean tie on a specific routing question, and a prediction we registered with an independent third party before its outcome was known. The registered prediction — not any retrospective story — is the load-bearing evidence. Everything descriptive is context about the plane, offered as co-occurrence and design intent, never as demonstrated cause.

The control plane under study

The studied system is a portfolio-governance fleet of LLM agents in persistent terminal sessions, coordinating through a shared append-only ledger and a typed state store under a common set of governance canons. For neutrality we refer to operating seats by function — Orchestrator, Verifier, Builder, Certifier, Machine-Marshal — not by internal names.

The harness is the set of mechanisms whose shared, defining property is pre-hoc machine enforcement (checked and blocked before an action, not audited after). Listed as present in the plane, not as ranked causes of any outcome:

1. Probe-over-narrative at the authority source — state claims are checked against the authoritative instrument (post-fetch origin, a health command, the raw artifact), not a peer's message or an agent's memory.
2. Independent verification as gate structure — the seat that produces an artifact never grades it; a second seat reproduces or recomputes before "verified" is assertable.
3. Public self-correction at low status cost — a seat retracts its own wrong claim in-channel rather than defending it.
4. Binding contracts with named instruments — improvement gates pinned to endpoint, parameters, sample size, and a fixed gold set, ratified before treatment data exists.
5. Typed deterministic state over prose — backlog, promise, and handoff state live in validated typed ledgers, not conversational text.
6. Hard hooks that block, not advise — pre-action guards that deny disallowed writes deterministically.
7. Distinct seats with explicit responsibility assignment.
8. Canonical-close discipline — one authoritative close per loop; later acknowledgements link, not restate.
9. Sequencing binds — fixed action ordering (observability before treatment; baseline snapshot before treatment measurement) so ordering artifacts cannot masquerade as effects.
10. Event-gated commitments — background watchers with completion callbacks and time filters, replacing prose "I will monitor" promises.

This inventory describes the plane. Whether any mechanism is causally responsible for any outcome is not established here and is not claimed.

What the plane holds (descriptive context, not proof)

All of the following is descriptive — an account of what we observed while the plane ran, on periods we chose to look at, with no control condition. None is offered as causal evidence. Where a mechanism is named alongside an outcome, the relationship is co-occurrence and design intent, never demonstrated cause.

A catch-count from one documented period. During one stretch of operation the fleet recorded roughly a dozen wrong or nearly-wrong intermediate claims, each identified and corrected in-channel before any reached a durable artifact. Plain examples: a claim that some stuck queue rows should be discarded (an authority-source check showed they had already been processed); a "this file is unprotected" alarm (a check against the canonical source, not a stale local copy, showed it was fine); a "the feature is built" claim the author walked back mid-turn as still a roadmap item.

This is a caught-count, not a rate. "Twelve of twelve caught" is not a success rate: the denominator — errors made and not caught — is unobservable by construction; an uncaught error is definitionally one nobody recorded. We report a count during a described period and draw no rate from it. Saying this plainly matters — quoting it as a rate is exactly the trap.

Independent convergence across the team. Several seats separately wrote up the same period without coordinating on content and landed on a compatible diagnosis of what had changed. Independent agreement is the strongest thing a single-organization account can offer — and it is still internal agreement, not external measurement.

A longitudinal outcome dataset — the plane pruning its own overhead. We hold a larger, quantified log of graded outcomes accumulated over time in append-only ledgers joined by outcome id. Read honestly, its most interesting property is that it shows a harness component running mostly as overhead: of 570 recorded pulse outcomes, 5.4% were productive (31/570) and 93.3% were waste (532/570) — ignored, or acknowledged without follow-through — and the component was switched off per the pre-stated read of the instrument. (Outcomes recorded on the current-generation fleet, June–July 2026.) This cuts against a naive "the plane always helps" story: some machinery is noise, the data says so, and the plane measured and stood down its own dead weight.

A review-integrity exhibit. In the design phase of the follow-on experiment, our workflow required an adversarial review pass the original author could not waive. On the record, two reviewers refuted their own earlier findings — one discarded her own proposed primary metric after showing a system that simply never ships could game it; one demoted his own optimistic sample-size figure after accounting for the correct statistical test, making his own experiment more expensive to run. Three further findings in the same review were additive catches, not self-refutations, and we keep those tiers separate on purpose — collapsing "two self-refutations" and "three additional catches" into one impressive "five self-catches" is precisely the inflation an honest reviewer would flag. The write-up of the episode was then checked by a reviewer built on a different base-model family, which first returned a needs-fix verdict, named four specific overclaims, and certified only after each was narrowed. That cross-family check reduces same-family framing risk on that one artifact only — a partial mitigation, not a resolution; it graded the write-up's claims, it did not independently reproduce the underlying findings.

The registered prediction (the actual rigor engine). The value claims this plane makes are committed as pre-registered predictions — full design, metric, analysis plan, thresholds — deposited to an independent public registry under embargo before their outcomes are known. Two registrations anchor this line of work: 8mh2x (the routing experiment reported below) and 2nker (the live interception bet, pending). Both are embargoed and not reader-accessible yet. Their function is not to be read today; it is ordering — an independent, immutable record that the predictions preceded the outcomes. A commit hash in our own repository would establish authorship, not pre-registration; that distinction is the answer to the sharpest objection a reviewer can raise against a self-hosted study.

The honest first scar: a seat-economics routing experiment, reported in full

The first registered result to come off this plane is not about the plane's orchestration value at all. It is a seat-economics / routing experiment — the kind of question a control plane exists to answer about itself: given a fleet with more than one model available as a "seat," can the plane run a cheaper seat on a class of directive work and match the pricier seat, so it can route accordingly? We report it here, in full, with every rigor element intact, precisely for that reason — it is a scar, not a trophy: it is how we prove we are not bullshitting.

We frame it exactly as it was registered. This was not a test of the orchestration thesis, and not a test of one-shot artifact quality as ASIF's value — nobody registered a claim that static harness context makes a single seat's output prettier. The registered question (8mh2x) was a routing question: can sonnet-5 in-harness match opus-4-8 bare on this directive class? Its outcome-actions were all model-allocation routing decisions. Read it as an economics datapoint for the plane.

The spine of this result: the control plane does not make a single seat's one-shot output prettier — it was registered to answer whether the plane can run cheaper seats, and that question needs N≥24. What it returned at n=12 is a tie: no measurable difference, decision-grade only for a reversible routing choice, and explicitly not enough to move any canon-wide allocation policy. The null shows that ASIF did not improve isolated one-shot artifact polish — and that taught us the real measurement target is portfolio orchestration, not one-shot output.

Design. We ran a pre-registered, blinded 2×2 comparison — {sonnet-5, run at max effort; opus-4-8, run at xhigh effort} × {full harness, bare} — on a fixed set of held-out software-directive tasks. (Model names are publishable context, not a secret; every result carries its model-generation context per the recency-audit rule — outcomes measured on current-generation models, June–July 2026.) Predictions, metric, analysis plan, and tie thresholds were deposited to a public registry before any result was measured; an independent seat that ran none of the arms computed the outcome against the deposited predictions.

On the taskset codename. The frozen taskset is codenamed "Sonnet 5 × ASIF Harness," which reflects the motivating routing question — whether a Sonnet-tier seat in-harness can match an Opus-tier seat bare. The actual design is a two-model 2×2: (opus-4-8 xhigh, sonnet-5 max) × (harness, bare). The frozen taskset file is read-only and unchanged; this line is the only reconciliation of codename vs. design.

The result was a tie. Within-model, the harness effect on final-deliverable quality was statistically indistinguishable from zero — for both models — with point estimates in fact slightly negative for both (harness lift −0.0288 for opus-4-8, −0.0217 for sonnet-5; both inside the pre-registered ±0.05 tie/infra-noise band; we state the sign rather than round it away). The two models scored as a tie under the harness (gap O−S = +0.0054, deep inside the band, robust to a single-task drop). A methods finding sharpens it: the two pinned pooled decision-metrics are algebraically one number — the "crossing" statistic reduces to the sign of (harness_sonnet − harness_opus) once the baselines cancel — so "which seat crosses ahead" collapses to a sub-noise sign change and is simply not answerable at this sample size.

What the tie does and does not weaken. This weakens the claim that static harness context improves one-shot work-product quality; it does not test live interception, memory, or governance under execution. An independent certification confirmed a further scope point: the tested "harness" arm exercised the static context payload only (canon and pre-action guards in the pinned worktree); the dynamic organs — persistent memory recall, the live inter-agent verification loop — were structurally absent from a single-turn headless arm and are not what this test measured. The slightly-negative lifts are directionally consistent with a context-overhead reading (hypothesis (c) below), where added static context costs attention without a matching one-shot payoff — but at this resolution that direction is itself within noise.

We are precise about what this result is a null of: no measurable lift in the quality of the final deliverable, on this task set, at this sample size — and therefore no decision-grade crossing that would let the plane route a cheaper seat onto this directive class. It is not a measurement of coordination, error-interception, or determinism, and says nothing about them. Reading it as "the plane is useless" would be as wrong as reading it as "the harness works." What makes it a finding rather than a failure-to-find is the ordering: thresholds and comparison arms were committed to an immutable third-party record before any treatment was measured. Absent that ordering, a null is just noise. The routing verdict is not decision-grade for any canon-wide allocation change at n=12; any such change requires an N≥24 replication on fresh tasks.

Table 1 — Per-task deliverable score: bare vs. harness vs. delta

Composite impact score (0–1), current-generation models (June–July 2026). Delta = harness − bare, per model; pooled = mean of the two model deltas. Descriptive; co-occurrence, not cause.

Table 2 — Per-stratum lift and the routing (crossing) signal

Per-stratum n = 4 (directional only, underpowered, not decision-grade). Lift = harness − bare. Gap = harness(O) − harness(S) — the crossing/routing quantity tied to the registered P1/P2/P3 predictions, not a lift. Verified against the independent certifier.

Reading (the cheaper-seat routing signal — directional only, underpowered). Mean lift is ≤ 0 in every stratum. The gap (O−S) column (−0.0525 easy, +0.0375 medium, +0.0312 hard) is the crossing-direction quantity — which seat is ahead under the harness — that the registered routing predictions P1/P2/P3 pinned; it is not a lift, and per the pre-registration each per-stratum n=4 value is directional signal only. The registered split predicted crossing at easy and medium and not at hard. The certifier proved this gap is algebraically the "crossing" statistic and that every value sits well inside the ±0.05 noise band, so no stratum supports a "cheaper seat overtakes the pricier seat" reading. The registered hard-does-not-cross prediction held; the medium directional prediction missed (predicted a cross, did not occur — the pricier seat gained +0.0175 at medium while the cheaper seat lost −0.020). Nothing here is a "harness helped" reading — it is a routing signal, directional and underpowered.

Table 3 — Per-task qualitative reading (descriptive co-occurrence)

Descriptive co-occurrence only — a one-line reading of each per-task delta, not a cause.

Candidate explanations for the tie (labeled hypotheses, not findings)

The tie is a clean answer to the routing question and a clean non-answer to any orchestration claim. We do not claim to know why the point estimates came in slightly negative, or which of these holds:

• (a) The metric may be too coarse. A final-quality rubric might not register intermediate errors intercepted before they reached the deliverable — the intercepted error never appears in the graded artifact.
• (b) A strong base model may already self-correct. On small tasks a capable model may recover from its own missteps well enough to mask any harness contribution.
• (c) The task set may not stress what a harness helps most with — and static context may add overhead. Harness value could concentrate in durable memory, typed state, evidence trails, producer/grader separation, and multi-step continuity that a fixed set of held-out tasks does not exercise; and rich static context can cost attention without a one-shot payoff (the slightly-negative lifts, largest on easy tasks, are directionally consistent with this).
• (d) Overhead could cancel benefit. Harness machinery may cost as much as it saves; this report's own longitudinal pulse data shows one component running mostly as waste.
• (e) The comparison may be underpowered. At n = 12 a small real effect may be indistinguishable from zero regardless of whether one exists — which is exactly why any allocation change is gated on an N≥24 replication.

Each might explain the point estimates; none is demonstrated, and we do not claim which — if any — holds.

What we are NOT claiming

• Not that the harness causes convergence, quality, or error-catching. The one place a causal claim is licensed is a registered prediction, and the interception prediction is unresolved.
• Not an error-correction rate. We have a count during a chosen period and no denominator.
• Not that a cheaper seat inside a good harness reaches a pricier seat's absolute performance. Our own routing test found a tie, not a crossing, and current external evidence does not support a tier-equalization claim.
• Not a canon-wide allocation change on the routing result — that is gated on an N≥24 replication; the n=12 verdict is decision-grade for a reversible routing choice only.
• Not a generalization beyond this single fleet in this single organization.
• Not that "we achieved harmony." We documented one period and one routing tie; the interesting orchestration claim is still a registered bet awaiting its outcome.

The honest weaknesses

Stated as first-class sections, not as anticipated objections.

Sample of one. One fleet, one organization, one documented period. Everything descriptive is a case study, and case studies do not establish general causes. (In an experience-report genre this n = 1 is the contribution — a described plane others can compare against — but it is not a general causal claim, and we do not dress it as one.)

No denominator, and the period was selected for its outcome. The described stretch exists as an artifact only by virtue of being a good stretch — selecting a good day and reporting how good it was is selection on the outcome, and no causal inference survives it. To give the error-catching account a denominator, we pre-committed to sampling non-selected operating days, chosen blind to how they went, and coding both catches and misses (misses proxied by errors later discovered and traced back). We pre-register that this base-rate ratio may be unfavorable to us, and commit to reporting it either way.

Same-model self-grading. The errors were made, caught, and written up by members of the same fleet, and the graders share a base-model family with the agents under study. The literature we lean on is also the weapon against us: same-family models can agree without being right, and relabeling one model's reasoning as an "independent" seat can manufacture the appearance of independent correction. Until we have external human raters and a calibrated judge with divergence statistics, every internal-agreement number is internal consistency, not measurement — and we label it as such.

Underpowered routing test. The routing test pooled 12 tasks in strata of 4. Pre-registering a design does not create statistical power it lacks. We named the test in advance, report only at the resolution the data supports (pooled decision-grade for reversible routing; strata directional-only), and pre-committed a larger confirmatory replication on ≥ 24 fresh held-out tasks before any finer or canon-wide claim.

Static vs. dynamic harness. The tested arm exercised the static context payload only; the dynamic organs (live memory recall, the inter-agent verification loop) were structurally absent from a single-turn headless arm. The stage-2 replication is designed to measure exactly the plane layer this one-shot test could not.

Tone. Internal write-ups of this material are in a house voice full of narrative and persona. That voice reads as marketing to a skeptical reviewer, so this report strips it. The claims stand on the registrations and instruments, not the storytelling.

The live bet: interception, reveal #1 of a series

The routing tie clears the ground. It says, cleanly and on the record, that a control plane's job is not to make one seat's one-shot answer prettier — the real measurement target is portfolio orchestration, and orchestration is what we always registered the plane's value on: interception, coordination, typed governance, continuity.

Final-deliverable quality and mid-stream error-interception are different quantities; a plane can leave the first flat while changing the second. Interception — does the plane catch errors along the way, before they reach anything durable? — is the first of a deliberate series of registered reveals. It is committed as a second pre-registered prediction (2nker) whose outcome is not yet resolved. Nothing here should be read as evidence the plane intercepts errors: that is the open question, registered and pending. If it resolves against us, we will report that too.

The strategy from here is slow serial revelation — one registered result after another, each betting the plane's orchestration value on an immutable, third-party-timestamped prediction before we know how it turns out: stage-b interception (2nker), then a shadow replication, then the stage-2 N≥24 routing replication that the routing tie explicitly requires, onward. The disclosure of this task set is sanctioned precisely by design — each next stage runs on fresh held-out tasks.

The headline this report can carry, and no more: ASIF is a global multi-agent orchestration control plane that measures itself, catches its own waste, and is now being tested — in public, on the record — on live interception. The reason to take it seriously is not the retrospective story. It is that the claims we are actually betting on are written down in full, with an independent third party, before we know how they turn out, in a way we cannot quietly revise. That commitment — made before the outcome — is the whole point.

Descriptive claims are co-occurrence and design intent only; the single licensed causal claim (the interception prediction, 2nker) is registered and unresolved. Registry IDs cited for pre-registration ordering only, embargoed.

Methods at a glance

The cheap-to-disclose half of the method. Fuller methods — raw model outputs and the pinned registry and configuration contents — are held in the embargoed pre-registration and the stage-(b) preprint. The task SET is disclosed here (Tables 1–3); its disclosure is sanctioned given that the replication uses fresh held-out tasks.

• Arms (2×2): opus-4-8 (Opus-tier stronger arm, xhigh effort) and sonnet-5 (Sonnet-tier near-peer, max effort), each crossed with {full harness, bare}. Effort is pinned per model and constant across both arms (not a lift confound; certifier-verified).
• Sample size: 12 held-out software-directive tasks, pooled; balanced 4 / 4 / 4 across easy / medium / hard strata (per-stratum n = 4, directional only).
• Held-out discipline: each task is an open problem or a work product never before created; the graded outcome is not retrievable from any harness-reachable store (dual-probe leak-gate per task; prior methods/templates being retrievable is legitimate lift, a retrievable outcome is excluded).
• Metric / rubric: a composite impact rubric plus a mechanical used-rate control metric; composite deltas inside the pre-registered ±0.05 band are treated as ties.
• Statistical test: paired per-task deltas with a permutation test on the pooled comparison; strata underpowered, reported directional-only.
• Grading: blinded; the generator never grades itself; the verdict was computed by an independent seat that ran no arms, recomputed from 48 raw judge scores against the deposited predictions (arithmetic reproduced to < 0.0001; 48-row integrity parity clean).
• Routing scope: the pooled verdict is decision-grade only for a reversible routing decision; any canon-wide model-allocation change requires an N≥24 replication on fresh tasks.

Appendix: Data, Code & Registration Availability

Pre-registration. Both predictions in this report are pre-registered with the Open Science Framework under embargo, timestamped before their outcomes were known: the routing experiment reported here (OSF 8mh2x) and the pending interception prediction (OSF 2nker). Each is private under embargo and becomes publicly readable on its release date; the registrations exist as an independent, immutable record that the predictions preceded the outcomes.

Data. All underlying data is deposited as citable, versioned Zenodo archives: the redacted public task set at doi:10.5281/zenodo.21229473; the per-task evidence and crossing analysis behind Tables 1–3 (12 tasks × 2 models × 2 arms) at doi:10.5281/zenodo.21229466; and the longitudinal pulse-outcome dataset (570 rows) behind the self-pruning result at doi:10.5281/zenodo.21229464. The pulse figures (5.4% productive, 93.3% waste) are computed from the deposited append-only ledgers and reproduce exactly. In the released data, lane and machine identifiers are anonymized; instrument and method labels are retained so the public dataset stays join-able to later results in the series.

Reproducibility. The pre-registered analysis — paired per-task deltas, a permutation test on the pooled comparison, and the ±0.05 tie band — is specified in full above and in the deposited registrations. Scoring was blinded and computed by a party that ran none of the experimental arms; the pooled statistics reproduce from the raw scores. Because generation runs on proprietary current-generation models, the released artifacts make the analysis reproducible even where the generation is not.

External citations. Every external result cited in support of an orchestration framing carries its model-generation context; results predating current-generation models reflect weaker base models and are weaker support for a claim about today's systems.