Do AI-agent companies do the job reliably? A setup-qualification study of three Fabri companies

1The question

Before asking whether a company of agents is capable, we ask whether it is dependable: given the same task and a fresh, isolated environment, does it finish the work — and finish it correctly — on every attempt? We hold each company to a strict bar: to qualify, a setup must pass end-to-end on 100% of its scheduled replicas. Anything less is not a qualification.

2Method

1. Isolation. Every replica gets a fresh compile of the company and its own disposable home directory, so no state leaks between runs.
2. Two separate measurements. Completion (did the run finish and produce output) and rubric (was the output correct and safe) are scored independently and never averaged together; an incomplete run receives no rubric verdict.
3. Deterministic rubric. The output is checked for required concepts and forbidden phrases fixed in advance — e.g. an incident update must mention a rollback and a follow-up, and must not claim a fix was deployed without evidence.
4. Recursive cost. Every delegated sub-agent's spend rolls up into the parent, so a reported cost is the whole company's tree, not just the top.
5. Bounded candidate search with no-op rejection. A proposed configuration change that preflight shows would do nothing is rejected before any model call — a no-op costs nothing.

3Results

4Discussion

Small samples overstate reliability. Support HQ's 3/3 release gate (95% CI 44–100%) read as flawless, yet ten runs of the identical setup produced 9/10 (90%, 95% CI 60–98%). The additional replicas did not refine the estimate so much as overturn it: the true rubric reliability is near 90%, and the 100% gate is not met.

The dominant failure is over-claiming. Two of three companies failed by asserting facts the evidence did not support — a deployed fix, a customer result, buying intent. These are precisely the errors the rubrics forbid, and the harness caught each one. That completion was near-total throughout underscores the central point: a run finishing cleanly is not evidence that its output is correct.

5Limitations

• Setup only. This section measures setup reliability. Whether Fabri's trace-backed memory improves a company is a separate controlled experiment, summarized in §6.
• Narrow scope. Three companies, one workload each, modest replica counts. The results do not generalize to other tasks or configurations.
• As-shipped, not optimized. Each company was evaluated as delivered; a single bounded configuration hypothesis was tested. The point is to measure honestly first, then improve.

6Companion study: memory vs. control

A separate controlled experiment asks whether Fabri's trace-backed memory earns its cost. The company is trained on a related task — writing its learned guidelines to memory — then given a fresh holdout task twice: once with that memory copied into a clean compile, once with empty memory (the control). Across ten replicas of Support HQ, memory retrieved 2 trained guidelines on every run (control: 0) — the mechanism fires reliably. Yet the memory arm passed the holdout rubric only 7/10 (70%, 95% CI 40–89%), against 9/10 (90%, 95% CI 60–98%) for control. That is a descriptive −20-point gap. The Wilson intervals describe each arm separately, and no paired significance test was run, so this does not establish a statistically significant disadvantage for memory. A preliminary three-replica pilot had shown the opposite sign (memory 3/3 vs control 2/3, both with CIs spanning roughly 20–100% at n=3 — barely more than a coin flip's worth of evidence); the larger sample did not so much shrink that gain as flip its point estimate — the same small-sample fragility documented in §3. On this workload the self-improvement loop runs and is not shown to pay: it retrieves lessons reliably, and the runs that received them did not measurably do better.

The pattern is not unique to Support HQ. A ten-replica run on Reliability Labs shows the same shape: memory retrieved 2 guidelines every run (control: 0) yet passed 6/10 (60%, 95% CI 31–83%) against control's 7/10 (70%, 95% CI 40–89%) on the same 10-replica basis (control completed only 9 of its 10 scheduled replicas; the completed-only rate is 7/9, 78%, but that is not on the same denominator as memory's 7/10 — comparing like-for-like, memory trails by 10 points, and it cost ~19% more). As with Support HQ, this is a descriptive difference; no paired significance test was run. Across the two companies measured so far, trace-backed memory fires reliably but is not shown to improve reliability, and both point estimates run slightly the other way. Varying the retrieval configuration does not rescue it: raising top_k from 5 to 10, and switching to MMR-diversified selection, left the retrieved evidence unchanged — memory returned the same 2 guidelines every run in all three configurations, because the trained memory only contains ~2 guidelines. You cannot pull more than exists, so retrieval config is not the lever. The bottleneck is upstream — how many good, distinct lessons a training run mines and promotes — which is the next experiment worth running.

6.1Retrieval-configuration sweep

If memory's shortfall were a retrieval problem, widening or diversifying what gets pulled back should change the evidence the agent sees. Re-running the Support HQ memory arm with top_k raised from 5 to 10, and again with hybrid+mmr selection, tested exactly that.

Honesty / provenance (verbatim from the source result): "The retrieval-count invariance above is deterministic and was observed on every completed pair. A full ten-replica rubric re-measurement for each variant was not completed — repeated infrastructure interruptions cut the runs short (8 memory / 7 control completed pairs per variant before interruption). That does not affect the conclusion: the knob demonstrably does not change the retrieved evidence, so it cannot change the outcome."

7Reproducibility

Each published result records its own roster revision, company-source SHA-256, and Fabri version. Prompts, traces, session IDs, and raw model output are withheld; only aggregate metrics and rubric outcomes are released.