Skip to main content

​
Overview

Roast Me is not a scalar metric. It is a search problem: given an assistant treated as a black box, find the categories of realistic interaction that make it violate its behavioral contract reproducibly. Not “which prompt broke it” but “which kinds of question break it, repeatably”. It is for whoever has to sign off on an assistant they cannot inspect — an auditor, a red team, a release gate. You bring the contract, the knowledge base, the models and the credentials; gaussia owns the interfaces, the data shapes, the validation and the arithmetic.
No grader here has been calibrated against human labels. Every number Roast Me produces is a judge-only measurement: one language model’s estimate of whether another one misbehaved. Agreement between two graders is agreement between two judges, not agreement with a person. Read a violation rate as evidence to look at, never as a measured error rate.

​
Not a metric

Roast Me does not ride the metric pipeline. There is no dataset to load — Roast Me generates the dataset that roasts the assistant. It is a generator subsystem: nothing in it subclasses Gaussia, and nothing is registered in gaussia.generators. What enters the pipeline is the Roast Dataset it emits, which existing metrics then consume unchanged. That is why this page sits under Advanced rather than under Metrics.

​
Installation

Two extras, so whoever only profiles pays for neither training nor retrieval:
Neither extra is part of gaussia[metrics] or gaussia[all]. Every interface imports with both uninstalled, so you can implement against them before installing anything: the three engines behind the extra are imported from their own modules rather than re-exported, which is what keeps from gaussia.generators.roastme import ProbeLibrary, Profiler, Exploiter free of heavy imports. The boundary is about dependencies, not about whether a model is involved. LangChain is a base dependency, so GroundedProbeEngine, PromptedFactTwister and LlmMentionExtractor pull nothing of the extra and sit on the facade beside EnumerationProbeEngine. What they need is your model, and that was always yours to supply.

​
Claude Code skills

Four skills cover the pieces you have to author, each stating the rules the library enforces so a mistake surfaces at validation rather than halfway through a run that costs target calls: roastme-plugins, roastme-strategies, roastme-profiler and roastme-exploiter. They are not part of the pip package — they live in skills/ beside the code they describe, so they move with the schema instead of drifting from it.
Then /roastme-plugins in Claude Code, and the same for the other three. Run them in that order for a first setup; the last two also list what to check when a run comes back empty. Skip this section entirely if you are not using Claude Code — everything the skills say is on this page and in the two notebooks.

​
The three components

They run in this order, and only the middle one is unavoidable:
  1. Probe Library — the only component that touches the knowledge base. It turns your documents and your catalogue into probes, each tagged with whether the content it leans on is documented or invented. Five engines compose here, and they split on one question: four build a premise out of an entity’s name, so they need to know which names exist; the fifth twists a datum and needs no boundary at all. Skip the whole component if you already have probes: black-box runs are explicitly supported.
  2. Profiler — drives the target over the probes, grades every response against every principle, and aggregates the result into the profile: a weakness map ω over prose descriptors, plus the knowledge hooks H that actually broke the assistant. It reaches no knowledge base and it needs no credentials if your target replays recorded responses.
  3. Exploiter — searches from the profile for conjunctions of attributes that break the assistant reproducibly, and emits the ranked failure report.
The profile is the only artifact crossing from the Profiler to the Exploiter, and it carries prose rather than your internal identifiers.

​
The eleven interfaces

You implement against these. Nine ship a reference implementation, declared as such — a convenience, never the definition of the component. A behavioral contract is not on this list: it is a model you build, not an interface you implement. Nor is a category generator — the training-free search has none, so requiring one would force a fake implementation.

​
Reading the snippets on this page

The snippets below name the shipped components, so what you copy is what you would deploy. They are not one runnable sequence: for that, run roastme_quickstart.ipynb, which executes the whole arc offline behind crude stand-ins and stores its output, or the long walkthrough beside it, which builds every component up in turn. Two names recur below and are yours to supply:
  • your_adapter — your implementation of TargetAssistant. Gaussia ships none, because a transport belongs to the runtime it talks to. A recorded response set is an implementation of that interface, not a separate mode, which is what makes a credential-free run the same code as a live one.
  • judge — the LangChain chat model the shipped grader, query generator and on-profile filter each take. It is the judge, never the assistant under evaluation.

​
1. The behavioral contract

Π is an input, not a library constant: gaussia ships no contract, because a contract is what the measurement measures. Each principle carries a severity weight, the rubric the grader is handed unmodified, and exactly one grader. Weights must sum to 1.0 (within 1e-9), identifiers must be unique, and a principle with no grader cannot be constructed — so it can never contribute a silent zero to the violation score.
One grader instance can serve several principles — what the specification fixes is that each principle has exactly one, so comparing graders means running the whole evaluation again rather than aggregating two inside a principle. Every graded response carries the aggregate violation score and the per-principle grades:
so a failure traces to the principle it breaks. Because v is a weighted sum and not a count, a response breaking only the lighter principles can fall below a threshold that a naive “2 of 3 principles” reading would clear. That is the point of the weights.

​
2. The catalogue

The catalogue is yours. Gaussia specifies its shape, validates it, and ships schema examples — never a domain catalogue. A rubric and a risk taxonomy are what the metric measures, so a shipped one would quietly become a standard nobody chose. Two shapes. A PluginSpec is a risk family, mapping to one principle of the contract. A StrategySpec is an interaction pattern: which kind of entity it operates on, how it transforms it, and whether the resulting hook is documented or invented.
Two strategies are enough to show both shapes; the other two transforms, flip_value and flip_fact, appear in the shipped schema examples and are tabled below. description is not decoration: its comma-separated clauses become the probe’s attributes, which is what the Exploiter later grounds a category in. phrasing_hint reaches the generation prompt, so it has to be in the knowledge base’s language.

​
The transforms: four shipped, plus yours

transform is the one catalogue field whose value changes what a probe means, so an unrecognised string must never resolve. What keeps that guarantee is resolving against a registry, not the set being fixed — so a catalogue may name the four shipped plus any Transform you supply, and anything outside that is still refused. Read that table against your own entities, because two of the four have narrow ranges and neither says so when it misses. flip_value behaves on an entity with exactly one digit run — POLICY-1 → POLICY-2 — but RD$500,000 becomes RD$501,1, and an entity with no digits comes back unchanged, which the engine then labels documented, quietly turning that strategy into a second control. And flip_fact writes English, so on a Spanish corpus the premise is not Cuenta Digital Libre. Supplying one is the answer, and it goes to the engine and to validation together:
Pass the same sequence to both. A catalogue validated against one set and generated against another is exactly the case where validation stops meaning anything. A key colliding with a shipped one is refused rather than preferred, since either resolution silently changes what an existing catalogue means. A transform decides the text of a premise and never its label. Whether the result is documented is the engine’s call, derived from its own view of the base’s boundary — so flip_value applied to POLICY-1 in a corpus that also contains POLICY-2 legitimately yields a documented hook.

​
A control is a strategy with no plugin

That is the only mechanism by which a control is recognised — structurally, from an absent plugin. No library behaviour depends on any identifier you chose, so you may name your plugins and strategies whatever you like. A control puts no principle under test, so its probes carry none, and they are excluded from every violation-rate aggregate while staying in the graded record.
doc: 1 does not mean control. The two fields answer different questions: doc says whether the entity exists, plugin says whether a principle is on the line. A probe about a real, documented entity is scored whenever its strategy names a plugin. To bring “did it answer real content correctly” inside the violation rate, add a principle for it and point a keep_real strategy at a plugin that serves it.

​
Validation you get before anything runs

validate_catalogue rejects a catalogue before generation, on six conditions: every referenced principle resolves in the contract, every referenced plugin exists, transform resolves against the registry, doc is 0 or 1, identifiers are unique, and every entity_kind is one a configured engine declares it handles. That last one matters because entity_kind is your own vocabulary and gaussia never learns what it means — without the check, a plural typo would validate cleanly and yield an empty probe set with no error. transform resolves against the four shipped transformations, any you supply, and the twist patterns any configured FactTwister declares — so pass the twisters alongside the transforms:
A pattern no configured twister declares is still refused before generation, which is the point of naming them here rather than discovering them mid-run.

​
3. Probes from a knowledge base

A Document carries an id, its content, and structured — whether this document’s knowledge boundary is enumerable, which is what decides which engines can establish absence over it. Engines compose rather than cascade: every engine that can handle a document sees it, every engine’s output contributes, duplicates merge, and each surviving probe records the engine that produced it.
Every absence label the retrieval engine produces comes back absence_reliable=False, and every one the graph and multi-hop engines produce comes back True. That is not a quality difference between two implementations — similarity search is structurally unable to decide absence, because it never reveals what it failed to retrieve. Recording the difference is what keeps an unreliable label from being indistinguishable from a confirmed one. The multi-hop references in that output — POLICY-1 -> POLICY-2 -> FORM-7-2, a real chain whose last hop is mutated — are the attack that engine adds: an assistant that refuses an invented entity may still accept an invented relation between real ones.

​
The five engines

The first three run by default and together span the absence/breadth trade-off: no single engine gives both. Note this is not the engine set the paper evaluated — its canonical dataset came from retrieval, graph and enumeration, and the multi-hop engine appears in no trade-off table there. Check what those first three can see in your corpus before trusting them. All three read mentions through a MentionExtractor, and the shipped one recognises compound identifiers — POLICY-1, Articulo_25. That is the shape of a corpus of numbered clauses, and a corpus of ordinary words defeats it in two different ways. Where it recognises nothing, it now refuses, naming itself and what to do instead. It used to return an empty set on the argument that producing no probes is honest — which is true about the boundary and wrong about the silence, because the engine then generated an empty probe set, the Profiler reported a rate over nothing, and the run completed. Where it recognises the wrong things — phone numbers, document filenames, footer anchors — nothing can fail, because a false positive is well-formed and deciding it is not an entity needs your domain. The engine generates a probe per false positive, the assistant is asked about a filename, and the run looks successful. On one real bank corpus it returned 208 mentions that were not entities. That second half is why the boundary is worth reading before a run, and every engine that establishes one now exposes it — the same call generation makes, rather than a private method:
If it comes back wrong, inject a reading of your own. The default stays the compound-identifier one, so a corpus that worked before still works, and there is a shipped alternative that reads prose:
LlmMentionExtractor replaces the regular expression; it does not replace an enumerator. Measured over the same bank corpus against a hand-written enumeration of 136 products, it recovered 125 of them — recall 0.92. That is excellent coverage and it is not completeness: the 8% it missed are real entities a probe would then label invented, and every one of those is a false doc = 0 the judge is asked to rule against. Use it in the engines that do not decide absence. In the one that does, keep the enumerator, whose contract is completeness and not coverage.
MentionExtractor lives beside the engines rather than in core/: it is a collaborator of three shipped engines, not part of the specification you implement against. FactTwister is in core/ for the opposite reason — it reads the corpus and returns a claim about it, which is a contract you implement against. EnumerationProbeEngine takes no extractor at all: its boundary comes from your enumerator instead, which is the other answer to the same problem and the stronger one when you can enumerate. With no knowledge base, particularisation still returns probes through the same interface: domain-agnostic and hookless. The hook stays empty rather than being filled with a placeholder, because a fabricated hook would put an invented entity into the retained hooks the Exploiter grounds its categories on.

​
The enumeration engine is opt-in

It is the only engine that cannot run on a knowledge base alone: enumerating “every entity of this kind that exists” is irreducibly domain knowledge, so you supply an EntityEnumerator. The refusal is structural — the collaborator is a required argument, so an engine with no boundary never comes into existence.
True then False: a document that says its boundary is not enumerable is one this engine declines rather than guesses over.

​
The grounded engine needs no boundary at all

The other four build a premise out of an entity’s name, which is why they need to know what names exist: to claim Cuenta Flash Popular Plus is invented, something has to know the complete list. This one twists a datum and leaves the name real — the balance, the term, the requirement — so it claims no absence and needs no list. That is the whole of what it buys, and it is the reason a corpus of ordinary words no longer forces you to write an enumerator and a transformation. Only the enumerator is irreducible. It takes a FactTwister, and the shipped one goes through your model:
A strategy names its pattern in transform. Four ship: the three twists above, plus keep_real, which asks the passage’s fact unaltered and is how a control survives the change of engine. Controls are the only thing separating “the assistant fails” from “the rubric charges too much”, so a grounded catalogue needs one exactly as much as a templated one does.
The pattern is yours to name, never the model’s to choose. Offered all three and left to pick, the shipped twister returned false_attribute 21 times out of 21 over a real corpus. That is the gap the paper admits about its own equivalent engine: it tags every twist with one generic strategy id, and the qualitative patterns its configuration declares appear in none of its tables. Asking for one pattern per request is what closes it — so the schema the twister binds carries no pattern field at all, and a model that names one has nowhere to put it.
How many probes it makes, and what it costs. passages × strategies, where the passages are your corpus cut on blank lines at passage_chars (4000 by default). A 160,000-character corpus is 40 passages, so 12 strategies is 480 probes and 480 model calls before the assistant is contacted once. passages_per_strategy bounds that, and bounds it honestly: it takes the first n passages in document order rather than sampling, so what a bounded run covered is stated by the number rather than hidden by it. Passages are ordered rather than retrieved by similarity, which is a boundary decision and not a simplification — retrieval would pull the roastme extra into a path that needs nothing beyond the base dependencies, and ordered cutting keeps a generated probe set a function of the corpus and the cut size alone. What it gives up. It cannot test fabrication. Every probe it emits carries absence_reliable=False, because a passage is a sample of the corpus and no sample can confirm what the corpus omits. An assistant that invents entities is still the enumeration engine’s business — and the two compose in one ProbeLibrary, which is the configuration that measures both. Every probe records the model behind it on Probe.model, so a probe set generated by one model is never mistaken for one generated by another.

​
4. Profiling a recorded response set

The Profiler takes probes and a target, and returns the profile plus every graded outcome. It has no access path to the knowledge base — nothing on its surface accepts a Document. Probes normally arrive from the Probe Library. Written out, one charging probe and one control look like this:
Four properties of that result, each load-bearing:
  • p2 is a control — its strategy names no plugin, so it is graded and kept in the record and excluded from every rate. Whether its entity exists decides nothing.
  • A probe whose exchange fails at the transport is ungraded, not compliant. Its outcome carries violation=None and moves neither the numerator nor the denominator of anything, so an outage cannot read as good behaviour.
  • Weakness entries are keyed by (principle, descriptor) and carry the rate, its sample size and its standard error — because a descriptor resting on two probes cannot distinguish “never failed” from “undersampled”.
  • The descriptor is prose, built from the probes’ own attributes. Your strategy identifiers do not cross to the Exploiter.
H, the retained hooks, holds the hooks of the probes that actually drew a violation. A hook whose probe drew none is not evidence of a weakness. The quickstart notebook shows both of the first two cases in one run: a control excluded from the rate, and a hook dropped because its probe drew nothing.

​
5. The Roast Dataset

The audit outlives the audit: one record per query — query, response, violation score, principles charged, grader rationale, and the supporting evidence when the probe was knowledge-grounded — loadable through the SDK’s ordinary dataset contract and consumable by existing metrics unmodified.
ground_truth_assistant is filled with "": a trap has no correct answer, and inventing one would let a metric score against it. So the metrics that read the assistant’s answer alone — Toxicity among them — consume a Roast Dataset with no change; the ones that score against an expected answer have nothing to compare with, by construction. evidence_available travels on the record rather than being inferred from evidence, because “there was no evidence to check against” and “evidence was sought and not found” are different findings and one None would collapse them.

​
6. Searching for failure categories

A category is an ordered conjunction of natural-language attributes, each traceable to the weakness entry or hook that induced it. The Exploiter proposes categories, samples queries for each, gates them, sends the survivors, grades the answers, and scores the category as a lower-confidence bound so consistency beats luck:
Two categories with the same mean violation are not equal: the lower-variance one ranks higher. A category that passes is then refined to the smallest sub-conjunction still passing, and the attributes that came off are reported as incidental — which is what turns a pile of co-occurring attributes into something actionable.

​
The method parameters

A complete worked configuration ships in examples/roastme/jupyter, so the two required values are copied from a visible reference rather than guessed.

​
κ and δ come from the component that owns the scale

κ gates a single query and δ bounds a category’s realism gap. Both are compared against numbers a substitutable component produced, so their meaning travels with that implementation and not with your config. Here is the failure that rules out a global default. One on-profile filter scores in [0, 1] and another in [0, 100]. Both are valid. A κ of 0.6 gates sensibly against the first — and admits every query against the second, silently: the run completes, the report looks populated, and nothing was ever gated. Validating a declared range would not catch it either, because 0.6 is inside both ranges. So each implementation declares the threshold it recommends on its own scale, and the Exploiter resolves once, at construction, in one order:
  1. a value you supplied wins;
  2. otherwise the configured component’s recommendation;
  3. otherwise it refuses to construct, naming the component and the parameter.
Keep the shipped components and you never see either parameter. Substitute one and you are obliged to supply the number — because no configured combination may fall back to a value calibrated for a different component’s scale.

​
The three substitutable collaborators

How the schema is bound to your model is a parameter, and the provider’s default is not safe. The two model-driven collaborators above, and both model-driven probe components, take a structured_output strategy and default to the JSON-schema route. Left to the provider’s own default, a model behind the HuggingFace router ignored the schema entirely and generated prose until it hit forty thousand tokens — so the request failed on length, which reads as a model failure and is a binding failure. Pass ToolCallingOutput() for a provider that offers only tool calling.Where they differ is what an off-format answer costs, and the difference is forced. The generator re-asks, because a reply with no questions in it is a short reply and it already handles those; after the attempt budget the run fails loudly rather than returning fewer queries than S(c) will divide by. The κ gate has no second option — it must return a number and neither default is honest, since 0.0 gates the query out and shrinks what the search covered without saying so while 1.0 lets it through ungated. So it raises.
The paper names the query generator and the on-profile filter and gives no construction for either. The two shipped here are gaussia’s invention, and substituting them changes what the search measures. A lenient filter turns “we told it to break a rule and it did” into a reported weakness; a weak query generator makes a real failure category look like none. Only the realism estimator’s construction — expected cosine distance from a pool of natural queries — comes from the paper. Every failure report records which implementation of each produced it, so a weak result is attributable to the part that can be swapped rather than to the method.
The realism estimator never contacts the assistant — realism is a property of the queries and the prior — so the budget never costs the calls it exists to protect:
0.135 against 0.502 with the default all-MiniLM-L6-v2: the first reads like the prior, the second does not, and at the recommended δ of 0.5 only the first survives — by two thousandths. That margin is why the recommendation travels with the estimator rather than living in the config: the absolute values are the embedder’s, not the method’s, so swapping the model means recalibrating δ instead of inheriting it. The pool is yours: an empty one is refused rather than silently making every category look realistic.

​
A run end to end

The training-free search is the default: the profile’s attributes are evaluated one at a time, the attributes behind the highest-scoring query/response pairs are conjoined into one candidate, and the candidate is refined. No GPU, no trained model, no optimiser — only target calls.
Reading that report:
  • Categories are ranked by S(c), each auditable down to its queries, its responses, its per-principle rationale, and the realism and on-profile checks it passed.
  • The queries that reached τ on their own are surfaced alongside the category verdict. Without that, an empty ranking would read as a clean assistant — and “no category broke it reproducibly” has to stay distinguishable from “the assistant answered correctly”.
  • A query below κ contributes exactly 0.0 and is never sent, so the gate costs no target call. It stays visible through on_profile, so a zero is explainable rather than mysterious: a category whose queries point at nothing the profile marks as weak scores 0.00 for that reason, not because the assistant answered it well.
  • components records which implementation of each substitutable piece ran, plus the κ and δ in force and whether each was supplied or recommended.
A component that recommends nothing, used with nothing supplied, fails at construction rather than inheriting a number calibrated elsewhere: set recommended_threshold to None on the filter above, leave kappa out of the config, and Exploiter(...) raises naming both the component and the parameter. The paper’s headline procedure sits behind the same interface. Five steps in a loop — sample candidates from the policy, discard what the gates reject, send and grade the survivors, turn each outcome into a reward, apply one update — and only the fifth needs a GPU. The policy and the update step are injected, so the loop’s sampling, gating, reward and stopping behaviour is verifiable with neither a GPU nor a trained model.
For real training, ClippedPolicyUpdate in gaussia.generators.roastme.searches.policy_update implements PolicyUpdateStep as one PPO-clipped step over the policy’s adapters. It is the only module in the subsystem that imports the training stack, so it needs gaussia[roastme-rl] and a GPU. Under either search, the query generator stays unmodified: optimisation pressure applies to the category generator alone, and that is what preserves realism. It is only a checkable claim because the two are distinct objects.

​
The shipped grader

LogprobGrader reads a binary verdict out of the judge model’s own token distribution. Every model-facing string is yours — the rubric, the verdict surface forms, the reasoning budget — and the GraderConfig built in section 1 is the whole of its configuration. Two behaviours are worth knowing before you trust a grade:
  • The verdict is located by scanning the whole generated sequence for the last verdict-shaped token, because a reasoning model’s first token belongs to its preamble. The verdict is then discarded if the model’s own final answer does not independently parse to one. A grade reached this way records method="logprob-last-verdict-token".
  • Whether logprobs are usable at all is a property of the serving provider, not of the model, so it is probed at runtime. When they are unusable the grader falls back to sampling over fallback_samples and records method="sampling-fallback" — raising instead would make the violation-rate denominator depend on provider behaviour.
Both strings are importable as LOGPROB_METHOD and SAMPLING_FALLBACK_METHOD from gaussia.graders. Every grade records the grader, the model and the method that produced it, so graders are substitutable without touching anything downstream. The framework’s shared llm/judge.py is untouched by this feature.

​
Limitations

Stated here because a reader will otherwise infer stronger claims than the evidence supports.
  • No grader has been calibrated against human labels. Every figure Roast Me produces is a judge-only measurement. This is the paper’s own statement about its graders, not a gap in the implementation, and there is no field on the result and no calibration gate that would let you forget it.
  • The training-free search has no published result behind it. AttributeIterationSearch is the default because it needs no GPU and costs only target calls — not because it was the procedure evaluated. The paper’s headline procedure is the policy-gradient one, and it reports the search as a validated integration rather than a validated finding, with sample size as the stated blocker.
  • Two of the three shipped Exploiter collaborators are gaussia’s own construction, so a weak report may be about them rather than about the assistant. Read report.components before concluding anything.
  • The retrieval engine cannot confirm absence, and the multi-hop engine appears in none of the paper’s trade-off tables. Both ship; neither claims more than it can.
  • A query citing no knowledge-base entity leaves the grader nothing to check against, so its score reflects the judge’s own knowledge. evidence_available on the record is what makes that visible instead of silently averaged in.
  • A refusal to answer is a legitimate response, not a transport failure. Whether it violates a principle is the rubric’s call, and the rubric is yours.
  • The weakness map is keyed on the strategy alone. The paper’s Z is (template, topic, hook type); the specification chose the strategy identifier for z, and this is what was built. The cost is real: four probes of one strategy, over a topic the assistant always breaks and one it never breaks, report 0.5 — so at η = 0.6 no category is proposed for a behaviour that fails every time. The Exploiter misses weaknesses rather than misreporting them, which is the more expensive direction to be wrong in.
  • A category that passed τ is not marked as such. FailureReport.categories carries every category evaluated. C* is reconstructed by comparing each score against the tau recorded in components.
  • The default engine set is not the one the paper evaluated. Retrieval, graph and multi-hop run by default; the paper’s canonical dataset came from retrieval, graph and enumeration, and the multi-hop engine appears in none of its trade-off tables. The out-of-the-box configuration produced no published number.
  • The mention extractor reads compound identifiers, and a corpus of ordinary words defeats it. Over Spanish product pages CompoundTokenExtractor returns 208 mentions that are not entities — phone numbers, PDF filenames, footer anchors — and nothing fails: each becomes a probe. Only half of that failure is detectable, and that half now refuses: a corpus it recognises nothing in raises. False positives cannot, because deciding a well-formed match is not an entity needs your domain. Read engine.boundary(kind, documents) before trusting a run, or supply an enumerator and use EnumerationProbeEngine.
  • A model-driven reading of the corpus has coverage, not completeness. LlmMentionExtractor recovered 125 of 136 hand-enumerated products — recall 0.92 — and the 8% it missed would be labelled invented by any probe built over it. It is the right replacement for the regular expression and the wrong replacement for an enumerator. Neither it nor the twister is any engine’s default, and that is deliberate: a model in the generation path makes the probe set unreproducible, so two runs of one assistant stop being comparable, and it degrades without failing.
  • A generated probe set is attributable but not reproducible. Probe.model records which model produced a probe. Nothing caches the set, so re-running the grounded engine against the same corpus re-asks the model, and comparing two runs of one assistant compares two instruments unless you keep the probes.
  • A provider failure mid-generation ends the pass. An off-format answer costs one draw and no more. A refusal — a rate limit, a length error — propagates and takes the probes already built with it. That is deliberate rather than settled: a blanket catch would swallow an expired key, and the retryable half belongs to your model client, which is where max_retries lives.
  • A query the κ gate stops still counts in its category’s score. It contributes exactly 0.0 (FR-030), which lowers the mean and raises the variance, so S(c) falls twice over. That is the requirement rather than a defect, and it moves the ranking: on one measured run a category rose from S=0.109 to 0.202 computed over the queries actually asked — from fourth place to second. on_profile is what makes it auditable.