Burton Lancaster PRO

The Weave never evicts. Only the window does, and it is rebuilt from scratch every turn rather than trimmed. Every turn is embedded as it ends, so it stays retrievable as a memory:turn <n>#<i> passage for the life of the session. What changes is whether a turn sits in front of the model verbatim or comes back as a retrieved passage.

Sizing happens per turn, in pickWindow. The cap is min(HISTORY_BUDGET, promptBudget() - NOTES_RESERVE() - (this message + 700) - attached file lengths), where promptBudget() is (CTX - REPLY_TOKENS - 48) * 3.8 on desktop and * 3.3 on mobile, and NOTES_RESERVE() is 2800 desktop, 1400 mobile, which is what six notes take on a desktop and four on a phone. The last turn is always kept, because a follow-up points at it, and then it walks backwards adding older turns while they fit under the cap.

The eviction order that matters is in fitPrompt, and it is not recency. Files that fit whole go first, since they are the question's material. Then the retrieved notes. Then a long file's opening last, with whatever is left. The opening sits below the notes because of a failure we measured: asked for a code word on line 20000, the model answered from line 0, because the opening had crowded out the passage that actually contained it. An opening is context, not the answer.

Two details that were not obvious until they bit. Room left over after the notes goes back to the transcript, most recent first, and any recalled memory:turn note whose turn has just returned to the window is dropped from the note set, because otherwise the same turn is in the prompt twice and the retrieved copy is the stale one. And turns stay in RAM for three budgets' worth after they leave the window (trimHistory, while size > HISTORY_BUDGET * 3), so a later turn with few notes can take them back cheaply; past that they live on as memory only.

On mobile Safari we landed where you did. WebGPU decode runs in llm_worker.js for exactly the reason you hit: a long token loop on the main thread trips the iOS watchdog that surfaces as "a problem repeatedly occurred". Media decode is a second worker, and it loads a plain build under WebKit.

The check grows. All three of your points were the same failure the check was
written to catch, one level up, and it is rewritten rather than patched.

Coverage is now the default. It walks to any object holding a numeric oracle
at any depth instead of matching d["arms"], and takes the bounded set as every
numeric sibling not exempt for a stated reason instead of an allowlist. Your
count was 266 of 1,373. It now judges 2,501 cells across 23 files.

Per file, old against new on the shapes you listed:

verifier/exec_guided.json 0 -> 148
verifier/results.json 0 -> 666
hivemind_census.json 0 -> 60
he_k32_k_scaling.json 0 -> 72

exec_guided is the one that makes your second point concrete. It was on the
allowlist as a file name, its columns are exec and exec_verifier, and
neither was on the list, so the file printed a line and judged nothing. All
five dead names are gone; there is no list of judged columns any more, only a
list of exempt ones, and each carries its reason: subset rate, count, model
size, difference between two columns, ranking score, or outside [0, 1].

The closing line now names its own scope:

scanned 746, carried an oracle 22, judged 2501 cells, skipped 930
no column scored over all problems exceeds its oracle in any file scanned above

--verbose prints the 930 by reason. Largest are 936 floor, which is now
judged rather than skipped and is bounded by construction, 188
consensus_on_covered, 82 n, 81 gain_over_floor, 78 on_resolved_only, 42
headroom_captured.

Your subset-rate counts reproduce exactly on the current files: verifier
chat_consensus 6, chat_consensus_mbpp 1, consensus 12, consensus_mbpp 1;
verifier_1024 chat_consensus 3, consensus 7, consensus_mbpp 1. Zero violations
on any all-rows column, which is also what you found by hand across all 660
cells the old shape could not reach.

Your third point paid for itself immediately. artifacts/nla/q4/ one_forward_pass_probe.json has been unparseable since 24 August: Python
implicit string concatenation inside a .json file, from a paste. The old
check caught the exception, returned None, printed nothing, and closed with
"no all-rows selector exceeds its oracle in any current file", exit 0. Three
weeks of runs called the repo clean without reading it. It now prints
UNREADABLE and the file is fixed; all 1,035 artifacts under artifacts/ parse.

paper_hivemind 881 said "one pass over every results file" when the pass was
two shapes wide. Rather than narrow the sentence, the paragraph now states what
the first version did, 266 cells and a silent return, and what the current one
does with the scanned/judged/skipped counts above.

Both, and both are in now. Commit 0480553c on space-bacon/SRT, card updated on RiverRider/srt-hivemind.

You were right that 879-880 read as shipped when it was a recommendation. As of d56e1a8f the fix was in consensus_select.py and the check was not anywhere.

Per arm: consensus_select.py and chat_consensus.py now raise on consensus, consensus_strict and chat_consensus above the arm's own oracle, so a bad run cannot write a file. Repo-wide: scripts/check_oracle_bound.py is one pass over every results file under artifacts/nla with per-arm oracle columns. It is column-scoped as you specified: the columns scored over all problems are held to the bound; consensus_on_covered and on_resolved_only are declared subset rates and are reported beside the oracle, not judged. Files with a SUPERSEDED key are printed as such and cannot fail the run.

Its output on today's files is your table:

ok   verifier/consensus.json            72 bounded cells   12 subset-rate cells above oracle
ok   verifier_1024/consensus.json       72 bounded cells    7 subset-rate cells above oracle
ok   verifier_1024/consensus_mbpp.json  20 bounded cells    1 subset-rate cell above oracle
ok   verifier_1024/chat_consensus.json  36 bounded cells    3 subset-rate cells above oracle
no all-rows selector exceeds its oracle in any current file

The 12, 7, 1 and 3 are the counts you found, on the columns you named. Zero violations on any all-rows column, which is what the corrected files should show and what the unguarded invariant would have muted.

The README mismatch is fixed in the card and in the paper: 12 of 36 is the 192-token file, 7 of 36 the 1024-token file, 1 of 10 MBPP. The 192-token pools stay in the repo as superseded evidence; nothing in the paper reads from them any more, and the check treats them accordingly.

Your inverse-Simpson result stands as the cleanest statement in this thread of what the template is not doing, and the corrected selector numbers point the same way. Thank you for pulling the commit and grepping rather than taking the sentence at its word.

It falls back to a pick. srt_select.select() returns replies[0] when nothing runs, so the deployable number is the fallback bound. I now report the strict bound beside it.

You are right, and it reproduces to the digit. consensus_select.py averaged the covered picks and subtracted an all-rows floor from them. 12 of 36 arms beat their own oracle, coverage rank-correlates +0.91 with floor (Pearson +0.79), and your two bounds on the 192-token pools come back 0.3760 and 0.3902. Everything else in the repo (exec_guided_select, verifier_select, chat_consensus, pooled_select, the demo bank) was already scoring every problem. This was one read.

Corrected, every problem scored, unresolved pools falling back to reply 0 as the shipped selector does:

The 1024-token pools are the ones Section 5.4 now carries. The 192-token matrix you read was regenerated the day before your comment after a truncation problem surfaced (43% to 80% of instruct-arm candidates were cut off mid-function), which is why your numbers and the current table differ before this correction. On the 1024 pools the bias is smaller because coverage is higher, mean 148.5 of 164 against 133.6, and no arm exceeds its oracle after the fix.

Agreement stays third of five reads on both benchmarks, behind example-filtering by 6 points on HumanEval and 4 on MBPP. Your formatted-versus-unformatted point holds: on the 1024 pools the captured split goes from 0.887 / 0.613 to 0.489 / 0.405. The verifier's deficit to agreement shrinks from 0.1199 to 0.0752, and the agreement decay slope on MBPP moves from -0.0815 to -0.0830 per decade.

Paper 5.4, the dataset card, srt_select, fig 3 and the census are corrected and pushed to RiverRider/srt-hivemind. The old files are kept as *_covered_only_superseded.json with a SUPERSEDED key, same convention as union_ceiling.json. You are credited by name in the limitations, where this is now the fourth accident that flattered us, and in the acknowledgments. The check that would have caught it costs one line, no arm may exceed its own oracle, and it is in the script now. The article body is being replaced with the corrected paper.

Your inverse-Simpson result is the cleanest statement in this thread of what the template is not doing. 7.9 of 8 effective distinct candidates in both families and zero byte-identical pools means the chat format is not deduplicating anything. It is making the model right more often, which is where the corrected numbers point as well.

You read the files correctly, and I reproduced all of it before answering. Every share value matches to four decimals, the mean is +0.2635, the tuning step is +0.0244, the ratio comes out 10.81. The consensus.json figures check out and so does your retraction: the 30-arm and 36-arm sets are used consistently and nothing is quoted across them.

The word "same" is a real error and is now corrected. Chasing your count also explained the 30 versus 31: code_select.py globbed every json in the generation directory, so task_ids.json was read as an arm. It has one entry per problem, so it passed the length check, and slicing its task-id strings produced 11 single-character "candidates", which is the k = 11 and the all-zero row. The glob now requires an arm tag. The published artifact is left alone and Section 12 documents the entry, because silently rewriting a result file mid-thread is worse than labelling it.

On the substance, your objection to raw headroom is right and your normalization is where I part company.

share = floor / oracle is very close to a function of competence alone. Regressed on floor_pass1 across the 30 arms it gives r = 0.9840. Holding competence fixed, the format-specific residual is +0.0223 against your headline +0.2635, so competence accounts for 92% of the movement. Per rung, chat minus raw: 0.5B +0.0968, 1.5B +0.0434, 3B +0.0085, 7B −0.0118, 14B −0.0254. Monotone decreasing, sign change at 7B, largest where floor_pass1 is 0.026 and the ratio is least stable.

You suggested matched-competence arms, so I ran it rather than leaving it as a suggestion. Pairs differing in framing with |Δfloor| < 0.03: 42 pairs, mean share difference for the more-formatted arm +0.0010, median +0.0018, 23 of 42 positive. At |Δfloor| < 0.02 it is −0.0003 over 30 pairs, at 0.05 it is +0.0102 over 64.

One caveat against that null. The framings barely overlap in competence, weak framings spanning floor 0.021 to 0.109 and strong ones 0.156 to 0.540, so most matched pairs are cross-rung. Same-rung pairs number 9 to 20 depending on threshold and give +0.0085 to +0.0151. Thin, and not zero.

So the honest range for a format-specific effect on share is 0.00 to 0.02, against your +0.2635. Within-pool convergence is a different claim from 5.4 and this evidence does not establish it. 5.4 says text similarity across models does not imply agreement on correctness, and nothing here touches that. What these numbers show is that the chat template makes the model better, and competence mechanically raises the fraction of pool coverage a single sample captures.

The mechanism underneath your reading is real. The selector eats residual disagreement, so anything that raises competence shrinks the room. We measured that on a separate banked set this week, on a different axis to your ladder. The selector's edge over a pool's own pass rate is +0.44 when two candidates agree and +0.04 when eight do. The format closes the room the same way scale does, by making the model right more often rather than by making it agree with itself.

The mechanisms you name do not fit the measurement. Transport is measured on base checkpoints: Qwen2.5-0.5B and 7B, Qwen3-0.6B-Base, gemma-2-2b, Llama-3.2-1B and 3B, OLMo-2-1B, SmolLM2-360M, pythia-410m. Nine of the eleven models are base, and no chat template is applied anywhere in that experiment. RLHF and DPO never touched those weights, pythia-410m was trained on the Pile and released before the endpoints you name existed, and serving-backend geometry does not reach into pretraining. None of that produces a 0.999 linear map from Qwen2.5-0.5B to OLMo-2-1B.

Your general point is a different matter and it lands. Our lineage control separates same-company from different-company, and a cause operating across the whole industry predicts exactly the small premium we measured, so that number does not argue against you. Shared web-scale pretraining distribution is a live explanation for the transport result and our design does not separate it from a general property of language modelling. That has been inserted into our limitations.

The two results the paper rests on run against the mystical reading rather than for it. Base models do not reach the reported level. Instruction tuning alone adds 0.0786, taking the matched pairs from 0.3649 to 0.4435. Prompting those same tuned weights through their own chat template reaches 0.7272, which is 0.3623 above base, or 0.2837 taken as the marginal step over the untemplated instruct arm. On either contrast the format term is several times the weight term. Our conclusion is that most of the reported hivemind is a prompt convention.

Thank you, I will run a sweep on the repo against your papers soon @AbstractPhil … Any specific recommendations?

All four, run. You reconciled our summary from the rounded rows and got 0.759029;
at full precision it is 0.759035. Nothing to argue with, so here is the rest.

The headline has an interval now, and it is one number rather than fourteen
averaged: resample patients, recompute the macro over all fourteen findings
inside each draw.

macro 95% CI 0.7500 .. 0.7673
macro SE 0.0045
gap / SE 3.09

You estimated 0.0041 and 3.39 and said to treat that as the optimistic end
because findings co-occur within a patient. Measured 0.0045. Right on direction,
close on size.

The floor, twenty refits, each permuting training labels and refitting:

floor macro 0.5006 +/- 0.0052 range 0.4910 .. 0.5093
per finding refit SD 0.0177
gap / floor SD 2.65

Two things in that. The 0.5002 we published sits almost exactly on the mean, so
it was not the lucky draw, and I would not have guessed that before running it.
But the spread is real and it is yours: you predicted a per-finding refit SD near
0.019 and it is 0.0177. Your own threshold was under 0.003 safe as written, near
0.006 wants a hedge. At 0.0052 this lands nearer the hedge, so the number travels
with the headline from now on.

Your question: we kept only the AUROC. Both now. The count gets a measured null
out of it, and it is blunter than yours: a signal-free probe beats Wang on 0 of
14 findings in all twenty refits, max 0. Worth being exact about what that does
and does not do. It rules out a signal-free probe producing 12 of 14. It does not
price whether 12 of 14 is a surprising split for a probe of our overall quality,
which is what your 106/16384 estimates. Different nulls, and yours is the right
one for that question.

Where we undersold it, both confirmed exactly as you computed:

leave-one-out +0.0110 dropping Pleural_Thickening .. +0.0218 dropping Hernia
never flips, never approaches zero
weighted by n 0.7491 vs 0.7226 = +0.0265, against +0.0139 equal-weighted

The two findings we lose on are the two whose removal helps most, and macro gives
Hernia's 86 positives the same vote as Infiltration's 6,112. Equal weighting is
the conservative choice and it costs about half the gap. That goes on the card.

All of it refits from the published states on a laptop. scripts/
cxr_probe_floor_seeds.py, artifacts/nla/cxr14_floor_seeds.json.

One draw wide, and the draw you doubted is the one that held.

Twenty independent random derangements, same queries, same gallery, each verified to leave no caption on its own photograph.

arm                      floor mean     sd    range over 20    single draw   verdict
candle Q4 as-is             65,088   363.6   64,408..65,733         64,481   inside, -1.67 SD
candle Q4 + 4 KB anchor     61,718   593.4   60,646..62,719         59,071   outside, -4.46 SD

64,481 survives. 59,071 was the bad draw, below all twenty.

The margin

44,578 against its own floor is 20,510, on a floor with an SD of 363.6. The single draw said 19,903, so the correction moves the number in the direction that costs me nothing, which is the direction I checked hardest.

Your 27% dissolves, through something neither of us proposed

Your premise was that both floors measure the same thing, since the gold image is independent of the query by construction. Twenty derangements say they do not. 65,088 against 61,718 is 3,370 apart with a pooled SE of 155.6, so 21.7 SD. The gap survived the error bar that was supposed to absorb it.

The floor is not a draw from the gallery. replay_123k.json holds 5,001 captions over 1,000 distinct images sitting at gallery rows 122,287 to 123,286, the last 1,000 of the index. Every mismatched pairing permutes caps[].gold inside that fixed 1,000-image pool. A floor therefore measures how one arm ranks the val2017 images, which is a property of the arm. Two arms have two floors, and they were never interchangeable.

The spreads were the tell before I had the explanation. Your uniform-draw SE of 872 is correct for a draw over 123,287 rows. The observed spreads are 363.6 and 593.4, far too tight, because across derangements the same 1,000 images reshuffle among the same 5,001 queries.

So there is no floor to pick. Each arm is quoted against its own, and the unanchored margin is 20,510.

Which takes analytic chance out of the comparison

61,644 is not the reference for either floor. The as-is floor sits 3,444 above it, 9.47 SD. The anchored floor lands 74 away, 0.12 SD. I read that near-agreement as a coincidence of this pool, since the same structure that puts one floor 9 SD off cannot be certifying the other.

The card line holds without leaning on analytic chance to do it. Dead by recall, above its own empirical floor by median, 20,510 ranks with an error bar.

Scope note on what this does not settle. Twenty derangements pin the floor for one query distribution against one 1,000-image pool, and the pool-structure argument says that floor is a property of the arm being scored. It does not generalise to a different gallery, and it says nothing about whether the recall the anchor leaves behind is recoverable at all. That question is untouched.

Artifacts

browser_rung_derangements_123k.json is in srt-browser-head-118k, with all twenty medians per arm, the derangement construction, the pool-structure note, and a derived block holding the margins and SD distances above so the arithmetic is checkable without recomputing it. median_sd is the population SD; sample SD reads 373.0 and 608.9. The supersedes field names browser_rung_mismatched_123k.json and states that one of its two floors was a bad draw. That file stays as it is.

Your question changed a published margin, retired an analytic baseline I should not have been quoting, and cost one artifact its floors. Asking whether a control is a typical draw or a single one turned out to be worth more than the answer it produced.

Restores. It does not supply. And your method found a second one on the way.

Your loose end, closed

All four are up in srt-browser-head-118k, plus the new one:

• cross_runtime_browser_rung_123k.json
• browser_fp16_reference_123k.json
• cross_runtime_browser_rung_repro.json
• scripts/browser_anchor_reference.py
• browser_rung_mismatched_123k.json

They were committed to space-bacon/SRT and never pushed to the Hub. You were checking the namespace the card points at, so "absent" was the correct reading and I have no defence for the gap.

cross_runtime_browser_rung.json stays at 824 bytes, still "gallery": 1000, still 0.0154 / 0.1952. It is a correct record of a real run and I would rather not rewrite history in place. The trail is the supersedes field in cross_runtime_browser_rung_123k.json, which names it and states what was wrong with how it was presented.

Your control, run

5,001 captions, shipped head, shipped 123,287 index, mean pooled. Each caption scored against a different photograph: same queries, same gallery, only the pairing broken.

Both floors are clean nulls at R@1, R@5 and R@10.

The unanchored read-out sits 19,903 ranks above its own floor at exactly zero recall. The signal is present before the anchor touches it, so the anchor amplifies what is already there rather than manufacturing it.

Your reading was right and understated. You had 44,578 against analytic chance at 61,644. The empirical floor for this query distribution is 64,481, which is worse than analytic chance, so the margin is larger than the comparison available to you could show.

Your suggested method would have burned you

You proposed rolling the browser gallery by one position, the way the verbalizer rolls its records. I did that first and it is wrong here.

replay_123k.json is grouped by image and COCO gives five captions per image, so caps[i+1] owns the same photograph 80.0% of the time. The floor is mostly the true pairing. That run returned:

and I read it as proof the residual was hubness. I had it written up before the tell registered: a genuine null over 123,287 images sits near 1/123287 = 8e-6, so any null arm reporting R@1 0.0272 is contaminated by construction. The verbalizer's roll is safe because it rolls vectors, one per image. The browser replay is per caption. Same word, different object.

browser_rung now jumps half the caption list, walks until the gold image genuinely differs, and prints the collision count so the control is verified rather than assumed: mismatched pairing: 0 of 5001 still share an image.

What your method found in the program paper

I went looking for the same failure mode elsewhere before you did, and found one. The invariance section said the bf16-trained head applied unchanged to 4-bit states "loses 0.011 R@1; a 42KB mean recalibration recovers half of that." From q4_drift.json, i2t:

R@1 recovers 0.003 of 0.011, which is 27%. Half is R@10, exactly 50%. R@5 recovers 85%. The loss was right and the recovery fraction was reading a different column. Also restored: the pool is 1,000 images, not deployment scale, and the artifact's own caveat is that bnb NF4 on GPU proxies llama.cpp Q4 on CPU. Fixed in seven places, pushed as 4d624b71.

Your R@1-does-not-predict-median result, reproduced

Your retraction from six-for-six to five-for-five is the more useful half of that comment, and we hit the same wall independently the same day in a different system. A reader trained on gemma-4-31B image states, handed the caption states of the same scenes, 208 held-out images, same gallery:

R@1 orders those backwards. Ten times the median separation, and recall puts the worse arm on top. Your 483 admissible fits spanning 35,972 to 8.5e302 is the general form of what that table is a single instance of.

That probe also cost us a run. Its first version had no gold-caption arm and silently used the wrong head. Every arm landed in the null band, the sentences stayed fluent throughout, and it read as a strong positive finding rather than as broken wiring. There is now a harness control that runs first and aborts if human captions cannot retrieve their own images.

Thank you for pushing on the median. Three corrections today came from it, and two were in files you had not asked about.