Methodology

Three question types — the contract follows the question

Not every question admits the same kind of answer, so each debate carries a question type that fixes its answer contract. No separate option list is ever supplied in any prompt, for any type.

Every debate then runs the same four stages:

Replies use a structured JSON contract. Parsing, validation, scoring, and aggregation are deterministic; current runs use no LLM judge.

Canonicalisation & validation

Free-text answers need careful aggregation: “Jordan”, “MJ”, “Michael Jordan”, and “乔丹” should count together, while a genuinely new answer must not be forced into a familiar name.

1. Clean the answer. Surrounding quotation marks and punctuation are removed without changing the model’s wording.
2. Check the published alias table. data/aliases.json maps spelling variants, last-name-only forms, abbreviations, and Chinese forms to canonical names.
3. Use normalised matching. Matching is case-insensitive and strips diacritics. A compact, space-free comparison handles Chinese names written with different punctuation or spacing.
4. Allow a tight token match. A short form can match a longer reference name, or a reference name can match an answer with at most one extra token. Sentence-like non-answers are not rewritten into clean picks.
5. Preserve everything else exactly. If no rule matches unambiguously, the cleaned answer remains verbatim and is counted under the model’s own wording.

For open debates, the stored reference list is used only for canonical matching and the visitor-vote options; it is never shown to models, which can answer with any name they choose. For closed debates (yes/no and choice), the debate’s stored list is exactly the alternatives named in the question itself, and it doubles as the validation set and the visitor-vote options.

Closed-answer validation

Canonicalisation still applies to closed answers — for spelling and language variants only (Coke → Coca-Cola, 猫 → Cats, 是 → Yes). After canonicalisation, a closed answer is valid only when the pick is one of the alternatives named in the question; the ranking is an exact permutation of those alternatives, with the pick first; and the points keys are exactly that set. Anything else — a third option the question never named, a hedge like “Maybe”, a topping, a sentence — is invalid. The runner permits one corrective retry that names the problems; an answer still invalid after that retry fails openly and is logged.

Points validation

For open debates, an answer is valid only when the pick is one short, non-empty name; the top three starts with that pick; points keys belong to the model’s own top three; at least two entries are scored, including the pick; every value is finite and non-negative; and the pick has the strictly highest points. For closed debates, the pick must be a named alternative, the ranking a full permutation of them, and the points keys exactly that set, with the same finite, non-negative, strictly-highest rules. Valid points are normalised to 100 for display, with the original sum stored. An answer still invalid after its corrective retry fails openly and is logged — it is never silently repaired or fabricated.

Scoring definitions

Criteria audit

Each pick response must return exactly three criteria that the model says it weighted most. Bias Gauge displays those stated criteria beside the result and treats them as part of the answer, not as independently verified reasoning. The audit can reveal which considerations models volunteer — titles, peak performance, longevity, influence, taste, and so on — but it does not prove those criteria caused the pick.

No judge model extracts criteria in the current runs. If a judge is introduced later, its identity and role will be disclosed here before any dataset produced with it is presented as live.

The menu-era confound

The first live design used forced multiple choice: every prompt supplied a fixed candidate menu, and the model had to choose from it. After that run, Linshuo identified the central problem — the models were being given the options. A check of the stored results found that the consensus pick equalled the first-listed candidate in 24 of 24 debates.

That pattern cannot be interpreted cleanly. The debate author had generally listed the conventional favourite first, and language models can also favour early options. Author ordering and model position bias were entangled, so the menu results could support same-menu comparisons between models, but not strong claims that a winner was each model’s spontaneous answer.

The menu design was retired rather than patched with a shuffled list. The current design removes the menu entirely, preserves each model’s own answer wording, and publishes the earlier result as a cautionary comparison. Menu-era data and the open-ended comparison are retained in the run reports under data/runs/.

A second refinement followed the same day: the first open-ended run applied one contract to every question, and closed questions produced out-of-contract answers — a pineapple top three of “Yes / Pepperoni / Mushroom”, a pick that was literally “Pineapple”. Those debates were re-run under the question-type contracts described above; the open debates’ answers were carried over unchanged.

Limits & caveats

• One run is a sample, not a distribution. A single answer per model and debate cannot show how often that model would give the same answer across repeated independent runs.
• Answers are elicited results, not beliefs. Wording, language, decoding settings, provider routing, and model updates can all change an answer.
• No ground truth exists for these questions. Agreement is a descriptive result, not evidence that the consensus answer is correct.
• Canonicalisation is fallible. Aliases and token rules are published so ambiguous matches can be audited; unmatched answers remain visible under their own wording instead of disappearing.
• Hedging is under-measured in JSON replies. The detector can only score language present in the stored raw answer.
• Human votes are context, not a controlled baseline. Visitors self-select and vote from the reference list (for closed debates, exactly the alternatives named in the question). Models answer freely on open debates and within the named alternatives on closed ones.

The English and Chinese prompt templates are published in src/data/prompts/, the alias table is published as data, and scoring code is in src/lib/scoring.ts.

Publication, cadence & cost

Before the first live run, the site used clearly labelled deterministic sample data. A live dataset now replaces the previous dataset as a unit through POST /api/runs/ingest only when at least 95% of attempted model–debate pairs succeed. Failed pairs, validation problems, retries, and known costs are recorded in the run report rather than hidden.

There are no scheduled runs: a re-run happens when one is commissioned. Every response is stamped with a runId and an answeredAt timestamp captured when that answer was obtained, each run is preserved as a snapshot under data/history/, and data/runs-registry.json indexes the runs — so answers are effective-dated and comparable over time. The runner estimates cost in advance and stops at the configured RUNNER_MAX_USD budget cap. Actual usage-accounted cost is reported for each run when the provider returns it.