Leaderboard CMASS v1.0 · v2026.10.25

Search agents on CMASS

Each agent runs in the Piika harness with two tools, search and read_document, over a BM25 index of the 553M ClimbMix documents. Closed-book runs answer the same questions with every tool disabled. Within each dataset, rows are ranked by accuracy with retrieval tools.

#	Model	Harness	With retrieval tools					Closed-book
#	Model	Harness	Acc.	P@all	R@all	H.G.	T.C.	Acc.
BCP80 questions · 11,450 qrels
1	GPT-6.1 SolAPI OpenAI · high reasoning effortHarness: Piika	Piika	85.0	38.2	12.1	75.5	21.5	52.5
2	Gemma 4 31Bopen weights Google DeepMind · IT, thinking enabledHarness: Piika	Piika	27.5	10.9	4.2	31.4	23.5	16.3
3	Qwen 3.5 9Bopen weights Qwen · thinking enabledHarness: Piika	Piika	16.3	9.2	4.4	31.4	49.9	5.0
TRQA68 questions · 8,307 qrels
1	GPT-6.1 SolAPI OpenAI · high reasoning effortHarness: Piika	Piika	94.1	70.9	10.6	79.7	8.6	86.8
2	Gemma 4 31Bopen weights Google DeepMind · IT, thinking enabledHarness: Piika	Piika	66.2	59.2	17.0	82.7	12.4	50.0
3	Qwen 3.5 9Bopen weights Qwen · thinking enabledHarness: Piika	Piika	57.4	59.4	18.6	85.8	19.7	39.7
DSQA71 questions · 9,889 qrels
1	GPT-6.1 SolAPI OpenAI · high reasoning effortHarness: Piika	Piika	90.1	50.5	15.0	75.7	17.9	70.4
2	Qwen 3.5 9Bopen weights Qwen · thinking enabledHarness: Piika	Piika	39.4	35.3	19.8	72.0	37.1	12.7
3	Gemma 4 31Bopen weights Google DeepMind · IT, thinking enabledHarness: Piika	Piika	38.0	33.2	17.4	68.1	21.5	23.9
HotpotQA204 questions · 11,090 qrels
1	GPT-6.1 SolAPI OpenAI · high reasoning effortHarness: Piika	Piika	94.6	57.9	17.2	90.2	6.7	89.7
2	Qwen 3.5 9Bopen weights Qwen · thinking enabledHarness: Piika	Piika	70.1	44.4	19.7	83.9	17.0	27.5
3	Gemma 4 31Bopen weights Google DeepMind · IT, thinking enabledHarness: Piika	Piika	65.7	44.4	18.1	78.4	10.2	36.8

Table 3 of the CMASS paper plus P@all, on CMASS v1.0 (release v2026.10.25). All values are percentages except T.C., a mean count. Bold marks the best value per dataset; tool calls are not ranked. One run per model, dataset and condition.

Metrics

What each column measures

Acc.: Percentage of all questions answered correctly; unanswered questions count as failures. DSQA uses strict exact-set equality: every gold member and no incorrect additions, in any order.
P@all: Precision@all: the percentage of the distinct documents in a question's search results that are qrels, averaged over questions. Unjudged documents count as non-relevant, so it is a lower bound.
R@all: Recall@all: the percentage of a question's qrel documents that appear in any of its search results, each counted once, averaged over questions.
H.G.: Hop groundedness: the percentage of a question's hops supported by at least one retrieved document, averaged over questions. It uses hop-level judgments that are not released.
T.C.: Mean tool calls per question: searches plus document reads.
Closed-book: Accuracy from the question alone, with all tools disabled at the API level.

Setup

How the runs were made

Agent harness: Piika, a search-agent framework that follows the agent design of Pi-Serini. The agents are the models below.
Tools: search returns at most five documents with previews of at most 500 characters; read_document returns the first 200 lines of a document.
Retriever: BM25 over ClimbMix through the Pyserini 2.4.0 REST backend (k₁ = 0.9, b = 0.4).
Models: GPT-6.1 Sol with high reasoning effort; Gemma 4 31B IT and Qwen 3.5 9B with thinking enabled, served with vLLM.
Version: CMASS v1.0, released on 25 October 2026 as v2026.10.25. P@all, R@all and H.G. depend on the qrels, so report the version you used.

Prompts, settings and scripts: REPRODUCING.md and evaluation/.

Submit

Add your system

Submission instructions are coming soon.