rag_framework/
Shared infrastructure
Language-model client, document search, and the shared query and document types.
Preference-Learning RAG
A research system that writes cited reports, compares them, and uses what the better report covered to improve the next draft.
Method
Compare two reports. Teach the next one.
Standard RAG writes once and stops. This system starts from a nugget bank — short questions the report should answer — and writes two drafts, one more abstractive and one more extractive. A judge picks the stronger draft, then writes new questions for facts the winner covered and the loser missed. Those questions join the bank. The next draft has to answer them.
The first round produces the pair. Later rounds write one challenger against the current champion. The loop stops at a round limit, when no new questions appear, or when the same report keeps winning.
Closed loop
Click a step, or use the arrow keys
01 · Retrieve
Pull supporting documents for the topic. The system queries a search API, so the full corpus does not need to live on disk.
Framework
Separate generation from judgment.
Report writing and preference judging are separate packages. A third package runs the loop between them.
crucible/
Report generation
Builds the question bank, finds supported answers, assembles a cited report, and cleans citations.
prefnugget/
Preference judging
Ranks two reports, extracts the questions that separate them, and grades coverage from 0 to 5.
research/
Loop and evaluation
Runs the preference loop, decides when to stop, and compares five systems on the same topics.
Configs live in experiments/, command-line entry points
in examples/, and dataset setup in
scripts/.
Benchmark
Five systems. Same evidence.
Each system answers the same topics with the same retrieved documents, so differences come from how the report is written, not from a different search set.
Proposed method
preference_loop_full
Proposed method. Full loop: two opening drafts, contrastive questions, then challenger rounds until stop.
Gold coverage
Share of human-written gold questions graded 4 or 5.
Gold mean / max grade
Average and best 0–5 grade against that gold rubric.
Query-only coverage
Coverage of generic questions derived from the query, used as a control.
Citation coverage
Share of report sentences that include a citation.
Citation validity
Share of citations that point to a retrieved document.
Coverage–citation F1
Harmonic mean of gold coverage and citation validity.
Pairwise win rate
How often the full loop is preferred over each baseline.
Ideas
Why close the loop.
01
Feedback that can be used
A single quality score does not say what to fix. A pairwise judgment can name the missing facts and turn them into questions for the next draft.
02
Coverage a single pass misses
One generation pass often skips facets of the request. Adding contrastive questions is meant to raise gold coverage and citation quality against one-pass CRUCIBLE and a query-only baseline.
03
Prior work
CRUCIBLE (ECIR 2026) generates cited reports from nuggets. PrefNugget / Too Many Questions (SIGIR 2026) derives contrastive questions from preferences. Insider Knowledge (ECIR 2026) argues for reporting gold coverage separately. This project connects those pieces into one generation–evaluation loop.