ReMaRk
arXiv preprint · 2026

ReMaRkMany Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

Xin LiHao JiangJinghao GuoWeisi LinChau Yuen

Nanyang Technological University, Singapore

Code arXiv PDF BibTeX Models Data
TL;DR A Reddit request draws several replies that name different titles. Rank-GRPO rewards each generated list against one drawn reply, so the advantage mostly records which reply was drawn (85% of its variance). ReMaRk scores every list once against the union of all replies and samples each request equally, and reaches the validation recall of 2,000 per-reply RL steps in 25–53% of the steps.
One request, five responders and one generated list. Rank-GRPO credits the list against one drawn responder, so its reward depends on which one was drawn; ReMaRk credits it once against the union of all responders' items.

01Key findings

Train on the request, not on each answer.

Nothing else in Rank-GRPO changes: same model, data, budget and length control. Only the training unit and its reward differ, and that decides how much of the RL signal is noise.

RL steps to the same recall
25–53%

of the steps reach the 2,000-step validation recall of the per-responder baseline, in all ten backbone and dataset settings.

Lead at 2,000 steps
+0.8–1.6pts

test recall@20 over the better per-responder arm, on seven backbones (Qwen3.5 2B to 35B-A3B, Llama 3B and 8B) and three datasets.

Responder noise in Rank-GRPO
85%

of the variance of the advantages comes from which responder was drawn, and in more than half of the comparisons every list gets the same score.

Beyond recommendation
+3.67R@10

on ProtoQA, where about 100 people answer each question, at 250 steps (three seeds).

02Abstract

A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is.

We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours.

The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.

03Method

One row per request, one reward against all responders.

Rank-GRPO turns each responder into a separate training row and rewards a list only for that responder's titles. ReMaRk keeps the online GRPO update and changes only the training unit.

Rank-GRPO scores each sampled list against one drawn responder; ReMaRk scores it once against all of them. Top: one row per responder, drawn by row sampling. Bottom: one row per request, drawn uniformly, with 0/1 item rewards against the union of the responders' items.
Rank-GRPOOne row per responder
  • Rank-GRPO. A list earns rank-level credit for every title the one drawn responder named; requests with more responders are drawn more often.
  • The problem. Responders rarely agree: most suggested titles are named by a single person, so a good list scores high or low depending on who was drawn.
ReMaRkOne row per request
  • Union reward. Each position of a list earns 0/1 credit if any responder of the request named that title.
  • Request sampling. Every request is drawn equally, once instead of once per responder; per-group advantage scaling stays bounded for 0/1 rewards.

04Why it works

The responder is a nuisance variable.

Which responder ends up in a training row is an accident of how the data were collected. Averaging the reward over the responders is a Rao–Blackwellization: at a fixed advantage scale the expected update is unchanged and a variance term disappears. GRPO's per-group scaling can amplify one responder's vote when rewards are fractional, so ReMaRk keeps a 0/1 reward on the union, whose scale stays bounded.

  • The gain follows the variance removed. It grows with the number of responders and with the share of responder noise removed.
  • Pooling matters, not the aggregate. Once list lengths are controlled, the averaged reward is level with the union at 2B.
  • Same end point, reached sooner. Lower variance predicts faster progress, so we measure the RL compute needed to reach a target.

05Results

What the experiments show.

R1 · COMPUTE

The same recall with less RL compute.

Best validation recall so far against training GPU-hours for pairs trained until neither improves. Trained this long, ReMaRk is never significantly behind on test, and in four of seven pairs it reaches Rank-GRPO's best with 18 to 63% of the GPU-hours.

Best validation R@20 so far against training GPU-hours (Reddit-v2); each line ends at the selected checkpoint.

Where it does not help. On requests with a single responder every aggregation equals the per-responder reward, so on datasets with one recommender per request ReMaRk reduces to Rank-GRPO. Its value grows with the share of requests that have several responders.

R2 · EVERY SETTING

A lead in every setting.

Test gain over the better per-responder baseline at 2,000 steps, with 95% confidence intervals, for every backbone, dataset and seed.

Test gain in recall@20 points (95% CIs). Filled: per row (seed 3407); small dots: other seeds; open: per request.
R3 · NOISE REMOVED

The gain follows the responder noise removed.

(a) Test gain of ReMaRk over Rank-GRPO by the request's number of responders (95% CIs). (b, c) Dose-response at 2B on validation: recall when the reward averages m = 1, 2, 4 or all responders, spaced by the share of noise removed, for all and for single-responder requests.

06Citation

BibTeX

li2026remark.bib 
@misc{li2026rightanswerstrainingexample,
      title={Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders},
      author={Xin Li and Hao Jiang and Jinghao Guo and Weisi Lin and Chau Yuen},
      year={2026},
      eprint={2610.17771},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.17771},
}

07Resources

Code, data and paper.

Code · GitHubLive ↗
LiXin97/ReMaRk

Training and evaluation code for ReMaRk and Rank-GRPO, the reward functions, configs, and Reddit-Books and Reddit-Music as IDs and labels.
Paper · arXivLive ↗
arXiv:2610.17771

Full paper with the appendix: proofs, per-seed results, compute, length control and the dataset construction.
Models · Hugging FaceLive ↗
ReMaRk collection

23 checkpoints: SFT, ReMaRk and Rank-GRPO on Qwen3.5 2B, 4B and 9B, Llama-3.2-3B and Llama-3.1-8B, for Reddit-v2, Reddit-Books and Reddit-Music.
Data · Hugging FaceLive ↗
XINLI1997/ReMaRk-data

IDs, labels and SFT targets (as item ids) of Reddit-Books, Reddit-Books-Late and Reddit-Music; CC BY-NC 4.0.