Do not keep pushing a failure away forever.
The reference-relative NPO loss suppresses a negative response, then attenuates its gradient as that response becomes sufficiently unlikely relative to the reference policy.
NeurIPS 2026 Spotlight · ~0.95% of submissions
1Nanyang Technological University 2Skywork AI
You can learn from a clear failure
without knowing what success looks like.
When checks can expose mistakes but cannot certify correctness,
NOPO learns from trusted negatives and leaves passing responses unlabeled.
Passing a partial check is evidence of non-falsification, not a positive training label.
01 / The problem
A failing unit test exposes a bug. A passing test suite can still miss one.
We call this asymmetry one-sided verifiability (OSV).

Rewarding every response that passes an incomplete check can reinforce shortcuts. NOPO uses the reliable side of the feedback: the failures the verifier can actually identify.
02 / The method
Negative-only learning needs two safeguards: one for each response,
and one for the sampled group.

The reference-relative NPO loss suppresses a negative response, then attenuates its gradient as that response becomes sufficiently unlikely relative to the reference policy.
Scale the loss by (c/G)γ, where c of G samples pass the checks. With the default γ = 1, all-failed groups receive zero weight. Passing samples supply a count, not a positive gradient.
03 / The evidence
Evaluated with Qwen3-4B and Qwen3-8B across code, logic, and math,
without oracle correctness labels during training.
Across five benchmarks
+6.01 ppMean pass@16 over BaseAcross five benchmarks
+7.19 ppMean pass@16 over BaseAt 512 samples
+15.90 ppCodeContests pass@512 over Base
The gains above use Tables 2 and 3. NOPO leads in 9 of 10 benchmark–backbone settings in Table 2; GRPO is stronger on OlympiadBench with Qwen3-8B. Pass@k measures candidate coverage, not practical answer-selection accuracy.
Selected Base and NOPO rows across TACO, CodeContests, LiveCodeBench, ZebraLogic, and OlympiadBench. Each benchmark uses its corresponding adaptation setting. Avg. is the mean across the five benchmarks. The full paper also reports GRPO, EM-RL, and w-Reinforce.
| Model / method | TACO | CodeContests | LCB | ZebraLogic | OlympiadBench | Avg. |
|---|---|---|---|---|---|---|
| Qwen3-4B · Base | 13.62 | 41.18 | 34.44 | 71.25 | 73.78 | 46.85 |
| Qwen3-4B · NOPO | 19.52 | 50.47 | 39.61 | 79.37 | 75.33 | 52.86 |
| Qwen3-8B · Base | 16.12 | 43.10 | 34.86 | 56.25 | 74.37 | 44.94 |
| Qwen3-8B · NOPO | 19.98 | 55.04 | 42.86 | 66.25 | 76.52 | 52.13 |
In the authors’ words
Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives.
This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses.
Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.
NOPO relies on trustworthy negative judgments; it does not correct arbitrary verifier noise. LLM self-verifiers can reject correct answers, and low failure-detection recall weakens supervision. Suppressing failures still leaves ambiguity about where probability mass should move. With γ > 0, an all-failed group supplies no update.
Read, build on, cite
Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, and Bo An.
Negative-Only Policy Optimization for One-Sided Verifiable Rewards.
NeurIPS 2026 · Spotlight. The citation below identifies the public arXiv version.
@article{xu2026nopo,
title = {Negative-Only Policy Optimization for One-Sided Verifiable Rewards},
author = {Jiacheng Xu and Shuo He and Fuxiang Zhang and Chaojie Wang and Bo An},
journal = {arXiv preprint arXiv:2611.17287},
year = {2026},
url = {https://arxiv.org/abs/2611.17287}
}
First submitted on 12 November 2026. PDF: unchanged arXiv v1, 18 pages. Figures 1–3 are cropped from the same paper by the authors listed above, under CC BY 4.0. The introductory diagram is a simplified explanation of the method.