Jiacheng XuHomepage

NeurIPS 2026 Spotlight · ~0.95% of submissions

NOPONegative-Only Policy Optimization
for One-Sided Verifiable Rewards

Jiacheng Xu1,2, Shuo He1, Fuxiang Zhang1,2, Chaojie Wang2, Bo An1

1Nanyang Technological University   2Skywork AI

You can learn from a clear failure
without knowing what success looks like.

When checks can expose mistakes but cannot certify correctness,
NOPO learns from trusted negatives and leaves passing responses unlabeled.

Sample responsesOne prompt.
Multiple candidates.
Incomplete verifierTests · Constraints
Local error checks
✕   A failure is foundSuppress this response
✓   No failure is foundLeave it unlabeled

Passing a partial check is evidence of non-falsification, not a positive training label.

01 / The problem

It is easier to find a mistake
than to prove an answer right.

A failing unit test exposes a bug. A passing test suite can still miss one.
We call this asymmetry one-sided verifiability (OSV).

Three examples of one-sided verification: a boundary test exposes a code bug, domain constraints reject a logic assignment, and an LLM identifies an arithmetic error. Passing each check does not establish full correctness.
Three settings, the same asymmetry. Executable checks, constraint checks, and LLM error detection. Figure 2 from the paper. Enlarge ↗
Why standard reinforcement can go wrong

Rewarding every response that passes an incomplete check can reinforce shortcuts. NOPO uses the reliable side of the feedback: the failures the verifier can actually identify.

02 / The method

Suppress failures.
Control how far the policy moves.

Negative-only learning needs two safeguards: one for each response,
and one for the sampled group.

Figure 3 compares off-policy NPO, naive on-policy NPO, and NOPO. NOPO scales negative updates by non-falsified support and skips all-failed groups, while retaining reference-relative suppression.
From fixed-data suppression to controlled on-policy learning. NOPO adds a support gate to each group of newly sampled responses. Figure 3 from the paper. Enlarge ↗
Sample-level safeguard

Do not keep pushing a failure away forever.

The reference-relative NPO loss suppresses a negative response, then attenuates its gradient as that response becomes sufficiently unlikely relative to the reference policy.

Group-level safeguard

Use non-falsified alternatives as support.

Scale the loss by (c/G)γ, where c of G samples pass the checks. With the default γ = 1, all-failed groups receive zero weight. Passing samples supply a count, not a positive gradient.

03 / The evidence

Better coverage. Transferable gains.
Benefits that persist at larger k.

Evaluated with Qwen3-4B and Qwen3-8B across code, logic, and math,
without oracle correctness labels during training.

Across five benchmarks

+6.01 ppMean pass@16 over Base
Qwen3-4B · 46.85 → 52.86

Across five benchmarks

+7.19 ppMean pass@16 over Base
Qwen3-8B · 44.94 → 52.13

At 512 samples

+15.90 ppCodeContests pass@512 over Base
Qwen3-4B · 51.05 → 66.95
Figure 1 shows mean pass@16 improvements on two Qwen3 backbones, transfer from CodeContests to TACO, LiveCodeBench and CodeForces, and pass@k scaling on CodeContests up to 512 samples.
Left: mean pass@16. Middle: Qwen3-4B transfer after CodeContests adaptation. Right: CodeContests inference-time scaling. Figure 1 from the paper. Enlarge ↗

The gains above use Tables 2 and 3. NOPO leads in 9 of 10 benchmark–backbone settings in Table 2; GRPO is stronger on OlympiadBench with Qwen3-8B. Pass@k measures candidate coverage, not practical answer-selection accuracy.

Inspect the benchmark scores Table 2 · pass@16 (%)

Selected Base and NOPO rows across TACO, CodeContests, LiveCodeBench, ZebraLogic, and OlympiadBench. Each benchmark uses its corresponding adaptation setting. Avg. is the mean across the five benchmarks. The full paper also reports GRPO, EM-RL, and w-Reinforce.

Base and NOPO comparison: pass@16 (%)
Model / methodTACOCodeContestsLCBZebraLogicOlympiadBenchAvg.
Qwen3-4B · Base13.6241.1834.4471.2573.7846.85
Qwen3-4B · NOPO19.5250.4739.6179.3775.3352.86
Qwen3-8B · Base16.1243.1034.8656.2574.3744.94
Qwen3-8B · NOPO19.9855.0442.8666.2576.5252.13

In the authors’ words

Abstract

Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives.

This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses.

Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.

Scope and limitations

NOPO relies on trustworthy negative judgments; it does not correct arbitrary verifier noise. LLM self-verifiers can reject correct answers, and low failure-detection recall weakens supervision. Suppressing failures still leaves ambiguity about where probability mass should move. With γ > 0, an all-failed group supplies no update.

Read, build on, cite

Citation & resources

Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, and Bo An.
Negative-Only Policy Optimization for One-Sided Verifiable Rewards.
NeurIPS 2026 · Spotlight. The citation below identifies the public arXiv version.

@article{xu2026nopo,
  title   = {Negative-Only Policy Optimization for One-Sided Verifiable Rewards},
  author  = {Jiacheng Xu and Shuo He and Fuxiang Zhang and Chaojie Wang and Bo An},
  journal = {arXiv preprint arXiv:2611.17287},
  year    = {2026},
  url     = {https://arxiv.org/abs/2611.17287}
}

First submitted on 12 November 2026. PDF: unchanged arXiv v1, 18 pages. Figures 1–3 are cropped from the same paper by the authors listed above, under CC BY 4.0. The introductory diagram is a simplified explanation of the method.