Ask how programs behave.
Generate inputs from the problem statement without predicting correct outputs. Execute every candidate on the same probes.
EMNLP 2026 Oral · ~2.6% of submissions
Nanyang Technological University, Singapore
Let programs vote through behavior.
ERPO adapts a code model at test time using shared input probes,
without needing their correct outputs.
01 / The method
Different programs can implement the same solution. Comparing their outputs gives a learning signal—but shared bugs can also create consensus.
Generate inputs from the problem statement without predicting correct outputs. Execute every candidate on the same probes.
Probe Consensus Reward (PCR) counts agreement with per-probe majority outputs. It is a surrogate signal, not a correctness certificate.
Mask the top-ranked half and retain the lower half’s signed normalized advantages. An entropy ceiling penalizes excessive entropy growth.
02 / The evidence
Qwen3-4B and Qwen3-8B, non-thinking mode. Adaptation uses unlabeled LiveCodeBench v6; transfer benchmarks receive no updates.
Qwen3-4B · LiveCodeBench
+10.7 pppass@1 over BaseQwen3-4B · LiveCodeBench
+11.9 pppass@16 over BaseQwen3-4B · transfer mean
+13.6 pppass@16 over Base
Headline values are from Table 1. Transfer mean averages CodeContests, CodeForces, and TACO. Pass@k measures oracle candidate coverage; it does not measure practical answer-selection accuracy.
| Model / method | LCB pass@1 | LCB pass@16 | Transfer mean pass@1 | Transfer mean pass@16 |
|---|---|---|---|---|
| Qwen3-4B · Base | 26.0 | 34.4 | 13.8 | 25.7 |
| Qwen3-4B · ERPO | 36.7 | 46.3 | 25.4 | 39.3 |
| Qwen3-8B · Base | 25.6 | 34.6 | 14.6 | 27.6 |
| Qwen3-8B · ERPO | 35.1 | 50.9 | 22.9 | 38.8 |
The full paper also compares GRPO with public-test rewards, GRPO with PCR, and negative-sample reinforcement with PCR. Table 1 uses 16 validation samples per problem; the separate scaling experiment uses up to 256.
03 / Interpretation
PCR can reward shared bugs and miss correct minority solutions. ERPO controls updates; it does not turn consensus into a trusted verifier. Experiments cover competitive-programming-style tasks, not project-level or industrial software. A full training run takes approximately 12 hours on eight 80 GB A100 GPUs.
In the authors’ words
Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal.
To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus.
We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
Explore the work
Accepted to EMNLP 2026 as an Oral presentation (~2.6% of submissions). The public manuscript was first submitted to arXiv on 8 September 2026. The PDF is an unchanged copy of arXiv v1; the method image is rendered from its original figure. Paper and figure by Jiacheng Xu, Feng Chen, Xiuneng Xu, and Bo An, available under CC BY 4.0.
Build on this work
Jiacheng Xu, Feng Chen, Xiuneng Xu, and Bo An. “Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation.” 2026. arXiv:2609.09135.
The citation below identifies the public arXiv version.
@article{xu2026erpo,
title = {Entropy-Regularized Rank-Masked Policy Optimization for
Test-Time Reinforcement Learning in Code Generation},
author = {Jiacheng Xu and Feng Chen and Xiuneng Xu and Bo An},
journal = {arXiv preprint arXiv:2609.09135},
year = {2026},
url = {https://arxiv.org/abs/2609.09135}
}