Jiacheng XuHomepage

EMNLP 2026 Oral · ~2.6% of submissions

ERPOEntropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

Jiacheng Xu, Feng Chen, Xiuneng Xu, and Bo An

Nanyang Technological University, Singapore

Let programs vote through behavior.

ERPO adapts a code model at test time using shared input probes,
without needing their correct outputs.

ERPO generates output-free inputs, scores candidate execution consensus, and updates low-ranked responses with entropy control.
Probe → execute → adapt. Figure 2 from the paper. View full-size figure ↗

01 / The method

Turn execution agreement into a cautious update.

Different programs can implement the same solution. Comparing their outputs gives a learning signal—but shared bugs can also create consensus.

Probe inputs

Ask how programs behave.

Generate inputs from the problem statement without predicting correct outputs. Execute every candidate on the same probes.

Behavioral reward

Measure agreement.

Probe Consensus Reward (PCR) counts agreement with per-probe majority outputs. It is a surrogate signal, not a correctness certificate.

Conservative adaptation

Mask ranks. Cap entropy.

Mask the top-ranked half and retain the lower half’s signed normalized advantages. An entropy ceiling penalizes excessive entropy growth.

02 / The evidence

Improvement on target tasks—and beyond.

Qwen3-4B and Qwen3-8B, non-thinking mode. Adaptation uses unlabeled LiveCodeBench v6; transfer benchmarks receive no updates.

Qwen3-4B · LiveCodeBench

+10.7 pppass@1 over Base
26.0 → 36.7

Qwen3-4B · LiveCodeBench

+11.9 pppass@16 over Base
34.4 → 46.3

Qwen3-4B · transfer mean

+13.6 pppass@16 over Base
25.7 → 39.3
ERPO performance overview: training dynamics on Qwen3-8B, Qwen3-4B transfer averages, and Qwen3-4B LiveCodeBench pass@k scaling.
Learning dynamics, transfer, and scaling. Left: Qwen3-8B on LiveCodeBench. Middle: Qwen3-4B mean transfer. Right: Qwen3-4B LiveCodeBench scaling. Figure 1. View full-size figure ↗

Headline values are from Table 1. Transfer mean averages CodeContests, CodeForces, and TACO. Pass@k measures oracle candidate coverage; it does not measure practical answer-selection accuracy.

Inspect the benchmark scores Table 1 · percentages
Base and ERPO comparison (%)
Model / methodLCB
pass@1
LCB
pass@16
Transfer mean
pass@1
Transfer mean
pass@16
Qwen3-4B · Base26.034.413.825.7
Qwen3-4B · ERPO36.746.325.439.3
Qwen3-8B · Base25.634.614.627.6
Qwen3-8B · ERPO35.150.922.938.8

The full paper also compares GRPO with public-test rewards, GRPO with PCR, and negative-sample reinforcement with PCR. Table 1 uses 16 validation samples per problem; the separate scaling experiment uses up to 256.

03 / Interpretation

Consensus still needs care.

PCR can reward shared bugs and miss correct minority solutions. ERPO controls updates; it does not turn consensus into a trusted verifier. Experiments cover competitive-programming-style tasks, not project-level or industrial software. A full training run takes approximately 12 hours on eight 80 GB A100 GPUs.

In the authors’ words

Abstract

Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal.

To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus.

We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.

Explore the work

Resources

Accepted to EMNLP 2026 as an Oral presentation (~2.6% of submissions). The public manuscript was first submitted to arXiv on 8 September 2026. The PDF is an unchanged copy of arXiv v1; the method image is rendered from its original figure. Paper and figure by Jiacheng Xu, Feng Chen, Xiuneng Xu, and Bo An, available under CC BY 4.0.

Build on this work

Citation

Jiacheng Xu, Feng Chen, Xiuneng Xu, and Bo An. “Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation.” 2026. arXiv:2609.09135.

The citation below identifies the public arXiv version.

@article{xu2026erpo,
  title   = {Entropy-Regularized Rank-Masked Policy Optimization for
             Test-Time Reinforcement Learning in Code Generation},
  author  = {Jiacheng Xu and Feng Chen and Xiuneng Xu and Bo An},
  journal = {arXiv preprint arXiv:2609.09135},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.09135}
}