Step-level credit for long-horizon LLM agents. CCPO weights observation-matched peers by contextual similarity and pairs historical credit with future credit, so a step is credited for the progress it makes, not only for its trajectory’s eventual outcome. No value critic.
1Stanford University 2Texas A&M University 3Yale University 4Amazon 5Meta AI 6Microsoft †Project lead. Huiwen Han was a visiting student at Texas A&M University during this work.
Identical observations can occur after different preceding actions, reasoning, exploration depth and accumulated context. Peers that share an observation are not equally informative for the step being evaluated.
→ Weight observation-matched peers by contextual similarity, so peers from similar contexts contribute more to the step’s baseline.
A return comparison credits a step for belonging to a trajectory that ended well, but an eventual success can include redundant actions or detours.
→ Pair historical credit with future credit, the change in contextual outcome value over a short realized continuation.
CCPO credits each step of a rollout without a value critic. It finds the steps in other rollouts of the same task that saw the same observation, and weights these peers by how similar their context is. The weighted peers answer two questions: did this trajectory’s return beat theirs (historical credit), and did the expected outcome rise over the next few steps (future credit)? Their sum is the step’s advantage.
currentquery (i, t) peer(j, s) ∈ 𝒫i,t excluded(same trajectory) history(−KH steps) future(+KF steps)
oBYou arrive at countertop 1. On the countertop 1, you see an apple 1.
For a query step, exact peers are steps from other trajectories of the same task that saw the same observation. The query’s entire trajectory is excluded.
If no exact match exists, CCPO falls back to all same-task steps from other trajectories.
: LLM hidden state c : context summary stats
![\mathcal{B}_{i,t}[r]=\sum_{(j,s)\in\mathcal{P}_{i,t}}\bar w_{(i,t),(j,s)}\,r_{j,s}](static/eq/baseline.svg){kind=small}
Each step’s context is a frozen LLM’s final-layer hidden state at the last prompt token, concatenated with a trajectory summary: elapsed steps, distinct-observation count, novelty fraction and a revisit indicator. The prompt holds only the most recent steps, so the summary supplies the earlier history.
Peers closer to the query in this space contribute more; the bandwidth τB scales with the median pairwise context distance in the bucket.
![C^{\mathrm H}_{i,t}=Y_{i,t}-\mathcal{B}_{i,t}[Y]](static/eq/credit-h.svg){kind=small}
![C^{\mathrm F}_{i,t}=z_{x_i}\big[V_{i,t^+}-V_{i,t}\big]](static/eq/credit-f.svg){kind=small}
![V_{i,s}=\mathcal{B}_{i,s}[Z]](static/eq/value.svg){kind=small}
Historical credit compares the step’s return with the contextual peer baseline, so it depends on how the trajectory ends. Future credit asks whether the step itself moved the trajectory to a more valuable context over the next KF realized steps.
Future endpoints are used only during training.
![A^{\mathrm{CC}}_{i,t}=\mathcal{N}_{x_i}\big[\,C^{\mathrm H}+C^{\mathrm F}\,\big]_{i,t}](static/eq/advantage.svg){kind=small}
Historical credit and task-standardized future credit are added with unit coefficients and normalized within each task. The advantage weights the step’s response tokens in a dual-clipped PPO objective with a KL penalty toward the frozen reference policy.
An episode-augmented variant adds a GRPO-style episode advantage, ACC + AEP.
CCPO is evaluated on two multi-step benchmarks, ALFWorld (text-based household tasks) and WebShop (web shopping), with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. It is compared with off-the-shelf LLMs, prompt-based agents and RL training methods. CCPO leads the listed RL baselines in mean success, reaching 96.9% and 97.4% on ALFWorld and 75.0% and 79.4% on WebShop at 1.5B and 7B, and outperforms GRPO by up to 21.6 and 18.5 points.
Mean over three seeds. Whiskers show ±1 standard deviation where reported.
Success rate (%) per ALFWorld sub-task and overall; success rate (%) and score for WebShop. Means over three seeds with standard deviations. Within each section the best mean is shaded and the second best underlined. * Baselines reproduced, or off-the-shelf models evaluated, by the authors. Gray rows are results from the original papers and are excluded from ranking. Δ GRPO is the min–max gain over GRPO across both CCPO variants.
All ablations use Qwen2.5-1.5B-Instruct, with ACC on ALFWorld and ACC + AEP on WebShop as reference configurations. Contextual weighting outperforms uniform weighting, and historical and future credit serve as complementary learning signals: uniform weights lower success by 3.7 and 2.6 points, and either credit alone lowers it by up to 12.3.
Percentage points below full CCPO. Mean success of each ablation is shown at the right of its bar.
ALFWorld success as a five-step window shifts from all history to all future.
Symmetric windows of one, two and three steps each way.
Markers follow the paper: history step, current step, future step. The default two steps each way outperforms every tested asymmetric window and both one and three steps each way; widening to three reduces success from 96.9% to 91.4%. Window variations are restricted to ALFWorld, where successful 1.5B trajectories average 8.7 steps versus 5.1 on WebShop. Error bars: ±1 SD over three seeds.
H and F denote active historical and future credit spans, with W = H + F + 1. Zero marks a disabled credit channel. Channel-only rows retain KH = KF = 2. Temporal-window rows also change the actual window lengths. A dash means the configuration was not run on WebShop.
Evaluation trajectories, credit magnitudes over training, and credit preferences by action category, all on ALFWorld.
On this ALFWorld evaluation task, GiGPO, HGPO and G2PO keep navigating among places they have already visited and never acquire the target. CCPO reaches the same empty hanger only after picking the toilet paper up, and completes the task in 7 steps.
CCPO’s revisit rate across three ALFWorld evaluation draws, versus 26.8–41.0% for the baselines.
One sampled evaluation trajectory per method at training step 150, temperature 0.4, 50-step limit. Each cell is one step; a revisit is navigation to a previously visited location. CCPO uses ACC only.
Share of steps whose observation was already seen in the same rollout. Mean ± SD over three evaluation draws; lower means less revisiting.
Both advantages are computed from the same ACC-only ALFWorld rollouts; AEP is logged for comparison and never backpropagated. As validation success improves, the mean absolute episode advantage diminishes while the contextual advantage remains elevated, so a learning signal remains after most episodes succeed.
Mean absolute credit per training iteration on ALFWorld. Curves are centered 5-iteration means; bands are 95% trajectory-bootstrap confidence intervals. The strip below each panel traces validation success over the same iterations.
Historical credit dominates early. Future credit becomes more prominent later, even though the two are always combined with fixed unit weights. Within 10-iteration windows their magnitudes become less consistently correlated late in training, especially at 7B. The two components evolve differently, supporting complementary roles.
Future-to-history magnitude ratio at 1.5B, iterations 1–30 vs 121–150.
The same ratio at 7B: by the end, future credit is as large as historical credit.
Shaded spans mark iterations 1–30 and 121–150, over which the ratio E|AF| / E|AH| is computed. The contributions satisfy ACC = AH + AF.
Pearson r between unsmoothed |AH| and |AF| in non-overlapping 10-iteration windows.
Within groups matched on task, observation and iteration, both channels favor task operations (cleaning, heating, cooling). They differ on exploration: relative to other actions, AF favors new-location transitions more than AH. Variants without the context summary, and with (4, 0) or (0, 4) windows, show similar patterns.
Paired difference AF − AH for new-location transitions in full CCPO. Bonferroni-adjusted CI [0.001, 0.073] over 2,804 groups.
Relative credit preference Positive (+) Negative (−) — Insufficient matched groups (<60) * Significant off Inactive component
Mean category-relative credit within matched task–observation groups and iterations, from successful Qwen2.5-1.5B ALFWorld rollouts at iterations 101–150. Brackets give Bonferroni-adjusted confidence intervals; asterisks mark intervals excluding zero. Context markers denote current, history and future.
Each ablated policy is paired with full CCPO on the same 140 evaluation games, comparing the step at which an action type first occurs. Against full CCPO, the future-only policy first opens a receptacle earlier in 72.1% of paired games, and the history-only policy later in 54.3%.
Future-only policy: first open comes earlier than under full CCPO.
History-only policy: first open comes later.
Share of paired games (%) in which the ablated policy acts earlier, at the same step, or later.
Checkpoint 150, Qwen2.5-1.5B-Instruct, successful and failed episodes. Percentages condition on both policies performing the action; these are descriptive proportions. Categories with fewer than 12 pairs, and those not compatible with a first-occurrence analysis, are omitted.
Context summaries add accumulated trajectory statistics to the semantic representation: step count, distinct-observation count, novelty fraction and a revisit indicator. Adding them changes normalized peer weights by more than one percentage point for 33.8% of matched query–peer pairs. Removing them lowers mean success by 2.6 points on ALFWorld and 1.3 on WebShop.
| Change (pp) | > 0.25 | > 0.5 | > 1 | > 5 | > 10 |
|---|---|---|---|---|---|
| Pairs (%) | 62.1 | 48.7 | 33.8 | 8.6 | 3.6 |
@misc{han2026contextualcreditpolicyoptimization,
title = {Contextual Credit Policy Optimization},
author = {Huiwen Han and Ziyao Zeng and Zihao Zhu and
Sisong Bei and Hao Wang and Hongyu Xu and
Yan Han and Degui Zhi and Zhiwen Fan},
year = {2026},
eprint = {2610.19103},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.19103}
}