Many-Shot CoT-ICL: Making In-Context Learning Truly Learn
ICML
While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. We study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.
Increasing CoT demonstrations is unstable for non-reasoning LLMs and benefits mainly reasoning-oriented LLMs.
Retrieval remains useful, but standard semantic-similarity retrieval is unreliable for reasoning tasks because surface similarity poorly predicts procedural (CoT) compatibility.
Performance variance grows with more CoT demonstrations, making the order of demonstrations a first-class concern.
Viewing many-shot CoT-ICL as in-context test-time learning suggests that demonstrations should form a learnable curriculum, not merely a large retrieved set. We derive two practical principles:
Select demonstrations whose reasoning traces are easy for the target model to understand and follow. A demonstration is not useful merely because its question is semantically similar to the query.
Order demonstrations so their reasoning forms a smooth conceptual trajectory, avoiding abrupt shifts that increase path dependence and order sensitivity.
Motivated by the correlation between embedding-space curvature and performance, CDS constructs an ordering of many-shot CoT demonstrations that produces a smooth trajectory in embedding space, avoiding abrupt transitions between successive demonstrations. It approximates the combinatorial optimum with a lightweight TSP heuristic (nearest-neighbor + 2-opt), taking under a minute on a standard CPU for up to 128 demonstrations.
CDS seeks a low-curvature ordering of demonstrations, yielding a smoother conceptual progression.
CDS is robust across tasks, embedding models, and target LLMs (gpt-5.2 and Qwen3-14B), and across demonstration counts (16–128). CDSbge swaps the ordering embedder for bge-m3.
| Task | Model | Method | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|
| number_theory | gpt-5.2 | origin | 89.63 | 91.11 | 88.56 | 91.48 |
| CDS | 89.26 | 92.59 | 92.04 | 91.85 | ||
| Qwen3-14B | origin | 86.67 | 87.96 | 86.30 | 90.93 | |
| CDS | 85.56 | 87.85 | 87.78 | 90.74 | ||
| geometry | gpt-5.2 | origin | 75.99 | 74.74 | 75.37 | 75.78 |
| CDS | 81.21 | 78.08 | 80.79 | 75.99 | ||
| Qwen3-14B | origin | 66.18 | 65.76 | 65.14 | 73.07 | |
| CDS | 65.55 | 68.27 | 68.89 | 73.90 | ||
| DetectiveQA | gpt-5.2 | origin | 80.52 | 82.47 | 83.77 | 85.71 |
| CDS | 80.52 | 83.12 | 85.06 | 88.31 | ||
| Qwen3-14B | origin | 75.97 | 74.03 | 70.78 | 72.73 | |
| CDS | 76.62 | 75.32 | 73.38 | 75.32 |
Accuracy (%) at 16/32/64/128 demonstrations. Bold marks CDS improvements over origin.
Many-shot CoT-ICL is better understood as in-context test-time learning than as scaled pattern matching. Two practical principles follow: choose demonstrations that are easy for the target model to understand, and order them for a smooth conceptual progression. CDS operationalizes the second principle and turns the long context window into a structured curriculum.
@article{chung2026manyshot,
title = {Many-Shot CoT-ICL: Making In-Context Learning Truly Learn},
author = {Chung, Tsz Ting and Liu, Lemao and Yu, Mo and Yeung, Dit-Yan},
journal = {arXiv preprint arXiv:2605.13511},
year = {2026}
}