Implicit Reasoning in Large Language Models: A Comprehensive Survey

The official GitHub page for the survey paper "Implicit Reasoning in Large Language Models: A Comprehensive Survey".

[arXiv][https://arxiv.org/abs/2509.02350]

[PaperWeekly][https://mp.weixin.qq.com/s/YPVtFIFuEDW90CMfKaup_w?scene=1&click_id=4]

1. Introduction

2. Preliminaries

3. Technical Paradigms for Implicit Reasoning

3.1 Latent Optimization

3.1.1 Token-Level

1. 2025_arXiv_CoCoMix_LLM Pretraining with Continuous Concepts. [arXiv] [Github] [HuggingFace] [YouTube] [Bilibili]

2. 2025_arXiv_Latent Token_Enhancing Latent Computation in Transformers with Latent Tokens. [arXiv] [HuggingFace] [YouTube]

3. 2025_ICML_LPC_Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes. [ICML] [arXiv]

4. 2025_ICML_Token Assorted_Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning. [ICML] [arXiv] [HuggingFace]

5. 2025_arXiv_Latent-SFT_Latent Reasoning in LLMs as A Vocabulary Space Superposition. [arXiv] [GitHub]

3.1.2 Trajectory-Level

3.1.2.1 Semantic Anchoring

1. 2024_arXiv_CCoT_Compressed Chain of Thought: Efficient Reasoning through Dense Representations. [arXiv] [HuggingFace]

2. 2024_arXiv_HCoT_Expediting and Elevating Large Language Model Reasoning via Hidden Chain-of-Thought Decoding. [arXiv] [HuggingFace]

3. 2025_arXiv_CODI_CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation. [arXiv] [GitHub] [HuggingFace]

4. 2025_arXiv_SynAdapt_SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thought. [arXiv]

5. 2025_arXiv_SIM-CoT_SIM-CoT: Supervised Implicit Chain-of-Thought. [arXiv] [Github] [HuggingFace]

3.1.2.2 Adaptive Efficiency

1. 2025_arXiv_LightThinker_LightThinker: Thinking Step-by-Step Compression. [arXiv] [Code--Github] [HuggingFace] [YouTube]

2. 2025_arXiv_CoT-Valve_CoT-Valve: Length-Compressible Chain-of-Thought Tuning. [arXiv] [Code--Github] [HuggingFace]

3. 2025_arXiv_CoLaR_Think-Silently-Think-Fast=Dynamic-Latent-Compression-of-LLM-Reasoning-Chains. [arXiv] [Homepage] [Github]

3.1.2.3 Progressive Refinement

1. 2024_arXiv_ICoT-SI_From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step. [arXiv] [Github] [HuggingFace] [YouTube] [Bilibili]

2. 2024_arXiv_Coconut_Training Large Language Models to Reason in a Continuous Latent Space. [ICLR] [arXiv] [Github] [HuggingFace] [YouTube]

3. 2025_arXiv_Heima_Efficient Reasoning with Hidden Thinking. [arXiv] [Code--Github] [HuggingFace] [YouTube]

4. 2025_arXiv_PonderingLM_Pretraining Language Models to Ponder in Continuous Space. [arXiv] [Github]

5. 2025_arXiv_BoLT_Reasoning to Learn from Latent Thoughts. [arXiv] [GitHub] [HuggingFace] [YouTube]

6. 2025_arXiv_SpiralThinker_SpiralThinker: Latent Reasoning through an Iterative Process with Text-Latent Interleaving. [arXiv]

3.1.2.4 Exploratory Diversification

1. 2024_arxiv_LaTRO_Unlocking Latent Reasoning Capabilities via Self-Rewarding. [arXiv] [Github] [HuggingFace]

2. 2025_arXiv_Soft Thinking_Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space. [arXiv] [HomePage] [Github] [HuggingFace] [YouTube] [Bilibili]

3. 2025_arXiv_SwiReasoning_SwiReasoning=Switch-Thinking-inLatent-and-Explicit-for-Pareto-Superior-Reasoning-LLMs [arXiv] [HomePage] [GitHub]

4. 2025_arXiv_Randomized Soft Thinking_LLMs are Single-Threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking. [arXiv] [Github]

5. 2025_arXiv_Soft Tokens, Hard Truths. [arXiv]

6. 2025_ACL_SoftCoT_SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. [arXiv] [Github] [HuggingFace] [Data-HuggingFace]

7. 2025_arXiv_SoftCoT++_SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning. [arXiv] [Github] [HuggingFace]

8. 2025_arXiv_LatentTTS_Parallel Test-Time Scaling for Latent Reasoning Models [arXiv] [GitHub]

9. 2025_arXiv_CoT2_Continuous Chain of Thought Enables Parallel Exploration and Reasoning. [arXiv] [HuggingFace]

10. 2025_arXiv_EBM-CoT_Think Consistently Reason Efficiently: Energy-based Calibration for Implicit Chain-of-Thought [arXiv]

3.1.3 Internal-State-Level

1. 2023_arXiv_ICoT-KD_Implicit Chain of Thought Reasoning via Knowledge Distillation. [arXiv] [Microsoft] [Github] [HuggingFace] [YouTube]

2. 2024_NeurIPS Workshop_Distilling System 2 into System 1. [NeurIPS Workshop] [arXiv] [YouTube]

3. 2025_arXiv_ReaRec_Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation. [arXiv] [GitHub] [HuggingFace] [YouTube]

4. 2025_arXiv_Beyond Words_Beyond Words: A Latent Memory Approach to Internal Reasoning in LLMs. [arXiv]

5. 2025_arXiv_System-1.5 Reasoning_System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts. [arXiv] [HuggingFace]

6. 2025_ICML_LTMs_Scalable Language Models with Posterior Inference of Latent Thought Vectors. [ICML] [arXiv] [Homepage] [HuggingFace]

7. 2025_arXiv_HRPO_Hybrid Latent Reasoning via Reinforcement Learning. [arXiv] [Github] [HuggingFace]

8. 2024_arXiv_Copressor_Deliberation in Latent Space via Differentiable Cache Augmentation. [arXiv]

9. 2025_arXiv_System 1-2 Communication_Exploring System 1 and 2 Communication for Latent Reasoning in LLMs. [arXiv]

3.2 Signal-Guided Control

3.2.1 Single-Type Signal

1. 2024_arXiv_thinking-tokens_Thinking Tokens for Language Modeling. [arXiv] [HuggingFace]

2. 2024_ICLR_pause-token_Think Before You Speak: Training Language Models with Pause Tokens. [ICLR] [arXiv] [HuggingFace] [YouTube]

3. 2024_COLM_Quiet-STaR_Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking. [CoLM--OpenReview] [arXiv] [Github] [HuggingFace] [YouTube]

4. 2024_CoLM_FillerTokens_Let's Think Dot by Dot: Hidden Computation in Transformer Language Models. [CoLM--OpenReview] [arXiv] [GitHub] [HuggingFace] [YouTube]

5. 2024_CoLM_planning-tokens_Guiding Language Model Reasoning with Planning Tokens. [CoLM--OpenReview] [arXiv] [Microsoft] [Github]

6. 2025_arXiv_LatentSeek_Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space. [arXiv] [HomePage] [Github] [HuggingFace]

7. 2025_ACL_DIT_Learning to Insert [PAUSE] Tokens for Better Reasoning. [arXiv] [Github]

3.2.2 Multi-Type Signal

1. 2025_ACL_Memory-Reasoning_Disentangling-Memory-and-Reasoning-Ability-in-Large-Language-Models. [arXiv] [Github]

2. 2025_arXiv_Thinkless_Thinkless: LLM Learns When to Think. [arXiv] [Github]

3.3 Layer-Recurrent Execution

1. 2025_ICLR_looped-Transformer_Reasoning with Latent Thoughts: On the Power of Looped Transformers. [arXiv] [ICLR] [Poster] [Youtube]

2. 2025_arXiv_ITT_Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking. [arXiv] [HuggingFace] [YouTube]

3. 2025_ICLR_CoTFormer_CoTFormer: A Chain-of-Thought Driven Architecture with Budged-Adaptive Computation Cost at Inference. [NeurIPS Workshop] [ICLR] [arXiv] [Github]

4. 2025_arXiv_TaH_Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models [arXiv] [GitHub]

5. 2025_arXiv_RELAY_Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning. [arXiv] [GitHub]

6. 2025_arXiv_Huginn_Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. [arXiv] [HuggingFace] [Github] [YouTube]

7. 2025_arXiv_Ouro_Scaling Latent Reasoning via Looped Language Models [arXiv] [HomePage]

3.4 Discussion

4. Mechanistic and Behavioral Evidence

4.1 Layer-wise Structural Evidence

1. 2024_LREC-COLING_Jump to Conclusions: Short-Cutting Transformers with Linear Transformations. [ACL-LREC-COLING] [arXiv] [Github] [HuggingFace]

2. 2025_arXiv_LM Implicit Reasoning_Implicit Reasoning in Transformers is Reasoning through Shortcuts. [arXiv] [Github] [HuggingFace]

3. 2025_arXiv_Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMs. [arXiv] [GitHub]

4. 2025_arXiv_Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought. [arXiv]

5. 2025_arXiv_To CoT or To Loop? A Formal Comparison Between Chain-of-Thought and Looped Transformers. [arXiv]

4.2 Behavioral Signatures

1. 2024_NeurIPS_Grokked Transformer_Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization. [ICML Workshop--OpenReview] [NeurIPS] [arXiv] [Github] [HuggingFace] [YouTube]

2. 2024_ACL_latent multi-hop reasoning_Do Large Language Models Latently Perform Multi-Hop Reasoning. [ACL] [arXiv] [Github] [HuggingFace]

3. 2024_NeurIPS_step-skipping_Can Language Models Learn to Skip Steps. [NeurIPS] [ACM NeurIPS] [arXiv] [Github] [HuggingFace]

4. 2025_arXiv_Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models. [arXiv] [HuggingFace]

4.3 Representation-Based Probing

1. 2023_EMNLP_MechanisticProbe_Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models. [EMNLP] [arXiv] [Github] [HuggingFace]

2. 2024_arXiv_TTT_Think-to-Talk or Talk-to-Think: When LLMs Come Up with an Answer in Multi-Step Arithmetic Reasoning. [arXiv] [GitHub]

3. 2024_arXiv_Do LLMs Really Think Step-by-Step in Implicit Reasoning. [arXiv] [GitHub]

4. 2024_arXiv_Distributional Reasoning_Distributional Reasoning in LLMs: Parallel Reasoning Processes in Multi-Hop Reasoning. [arXiv] [YouTube 1] [YouTbue 2]

5. 2024_ACL_backward chaining circuits_A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task. [ACL] [arXiv] [GitHub] [HuggingFace]

6. 2025_ICLR Workshop_steering vector intervention_Uncovering Latent Chain of Thought Vectors in Large Language Models. [ICLR Workshop] [arXiv]

7. 2025_ICLR_CoE_Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation. [ICLR] [arXiv] [Github]

8. 2025_arXiv_Huginn-Latent-CoT_Latent Chain-of-Thught: Decoding the Depth-Recurrent Transformer. [arXiv] [Github]

5. Evaluation Methods and Benchmarks

5.1 Metrics

5.2 Benchmarks

5.2.1 General Knowledge and Commonsense Reasoning Benchmarks

5.2.2 Mathematical Reasoning and Programming Benchmarks

5.2.3 Language Modeling and Reading Comprehension Benchmarks

5.2.4 Complex Multi-hop and Multidisciplinary QA Benchmarks

5.2.5 Multi-modal Reasoning Benchmarks

Related Survey

Reasoning

1. 2023_arXiv_Survey_A Survey of Reasoning with Foundation Model. [arXiv] [Github]

2. 2024_EACL_Survey_Large Language Models for Mathematical Reasoning: Progresses and Challenges. [ACL] [arXiv]

3. 2025_arXiv_Survey_Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models. [arXiv]

4. 2025_arXiv_Survey_Towards Reasoning Era: A Survey of Long-Chain-of-Thought for Reasoning Large Language Models. [arXiv] [Github] [Homepage]

5. 2025_arXiv_Survey_From System 1 to System 2: A Survey of Reasoning Large Language Models. [arXiv] [Github]

6. 2025_arXiv_Survey_Thinking with Images for Multimodal Reasoning= Foundations, Methods, and Future Frontiers [arXiv] [Github]

Efficient Reasoning

1. 2025_arXiv_Survey_A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond. [arXiv] [Github]

2. 2025_arXiv_Survey_Efficient Inference for Large Reasoning Models: A Survey. [arXiv] [Github]

3. 2025_arXiv_Survey_Efficient Reasoning Models: A Survey. [arXiv] [Github]

4. 2025_arXiv_Survey_Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models. [arXiv] [Github]

5. 2025_arXiv_Survey_Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. [arXiv] [Github]

6. 2025_arXiv_Survey_Don't Overthinking It: A Survey of Efficient R1-style Large Reasoning Models. [arXiv] [Github]

Latent Reasoning

1. 2025_arXiv_Survey_Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning. [arXiv] [Github]

2. 2025_arXiv_Survey_A Survey on Latent Reasoning. [arXiv] [Github]

Related Reposority

Reasoning

1. The-Martyr / Awesome-Multimodal-Reasoning. [Github]

2. atfortes / Awesome-LLM-Reasoning [GitHub]

Efficient Reasoning

1. hemingkx / Awesome-Efficient-Reasoning. [Github]

2. Blueyee / Efficient-CoT-LRMs. [Github]

3. Hongcheng-Gao / Awesome-Long2short-on-LRMs. [Github]

4. zcccccz / Awesome-LLM-Implicit-Reasoning. [Github]

5. zzli2022 / Awesome-System2-Reasoning-LLM [Github]

📖 Citation

@article{li2025implicit,
  title={Implicit Reasoning in Large Language Models: A Comprehensive Survey},
  author={Li, Jindong and Fu, Yali and Fan, Li and Liu, Jiahong and Shu, Yao and Qin, Chengwei and Yang, Menglin and King, Irwin and Ying, Rex},
  journal={arXiv preprint arXiv:2509.02350},
  year={2025}
}