Open-RS: Enhancing Reasoning in Small LLMs with Reinforcement Learning

As a researcher passionate about advancing artificial intelligence, I’m thrilled to spotlight "Open-RS," an innovative open-source project hosted at GitHub. This project, tied to the paper "Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn’t," tackles a fascinating challenge: improving reasoning in small large language models (LLMs) using reinforcement learning (RL). On this day, April 2, 2025, I’m adding this post to my personal website to share its significance with fellow researchers, students, and AI enthusiasts.

Project Overview

"Open-RS" is all about making small LLMs—those with fewer parameters—reason better under tight resource constraints. The project focuses on a 1.5-billion-parameter model, DeepSeek-R1-Distill-Qwen-1.5B, and uses RL to push its reasoning capabilities to new heights. Why small models? They’re lighter, faster, and more practical for real-world applications where computational power is limited. The team behind Open-RS adapted the Group Relative Policy Optimization (GRPO) algorithm and trained it on a compact mathematical reasoning dataset, achieving impressive results with minimal resources.

The repository provides the code, datasets, and detailed instructions to replicate their experiments. It’s a treasure trove for anyone interested in RL, LLMs, or efficient AI training methods. Key highlights include:

• Training on 4 NVIDIA A40 GPUs (48 GB VRAM each) in just 24 hours.
• A cost of ~$42 using only 7,000 samples, compared to thousands of dollars for larger models.
• Significant performance gains, like 80% accuracy on AMC23 and 46.7% on AIME24, outperforming models like o1-preview.

Getting Started

To dive into Open-RS, you’ll need a Python environment with PyTorch 2.5.1 (crucial for vLLM compatibility) and a few other dependencies. Here’s a quick setup guide based on the repo’s instructions:

1. Clone the Repository:

   git clone https://github.com/knoveleng/open-rs.git
   cd open-rs
   

2. Install Dependencies: Install PyTorch 2.5.1 and additional tools like accelerate and lighteval. A basic setup might look like:

   pip install torch==2.5.1 accelerate lighteval
   

   Check the repo’s README for the full list tailored to your use case.

3. Run an Experiment: For Experiment 3 (cosine scheduling), use:

   ACCELERATE_LOG_LEVEL=info accelerate launch \
     --config_file recipes/accelerate_configs/zero2.yaml \
     --num_processes=3 \
     src/open_r1/grpo.py \
     --config recipes/grpo.yaml \
     --cosine_max_len 3584
   

4. Evaluate a Model: Test Open-RS3 on AIME24 with:

   MODEL=knoveleng/Open-RS3
   MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
   OUTPUT_DIR=data/evals/$MODEL
   TASK=aime24
   lighteval vllm "$MODEL_ARGS" "custom|$TASK|0|0" \
     --custom-tasks src/open_r1/evaluate.py \
     --use-chat-template \
     --output-dir "$OUTPUT_DIR"
   

You’ll need at least one GPU for evaluation, but the repo supports multi-GPU setups too. The code is well-documented, with scripts like eval.sh to tweak tasks.

Key Findings

The Open-RS team ran three experiments, each building on the last. Here’s what they discovered:

• Open-RS1: Baseline RL setup, scoring 53.0% average across tasks.
• Open-RS2: Improved to 55.7%, hitting 80% on AMC23—a leap from 63% pre-training.
• Open-RS3: Reached 56.3%, with 46.7% on AIME24, beating o1-preview’s 44.6%.

These gains came from GRPO, a curated dataset of 7,000 samples (yielding 42,000 outputs), and clever tweaks like cosine scheduling. Compared to 7B models (e.g., Qwen2.5-7B-SimpleRL at $1,633)orother1.5Bmodels(e.g.,DeepScaleR-1.5B-Previewat$3,629), Open-RS is a budget-friendly powerhouse.

Why It Matters

Small LLMs are the future for edge devices, low-resource settings, and democratizing AI. Open-RS shows that RL can unlock reasoning potential without breaking the bank or needing massive hardware. For researchers, it’s a blueprint for efficient training; for developers, it’s a practical starting point for reasoning-focused applications like tutoring systems or automated problem-solving tools.

Technical Deep Dive

The project leverages:

• GRPO Algorithm: A variant of policy optimization tailored for LLMs, balancing exploration and exploitation.
• vLLM: For efficient inference, with parameters like max_model_length=32768 to handle long outputs.
• Hugging Face Integration: Models like Open-RS3 are hosted there, making them accessible for fine-tuning or direct use.

The dataset focuses on mathematical reasoning (e.g., AMC23, AIME24, MATH-500), curated to maximize learning from minimal samples. The repo’s src/open_r1 folder houses the core logic, with grpo.py driving training and evaluate.py handling custom tasks.

Limitations

It’s not all perfect. The project assumes GPU access, which might limit solo researchers. The focus on math reasoning leaves other domains (e.g., natural language reasoning) unexplored—though the framework could adapt. Also, PyTorch 2.5.1 is a strict requirement; deviating risks vLLM issues.

My Research Perspective

As I build my website, Open-RS aligns with my interest in efficient, scalable AI solutions. Its open-source nature invites collaboration, and I’m considering extending it to other reasoning tasks or integrating it into my own RL experiments. The cost-efficiency ($42 vs. thousands) is a game-changer for academic budgets, and the performance metrics inspire confidence in small-model potential.

Conclusion

"Open-RS" is a gem for anyone in the RL or LLM space. It’s practical, well-executed, and forward-thinking—proof that small models can punch above their weight with the right approach. Explore it at https://github.com/knoveleng/open-rs, and if it sparks your work, cite the paper: Dang & Ngo (2025). Let’s keep pushing the boundaries of what small LLMs can do.