NeurIPS LLM Efficiency Challenge 2023

Our team achieved Top 3 ranking in the NeurIPS 2023 LLM Efficiency Challenge (4090 Track) and Top 8 in the A100 Track, outperforming submissions from leading AI research labs.

---

Challenge Overview

The NeurIPS LLM Efficiency Challenge tasked participants with developing highly efficient large language models under strict constraints:

• Time Limit: Train base models within one day on a single GPU
• Hardware Tracks: NVIDIA RTX 4090 (24GB) and A100 (40GB)
• Goal: Maximize performance across diverse NLP benchmarks while respecting computational constraints

---

Our Approach

Parameter-Efficient Fine-Tuning

We implemented QLoRA (Quantized Low-Rank Adaptation), reducing trainable parameters by 90% while maintaining model quality. This allowed us to effectively fine-tune larger base models within the GPU memory constraints.

Data-Centric Strategies

• Curriculum Learning: Structured training from simple to complex examples
• High-Quality Data Curation: Carefully selected and cleaned training data
• Chain-of-Thought Augmentation: Enhanced reasoning through CoT demonstrations

Key Optimizations

• Mixed-precision training with gradient checkpointing
• Efficient data loading and preprocessing pipelines
• Hyperparameter optimization under time constraints

---

Results

Track	Ranking	Notable Achievement
4090 Track	Top 3	Outperformed major AI research labs
A100 Track	Top 8	Strong performance on reasoning tasks

Our models demonstrated particularly strong performance on:

• Mathematical reasoning benchmarks
• Code generation tasks
• General knowledge QA

---

Key Learnings

1. Data quality trumps quantity when training time is limited
2. Parameter-efficient methods enable training larger models on consumer hardware
3. Curriculum design significantly impacts final model capabilities

---

Repository

Full implementation and training scripts available at: https://github.com/knovel-eng/neurips-llm-2023