This project introduces a workload-aware prefill optimization layer for LLM serving. It adapts batch timing from live tail-latency feedback and packs prompts by length before they enter a continuous batching engine.
Project Overview
- Built an AIMD controller using real-time p95 time-to-first-token feedback.
- Implemented greedy and dynamic-programming prompt packing to reduce padding waste across heterogeneous requests.
- Integrated the controller as a lightweight queue layer before a vLLM-style continuous batching engine.
- Evaluated on production-style DynamoLLM traces, achieving up to 20% lower TTFT.
Read the technical report or explore the implementation.