NEO moves attention work and KV cache to host CPUs to fit bigger batches, delivering large gains on T4s but much smaller gains on H100s.
ORIGINAL PAPER
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
Moving AI memory to CPUs could unlock bigger batches without new GPUs
Throughput improved up to 7.5× on T4, 26% on A10G, and 14% on H100 while maintaining latency in the reported workloads.
The gain is positively related to CPU memory bandwidth and is strongest when GPU memory constrains batch size.
vLLM integration, four- and eight-GPU production tests, throughput per watt, CPU cost allocation, and performance on H100-class memory-rich fleets.
What the paper supports
Supported findings
Throughput improved up to 7.5× on T4, 26% on A10G, and 14% on H100 while maintaining latency in the reported workloads.
Abstract; p. 11, ConclusionThe gain is positively related to CPU memory bandwidth and is strongest when GPU memory constrains batch size.
pp. 10-11, DiscussionLimitations
NEO falls back to GPU-only operation when GPU memory is sufficient.
The authors had not integrated NEO into production-grade vLLM for larger multi-GPU tests.
On-prem TCO and throughput per watt may be worse because extra CPU work consumes energy and may require paid resources.
Companies with a traceable link
Alibaba Group9988 HK · BABA UN
Supports LongUsing already-provisioned CPU resources can improve inference economics in memory-constrained fleets.
Value-chainTencent700 HK
Supports LongHeterogeneous serving can increase throughput where fleet memory is the bottleneck.
Value-chainDell TechnologiesDELL UN
Challenges ShortCPU-rich server design becomes more valuable if heterogeneous inference broadens.
Value-chainGDS Holdings9698 HK · GDS UQ
MixedHigher throughput improves tenant economics but adds CPU power use and can change density requirements.
Value-chain