Dynamic Input Pruning reduces the weights moved from flash memory, improving simulated mobile LLM speed with little perplexity loss.
ORIGINAL PAPER
Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking
Mobile AI got 40 percent faster without shrinking the model
On Phi-3-Medium, the reported setup reduced memory use 46% and increased throughput 40% with less than 0.1 perplexity loss.
Cache-aware masking improved the accuracy-throughput trade-off relative to ordinary cache policies in the reported simulation.
Run the method on shipping Xiaomi-class silicon and measure tokens per second, battery drain, thermals, memory footprint, and feature usage.
What the paper supports
Supported findings
On Phi-3-Medium, the reported setup reduced memory use 46% and increased throughput 40% with less than 0.1 perplexity loss.
Abstract; p. 10, ConclusionCache-aware masking improved the accuracy-throughput trade-off relative to ordinary cache policies in the reported simulation.
pp. 8-10, EvaluationLimitations
The headline throughput result comes from simulated hardware rather than a shipping phone.
Benefits vary by model architecture, sparsity level, memory configuration, and fine-tuning.
The paper leaves broader model and hardware testing to future work.
Companies with a traceable link
Xiaomi1810 HK
Supports LongLower memory traffic could make useful on-device LLM features feasible on more handsets.
Value-chain