FastTree reorganizes attention work for requests that share prompt prefixes and reports up to 2.2× SGLang throughput.
ORIGINAL PAPER
FastTree: Optimizing Attention Kernel and Runtime for Tree-Structured LLM Inference
Shared prompts more than doubled LLM throughput
FastTree uses tree-tailored attention kernels plus runtime grouping to reduce redundant KV-cache loads.
Reported SGLang throughput improved by up to 2.2×.
Share of production traffic with reusable prompt trees, end-to-end cost per request, tail latency, and capacity orders after deployment.
What the paper supports
Supported findings
FastTree uses tree-tailored attention kernels plus runtime grouping to reduce redundant KV-cache loads.
Abstract and p. 11, ConclusionReported SGLang throughput improved by up to 2.2×.
AbstractLimitations
The benefit requires tree-structured prefix sharing; ordinary workloads may not see the same gain.
Runtime grouping uses a greedy heuristic rather than a globally optimal solution.
The paper does not report cloud TCO, pricing, or sustained production traces.
Companies with a traceable link
Alibaba Group9988 HK · BABA UN
Supports LongShared-context serving efficiency can lower the cost of agent and reasoning products.
Value-chainTencent700 HK
Supports LongThe same mechanism can improve cloud and application-layer model economics.
Value-chainGDS Holdings9698 HK · GDS UQ
MixedEfficiency may raise workload density while reducing incremental capacity per request.
Value-chain