Researchers at Prefrontal AI and ETH Zurich propose latent sparse attention (LSA), which clusters keys and values and uses hierarchical search to reduce indexing overhead during the long-context prefill stage. On a 10 million-token task, D-LSA speeds up the prefill of the attention layers in GLM and Qwen by 22.7x and 72.4x respectively, though retrieval accuracy varies across models.
Source:TechTalks · Read the original
Leave a Reply