Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
Paper
•
2510.01817
•
Published
•
11
Sparse Query Attention (SQA) Research
Experimental models with Sparse Query Attention layers. Reducing training time/cost by ~3-10% compared to GQA & MQA, with the same level performance
-
-
ReactiveAI/sSQAT-mm
Text Generation • 0.0B • Updated -
ReactiveAI/SQAT-mm
Text Generation • 0.0B • Updated -
ReactiveAI/xSQAT-mm
Text Generation • 0.0B • Updated -
ReactiveAI/SQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/sSQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/xSQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/xSMQAT-m
Text Generation • 0.0B • Updated -
ReactiveAI/GQA-Ref-Micro
Text Generation • 0.0B • Updated -
ReactiveAI/MQA-Ref-Micro
Text Generation • 0.0B • Updated