Radar
Top stories
Importance 2 of 5Covered by 1 source

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context…