FlashAttention

What is FlashAttention?

Attention is the heart of the Transformer, but it has a bad habit: slow and memory-hungry. Make the sequence long and VRAM climbs like a leak. FlashAttention is a faster, cheaper attention algorithm that, through clever scheduling, brings both memory use and compute time down.

What problem does it solve?

Memory grows quadratically with sequence length
Traditional attention stores an absurdly large "attention matrix" in full. Double the sequence and memory can jump fourfold.
Constant memory traffic slows things down
A lot of back-and-forth between memory tiers is wasted motion.

How does it pull that off?

Tiling
Instead of computing the whole matrix at once, it splits the work into small blocks computed in faster on-chip memory, keeping only the results.
Recomputation
What backprop needs isn't stored — it's recomputed on the fly, trading a little compute for a big drop in memory.
Less data movement
Fewer round-trips through slow memory means the GPU spends its effort actually computing.

Why it matters

FlashAttention has become standard in nearly all mainstream large models. It makes longer contexts and bigger models practical — one of the quiet heroes behind "models keep growing while memory keeps shrinking."

Bottom line: FlashAttention slims down and speeds up attention — compute in blocks, move less data, and you get cheaper memory and faster runs.