Dense attention, often seen in models like Character AI and Gemma, can achieve low bytes per token by sharing context across layers. Sparse attention offers another approach by increasing parameters but dividing by a sparsity term, though excessive sparsity can degrade quality.

Impact: Medium. These different attention mechanisms offer trade-offs between efficiency and model quality, influencing architectural decisions for LLMs.

In the source video, this keypoint occurs from 01:41:02 to 01:44:09.

Sources in support: Dwarkesh Patel (Host)