The cost of re-creating (rematerializing) the KV cache from scratch is primarily compute-bound, while storing it in memory tiers like HBM incurs costs related to capacity and bandwidth. The choice depends on how long the cache needs to be held, with shorter holds favoring faster, more expensive tiers.

Impact: Medium. Understanding the cost dynamics of rematerialization versus storage is crucial for efficient cache management and overall model operational efficiency.

In the source video, this keypoint occurs from 01:51:56 to 01:55:05.

Sources in support: Dwarkesh Patel (Host)