API pricing often reveals significant cost differences between input (prefill) and output (decode) tokens, with output being substantially more expensive (e.g., 5x). This suggests that decode operations are heavily memory bandwidth-limited, while prefill can be more compute-limited.
Impact: High. The disparity in pricing between prefill and decode highlights critical performance bottlenecks and informs how models are optimized for different operational phases.
In the source video, this keypoint occurs from 01:44:09 to 01:47:56.
Sources in support: Dwarkesh Patel (Host)

