System Design: Por que o LinkedIn precisou criar o Kafka?
Kafka's Architecture: Topics, Partitions, and Replication
Kafka organizes data into topics, which are further divided into partitions to enable horizontal scaling. Each partition is an ordered, append-only log. Order is guaranteed only within a partition, allowing for parallel processing across multiple partitions and servers. Availability is ensured through replication, where follower brokers asynchronously replicate data from a leader broker; if a leader fails, a follower takes over, maintaining data integrity and system uptime.
Kafka as a Central Hub for Data Integration
Kafka acts as a central data hub, enabling a single write event to trigger multiple downstream processes without tight coupling. For instance, a profile update at LinkedIn generates one event that simultaneously updates search indexes, recalculates recommendations, and logs data for analytics. This decoupling allows consumers to operate at their own pace, recover from failures independently, and process data in either real-time or batch modes, transforming event processing from a bottleneck into a flexible pipeline.
