Kafka as a Platform: the Ecosystem from the Ground Up
A presentation at Kafka Summit London 2022 in in London, UK by Robin Moffatt
Kafka as a Platform: the Ecosystem from the Ground Up Robin Moffatt @rmoff #kafkasummit
EVENTS @rmoff
EVENTS @rmoff
• • EVENTS d e n e p p a h g n i h t e Som d e n e p p a h t a Wh
Human generated events A Sale A Stock movement @rmoff
Machine generated events Networking IoT Applications @rmoff
EVENTS are EVERYWHERE @rmoff
EVENTS y r e v ^ are POWERFUL @rmoff
K V
LOG @rmoff
K V
K V
K V
K V
K V
K V
K V
Immutable Event Log Old New Events are added at the end of the log @rmoff
TOPICS @rmoff
Topics Clicks Orders Customers Topics are similar in concept to tables in a database @rmoff
PARTITIONS @rmoff
Partitions Clicks p0 P1 P2 Messages are guaranteed to be strictly ordered within a partition @rmoff
PUB / SUB @rmoff
PUB / SUB @rmoff
Producing data Old New Messages are added at the end of the log @rmoff
package main import ( “gopkg.in/confluentinc/confluent-kafka-go.v1/kafka” ) func main() { topic := “test_topic” p, _ := kafka.NewProducer(&kafka.ConfigMap{ “bootstrap.servers”: “localhost:9092”}) defer p.Close() p.Produce(&kafka.Message{ TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: 0}, Value: []byte(“Hello world”)}, nil) }
Producing to Kafka - No Key Time Partition 1 Partition 2 Partition 3 Partition 4 Messages will be batched and randomly distributed across the partitions @rmoff
Producing to Kafka - With Key Time Partition 1 A Partition 2 B hash(key) % numPartitions = N Partition 3 C Partition 4 D @rmoff
Producers • A client application • Puts messages into topics • Handles partitioning, network protocol • Java, Go, .NET, C/C++, Python • Also every other language Plus REST proxy if not @rmoff
PUB / SUB @rmoff
Consuming data - access is only sequential Read to offset & scan Old New @rmoff
Consumers have a position of their own Old Victoria is here New Scan @rmoff
Consumers have a position of their own Old Victoria is here New Scan Tim is here Scan @rmoff
Consumers have a position of their own Rick is here Scan Old Victoria is here New Scan Tim is here Scan @rmoff
c, _ := kafka.NewConsumer(&cm) defer c.Close() c.Subscribe(topic, nil) for { select { case ev := <-c.Events(): switch ev.(type) { case *kafka.Message: km := ev.(*kafka.Message) fmt.Printf(“✅ Message ‘%v’ received from topic ‘%v’\n”, string(km.Value), string(*km.TopicPartition.Topic)) } } }
Consuming From Kafka - Single Consumer Partition 1 App1 Partition 2 Partition 3 Partition 4 @rmoff
Consuming From Kafka - Multiple Consumers Partition 1 A1 1 App Partition 2 Partition 3 App2 Partition 4 @rmoff
Consuming From Kafka - Grouped Consumers Partition 1 A App 1 App11 Partition 2 Partition 3 App2 Partition 4 @rmoff
Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 C2 App1 Partition 4 @rmoff
Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 C2 App1 Partition 4 @rmoff
Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 3 App1 Partition 4 @rmoff
Consuming From Kafka - Grouped Consumers Partition 1 Partition 2 Partition 3 C1 3 App1 Partition 4 @rmoff
Consumers • A client application App App11 A App2 • Reads messages from topics • Horizontally, elastically scalable (if stateless) • Java, Go, .NET, C/C++, Python, everything else Plus REST proxy if not @rmoff
BROKERS and REPLICATION @rmoff
Leader Partition Leadership and Replication Follower Partition 1 Partition 2 Partition 3 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff
Leader Partition Leadership and Replication Follower Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partition 2 Partition 3 Partition 3 Partition 3 Partition 4 Partition 4 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff
Leader Partition Leadership and Replication Follower Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partition 2 Partition 3 Partition 3 Partition 3 Partition 4 Partition 4 Partition 4 Broker 1 Broker 2 Broker 3 @rmoff
So far, this is Pretty good @rmoff
So far, this is Pretty good but I’ve not finished yet… @rmoff
Streaming Pipelines Amazon S3 RDBMS HDFS @rmoff
Streaming Integration with Kafka Connect syslog Sources Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon Sinks Google Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon syslog Google Kafka Connect Kafka Brokers @rmoff
Look Ma, No Code! { “connector.class”: “io.confluent.connect.jdbc.JdbcSourceConnector”, “connection.url”: “jdbc:mysql://asgard:3306/demo”, “table.whitelist”: “sales,orders,customers” } @rmoff
Extensible Connector Transform(s) Converter @rmoff
hub.confluent.io @rmoff
K V
K V
K V
K V
K V ? s i h t s ’ t a h w … t i a W
Lack of schemas – Coupling teams and services 2001 2001 Citrus Heights-Sunrise Blvd Citrus_Hghts 60670001 3400293 34 SAC Sacramento SV Sacramento Valley SAC Sacramento County APCD SMA8 Sacramento Metropolitan Area CA 6920 Sacramento 28 6920 13588 7400 Sunrise Blvd 95610 38 41 56 38.6988889 121 16 15.98999977 -121.271111 10 4284781 650345 52 @rmoff
Serialisation & Schemas JSON Avro Protobuf Schema JSON CSV @rmoff
Serialisation & Schemas JSON Avro Protobuf Schema JSON CSV 👍 👍 👍 😬 https://rmoff.dev/qcon-schemas @rmoff
Schemas Schema Registry Topic producer … consumer @rmoff
A Consumer 1 Consumer @rmoff
A1 Consumer Consumer @rmoff
{ @rmoff
.stream(“widgets”, Consumed.with(stringSerde, widgetsSerde)) .filter( (key, widget) -> widget.getColour().equals(“RED”) ) .to(“widgets_red”, Produced.with(stringSerde, widgetsSerde));
A1 Consumer Consumer @rmoff
Streams A1 App Streams App @rmoff
Stream Processing with ksqlDB Stream: widgets ksqlDB CREATE STREAM widgets_red AS SELECT * FROM widgets WHERE colour=’RED’; Stream: widgets_red @rmoff
{ @rmoff
FROM WIDGETS WHERE WEIGHT_G > 120 { SELECT COUNT(*) FROM WIDGETS GROUP BY PRODUCTION_LINE SELECT AVG(TEMP_CELCIUS) AS TEMP FROM WIDGETS GROUP BY SENSOR_ID HAVING TEMP>20 ‘connector.class’ = ‘S3Connector’, ‘topics’ = ‘widgets’ …);
ksqlDB or Kafka Streams? @rmoff Photo by Ramiz Dedaković on Unsplash
Standing on the Shoulders of Streaming Giants ksqlDB Powered by Ease of use ksqlDB UDFs Kafka Streams Powered by Producer, Consumer APIs Flexibility @rmoff
Summary @rmoff
@rmoff
K V @rmoff
K V @rmoff
The Log @rmoff
Producer Consumer The Log @rmoff
Producer Consumer The Log Connectors @rmoff
Producer Consumer The Log Connectors Streaming Engine @rmoff
Apache Kafka Producer Consumer The Log Connectors Streaming Engine @rmoff
Producer Security Schema Registry Consumer The Log Streaming Engine ksqlDB REST Proxy Connectors Confluent Control Center
F2 00 OF RM Free money! (additional $200 towards your bill 😄 ) Fully Managed Kafka as a Service fl fl
- T&C: https://www.con uent.io/con uent-cloud-promo-disclaimer
s e l c i t r a eep-dive D • • • • • ka? f a K e ds h c n a e r p T A d s i e lat e R What . s v g min a e per r e t e S K t o n o e Z v E ut o h t i w fka a a K k f n a i K s : e t f ante KRa r a u G & ns o i t c a s n a Tr ge a r o t S & ng Processi tals n e m a d Fun • • • • • e c n a m r o f r Kafka Pe a k f a K e v i t ms e t s y S Cloud-na e s ba a t a D g n Streami fka a K e h c a p ls a n Testing A r e t n I s fka’ a K e r o l Exp • • • • • Over 10 Apache K afka 101 Kafka Co nnect 10 1 Kafka Str eams 101 ksqlDB 1 01 Inside ks qlDB hours of • • • • f ree cou rses Spring F ramewo rk and K Building afka Data Pip elines wi Event So th Kafka urcing w ith Kafka Data Me sh 101 Plus: Hands-on Quick Starts and Client Language Guides + Event Streaming Patterns + More fl developer.con uent.io
#EOF @rmoff rmoff.dev/talks youtube.com/rmoff
Kafka has become a key data infrastructure technology, and we all have at least a vague sense that it is a messaging system, but what else is it? How can an overgrown message bus be getting this much buzz? Well, because Kafka is merely the center of a rich streaming data platform that invites detailed exploration.
In this talk, we’ll look at the entire streaming platform provided by Apache Kafka and the Confluent community components. Starting with a lonely key-value pair, we’ll build up topics, partitioning, replication, and low-level Producer and Consumer APIs. We’ll group consumers into elastically scalable, fault-tolerant application clusters, then layer on more sophisticated stream processing APIs like Kafka Streams and ksqlDB. We’ll help teams collaborate around data formats with schema management. We’ll integrate with legacy systems without writing custom code. By the time we’re done, the open-source project we thought was Big Data’s answer to message queues will have become an enterprise-grade streaming platform, all in 45 minutes.
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
🎥 Recording
-
☁️Confluent Cloud☁️
Managed Apache Kafka, ksqlDB, and Schema Registry. Use code
RMOFF200when you sign up! -
Kafka Internals - free training
Full length course written by Jun Rao, one of the original creators of Apache Kafka.
-
Apache Kafka 101
Free training course
-
Confluent Developer
The pre-eminent resource for learning Apache Kafka. There are free training courses, event streaming patterns, deep-dive articles, and language-specific client programming guides. Check it out!
Buzz and feedback
Here’s what was said about this presentation on social media.
