Apache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline!
A presentation at London Apache Kafka Meetup in in London, UK by Robin Moffatt
Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline! @rmoff #KafkaMeetup
Kafka is an Event Streaming Platform App App App @rmoff #KafkaMeetup App request-response changelogs App App KAFKA App App DWH Hadoop messaging OR stream processing streaming data pipelines Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Database Offload @rmoff #KafkaMeetup Amazon S3 RDBMS Kafka Connect Kafka Connect HDFS Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Processing with Apache Kafka and ksqlDB order events CDC RDBMS customer orders customer Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Real-time Event Stream Enrichment @rmoff #KafkaMeetup order events customer orders C D C RDBMS <y> customer Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Message Bus @rmoff #KafkaMeetup order events App <x> Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Processing order events Stream Processing App <x> Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Transform Once, Use Many @rmoff #KafkaMeetup order events Completed orders Stream Processing App <x> Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Evolve processing from old systems to new Existing App New App <x> C D C RDBMS Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Evolve processing from old systems to new Existing App New App <x> New App <y> C D C RDBMS Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Evolve processing from old systems to new Existing App New App <x> New App <y> C D C RDBMS Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Building a streaming data pipeline Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Integration + Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Integration Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Streaming Integration with Kafka Connect syslog Sources Kafka Connect Kafka Brokers Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Streaming Integration with Kafka Connect Amazon S3 Sinks Google BigQuery Kafka Connect Kafka Brokers Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Kafka Connect Kafka Brokers Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Look Ma, No Code! { “connector.class”: “io.confluent.connect.jdbc.JdbcSourceConnector”, “jdbc:mysql://asgard:3306/demo”, “table.whitelist”: “sales,orders,customers” } https://docs.confluent.io/current/connect/ “connection.url”: Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Serialisation & Schemas Avro -> Confluent Schema Registry Protobuf JSON CSV https://qconnewyork.com/system/files/presentation-slides/qcon_17_-_schemas_and_apis.pdf Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup The Confluent Schema Registry Avro Schema Schema Registry Target Source Kafka Connect Avro Message Avro Message Kafka Connect Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Extensible Connector Transform(s) Converter Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Confluent Hub hub.confluent.io Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Integration + Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Processing Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
} “reading_ts”: “2020-02-14T12:19:27Z”, “sensor_id”: “aa-101”, “production_line”: “w01”, “widget_type”: “acme94”, “temp_celcius”: 23, “widget_weight_g”: 100 Photo by Franck V. on Unsplash { @rmoff #KafkaMeetup Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup SELECT * FROM WIDGETS WHERE WEIGHT_G > 120 { SELECT COUNT(*) FROM WIDGETS GROUP BY PRODUCTION_LINE } SELECT AVG(TEMP_CELCIUS) AS TEMP FROM WIDGETS GROUP BY SENSOR_ID HAVING TEMP>20 Photo by Franck V. on Unsplash “reading_ts”: “2020-02-14T12:19:27Z”, “sensor_id”: “aa-101”, “production_line”: “w01”, “widget_type”: “acme94”, “temp_celcius”: 23, “widget_weight_g”: 100 CREATE SINK CONNECTOR dw WITH ( Object store, ‘connector.class’ = ‘S3Connector’, data warehouse, ‘topics’ = ‘widgets’ RDBMS …); Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
ksqlDB @rmoff #KafkaMeetup The event streaming database purpose-built for stream processing applications. Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Stream Processing with ksqlDB @rmoff #KafkaMeetup Source stream Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Stream Processing with ksqlDB @rmoff #KafkaMeetup Source stream Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Stream Processing with ksqlDB @rmoff #KafkaMeetup Source stream Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Stream Processing with ksqlDB @rmoff #KafkaMeetup Source stream Analytics Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Stream Processing with ksqlDB @rmoff #KafkaMeetup Source stream Applications / Microservices Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Stream Processing with ksqlDB …SUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Let’s Build It! Rating events App Pro d uc e rA PI { “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” } Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Let’s Build It! Rating events App a k f a K t c e n n o C App u s n o C uc e rA PI Kafka Connect a fk t Ka ec n RDBMS Pro d I P A r e m Operational Dashboard Elasticsearch n Co User data Push notification ksqlDB Join events to users, and filter Data Lake SnowflakeDB/ S3/HDFS/etc Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Push notification Rating events App Pro d I P A r e m App u s n o C uc e rA PI Demo Time! a fk t Ka ec n RDBMS Operational Dashboard Elasticsearch n Co User data a k f a K t c e n n o C Kafka Connect ksqlDB Join events to users, and filter Data Lake SnowflakeDB/ S3/HDFS/etc Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Push notification Rating events App Kafka Connect a fk t Ka ec n RDBMS u s n o C uc e rA PI a k f a K t c e n n o C App Operational Dashboard Elasticsearch n Co User data Pro d I P A r e m ksqlDB Join events to users, and filter Data Lake SnowflakeDB/ S3/HDFS/etc Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Push notification to Slack Rating events App Pro d I P A r e m App u s n o C uc e rA PI Kafka Connect Operational Dashboard Elasticsearch n Co a fk t Ka ec n a k f a K t c e n n o C poor_ratings Data ratings ksqlDB User data RDBMS Lake Filter events S3/HDFS/ SnowflakeDB etc Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
{ “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” @rmoff #KafkaMeetup Filter all ratings where STARS<3 POOR_RATINGS } Producer API CREATE STREAM POOR_RATINGS AS SELECT * FROM ratings WHERE STARS <3 Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Push notification to Slack Kafka Connect Rating events App Kafka Connect a fk t Ka ec n RDBMS u s n o C uc e rA PI a k f a K t c e n n o C App Operational Dashboard Elasticsearch n Co User data Pro d I P A r e m Join events to users, and filter Data Lake SnowflakeDB/ S3/HDFS/etc Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
@rmoff #KafkaMeetup Producer API MySQL t c e n n o C a k f Ka m u i z e b e D Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Time The Stream/Table Duality Stream Account ID Amount 12345 + £50 12345
- £25 12345
- £60 @rmoff #KafkaMeetup Account ID Balance Table 12345 £50 Account ID Balance 12345 £75 Account ID Balance 12345 £15 Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
The truth is the log. The database is a cache of a subset of the log. —Pat Helland Immutability Changes Everything http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf Photo by Bobby Burch on Unsplash
{ “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” } Producer API @rmoff #KafkaMeetup Join each rating to customer data RATINGS_WITH_CUSTOMER_DATA t c e n n o C a k f a K { “id”: 3, “first_name”: “Merilyn”, “last_name”: “Doughartie”, “email”: “[email protected]”, “gender”: “Female”, “club_status”: “platinum”, “comments”: “none” CREATE STREAM RATINGS_WITH_CUSTOMER_DATA AS SELECT * FROM RATINGS LEFT JOIN CUSTOMERS ON R.ID=C.ID; } Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
{ “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” } Producer API t c e n n o C a k f a K @rmoff #KafkaMeetup Join each rating to customer data RATINGS_WITH_CUSTOMER_DATA Filter for just PLATINUM customers UNHAPPY_PLATINUM_CUSTOMERS { “id”: 3, “first_name”: “Merilyn”, “last_name”: “Doughartie”, “email”: “[email protected]”, “gender”: “Female”, “club_status”: “platinum”, “comments”: “none” CREATE STREAM UNHAPPY_PLATINUM_CUSTOMERS AS SELECT * FROM RATINGS_WITH_CUSTOMER_DATA WHERE STARS < 3 } Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
{ “rating_id”: 5313, “user_id”: 3, “stars”: 4, “route_id”: 6975, “rating_time”: 1519304105213, “channel”: “web”, “message”: “worst. flight. ever. #neveragain” @rmoff #KafkaMeetup CREATE TABLE RATINGS_BY_CLUB_STATUS AS SELECT CLUB_STATUS, COUNT(*) Join each rating to customer data FROM RATINGS_WITH_CUSTOMER_DATA Producer APWINDOW I TUMBLING RATINGS_WITH_CUSTOMER_DATA (SIZE 1 MINUTES) GROUP BY CLUB_STATUS; } t c e n n o C a k f a K { “id”: 3, “first_name”: “Merilyn”, “last_name”: “Doughartie”, “email”: “[email protected]”, “gender”: “Female”, “club_status”: “platinum”, “comments”: “none” } Aggregate per-minute by CLUB_STATUS RATINGS_BY_CLUB_STATUS_1MIN Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Kafka Connect → Elasticsearch @rmoff #KafkaMeetup Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Photo by on @rmoff #KafkaMeetup Want to learn more? CTAs, not CATs (sorry, not sorry) Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
kafka-summit.org Moffatt30 30% OFF* *Standard Priced Conference pass
Learn Kafka. Start building with Apache Kafka at Confluent Developer. developer.confluent.io
Confluent Community Slack group @rmoff #KafkaMeetup cnfl.io/slack Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Free Books! @rmoff #KafkaMeetup https://rmoff.dev/ldn-mar20 Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Resources @rmoff #KafkaMeetup #EOF • CDC Spreadsheet • Blog: No More Silos: How to Integrate your Databases with Apache Kafka and CDC • #partner-engineering on Slack for questions • BD team (#partners / [email protected]) can help with introductions on a given sales op Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
Kafka is a whole lot more than just a message bus! It includes stream processing and integration capabilities, and this talk will be mostly live coding to demonstrate what can be built around it, how to do it - and why to do it.
Have you ever thought that you needed to be a programmer to do stream processing and build streaming data pipelines? Think again! Apache Kafka is a distributed, scalable, and fault-tolerant streaming platform, providing low-latency pub-sub messaging coupled with native storage and stream processing capabilities. Integrating Kafka with RDBMS, NoSQL, and object stores is simple with Kafka Connect, which is part of Apache Kafka. ksqlDB is the source-available SQL streaming engine for Apache Kafka, and makes it possible to build stream processing applications at scale, written using a familiar SQL interface.
In this talk, we’ll explain the architectural reasoning for Apache Kafka and the benefits of real-time integration, and we’ll build a streaming data pipeline using nothing but our bare hands, Kafka Connect, and ksqlDB.
Gasp as we filter events in real-time! Be amazed at how we can enrich streams of data with data from RDBMS! Be astonished at the power of streaming aggregates for anomaly detection!
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
☁️Confluent Cloud
Fully Managed Apache Kafka, Schema Registry, KSQL, and Connectors
-
📚Free eBooks
Free eBooks to download, including “Kafka: The Definitive Guide”, and “I ❤️Logs”
-
📌From Zero to Hero with Kafka Connect
Learn all about Kafka Connect (including the connectors available with ksqlDB)
-
📌Introduction to ksqlDB
Learn more about ksqlDB beyond what was covered in this talk
-
📌No More Silos: Integrating Databases and Apache Kafka
Details on the specifics of streaming changes from a database into Kafka.
-
📌The Changing Face of ETL: Event-Driven Architectures for Data Engineers
-
✍️Kafka Connect Deep Dive – Converters and Serialization Explained
-
✍️Kafka Connect Deep Dive – Error Handling and Dead Letter Queues
-
👾Demo code
Code and script to try out the demo from the talk - you just need Docker Compose.
-
👥Kafka Summit London 2020
Two days of talks all about Apache Kafka - register with code
MOFFATT30for a 30% discount.
Buzz and feedback
Here’s what was said about this presentation on social media.
