đ€Building a Telegram bot with Apache Kafka and ksqlDB
A presentation at Big Mountain Data and Dev Conference by Robin Moffatt
đ€Building a Telegram bot with Apache Kafka and ksqlDB @rmoff Robin Moffatt
Whereâs my nearest carpark with available spaces?
How many spaces are available in this car park?
đĄTell me when a car park with spaces is available
đHow does occupancy vary over time?
$ whoami > Robin Moffatt (@rmoff) > Senior Developer Advocate at Confluent (Apache Kafka, not Wikis đ) > Working in data & analytics since 2001 > Oracle ACE Director (Alumnus) http://rmoff.dev/talks · http://rmoff.dev/blog · http://rmoff.dev/youtube @rmoff
Telegram @rmoff
Donât just tell me⊠show me! Demo code: https://rmoff.dev/carparks
carparks HTTP Kafka @rmoff
What are the key pieces of the design? @rmoff
Event Driven Alerts carparks HTTP Kafka @rmoff
K/V Lookups (materialised views) SELECT SPACES_AVAILABLE How many spaces are free at â FROM CARPARK WHERE NAME=âWESTGATEâ; Westgate carpark right now? ksqlDB â Kafka CARPARK_EVENTS 42 CREATE TABLE CARPARK AS SELECT LATEST(⊠GROUP BY NAME â There are 42 spaces free â @rmoff
A schema⊠carparks HTTP @rmoff
A schema⊠2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 ?! @rmoff
My kingdom for a schema! 2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 đ { âtsâ: â2020-10-14T12:28 UTC+1â, ânameâ: âBroadwayâ, âcapacityâ: 1132, âemptyâ: 921 } ⊠@rmoff
Applying a schema to streams of data source_topic ksqlDB CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); Kafka @rmoff
Applying a schema to streams of data source_topic ksqlDB Kafka derived_topic CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); CREATE STREAM myTargetStream WITH (VALUE_FORMAT=âPROTOBUFâ, KAFKA_TOPIC=âderived_topicâ) AS SELECT * FROM mySource; @rmoff
Integration carparks HTTP Kafka @rmoff
Streaming Integration with Kafka Connect syslog Sources Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 Google BigQuery Sinks Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Analytics @rmoff
Why build it this way? @rmoff
Events @rmoff
Streams of Events @rmoff
We want to react to them as they happen @rmoff
We want to build state from a stream of events @rmoff
We want to provide the latest data in our analytics @rmoff
Apache Kafka - an Event Streaming Platform Producer Connectors Consumer The Log Connectors Streaming Engine @rmoff
Why Kafka? @rmoff
Distributed, Immutable, Event Log Old New Events are added at the end of the log @rmoff
Consumers can seek to any point Read to offset & scan Old New @rmoff
Data is not deleted once read New Old Sally is here Scan @rmoff
Consumers are independent of each other New Old Fred is here Scan Sally is here Scan @rmoff
Consumers can be added later Rick is here Scan New Old Fred is here Scan Sally is here Scan @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream Analytics @rmoff
Stream Processing with ksqlDB Source stream Applications / Microservices @rmoff
Stream Processing with ksqlDB âŠSUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices @rmoff
Under the covers of ksqlDB Photo by @rmoff on
Kafka cluster consume produce ksqlDB @rmoff
JVM Kafka cluster consume produce ksqlDB Kafka Streams RocksDB @rmoff
k & ^ Kafka Fully Managed as a Service B D l q s
Running ksqlDB - self-managed DEB, RPM, ZIP, TAR downloads http://confluent.io/download Docker images ksqlDB Server confluentinc/ksqldb-server (JVM process) âŠand many more⊠@rmoff
Why Kafka? @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Flexible, event-driven applications Event-driven alerts Key/Value lookups Streaming ETL @rmoff
on Photo by Want to learn more? CTAs, not CATs (sorry, not sorry) @rmoff
Try it out for yourself https://rmoff.dev/carparks
60 DE VA DV $200 USD off your bill each calendar month for the first three months when you sign up https://rmoff.dev/ccloud Free money! (additional $60 towards your bill đ ) Fully Managed Kafka as a Service * T&C: https://www.confluent.io/confluent-cloud-promo-disclaimer
Learn Kafka. Start building with Apache Kafka at Confluent Developer. developer.confluent.io
Confluent Community Slack group cnfl.io/slack @rmoff
Further reading / watching https://rmoff.dev/kafka-talks @rmoff
#EOF https://talks.rmoff.net @rmoff
Imagine youâve got a stream of data; itâs not âbig data,â but itâs certainly a lot. Within the data, youâve got some bits youâre interested in, and of those bits, youâd like to be able to query information about them at any point. Sounds fun, right? Since I mentioned âquerying,â Iâd hazard a guess that youâve got in mind an additional datastore of some sort, whether relational or NoSQL. But what if I told youâŠthat you didnât need any datastore other than Kafka itself? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how. In this hands-on talk weâll walk through an example of building a Telegram bot in which ksqlDB provides the key/value lookups driven by a materialised view on the stream of events in Kafka. Weâll take a look at what ksqlDB is and its capabilities for processing data and driving applications, as well as integrating with other systems.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
âïžConfluent Cloud
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
- Use promo code 60DEVADV to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by Dec. 31, 2021, and to use it within 90 days after activation. Any unused promo value on the expiration date will be forfeited and there are a limited number of codes available, so donât miss out!
-
âïž Streaming data from Oracle into Kafka
Blog about the options for getting data from Oracle into Kafka
-
đ„ Integrating Oracle and Kafka
A conference talk covering the options for getting data from Oracle into Kafka
-
đKafka as a Platform: the Ecosystem from the Ground Up
Kafka 101 - introducing the concepts, the APIs, and the ecosystem
-
đŸDemo code
Try out the demo for yourself - all you need is Docker and Docker Compose.
-
đFree eBooks
Free eBooks to download, including Kafka: The Definitive Guide.
-
âčïž Confluent Developer
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
-
đ„ Kafka Connect tutorials on YouTube
-
𧩠Confluent Hub
Huge list of connectors for Kafka Connect
-
âïž Blog - Building a Telegram Bot Powered by Apache Kafka and ksqlDB
A fun blog showing what you can do with ksqlDB and Kafka
-
đŹ Confluent Community Slack group
-
đ Apache Kafka and ksqlDB in Action: Letâs Build a Streaming Data Pipeline!
-
đ Introduction to ksqlDB
Learn all about ksqlDB in this 45 minute talk & live demo
-
đFrom Zero to Hero with Kafka Connect
Learn all about Kafka Connect (including the connectors available with ksqlDB)
-
đThe Changing Face of ETL: Event-Driven Architectures for Data Engineers
-
đOn Track with Apache Kafka: Building a Streaming Platform solution with Rail Data
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.
Buzz and feedback
Hereâs what was said about this presentation on social media.
