đ€ Building a Telegram bot with Apache Kafka, Go, and ksqlDB
A presentation at Blueprint LDN by Robin Moffatt
đ€Building a Telegram bot with Apache Kafka, Go, and ksqlDB Robin Moffatt @rmoff
Whereâs my nearest carpark with available spaces?
How many spaces are available in this car park?
đĄTell me when a car park with spaces is available
đHow does occupancy vary over time?
$ whoami > Robin Moffatt (@rmoff) > Senior Developer Advocate at Confluent (Apache Kafka, not Wikis đ) > Working in data & analytics since 2001 > Oracle ACE Director (Alumnus) http://rmoff.dev/talks · http://rmoff.dev/blog · http://rmoff.dev/youtube @rmoff
Telegram @rmoff
Donât just tell me⊠show me! Demo code: https://rmoff.dev/carparks
carparks HTTP Kafka @rmoff
What are the key pieces of the design? @rmoff
Event Driven Alerts carparks HTTP Kafka @rmoff
K/V Lookups (materialised views) SELECT SPACES_AVAILABLE How many spaces are free at â FROM CARPARK WHERE NAME=âWESTGATEâ; Westgate carpark right now? ksqlDB â Kafka CARPARK_EVENTS 42 CREATE TABLE CARPARK AS SELECT LATEST(⊠GROUP BY NAME â There are 42 spaces free â @rmoff
A schema⊠carparks HTTP @rmoff
A schema⊠2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 ?! @rmoff
My kingdom for a schema! 2020-10-14,12:28,Broadway,1132,921 2020-10-14,12:28,Kirkgate Centre,611,474 2020-10-14,12:28,Sharpe Street,98,63 đ { âtsâ: â2020-10-14T12:28 UTC+1â, ânameâ: âBroadwayâ, âcapacityâ: 1132, âemptyâ: 921 } ⊠@rmoff
Applying a schema to streams of data source_topic ksqlDB CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); Kafka @rmoff
Applying a schema to streams of data source_topic ksqlDB Kafka derived_topic CREATE STREAM mySource (date VARCHAR , time VARCHAR , name VARCHAR , capacity INT ) WITH (KAFKA_TOPIC=âsource_topicâ, VALUE_FORMAT=âDELIMITEDâ); CREATE STREAM myTargetStream WITH (VALUE_FORMAT=âPROTOBUFâ, KAFKA_TOPIC=âderived_topicâ) AS SELECT * FROM mySource; @rmoff
Integration carparks HTTP Kafka @rmoff
Streaming Integration with Kafka Connect syslog Sources Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 Google BigQuery Sinks Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Integration with Kafka Connect Amazon S3 syslog Google BigQuery Tasks Workers Kafka Connect Kafka Brokers @rmoff
Streaming Analytics @rmoff
Why build it this way? @rmoff
Events @rmoff
Streams of Events @rmoff
We want to react to them as they happen @rmoff
We want to build state from a stream of events @rmoff
We want to provide the latest data in our analytics @rmoff
Apache Kafka - an Event Streaming Platform Producer Connectors Consumer The Log Connectors Streaming Engine @rmoff
Why Kafka? @rmoff
Distributed, Immutable, Event Log Old New Events are added at the end of the log @rmoff
Consumers can seek to any point Read to offset & scan Old New @rmoff
Data is not deleted once read New Old Sally is here Scan @rmoff
Consumers are independent of each other New Old Fred is here Scan Sally is here Scan @rmoff
Consumers can be added later Rick is here Scan New Old Fred is here Scan Sally is here Scan @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream @rmoff
Stream Processing with ksqlDB Source stream Analytics @rmoff
Stream Processing with ksqlDB Source stream Applications / Microservices @rmoff
Stream Processing with ksqlDB âŠSUM(TXN_AMT) GROUP BY AC_ID AC _I D= 42 BA LA NC AC E= _I 94 D= .0 42 0 Source stream Applications / Microservices @rmoff
Under the covers of ksqlDB Photo by @rmoff on
Kafka cluster consume produce ksqlDB @rmoff
JVM Kafka cluster consume produce ksqlDB Kafka Streams RocksDB @rmoff
k & ^ Kafka Fully Managed as a Service B D l q s
Running ksqlDB - self-managed DEB, RPM, ZIP, TAR downloads http://confluent.io/download Docker images ksqlDB Server confluentinc/ksqldb-server (JVM process) âŠand many more⊠@rmoff
Why Kafka? @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Stream Store Process Integrate @rmoff
Flexible, event-driven applications Event-driven alerts Key/Value lookups Streaming ETL @rmoff
on Photo by Want to learn more? CTAs, not CATs (sorry, not sorry) @rmoff
Try it out for yourself https://rmoff.dev/carparks
RM OF F2 00 $200 USD off your bill each calendar month for the first three months when you sign up https://rmoff.dev/ccloud Free money! (additional $200 towards your bill đ ) Fully Managed Kafka as a Service * T&C: https://www.confluent.io/confluent-cloud-promo-disclaimer
Learn Kafka. Start building with Apache Kafka at Confluent Developer. developer.confluent.io
#EOF https://talks.rmoff.net @rmoff
Confluent Community Slack group cnfl.io/slack @rmoff
Further reading / watching https://rmoff.dev/kafka-talks @rmoff
Imagine youâve got a stream of data; itâs not âbig data,â but itâs certainly a lot. Within the data, youâve got some bits youâre interested in, and of those bits, youâd like to be able to query information about them at any point. Sounds fun, right? Since I mentioned âquerying,â Iâd hazard a guess that youâve got in mind an additional datastore of some sort, whether relational or NoSQL.
But what if I told youâŠthat you didnât need any datastore other than Kafka itself? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how.
In this hands-on talk weâll walk through an example of building a Telegram bot in which ksqlDB provides the key/value lookups driven by a materialised view on the stream of events in Kafka. Weâll take a look at what ksqlDB is and its capabilities for processing data and driving applications, as well as integrating with other systems.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Confluent Developer
Your destination for learning all about ConfluentâTutorials, examples, documentation, blogs, podcasts, and lots more!
-
đŸCode for you to try the full demo yourself
The demo is built on Docker so anyone can easily spin it up and give it a try.
-
âïžConfluent Cloud
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
- Use promo code RMOFF200 to get an additional $200 of free Confluent Cloud usage. Be sure to activate it by June 30 2021, and to use it within 90 days after activation. Any unused promo value on the expiration date will be forfeited and there are a limited number of codes available, so donât miss out!
-
đ Kafka as a Platform: the Ecosystem from the Ground Up
-
đ„ Kafka Connect tutorials on YouTube
Buzz and feedback
Hereâs what was said about this presentation on social media.
