A presentation at Big Mountain Data and Dev Conference by Robin Moffatt
Imagine you’ve got a stream of data; it’s not “big data,” but it’s certainly a lot. Within the data, you’ve got some bits you’re interested in, and of those bits, you’d like to be able to query information about them at any point. Sounds fun, right? Since I mentioned “querying,” I’d hazard a guess that you’ve got in mind an additional datastore of some sort, whether relational or NoSQL. But what if I told you…that you didn’t need any datastore other than Kafka itself? What if you could ingest, filter, enrich, aggregate, and query data with just Kafka? With ksqlDB we can do just this, and I want to show you exactly how. In this hands-on talk we’ll walk through an example of building a Telegram bot in which ksqlDB provides the key/value lookups driven by a materialised view on the stream of events in Kafka. We’ll take a look at what ksqlDB is and its capabilities for processing data and driving applications, as well as integrating with other systems.
The following resources were mentioned during the presentation or are useful additional information.
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors.
Blog about the options for getting data from Oracle into Kafka
A conference talk covering the options for getting data from Oracle into Kafka
Kafka 101 - introducing the concepts, the APIs, and the ecosystem
Try out the demo for yourself - all you need is Docker and Docker Compose.
Free eBooks to download, including Kafka: The Definitive Guide.
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
Huge list of connectors for Kafka Connect
A fun blog showing what you can do with ksqlDB and Kafka
Learn all about ksqlDB in this 45 minute talk & live demo
Learn all about Kafka Connect (including the connectors available with ksqlDB)
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.
Here’s what was said about this presentation on Twitter.