Companies new and old are all recognising the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka streaming platform. With Kafka, developers can integrate multiple sources and systems, which enables low latency analytics, event-driven architectures and the population of multiple downstream systems.
In this talk, we’ll look at one of the most common integration requirements - connecting databases to Kafka. We’ll consider the concept that all data is a stream of events, including that residing within a database. We’ll look at why we’d want to stream data from a database, including driving applications in Kafka from events upstream. We’ll discuss the different methods for connecting databases to Kafka, and the pros and cons of each. Techniques including Change-Data-Capture (CDC) and Kafka Connect will be covered, as well as an exploration of the power of ksqlDB for performing transformations such as joins on the inbound data.
Attendees of this talk will learn:
- why events, not just state, matter
- the difference between log-based CDC and query-based CDC
- how to chose which CDC approach to use
Resources
-
☁️Confluent Cloud
Fully Managed Apache Kafka, Schema Registry, ksqlDB, and Connectors. Use promo code 60DEVADV to get an additional $60 of free Confluent Cloud usage. Be sure to activate it by Dec. 31, 2021, and to use it within 90 days after activation. Any unused promo value on the expiration date will be forfeited and there are a limited number of codes available, so don’t miss out!
-
✍️ Streaming data from Oracle into Kafka
Blog about the options for getting data from Oracle into Kafka
-
🎥 Integrating Oracle and Kafka
A conference talk covering the options for getting data from Oracle into Kafka
-
📌Kafka as a Platform: the Ecosystem from the Ground Up
Kafka 101 - introducing the concepts, the APIs, and the ecosystem
- 📌 Kafka as a Platform: the Ecosystem from the Ground Up
-
👾Demo code
Try out the demo for yourself - all you need is Docker and Docker Compose.
- 🎥 Recording
-
📚Free eBooks
Free eBooks to download , including Kafka: The Definitive Guide.
-
ℹ️ Confluent Developer
Tutorials, videos, blogs, podcasts, and more - all for developers working with Apache Kafka and Confluent Platform
- 🎥 Kafka Connect tutorials on YouTube
-
🧩 Confluent Hub
Huge list of connectors for Kafka Connect
-
👾 Building a Telegram Bot Powered by Apache Kafka and ksqlDB
A fun blog showing what you can do with ksqlDB and Kafka
- 💬 Confluent Community Slack group
- 📌 Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
-
📌 Introduction to ksqlDB
Learn all about ksqlDB in this 45 minute talk & live demo
-
📌From Zero to Hero with Kafka Connect
Learn all about Kafka Connect (including the connectors available with ksqlDB)
- 📌The Changing Face of ETL: Event-Driven Architectures for Data Engineers
-
🚂On Track with Apache Kafka: Building a Streaming Platform solution with Rail Data
Apache Kafka and Confluent Platform in Action! Using live streams of rail movement data in all sorts of useful ways for analysis and applications.