The Changing Face of ETL: Event-Driven Architectures for Data Engineers

[paint a better picture - technology] zoom Photo by NASA on Unsplash • <city view from above> • It’s the same thing that happened. It’s the same piece of data. we just want different things from it • apps -> respond to something happening (an order was placed!) • analytics -> tell us what happened (how many orders were placed?) • Historically, technology was such you had to. OLTP/OLAP was a compromise; you can have quick data in or quick data out : choose one. • Batch ETL was the inevitable sticking plaster on top of that. Whilst you only had a few systems inhouse from which to get data and one to write it to this didn’t matter. But that’s no longer the case • This isn’t about a compromise, about crowbaring everything into a new shiny technology that I’ve found • this is about adopting a unified platform that enables BOTH apps and analytics to be better (lower latency, more flexible architecture, more scalable) • this is all enabled through events, implemented on a highly scalable, distributed technology with huge integration capabilities and universally-supported API @rmoff | #ConfluentVUG | @confluentinc

Further reading / watching • Kafka as a Platform: the Ecosystem from the Ground Up http://rmoff.dev/youtube • https://rmoff.dev/kafka101 • Apache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline! • https://rmoff.dev/ljc-kafka-01 • From Zero to Hero with Kafka Connect • https://rmoff.dev/ljc-kafka-02 • Introduction to ksqlDB • https://rmoff.dev/ljc-kafka-03 • Integrating Oracle and Kafka • https://rmoff.dev/oracle-and-kafka • The Changing Face of ETL: Event-Driven Architectures for Data Engineers • https://rmoff.dev/oredev19-changing-face-of-etl • 🚂On Track with Apache Kafka: Building a Streaming Platform solution with Rail Data • https://rmoff.dev/oredev19-on-track-with-kafka @rmoff | #ConfluentVUG | @confluentinc