🚂 On Track with Apache Kafka: Building a Streaming ETL solution with Rail Data

A presentation at Kafka Summit Europe 2021 by Robin Moffatt

As data engineers, we frequently need to build scalable systems working with data from a variety of sources and with various ingest rates, sizes, and formats. This talk takes an in-depth look at how Apache Kafka can be used to provide a common platform on which to build data infrastructure driving both real-time analytics as well as event-driven applications.

Using a public feed of railway data it will show how to ingest data from message queues such as ActiveMQ with Kafka Connect, as well as from static sources such as S3 and REST endpoints. We’ll then see how to use stream processing to transform the data into a form useful for streaming to analytics in tools such as Elasticsearch and Neo4j. The same data will be used to drive a real-time notifications service through Telegram.

If you’re wondering how to build your next scalable data platform, how to reconcile the impedance mismatch between stream and batch, and how to wrangle streams of data—this talk is for you!

Resources

The following resources were mentioned during the presentation or are useful additional information.

  • 👾Code

    Try out the code shown in this talk using Docker

  • Confluent Cloud

    Confluent Cloud provides fully managed Apache Kafka, connectors, Schema Registry, and ksqlDB. Try it out and use code RMOFF200 for money off your bill (T&C)

  • ✍️ Blog

    This blog is a write up of the material covered in this talk

Buzz and feedback

Here’s what was said about this presentation on Twitter.