🚒 All at Sea with Streams - Using Kafka to Detect Patterns in the Behaviour of Ships

A presentation at Kafka Summit Americas by Robin Moffatt

This is the recording of my Kafka Summit 2021 talk about doing interesting things with real-time streams of AIS data. It covers both β€œstandard” streaming ETL stuff to build fancy dashboards, as well as using stream processing to identify in real-time ships that may be engaged in transshipping.

If you’re interested in the topic but prefer the written word to video, the talk is based on these two blogs.


The great thing about streams of real-time events is that they can be used to spot behaviours as they happen and respond to them as needed. Instead of waiting until tomorrow to find out what happened yesterday, we can act on things straight away.

This talk will show a real-life example of one particular pattern that it’s useful to detectβ€”ships engaged in potentially suspicious behaviour at sea. Transhipping is often used for legitimate purposes to optimise efficiencies but can also be used for nefarious purposes such as illegal fishing.

By capturing streams of maritime AIS data in real-time into Kafka and processing it with ksqlDB, it’s possible to detect the kind of characteristics that could indicate behaviour of interest, such as ships moving slowly at close proximity for a length of time.

I’ll demonstrate how the data was ingested from a raw TCP feed, unified with reference data from CSV files, and then processed to spot patterns with the resulting real-time stream of matches written to a new Kafka topic for validation and analysis.

Video

Resources

The following resources were mentioned during the presentation or are useful additional information.

Buzz and feedback

Here’s what was said about this presentation on Twitter.