Building a system based on events triggers many questions. The two biggies target the essence of what you want to do:

• What is an event?

• What does an event look like?

These two questions have to be answered before we even start designing anything. Different use-cases have led to different interpretations of “the event” which makes our task even more difficult. After all, it feels a bit like cheating having to answer “it depends” to these, doesn’t it?

The event-driven mindset

So let’s first start with some context. We need to understand the difference between a system based on events and an event-driven system. An event-driven system follows a mindset based around the following statements:

• Reality is event-driven by nature

• Events cause events

• Events are immutable

• State is a consequence

• Time is relative

These statements are the driving force behind any decision made for an event-driven system, so it is important to understand what is meant by each.

Reality is event-driven by nature

We use computer systems to keep track of the reality surrounding us and we have been doing so for years by keeping track of the current state of things. This state is easy to reason about because it represents what we see, but it has its limitations. For example, there is no way to deduct what happened to reach this state. For that, we need to take an additional aspect into account; time. We have tried to solve this by capturing state at moments in time giving us a high level understanding of what happened by comparing one snapshot to the next, with mixed success. For example, changing a customer’s address might be because they moved or because of a typo. We are lacking the context to understand what really happened.

So is snapshotting state at regular intervals really the right thing to do? Wouldn’t it be easier to just … capture what happened? After all, isn’t that what happens all around us? Your alarm going off in the morning, a call arriving on your phone, a product being bought or even a baby being born; all are examples of events that happen and that change the world around us. The same reasoning holds true for our organizations where events are causing changes to the reality in which an organization operates; a product being sold, a customer being registered, a store being opened … 

Keeping track of events gives us some very nice benefits. Since every interaction we have with our environment leads to events, we have the ability to understand what our reality was at a given moment in time by replaying these events up to that moment at nanosecond accuracy. 

We also have the ability to register additional information as part of the event, to better understand the context in which the event occurred. No more guessing of what actually led up to where we are now, the event contains the information.

Events cause events

Events seldom stand on their own but serve as triggers for other events. Someone or something will react to an event and by doing so will cause new events to happen. For example, someone ringing your doorbell will cause you to act and open the door, causing the door to be opened:

DoorbellRang —[ you ]--> DoorOpened

In this case you are acting as a processor for an event by acting on it. A processor determines what happens by interpreting the event and sending out new ones. In the situation above, only a single event is being produced by the processor (you), but there can be many more:

DoorbellRang —[ you ]--> DoorOpened               [ you ]--> WordsSpoken(“Coming!”)

It might even be you and your wife both serve as processors of the same event:

               [ your wife ]--> WordsSpoken(“Who is it?”) DoorbellRang —[ you ]--> DoorOpened               [ you ]--> WordsSpoken(“Coming!”)

Last but not least, a processor is not obligated to produce new events:

DoorbellRang —[ you, choosing to ignore ]

Since a processor can choose to produce one or more events, we can create chains of processors linked to each other by events:

DoorbellRang —[ wife ]-> Spoken(“Did you get that?”) –[ you ]-> DoorOpened

Even complex interactions like buying goods online can be modelled as a sequence of events:

• ProductSelected

• ProductOrdered

• OrderPaid

• OrderPicked

• OrderShipped

So interaction leads to events, acted on by a processor, leading to events, acted on by a processor, leading to … 

Events are immutable

This statement is by far the most challenging one when designing event-driven systems. Our current systems for tracking state are mutable; every record can be changed at will, overwriting the previous one.

But this is not how reality works. When you say something to another person, there is no way of taking that back. The reality is that you spoke those words. The only thing you can do now is compensate for what you said. Once an event happens, there is no way of taking it back or pretending it didn’t happen.

This means mistakes are events too and have an impact on our reality for a period of time. If we want to create an accurate recollection of what happened in the past, we need to take these into account as well. Even a correction being applied to compensate for a specific event is an event by itself, with its own context.

State is a consequence

When we look around us, most of the time we agree on what a car is, what a dog is, or what a book is. These concepts are easy to reason about because they are obvious and tangible.

Within organizations however, we like to think in less tangible concepts like ‘Product’, ‘Customer’ or ‘Order’. Contrary to the real-world concepts above, these ones are much more up for interpretation. While everyone can agree that these concepts exist, every department will have a different idea of what a customer means for them. It can mean something different depending whether you are talking to the sales department or logistics department. Even within the same department, different use-cases could require different representations of what a product means. And that’s fine.

The state we utilize to implement a use-case is determined by the use-case itself. Once we determine what our use-case is all about, we need to understand how it integrates with the rest of the world. For that we need to ask ourselves which events we are interested in and how they affect our knowledge. A service for managing product stock for example will need to keep track of new products being delivered, others being sold as well as the people available to retrieve the stock.

So state is the result of interpreting events and each processor can decide what it considers as being part of its state. A processor responsible for calculating sales statistics might want to keep track of the number of sold items per product. Another processor responsible for hosting the products API might want to keep the last aggregates state per product. Both are relying on their state to perform their duties but their actual state differs significantly.

Time is relative

Everything we observe is in the present. As far as we can tell, there is only one timeline and we have little to no control over it. We can write down what happens, but then another question pops to mind; which timestamp should we associate with the event? After all, the moment we receive an event is not the same as the moment the event actually happened. Both timestamps can be minutes, hours or even days apart.

With events being immutable, once an event is registered, there is no way of updating it. This means we will need to register our events in the order we receive them. Time in that sense is relative to the observer; until an observer observes the event, it hasn’t happened yet. 

Other timestamps are kept as context on the event itself. This allows us to interpret our events and create accurate state-in-time, taking late arriving events into account.

The Event

So with all of this background information, we can start to formulate what an event actually is:

An event is a data-structure capturing something that happened as a result of an action. The action itself can occur as part of the system we are observing (a processor interpreting the event) or external to the system (an external action like someone ringing the doorbell).

While that sums up what we think an event is, it does not explain what an event looks like. We already know that an event contains the context in which something happened as well as what exactly happened. How that is modelled is up for interpretation, but let’s make a gentle suggestion.

An event has a descriptive name, something like “DoorbellRang”, “ProductPicked” or “CustomerEntered”. Beware that these names are all past tense, a common practice when naming events.

Apart from the name, there is some metadata to keep track of:

• The actor or subject who instigated the event

• The origin of the event; the system that created the event

Next there are the time related properties of the event:

• The event timestamp indicating when the event occurred

• The processing timestamp indicating when the event was processed by the system

• Other timestamps, probably functional in nature can be added here as well

From here on, things depend more on the type of event. For example, a “ProductPicked” event will include a reference to the product which was picked and who picked it. A “CustomerEntered” event on the other hand will include a reference to the customer and the location where he entered. The important thing to remember is that enough context needs to be included within the event to understand what happened, without adding too much context. It is a tricky balance but I haven’t found a golden rule for this yet (suggestions are welcome).

Making Things Real

Apache Kafka provides an excellent platform for building out your event-driven vision. Terminology can be confusing though; Kafka has the concept of a message, Kafka streams even talks about a record. But a message is not necessarily an event, right? Let’s get that confusion sorted out, shall we?

The atomic unit within Kafka is a message. It holds our data as well as some metadata used to let Kafka do its magic. These messages are stored on the partitions of a topic; Each time we publish a message, it will be added to the end of one of these topic partitions. The actual partition is determined using the message key. A topic partition to that effect is a stream of messages, ordered in the way we produced them.

It is important to note that messages on a topic are not ordered. Only within a single partition, messages are ordered by their offset (not by timestamp). The combination of topic name, partition and offset uniquely identifies a message.

The resulting stream of messages can either be a stream of events - like “DoorbellRang” or “RocketLaunched” - or a stream of state like “User” or “Product”. We call the former an event stream and the latter a changelog. A changelog can be compacted to only retain the last state for a given key. This proves to be a very powerful tool in our event-driven toolbox.

Conclusion

I hate getting an “it depends” answer, which is why I deliberately took an opinionated approach to what an event-driven system should look like. I believe we have reached a point in time where technology allows us to capture reality in a different way from what we are used to; focus on behavior instead of consequence. At the same time, I can’t seem to get rid of the idea that the interpretation of behavior is relative to the observer. This drives the interpretation of events into state.

As always, reach out with ideas and thoughts, digitally or in person. I love talking about this stuff.