Event-Driven Architecture

The article is translated from the source

Simple definition

Event-driven Architecture (EDA) is a software architecture in which we focus on creating and using events .

An event will represent a purposeful action . Usually, events can correspond to the creation or change of state of several entities. For example, when we buy a product on e-commerce sites, it is also an event. Customers who submit a review about a product receive it as an event.

Event never happened

One of the characteristics of events is that they do not communicate with the objects that use them. Event “just happens” only. However that is one factor that makes the event “very powerful” – the fact that the event will turn into a record contains information related to the event that just happened, as well as emitters and most importantly, it’s completely separate for handlers . The truth is that event producers are unaware of the existence of event consumers .

A record only contains information related to the event that just happened . As in the above example of buying things on an e-commerce site, the purchase event can be described with a JSON object as follows:

Note that: in practice records and events are two terms that can be used interchangeably. For example, the term “event” is used to refer to the “record” of that event.

Our application generates an event, but it does not know who will handle that event, when , where and how .

The event producer itself only ensures to provide enough information for the event consumer to process the event.

Therefore other information, such as information about stock count of the product or the product’s địa chỉ gửi đến will not be of interest to the event producer here.

Channeling Events

If event producers and event consumers don’t know each other, how will they communicate with each other?

Events are usually stored in a place called a log (Sometimes the term ledger may be used). Logs is a low-level append-only data structure where the producer stores events so that consumers can later access and retrieve the events. All log operations are performed by brokers – a middleware located between producers and consumers. When an event is published, all objects can use that event.

When implementing event-driven systems, we often use the term stream to refer to one or more logs. Log is a “physical” term as it uses files for storage, and a stream is logical when it is a sequence of events . Apache Kafka is a pretty versatile streaming platform – streams are referred to by the two terms topics & partitions .

The relationship between producers, consumers and streams can be described as the following model

• Events happen at different times
• Events exist as records . Events and records are two technically different terms. Event refers to an action or something happening. Record contains descriptive information about the event. Here we often use the term event to refer to its record.
• Producers detect events by publishing the records corresponding to them to the stream .
• Stream stores strings of records . Streams can be created based on disk-based logs or can also be database tables , …
• Brokers manage access to streams , track reads and write operations, process consumer state, and manage tasks on streams. For example, the broker can truncate the contents on the stream when the stream becomes overloaded with the current amount of records.
• Consumer reads from streams and processes records . Event handling can lead to side-effects such as: inserting an entity into the database or refactoring the state of a remote entity, …
• The roles of consumers and producers may overlap . For example, a consumer’s handling of one event can lead to the generation of another event.

Decoupling through asynchrony and generalization

Returning to the starting point, Why does EDA help the components in the system reduce their dependence on each other?

A simple definition of dependency is the degree to which one component affects other components .

A typical example is when a service calls another REST API and waits for the results to return from that API, if in case the other REST API has problems and cannot return results, the service calling the API will have to stop any processing behind that API call.

We say that components are tightly coupled if they are tightly dependent on each other and loosely coupled otherwise.

1. Again, events only happen . The event procder is unaware of the existence of other components. So the producer will still work even if the consumer is not, so the broker itself will periodically buffer the event records without any “push” from the producer.
2. Storing event records in the broker eliminates the definition of time. Specifically, the producer can publish the event at time T1 , the consumer will “digest” that event at time T2 , basically T1 and T2 will be a few milliseconds apart (if everything works fine) or a few hours ( if some consumers have problems).

EDA is not a “silver bullet” or “an elixir” that separates components from each other completely. In fact, when producer and consumer do not depend on each other anymore, they depend on the broker , leading to a new point of failure that is the broker, so the broker must ensure high performance as well as good fault tolerance .

Types of event handlers

They can be divided into 3 main groups as follows:

Discrete event processing

For example when posting a post on social networks. One of the characteristics of discrete event handling is that the presence of an event is unrelated to other events and can be handled independently.

Event stream processing

Process events in a threaded manner, taking into account the order of events, as the current event is related to the past event. A typical example is events that change to a business entity. These events will be processed in a certain order, then will save the business entity data in the database. Consumers also need to avoid concurrent changes to the same database record, causing data inconsistency.

Complex event processing

Complex event processing (CEP) identifies and renders complex event patterns based on a simple sequence of events. Let’s take the example of CEP for monitoring temperature and smoke from sensors to detect if a fire has occurred. The temperature data at one point in time may not mean too much, but with a “temperature cluster” with the rate of change of temperature, it is completely possible to predict whether a fire is happening. or not.

When to use EDA (Event Driven Architecture)

There are a few typical use-cases as follows:

1. Opaque consumer ecosystem : when producers know almost nothing about consumers.
2. High fan-out : scenario where an event can be handled by many different consumers.
3. Complex pattern matching : where events can be “chained” to produce more complex events.
4. Command-query responsibility segregation : CQRS is a pattern that separates read & write operations to the data store. The implementation of CQRS will make the system more scalable as well as increase the reliability of the system, in return, the data consistency may not be as complete as expected. This pattern often goes hand in hand with EDA.

Benefits of EDA

1. Buffering & fault-tolerance : Events can be handled at different rates depending on the application and the producer doesn’t have to “slow himself down” for the consumer to catch up.
2. Decoupling producer & consumer : reduce the interdependence between producer and consumer. From there we can easily add or remove consumers and producers into the system.
3. Easy to scale : we can easily divide events into different substreams and process them in parallel as well as add consumers to be able to handle a large number of events in a timely manner.

Disadvantages of EDA

1. Limitations on asynchronous handling : EDA is a pretty strong pattern in reducing interdependencies between components, but it has more or less limitations with asynchronous processing.
2. New complications arise : with the traditional client-server, request-response model, we only need 2 main factors , while with EDA we also need a broker to handle the interoperability. interaction between consumer and producer .
3. Failure masking : with interdependent systems, when one component fails, it affects other components because these other components will also know that there is a component that failed (this is perfect It’s not a good thing when one component affects the entire system), but with EDA when components are not dependent on each other, the failure of a component may not be known to other components. This inadvertently causes the error to not be known and handled (even though it doesn’t affect other components). So with EDA we need a logging and monitoring mechanism for each event-driven component , but this will increase the complexity of the system.

Things need to notice

EDA is not 100% perfect, like other tools, it also has its own downsides. Here are the disadvantages of EDA that developers and architects should pay attention to when designing and implementing an event-driven system.

1. Convoluted choreography : reducing the interdependence of components can make the architecture of the system like the Rube Goldberg machine where the business logic will be implemented by a series of side-effects “disguised” under the label. “event”: one component throws an event, triggers another component’s handling, another component fires an event and fires another component, and so on… This way of interacting between components is quick become difficult to understand and control.
2. Camouflage commands as event : event is merely describing something that just happened, it doesn’t talk about how the event or something should be handled. Or to put it another way, the command – directive is a direct instruction to a particular component. Since commands & events are short messages, it is easy to misunderstand that command is event.
3. It is difficult to predict what consumers want : the event should contain only the necessary information for how it is handled. But in fact we can add other “redundant” information to the event.

summary

Microservices architecture is a piece of the puzzle for building a more maintainable, extensible system. Microservices are really great when it comes to separating components from each other but it also has a lot of outstanding problems. Breaking the system down from “monolith” to “microservices” can get us right back to where we started, which is the problem of “distributed monoliths” – distributed monoliths.

To complete the unfinished puzzle and solve the problem of interdependencies between components, we look to event-driven architecture.

EDA is a good tool to separate components in a system by modeling their interactions using the concepts producers , consumers , events , streams . Event describes something that has just happened, it is generated and handled asynchronously by components that do not know anything about each other. EDA allows components to work independently of each other. EDA itself isn’t perfect, but it offers more benefits than problems. Therefore EDA can be seen as an integral component of any successful microservices system.