Posts tagged event-centric

Event sourced process managers

In the comments to one of my previous posts Jarek pointed out that the set of concepts I use (based on excellent work by Rinat Abdullin) reminds him, rightfully, of Boundary-Control-Entity pattern. A much better summary of the pattern (and related patterns) than I would be ever capable of writing has already been provided by Uncle Bob. TL;DR: put technology-agnostic things in the centre and technology-aware on the outside, use Dependency Inversion Principle to wire things up.

What I want to clear is the place of Controls component in my model. Intuitively is seems like a Command Handler plays this role but that turns out to be impossible for a couple of reasons. A Command Handler is only responsible for translating commands (messages) to method calls on Aggregates. I can’t even touch more than one aggregate when processing a command (due to scalability requirements). Last but not least, it is stateless. So if not Command Handler than what?

Meet Process Manager (a.k.a. Saga). A Process Manager coordinates long running processes by means of message exchange. In some technology stacks (e.g. NServiceBus) Sagas are a built-in mechanism that programmers can use out-of-the-box. In my custom-built stack, however, Saga is just another Aggregate. There is absolutely no difference in technology. The difference is in behaviour. While normal Aggregates are usually passive (they execute the business logic and emit events for state changes), a Process Manager is active in a sense that it usually expects a response for the events it emits. Process Manager-as-an-aggregate needs to accompanied by a bunch of helper components to perform it’s job, namely:

  • Receptor that reacts on outside events and turns them into commands understood by a Process Manager
  • Command Handler that translates those commands to method calls
  • Gateway that translates Process Manager’s state changes (events) into commands sent to other Aggregates

And there we arrived at Jarek’s quesion: why do we need Gateways at all? We could as well use a Receptor on receiving Aggregate’s side, right? Their raison d’être is subtle. In case of a Process Manager, the responsibility for generating a command (an order if you will) is on PM’s side and the receiving Aggregate can’t ignore it. If we used a Receptor (logically belonging to this other Aggregate), it would mean that PM’s event can be safely ignored.

VN:F [1.9.22_1171]
Rating: 4.3/5 (3 votes cast)

Modelling accounts, event driven part 2

In the previous post I discussed the various approaches for modelling account operations. The outcome of this discussion was the insight that, when using event sourcing, aggregates are the derived concept with the primary concept being the domain invariants. I used money transfer as an example. As Maksim pointed out, in such scenario usually there is another invariant which was missing in my model

The balance of an account should not be less than zero

It is clear that this invariant cannot be satisfied by our Transaction aggregate. We must re-introduce the Account as an aggregate. By the way, what do you think about the new requirement? It is pretty specific, isn’t it? This might be a sign that we are missing a more general concept in the model. How about this

Account can reject participation in a transaction

This is called in Domain-Driven Design a refactoring towards deeper insight. Because of on it we can implement now all kinds of accounts, e.g. credit-only accounts, debit-only accounts (useful on system boundary), credit card accounts (accounts which balance cannot be positive) and many more and we have much better understanding of the domain concepts.

Let’s try to sketch the event-command chain for a successful transaction that would honor both consistency requirements

  1. Transfer command creates new instance of a Transaction aggregate. As a result, a Prepared event is emitted.
  2. Account receptor reacts on the Prepared event emitting two commands, one for each account involved in a transaction: PrepareDebit and PrepareCredit
  3. Destination Account processes PrepareCredit command based on its internal state and rules. In case of normal (non-negative balance) account, it does nothing. As a result, CreditPrepared event is emitted.
  4. Source Account processes PrepareDebit command based on its internal state and rules. In case of normal (non-negative balance) account, it decrements the available funds value. As a result, DebitPrepared event is emitted. Alternatively, DebitPreparationFailed event can be emitted if account rejects participating in the transaction.
  5. Transaction receptor reacts on CreditPrepared and DebitPrepared events by emitting appropriate notification commands.
  6. Transaction aggregate processes notification. If any account rejected the operation, it immediately emits TransactionCancelled event. Accounts which successfully prepared themselves for this transaction should react on this event by undoing any changes. When any account completes preparation, appropriate Confirmed event is emitted to update Transaction state. If both accounts confirmed, a pair of Debited/Credited events are emitted in addition to the Confirmed event. All three events are emitted in one transaction ensuring that funds transfer is done atomically.
  7. Account receptor reacts on Debited and Credited events by emitting appropriate notification commands.
  8. Destination Account processes the notification command based on its internal state and rules. In case of normal (non-negative balance) account, it increments both available funds and balance values.
  9. Source Account processes the notification command based on its internal state and rules. In case of normal (non-negative balance) account, it decrements the value of balance.

The double entry accounting rule is satisfied by Transaction aggregate via emitting both Credited and Debited events in one transaction while veto requirement (a generalization of non-negative balance invariant) is satisfied by using a variation two-phase commit protocol. Despite the fact that the whole interaction is asynchronous (events causing commands to be sent to other aggregates) the externally visible state of the system is always consistent: either there are no entries or there is a pair of Credited and Debited entries. The consequence of eventual consistency is the fact that, in theory, if the settlement phase is lagging for some reason the available funds value is lower than it could be. It means that there might be some false positive rejections which is of course not a desirable thing. But this shows how important monitoring is in asynchronous systems.

VN:F [1.9.22_1171]
Rating: 4.5/5 (2 votes cast)

Modelling accounts, event driven

I’d like to share results of my research on modelling an account. It is based mainly on this discussion on CQRS mailing list and specifically by Greg’s example. I’ll start with a slightly naïve model.

Account as an aggregate

Let’s model Account as an aggregate. What interface would an Account need? How about Debit and Credit? Here’s a sketch of how a command handler would look like for a TransferFunds command


var uow = new UnitOfWork();
var from = accountRepository.GetAccountForNumber(fromNumber);
var to = accountRepository.GetAccountForNumber(toNumber);

from.Debit(amount);
to.Credit(amount);

uow.Complete();

If you don’t like this approach you are right. The problem with this model is it requires two aggregates to be modified in one transactions. This violates the good practice of treating aggregate boundaries as transaction and consistency boundaries. Also, many event stores would not allow even allow you to create such transaction in the fist place. So, what can we improve?

Eventual consistency

We can take advantage of the idea of eventual consistency. Let’s refactor the command handler code:

var from = accountRepository.getAccountForNumber(fromNumber);
from.TransferTo(amount, toAccount);
accountRepository.PersistChanges(from);

The effect of this code is a AccountDebited event published by source account aggregate. This event then needs to be dispatched to a handler that would look like this:

var to = accountRepository.GetAccountForNumber(evnt.DestinationAccount);
to.ReceiveFunds(evnt.Amount);
accountRepository.PersistChanges(to);

Can you spot a problem here? While eventual consistency is generally a great tool, as all great tools it can be easily misused. The code above ensures that eventually, given infinite time, the transaction will be balanced. Can we wait so long? Usually it will happen very quickly but sometimes, under heavy load, it can take a while. Can we tolerate this? The answer, as always, is, it depends. We need to go back to the business problem we are solving and ask ourselves a question if we are not violating aggregate’s invariants by introducing eventual consistency.

In our example the invariant states that at any point in time credit and debit operations must balance (sum to zero). Clearly, after the command is processed but before the event processing happens, this rule is violated. What can we do with it?

Event driven modelling

Instead of trying to identify aggregates, let’s start with commands and events. Here’s what happens in the system when we want to transfer money:

The domain expert tells us that the rule we must comply with states that AccountDebited and AccountCredited events always happen together and they always balance to zero. What aggregate can we create here so that the rule is honoured? Clearly not Account. The aggregate we are looking for is Transaction. The interaction can be pictured as

The new command handler code would look like this:

var tx = new Transaction();
tx.Post(amount, fromAccount, toAccount);
transactionRepository.Store(tx);

and here’s the brand new Transaction aggregate:

public void Post(decimal amount, string fromAccount, string toAccount)
{
   this.Apply(new AccountDebited(amount, fromAccount));
   this.Apply(new AccountCredited(amount, toAccount));
}

Now the following question arises: if account is not an aggregate, what is it? Clearly, end users want to see accounts, balances and history records. The answer is provided by CQRS approach. Accounts exist only on the read side of the system. There should probably be a couple of them, one for each screen.

Takeaways

The takeaway from this a bit lengthy blog post is, if you are using Event Sourcing approach you should not start by identifying the aggregates by rather by identifying sequences of commands and events. Only then you can ask business expert what are the consistency boundaries in particular sequence. The result will be the definition of an aggregate.

VN:F [1.9.22_1171]
Rating: 5.0/5 (4 votes cast)

Event sourcing the past and today

It’s been quite some time since I first read about event sourcing on Mark Nijhof‘s blog (Mark has moved his blog several times since then). My understanding of the subject has been evolving for few years. Same with other people in the community. Greg Young, the father of the event sourcing and CQRS ideas, might tell you that the biggest mistake community did during these years was building frameworks. To some extent I agree but on the other hand I think that bashing all frameworks is like throwing out the baby with the bathwater.

Problems with frameworks

The biggest problem with CQRS frameworks of the past (and I know this because I contributed to one of them) was in my opinion the fact that we focused on wrong things. We, as a community, didn’t understand the whole concept well enough. Sure, there were always people who did and tried to warn the others but we were not listening because we were too busy coding The Most Beautiful And Complete Implementation Of Aggregate. Standard CQRS framework from the past looked like this:

The most important concept is the aggregate. This is logical, right? It is the place where business logic lives so it must be the most important concept. Aggregates produce events which are also quite important because they are used to maintain the state of the aggregate. Events themselves are stored in an event store. In those early days there was no mature ready-made event store implementation so a framework had to include one. Event versioning and commands manage to attract some attention but not a lot. In case of commands, this attention was mostly focused not on commands themselves but rather on how we handle them. Concepts like messages, integration, duplication/transactions and read models were in most cases out of scope.

One can summarize this approach as storing business object state by serializing state-mutating method calls. We surely missed the whole point.

Value of frameworks

This is the baby and bathwater part. Although as I wrote, we missed the whole point, I still see value in what we created. Frameworks do lower the entry barrier for people who want to try a new approach. NCQRS, a framework I contributed to (kudos to Pieter for building it) did help some people to grasp the concept of event sourcing. Frameworks also (at least the good ones) remove the need of writing boilerplate code and let us focus on the business problem we try to solve. In case of event sourcing and CQRS the problem was that we initially misidentified the areas where framework would have most value. Frameworks did provide ready-made solutions for things that most people wanted to be custom-made while failing to help in tough and/or generic problems such as:

  • read models
  • integration
  • transactions and coordination

A better approach

While I won’t say that I know The Best Approach To CQRS now, I would argue that I at least know one that is better than the ones we used in the past. The diagram of today’s framework should look like this:

The whole thing is more balanced. No longer a single concept dominates the whole picture. An event store has been moved out of scope of the framework as we have at least two production-ready implementations provided by Jonathan Oliver and Greg Young. The former is an abstraction layer built on top of other data stores (e.g. RDBMS) while the latter is an event store build from the ground up.

The key concept is message — an atomic piece of information we can package and send between processes, machines and systems. Not all messages are created equal though. A smart framework would distinguish commands (messages that tell the recipient to perform some action) from events (ones that inform everybody that something has happened). Handling duplication of messages is key to build a robust message-driven system. A good framework should provide a solution for this.

Event versioning should get some attention also as it is critical for maintainability of the system in the long run. Another interesting bit is integration — how do we communicate with other systems? The general answer is we allow them to read our event store. This requires however that we have durable subscription feature somewhere in our stack, either within the store or the framework.

If we are building a CQRS framework, there need to be a decent support for the read side also. Problems like automatic view rebuilding when code changes are the ones that a framework should be able to take care of.

The last thing in the picture in an aggregate. Noticed the different from the previous diagram? In a modern approach is has a servant role. In modelling phase aggregates are a corollary of command-event chains. We simply group related command-event pairs into aggregates according to their consistency requirements. In the implementation phase aggregates provide deduplication capabilities which are essential to most business logic.

And beyond

All the things I’ve just mentioned can be built in less than a month by a single person so the effort can be easily accommodated by even a medium size project. So far not much value added, at least for people who know messaging and event sourcing sourcing inside-out. True, but the thing I described, if done right, would be a very solid platform that we can use to build value added services on top of it. I am thinking of things like:

  • enforcing SOA/EDA architecture using best practices preached by Udi Dahan
  • being able tell the software to upgrade service A from version X to Y and having all dependencies upgrade automatically (and one-by-one to limit any down time)
  • monitoring any aspect of both the infrastructure and the user code running on top of it
  • and many, many others
VN:F [1.9.22_1171]
Rating: 5.0/5 (6 votes cast)

Really Simple Architecture components

In the previous post I’ve mentioned the idea of a Really Simple Architecture. This time I’d like to discuss the components of RSA in more detail. Because the whole concept evolves in my mind, by no means treat as any form of reference architecture. At least not yet ;-) This is just a thought experiment for now.

Components

First, I must admit I borrowed most of the ideas from Rinat’s blog.

Receptor

Receptors come in two flavours: internal and external. Both look and work exactly the same.

The difference is the source of the events they receive. An external receptor processes events coming from other service (or services) while an internal receptor reads from its own event store. External event receptors are the only way services can exchange data: one service publishes events that the other service reads and acts upon. Internal receptors can be used to integrate multiple command handlers into a pipeline: they turn events published by one handler into a commands for another.

Command handler

Unfortunately I could not find a better name for this thing. At least it says exactly what it does: process commands and publish events as a result. In general command handlers can do anything to handle an event but there are two specialization that are especially worth mentioning.

A stateless handler executes logic that does not depend on any persistent state. It just transforms commands to events using some kind of an algorithm.

An aggregate-based handler uses Domain-Driven Design’s Aggregate pattern to structure the logic. An incoming command is mapped to a method call on aggregate object. The method results in generating on or more events. These events are then persisted in the event store in one transaction. Aggregate’s state is loaded into memory each time a command is processed so the processing logic can be based on past events.

View projection

This component updates so called view model (or read model) based on processed events.

As with receptors, view projections can be internal (as on picture above) or external (not shown).

Gateway

They too come in two versions. First transforms commands into events using some external system.

When coupled with an internal receptor, this type of gateway can be used to send commands to an external system. The flow would be following:

  • a command handler publishes SomethingRequested event
  • an internal receptor transforms this event to RequestSomething command
  • a gateway receives the command, sends a message to an external system (e.g. via SOAP), receives a response and publishes the result as an SomethingDone event
  • an internal receptor transforms this event to HandleSomethingDone command targeted to the command handler that initiated the communication

In short, this type of gateway can be used when external party we are dealing with can reject the message we are sending. Otherwise, the second version of gateway, which is easier to plug into the system, can be used

This type gateway combines two features. It processes an event stream and publishes the data to external systems and can also receive data from this system and inject it to our service in form of commands.

The above list is not complete nor final. It evolves constantly while I try to implement more and more use cases using my RSA. There are, however, much more stable concepts in the architecture. Let’s call them primitives. There are only four of them.

Primitives

Command

A command is a message that instructs somebody to perform some action. The architecture acknowledges that commands do get duplicated from time to time. Command handlers are expected to deal with this fact of life. To make it possible, a command always carries an ID.

Event

An event is a message that denotes something that has happened in the past. An event may (but is not required to) point to a command that caused it. Events are organized in streams that have business meaning e.g. responses from external party X or events related to service Y. Event streams may or may not guarantee to be free of duplicates. Allowing duplication is a property of an event stream that is dependent on kind of a command handler(s) that emits events to the stream.

Event Handler

An event handler is a thing that processes events from one or more logical event stores. In a result it can either emit a command or update a view. Event handlers can read from views in order to provide context for the processing. An event handler is responsible for assigning proper IDs to generated commands. In case of an event stream that guarantee no event duplication (most published event stream are like that), the easiest way to assign an ID to command is to use event’s unique ID. Otherwise, a business-level ID has to be used.

Command Handler

A command handler is a thing that processes commands from a queue. In a result it emits events to some stream. Event handlers can read from views in order to provide context for the processing. Command handler might provide no duplication guarantee for the event stream(s) it generates. Deduplication can be achieved by loading events generated in the past and checking if any one of them is related to a command being processed. If so, a command is considered a duplicate.

Usually handlers can’t afford loading whole history for each processed command so it leaves two optimization options. Command handler can use some business-level ID carried by a command to decide what stream to emit resulting events to. In such case resulting streams can be short enough to allow to load them before processing each command. In other words, a concept of aggregate can be used. As an alternative, a sliding deduplication window can be used to limit the number of past events that needs to be loaded.

View store

Anything that holds persistent state.

The Idea

As you may expect, these four primitives are based on something even more fundamental. The Idea behind RSA is that any message in a distributed system (as in real life) can be duplicated. RSA does not try to create an illusion that it is not the case. Rather than, it provides means for creating deduplicated bubbles within the solution where it makes sense.

VN:F [1.9.22_1171]
Rating: 5.0/5 (5 votes cast)