In the first part of our NATS Cluster Architectures series, we established robust foundations with regional clusters, covering single-node and three-node NATS deployments. Now, we're ready to expand our horizons and tackle the complexities of connecting your systems across geographical divides.

As applications scale and user bases globalize, bridging data centers or cloud regions becomes essential. NATS excels in these scenarios, offering uniquely flexible multiregional topologies. This installment moves beyond local networks to explore how NATS helps you truly connect the globe.

Here, we'll be diving into:

1. Three-Region Superclusters

2. Leaf Node Deployments

And for each topology, we'll examine:

• Key characteristics and design considerations

• Implications for JetStream persistence

• Tradeoffs in consistency, latency, and availability

• Ideal use cases

• Impact on application integration, particularly for C#/.NET clients

Disaster Ready: Multiregional Supercluster

Superclusters in NATS provide a very high degree of fault tolerance and scalability by interconnecting multiple independent NATS clusters. This allows for resilience across geographically dispersed regions. Here, we'll explore a Supercluster with three regions: Region1, 2 and 3.

In this topology, each region operates as a distinct, three node NATS cluster, as described in the previous section. These regional clusters are then interconnected through Gateways. Gateways establish connections between the regional clusters, allowing for the exchange of messages and subject interest information.

Characteristics at a Glance

• High Availability: Failure of an entire region can be tolerated, although some data loss is possible depending on JetStream configuration

• Global Scalability: Scales horizontally by adding more regions or adding servers to existing regions

• Geographic Distribution: Connects NATS deployments across wide-area networks and different geographical locations

• Increased Complexity: Multiple clusters, across multiple regions demands careful, thorough design and configuration with a high level of operational observability and expertise

• Potential for Increased Latency: Cross region communication will naturally have higher latency than local communication

The nature of a Gateway

Gateways operate on a dedicated port, separate from client or route ports. They gossip gateway nodes and any discovered gateways. Gateways have several very different characteristics from route connections.

• Named Connections: Gateway connections themselves are named, specifying their cluster

• Gateway Mesh: A full mesh is formed between clusters instead of between all individual servers

• Unidirectional Connections: Unidirectional connections bind gateways, though discovery makes them appear bidirectional

• Client Transparency: Gateways connections are not gossiped or propagated to clients

Gateway connections are designed to reduce the number of connections required between servers when joining clusters as compared to a full mesh of all servers across all clusters. For instance, with three clusters, each with three nodes; a full mesh routed cluster requires 36 connections, yet only 18 gateway connections are required.

Another key aspect of gateway connections is the optimization of interest graph propagation. This limits unnecessary cross region chatter. Interest propagation across gateways uses mechanisms like Interest-only Mode and Queue Subscriptions. Interest-only requires remote gateways to have explicitly registered interest in a subject. Queue Subscription semantics are honored globally, i.e. messages are delivered to a single subscriber in a queue group across the Supercluster, prioritizing local queue subscribers before failing over to the next lowest RTT cluster.

JetStream Considerations

JetStream deployed across a Supercluster provides global data distribution, high availability, and fault tolerance. However, resource placement, consistency boundaries and their guarantees must be well understood.

• Local Placement: Streams and Consumers can only be placed among servers within a cluster, tags across regions will fail to deploy

• Local Replication: Streams and Consumers can only replicate data within a cluster, i.e. across routes, not gateways

• Meta Group Dependence: The JetStream Meta Group is made up of the entire Supercluster, if quorum cannot be reached, new resources cannot be created, i.e. new Streams and Consumers

• Regional Resilience: Each region operates its own JetStream resources independently, each with their own RAFT group, even in the loss of a Metagroup quorum, reads/writes to existing resources succeed

Replication and Consistency

• Within a Region/Cluster: Strong consistency is ensured within each regional cluster with consensus among all replicas

• Mirrored & Sourced Streams: To achieve fault tolerance across regions Streams can be Mirrored or Sourced, with mirroring being a read-only copy of one Stream and sourced being a locally writeable copy of one or many Streams

• Eventual Consistency: Mirrored & Sourced Streams provide eventual consistency, data written to the source stream(s) will eventually be replicated to the mirrored/sourced stream

• Data Loss Possible: In the event of a catastrophic regional failure, data loss is possible, even with mirroring/sourcing, although loss is limited to the replication window

Ideal Use Cases - Globally Available and Disaster Tolerant

• Global Applications: When users and/or services are distributed across multiple regions

• Disaster Tolerance Scenarios: Where you need to ensure business continuity in the event of a regional outage

• Multi Cloud Deployments: When multiple cloud providers are required, supercluster gateways bridge independent provider clusters

NATS Server Config

Getting interesting! This shows how you might configure the servers within one region, region1. Extrapolating to other regions is trivial, and again “self” gateways are smartly ignored.

port: 4222
cluster: {
  name: region1
  listen: 0.0.0.0:6222
  routes: [
    "nats://region1-server1:6222",
    "nats://region1-server2:6222",
    "nats://region1-server3:6222"
  ]
}
gateway: {
  name: region1
  port: 7222
  gateways: [
    { name: "region1", url: "nats://region1-gateway:7222" },
    { name: "region2", url: "nats://region2-gateway:7222" },
    { name: "region3", url: "nats://region3-gateway:7222" }
  ]
}
jetstream: {
  store_dir: /data/jetstream
  max_memory_store: 1GB
  max_file_store: 10GB
}

Key Differences

• cluster.*: Each region has its own cluster configuration, i.e. cluster.name, cluster.listen, routes

• gateway: Section defines how the regional clusters connect

• gateway.name: Names the cluster, all gateways within a cluster must define the same name

• gateway.listen: The host and port for incoming gateway connections from other regions

• gateway.gateways: A list of the gateway addresses of other regions

NATS.Net Integration

Connecting to a NATS Supercluster from a C# application is similar to connecting to a single cluster. Clients connect to a server within their local region. The Supercluster handles the routing of messages across regions.

const string Region1 = "region1";
const string Region2 = "region2";
var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = loggerFactory.CreateLogger<Program>();

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
var subscribing = Region2JetStreamSubscribeAsync(cts.Token);
var publishing = Region1JetStreamPublishAsync(cts.Token);

await Task.WhenAll(publishing, subscribing);

async Task Region1JetStreamPublishAsync(CancellationToken token)
{
    var context = GetRegion1JetStreamContext();
    Placement placement = new() { Cluster = Region1 };
    var streamConfig = new StreamConfig($"{Region1}-stream", [$"{Region1}.>"])
    {
        NumReplicas = 3,
        Placement = placement
    };
    _ = await context.CreateOrUpdateStreamAsync(streamConfig, token);

    foreach (var id in Enumerable.Range(1, 10))
        await context.PublishAsync($"{Region1}.data", $"From {Region1}: {id}", cancellationToken: token);

    logger.LogInformation("{Region} client completed publishing", Region1);
}

async Task Region2JetStreamSubscribeAsync(CancellationToken token)
{
    var context = GetRegion2JetStreamContext();
    Placement placement = new() { Cluster = Region2 };
    SubjectTransform transform = new() { Src = $"{Region1}.>", Dest = $"{Region2}.from-{Region1}.>" };
    StreamSource source = new()
    {
        Name = $"{Region1}-stream",
        SubjectTransforms = [transform]
    };
    var streamConfig = new StreamConfig($"{Region2}-stream", [$"{Region2}.>"])
    {
        NumReplicas = 3,
        Placement = placement,
        Sources = [source]
    };
    var stream = await context.CreateOrUpdateStreamAsync(streamConfig, token);
    var consumer = await stream.CreateOrderedConsumerAsync(cancellationToken: token);

    await foreach (var msg in consumer.ConsumeAsync<string>(cancellationToken: token))
    {
        logger.LogInformation("{Region} client received: {Message}", Region2, msg.Data);
        await msg.AckAsync(cancellationToken: token);
    }
}

INatsJSContext GetRegion1JetStreamContext()
    => new NatsJSContext(BuildNatsConnection(Region1));

INatsJSContext GetRegion2JetStreamContext()
    => new NatsJSContext(BuildNatsConnection(Region2));

INatsConnection BuildNatsConnection(string region)
    => new NatsConnection(new()
    {
        Url = $"nats://{region}1:4222,nats://{region}2:4222,nats://{region}3:4222",
        LoggerFactory = loggerFactory
    });

Key Differences

This sample becomes a bit more complex but showcases much more functionality.

• Region1JetStreamPublishAsync

  • Creates a region local stream, region1-stream

  • Consumes all subjects matching region1.>

  • Finally, publishes 10 messages to region1.data that get ingested into region1-stream

• Region2JetStreamSubscribeAsync

  • Creates a region local stream, region2-stream

  • Consumes all subjects matching region2.>

  • Also, Sources region1-stream, replicating all data locally

  • Then Transforms all inbound region1.> subjects to an east-centric version, region2.from-region1.>

  • Finally, consumes all available messages via an ephemeral ordered Consumer

Localized Reliability: Leaf Nodes

Leaf Nodes extend an existing NATS deployment, be it single server, cluster, supercluster, etc., to conceptually edge deployments. This creates an isolation boundary and local resiliency with respect to the remote Hub. Leaf Nodes provide communication connectivity locally within their edge boundary even when disconnected from the remote. Unlike servers in the remote hub cluster, Leaf Nodes do not participate in the same routing mesh. Instead, they establish an account scoped point-to-point connection(s) to the remote. Further, Leaf Nodes themselves do not need to be reachable by clients but instead can be used to form any acyclic topology.

Client reachable Leaf Nodes present a discrete NATS context distinct from that of the remote. Clients connect against a local AuthN/AuthZ policy of the Leaf Node, and the Leaf Node forwards messages to and from the remote. Only the credentials required to form the Leaf Node to hub connection need be shared between the domains, allowing security and generally Operator domains to be bridged. The remote hub sees the Leaf Node as a single account scoped connection, carrying with it the permissions, exports, and imports of that account.

Characteristics at a Glance

• Network Segmentation: Leaf Nodes represent isolated segments, preventing unnecessary traffic from crossing network boundaries

• Connection Schemes: Leaf Nodes can connect to their remote via plain TCP, TLS, and even WebSockets

• Controlled Connectivity: Account/User level configuration provides fine grained control over which subjects are exchanged with the upstream

• Resilience: Leaf Nodes provide localized resilience, if the connection to the remote is lost, clients within the Leaf Node's network segment can still communicate with each other

• Distinct AuthN/AuthZ: Clients of the Leaf Node connect against the local auth policy, while the Leaf Node connection to the remote is a separate shared policy

JetStream Considerations

Leaf Nodes can either extend or isolate a JetStream Domain. The JetStream Domain is an independent namespace within JetStream that disambiguates requests between NATS deployments. Extending a central JetStream deployment through a leaf deployment would map to the same domain. While isolating JetStream deployments accessible to the same client would be done via unique JetStream domain names.

• Isolated Domains: Leaf Nodes can isolate JetStream domains, selectively sharing between the independent namespaces

• Data Resiliency: JetStream data is stored locally on the Leaf Node, ensuring that data is preserved even if the connection to a remote is temporarily lost

• Data Locality: Ensures data locality for compliance or performance reasons

• Consistency Separation: Leaf local streams can be immediately consistent, while streams shared across the Leaf node connection can be eventually consistent via sourcing/mirroring to/from the remote

Ideal Use Cases - Segmented or Intermittent Connectivity

• Remote Distribution: Connecting NATS servers in different, remote locations, such as disparate local offices, remote sites, etc.

• Organization Separation: Leaf Nodes bridge security and operator domains allowing zero trust between separate entities

• Edge Computing: Deploying Leaf Nodes at the edge of the network to collect and process data locally before forwarding it to a central system

• Network Segmentation: Isolating different parts of your network for security or performance reasons

• Data Localization: Keeping data within a specific region or network segment for compliance or performance

NATS Server Config

Leaf Node server config becomes a bit more involved and needs to be well planned. Here, we see a shared account between the Hub and Leaf, while each may have other respective local accounts.

accounts.shared.conf

accounts: {
  SYS: {
    users: [{ user: sys, password: sys }]
  },
  LEAF: {
    users: [{ user: leaf, password: leaf }],
    jetstream: enabled
  }
}
system_account: SYS

server.hub.conf

port: 4222
server_name: hub
include ./accounts.conf
jetstream: {
  store_dir: /data/jetstream
  domain: hub
  max_memory_store: 1GB
  max_file_store: 10GB
}
leafnodes: {
  port: 7422
}

server.leaf.conf

port: 4222
server_name: leaf-node
include ./accounts.conf
jetstream: {
  store_dir: /data/jetstream
  domain: leaf
  max_memory_store: 1GB
  max_file_store: 10GB
}
leafnodes: {
  remotes: [
    {
      urls: ["nats://leaf:leaf@0.0.0.0:7422"]
      account: "LEAF"
    }
  ]
}

Key Differences

• accounts.shared.conf: LEAF account is shared and used to connect the Leaf Node back to the Hub

• jetstream.domain: Defines separate isolated JetStream Domains, namespaces, for the Leaf Node and Hub

• Hub leafnodes.*: Enables the server to listen for Leaf Node connections on the specified port

• Leaf leafnodes.*: Configures a Leaf Node connection back to the Hub on the specified port, using the shared Account and User

NATS.Net Integration

Connecting to a Leaf Node from a C# application is much the same as connecting to any NATS server. The client authenticates with a local account and is unaware that it's connected to a Leaf Node or that there is anything ‘upstream’.

var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = loggerFactory.CreateLogger<Program>();

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
var opts = new NatsOpts
{
    Url = "nats://leaf:leaf@leaf-node:4222",
    LoggerFactory = loggerFactory
};

await using var connection = new NatsConnection(opts);
var context = new NatsJSContext(connection);

var streamConfig = new StreamConfig("local-stream", ["leaf.>"]);
var stream = await context.CreateStreamAsync(streamConfig, cts.Token);
var consumer = await stream.CreateOrderedConsumerAsync(cancellationToken: cts.Token);

await context.PublishAsync<string>("leaf.data", "Hello World");

var msg = await consumer.NextAsync<string>(cancellationToken: cts.Token);
logger.LogInformation("Leaf received: {Message}", msg?.Data);
await (msg?.AckAsync(cancellationToken: cts.Token) ?? ValueTask.CompletedTask);

await connection.PublishAsync("remote.data", "Message to Remote", cancellationToken: cts.Token);

Wrap Up

The second part of the NATS Topologies set expanded our focus and explored two key NATS topologies:

• Superclusters: Understood how Superclusters interconnect independent NATS clusters across regions, providing high availability and examined JetStream replication and consistency in a multi-regional context

• Leaf Nodes: Then we discussed how Leaf Nodes enable you to extend a central NATS deployment across different networks, and isolated security domains, also highlighting the unique implications of isolated JetStream domains

But there's still more to discover!

Next Up - Advanced Topologies!

In Part 3: Advanced Topologies, we'll push the boundaries of NATS even further, literally! First, we’ll be exploring the nature of a Stretch Cluster and how it differs from a Supercluster. Then, we’ll look at the Leaf Cluster, trading immediate consistency for greater resiliency. Get ready to master some of the most powerful and intricate NATS architectures!

Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!

NATS topology refers to the way your NATS servers are arranged and connected. Unlike many messaging systems with more rigid deployment models, NATS offers remarkable flexibility. You're not shoehorned into a one-size-fits-all approach; instead, you become the architect of your messaging fabric.

Your choice of topology, whether a single server, resilient cluster, leaf nodes, a supercluster, etc. will directly shape fault tolerance, message delivery guarantees, network segmentation, and overall system complexity. This means you can tailor your NATS deployment precisely to your needs, but it requires careful consideration.

This miniseries of posts will explore a variety of NATS server topologies, increasing in their complexity. Here, we’ll be covering:

1. The single instance deployment

2. A resilient three-node cluster

And for each:

• Key characteristics of each topology

• Tradeoffs in consistency, latency, and availability

• Potential use cases

• Impact on application integration, particularly within C#/.NET clients

The Singularity: Single NATS Server

A single NATS server is the most fundamental setup, deployed as a single instance of the NATS server. It's the easiest to grasp and get started with, making it a natural entry point for development and simple use cases. This is the same deployment we used in “Getting Started with NATS in C#”

In this topology, a single NATS server process handles all client connections, message routing, and any configured persistence, i.e. JetStream, if enabled. All publishers and subscribers within our system connect directly to this single endpoint.

Characteristics at a Glance

• Simplicity: Straightforward to configure and manage

• Low Overhead: Requires the least amount of infrastructure resources, i.e. CPU, memory, network

• Single Point of Failure: If the server becomes unavailable due to hardware failure, network issues, or software problems, the entire messaging system halts

• Limited Scalability: The performance and throughput of the system are bounded by the capacity of the single server instance

  • Note: Although, a single NATS server can still be quite performant, to the tune of millions of messages/sec depending on resources

• No Redundancy: Availability limited to the single node, no automatic failover or backup

• No JetStream Replication: Of course, with a single server, no JetStream resources can be replicated

Ideal Use Cases - Dev/Local/Non-Prod

• Local Dev and Testing: Perfect for a local dev NATS environment and local integration testing

• Demos, POCs: Proving new messaging patterns among microservices, demoing new features, etc.

• Learning and Exploration: Low barrier of entry to understand the basic concepts of NATS without the complexities of clustering

NATS Server Config

Couldn’t be simpler! This specifies that clients should connect on the default port 4222 and the server will be listening on 0.0.0.0:4222

port: 4222

Mounting a file for just this might be overkill, you can simply specify many options via the command line as well. While 4222 is the default, just for example, with Docker, this is equivalent to the server.conf above

docker run --rm --name my-nats -p "4222:4222" nats -p 4222

NATS.Net Integration

Connecting to a single NATS server will use a connection string pointing to its address, just the same as we saw in “Getting Started with NATS in C#”. For completeness, the NATS.NET client library provides the tools for this. Importantly, in this topology, our app becomes directly reliant on the availability of this single server. As a result, several factors will impact our app design.

• Connection Management: The NATS.Client.Core library does provide robust connection management, but with a single server, its primarily focused on the initial connection and handling disconnects

• Initial Connection: By default, the client will lazily connect to the server, and fail fast on the initial connection attempt, i.e. the initial attempt is not retried and a NatsException is thrown

• Reconnects: After an initial successful connection, any drop will result in a reconnect attempt up to a configured limit via NatsOpts.*Reconnect* properties

• No Automatic Failover: Reconnection attempts have no other route except the same single address

var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = loggerFactory.CreateLogger<Program>();

var opts = new NatsOpts
{
    Url = "nats://localhost:4222",
    LoggerFactory = loggerFactory
};

await using var connection = new NatsConnection(opts);

var rtt = await connection.PingAsync(CancellationToken.None);

logger.LogInformation("Ping successful - {RTT}ms to {@ServerInfo}",
    rtt.TotalMilliseconds,
    connection.ServerInfo);

Resilience Through Redundancy: Three Node Cluster

Clustering in NATS provides fault tolerance and increased capacity by connecting multiple NATS servers. A cluster of three nodes is a common starting point for production deployments and offers a balance between resilience and complexity.

In this topology, three NATS servers are interconnected via routes, and form a full mesh. Each server maintains bidirectional connections to the others, allowing them to fully propagate the Subject interest graph. Clients can connect to any server in the cluster, and the cluster will route messages appropriately, even if a server becomes unavailable.

Characteristics at a Glance

• Fault Tolerance & Increased Availability: The cluster can still serve Core NATS traffic even in the face of a two node failure, i.e. operates with only a single node

• Improved Scalability: Distributes the load across multiple servers, increasing the overall capacity of the system.

• Automatic Discovery: NATS servers in a cluster will automatically discover each other after reaching any ‘seed’ server, cluster gossip ensures a full mesh is maintained

• Message Routing: Servers route messages to the appropriate destinations within the cluster, cluster is transparent to publisher/subscribers beyond any connection string

• Increased Complexity: Although still minimal, configuring and managing a cluster does involve more complexity than a single server

JetStream Considerations

Clustering is crucial for JetStream's fault tolerance and data durability. JetStream leverages the Raft consensus algorithm to replicate data across multiple servers within a cluster. This ensures, to the limit of consensus, that the data remains available and consistent.

• Quorum: JetStream requires a quorum, \(1/2*n+1\), to be available in order to handle requests, e.g. reads, writes, acks, etc.

• Storage: Each server in the cluster needs its own storage for JetStream data

• Replication Latency: Replication adds some network overhead, low latency network connections are critical

Replication and Consistency

• Replication Factor: With three nodes, 3x replication of JetStream Stream and Consumer data/metadata can be achieved

• Consistency: Raft ensures strong consistency, meaning that all replicas agree on the order of the data

  • Note: Replication is set at the Stream level and either inherited or optionally set at the Consumer level, “In Sync Replicas” will always be equal to Replicas, i.e. acks = all

• Fault Tolerance: A replication factor of 3 allows the cluster to tolerate the failure of up to one server without losing any data, and continuing servicing requests

• Increased Durability: Storing multiple copies of the data significantly reduces the risk of data loss due to hardware failures or other issues

Ideal Use Cases - Single Region Production

• Single Region Production Environments: Essential for any production system that requires high availability and fault tolerance

• Durable, Highly Available Apps: Applications where data loss or downtime is unacceptable

• Scalable Systems: Systems that need to handle a high volume of messages and clients

NATS Server Config

A basic example of a server.conf, you could create three separate configuration files (server1.conf, server2.conf, and server3.conf), each with the appropriate settings for that particular server. Although, the config can be reused depending on your networking environment. Note that routes pointed to “self” are smartly ignored making config sharing a bit easier.

listen: 0.0.0.0:4222
cluster {
  name: my-cluster
  listen: 0.0.0.0:6222
  routes: [
    "nats://server1:6222",
    "nats://server2:6222",
    "nats://server3:6222"
  ]
}
jetstream {
  store_dir: /data/jetstream
  max_memory_store: 1GB
  max_file_store: 10GB
}

Key Differences

• cluster: Section defines the details of our cluster

• cluster.name: Uniquely identifies the cluster

• cluster.listen: The host and port to listen on for incoming clustering connections

• cluster.routes: A list of dedicated Routes to other servers

• jetstream: Configures JetStream, including storage target and limits

NATS.Net Integration

Connecting to a NATS cluster with the NATS.Client.Core and NATS.Client.JetStream libraries is straightforward. This time, you can provide a list of server URLs, and the client will automatically connect to the first available server. If that server becomes unavailable, the client will attempt to connect to another server in the list.

var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = loggerFactory.CreateLogger<Program>();

var opts = new NatsOpts
{
    Url = "nats://server1:4222,nats://server2:4222,nats://server3:4222",
    LoggerFactory = loggerFactory
};

var connection = await GetConnectionAsync(logger, opts);
var stream = await GetStreamAsync(logger, connection);

async Task<INatsConnection> GetConnectionAsync(ILogger logger, NatsOpts opts)
{
    await using var connection = new NatsConnection(opts);

    var rtt = await connection.PingAsync(CancellationToken.None);

    logger.LogInformation("Ping successful - {RTT}ms to {@ServerInfo}",
        rtt.TotalMilliseconds,
        connection.ServerInfo);
    return connection;
}

async Task<INatsJSStream> GetStreamAsync(ILogger logger, INatsConnection connection)
{
    var context = new NatsJSContext(connection);
    var config = new StreamConfig("my-first-stream", ["some.subjects.>"])
    {
        NumReplicas = 3
    };

    var stream = await context.CreateOrUpdateStreamAsync(config, CancellationToken.None);
    logger.LogInformation("Stream created/updated - {@StreamInfo}", stream.Info);
    return stream;
}

Key Differences

• NatsOpts.Url: Set to a comma separated list of server URLs

• NATS.Client.Core: Handles connecting among multiple servers, starting with the first available server and reconnecting to another server if the connection is lost

• NatsJSContext: Building on top of Core NATS, the JetStream Context accepts an existing NatsConnection

• my-first-stream: Configured with a replication factor of 3, ensuring that data is replicated across all three servers in the cluster

Wrap Up

In this first part exploring NATS topologies, from single to three node cluster, we've laid the groundwork for building robust and scalable messaging systems with NATS.

Next Up - Multiregional Clusters!

Moving beyond single region clusters, we're now ready to take our NATS deployments to the next level. In Part 2: Multiregional Clusters, we'll dive into how Superclusters and Leaf Nodes enable you to connect NATS deployments across geographical distances, building resilient and scalable messaging solutions that span the globe. Get ready to unleash the full potential of NATS for distributed, mission-critical applications!

Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!

In the world of distributed systems, event driven architecture, EDA, has emerged as a powerful paradigm for building scalable and resilient applications. At the heart of many EDA implementations lies a robust messaging system, and NATS is a compelling choice known for its speed, simplicity, and reliability.

In a crowded landscape of message brokers and event buses like Kafka, Event Hub, SQS, etc. NATS carves a unique niche by focusing on fundamental application connectivity. While offering straightforward, location transparent Pub/Sub as a core capability, NATS goes beyond simple message passing. It provides a versatile foundation for implementing complex messaging patterns, including basic Fan-Out/In, Work Queues, Request/Reply semantics, and much more. This flexibility allows us to rethink traditional communication methods, offering an elegant and often simpler alternative to REST based APIs and traditional complex ingress, and this is just scratching the surface of Core NATS before delving into its advanced features like JetStream persistence and other higher-level abstractions.

Where do we start?

Today, we'll explore how to integrate NATS Core into your .NET applications using C#. We'll walk through the process of setting up a console application, configuring the NATS connection using dependency injection, and then demonstrate the foundational patterns of Pub/Sub and Request/Reply.

We’re going to accomplish three main goals:

1. Stand up a C# console project with our NuGet dependencies

2. Integrate with the NATS server via the C# NATS client

3. Demo Pub/Sub and Request/Reply

Setting Up Your .NET Console Project

First things first, let's create a new .NET console application. Open your terminal or command prompt and execute the following command

dotnet new console -n NATSUnleashed.CSharpIntro

Next, we’ll need to add the official NATS.NET client library to our project along with the serializer package and usual hosting components. Run these commands

dotnet add package NATS.Client.Core
dotnet add package NATS.Client.Serializers.Json
dotnet add package Microsoft.Extensions.Hosting

Finally, open the solution and we’ll get started!

Creating the NatsService

We want a NATS service that will allow us to run our app as any of the following actors:

• publisher - Publishes to NATS Server on a given subject

• subscriber - Subscribes to the same subject as the publisher

• requestor - Makes requests to NATS server and expects a reply

• responder - Responds to requests from the requestor

NatsService - Config Options, Message Schema, Extensions

Configuration

First, we’ll create an options record to bind various configuration settings to.

public sealed record NatsConfig
{
    public string NatsUrl { get; set; } = default!;
    public string AppType { get; set; } = default!;
    public string Subject { get; set; } = default!;
    public string? QueueGroup { get; set; } = default!;

    public void Deconstruct(
        out string natsUrl,
        out string appType,
        out string subject,
        out string? queueGroup)
        => (natsUrl, appType, subject, queueGroup) = (NatsUrl, AppType, Subject, QueueGroup);
}

Simply a record that captures the server URL, the type of actor to run, our subject and queue group.

Message Schema

Next, a simple message schema to pass around

public sealed record ExampleMessage(
    int Id,
    string Message);

Extensions - Message Builders

Finally, a set of helpers to build messages with distinct random Id’s and static content

public static class Extensions
{
    public static ExampleMessage NextPubSubMessage(
        this Random random)
        => new(random.Next(100, 199), "Hello World");

    public static ExampleMessage NextRequestMessage(
        this Random random)
        => new(random.Next(200, 299), "Anyone Home?");

    public static ExampleMessage NextReplyMessage(
        this Random random)
        => new(random.Next(300, 399), "I'm home!");
}

With the basics in place, we can implement the core NatsService

NatsService - The Background Service

Our NatsService will be a BackgroundService that becomes configured as any one of our actor variations. We’ll switch on the AppType config to drive our behavior

public sealed class NatsService(
    ILogger<NatsService> logger,
    INatsConnection connection,
    IOptions<NatsConfig> options)
    : BackgroundService
{
    private readonly ILogger<NatsService> _logger = logger;
    private readonly INatsConnection _connection = connection;
    private readonly IOptions<NatsConfig> _options = options;

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
        => _options.Value.AppType switch
        {
            AppBuilding.Publisher => PublisherAsync(stoppingToken),
            AppBuilding.Subscriber => SubscriberAsync(stoppingToken),
            AppBuilding.Requestor => RequestorAsync(stoppingToken),
            AppBuilding.Responder => ResponderAsync(stoppingToken),
            _ => throw new NotSupportedException($"App type {_options.Value.AppType} is not supported.")
        };
}

Then we can implement each behavior variation.

Publishing

private async Task PublisherAsync(CancellationToken cancelToken)
{
    var (_, _, subject, _) = _options.Value;
    _logger.LogInformation("Publishing to {Subject}", subject);
    while (!cancelToken.IsCancellationRequested)
    {
        var message = Random.Shared.NextPubSubMessage();
        await _connection.PublishAsync(
            subject, 
            message, 
            cancellationToken: cancelToken);

        await Task.Delay(TimeSpan.FromSeconds(3), cancelToken);
    }
}

Subscribing

private async Task SubscriberAsync(CancellationToken cancelToken)
{
    var (_, _, subject, group) = _options.Value;
    _logger.LogInformation("Subscribing to {Subject}", subject);
    await foreach (var msg in _connection.SubscribeAsync<ExampleMessage>(
        subject,
        queueGroup: group,
        cancellationToken: cancelToken))
        _logger.LogInformation("Pub/Sub - Received [{Message}]", msg.Data);
}

Requesting

private async Task RequestorAsync(CancellationToken cancelToken)
{
    var (_, _, subject, _) = _options.Value;
    _logger.LogInformation("Requesting to {Subject}", subject);
    while (!cancelToken.IsCancellationRequested)
    {
        var message = Random.Shared.NextRequestMessage();
        var reply = await _connection.RequestAsync<ExampleMessage, ExampleMessage>(
            subject, 
            message, 
            cancellationToken: cancelToken);
        _logger.LogInformation("Req/Rep - Reply [{Reply}]", reply.Data);

        await Task.Delay(TimeSpan.FromSeconds(3), cancelToken);
    }
}

Responding

private async Task ResponderAsync(CancellationToken cancelToken)
{
    var (_, _, subject, _) = _options.Value;
    _logger.LogInformation("Subscribing to {Subject}", subject);
    await foreach (var msg in _connection.SubscribeAsync<ExampleMessage>(
        subject, 
        cancellationToken: cancelToken))
    {
        _logger.LogInformation("Req/Rep - Request [{Message}]", msg.Data);
        var message = Random.Shared.NextReplyMessage();
        await msg.ReplyAsync(message, cancellationToken: cancelToken);
    }
}

Dependency Injection

Now we’ll wire up our DI, to include our options config, the NatsConnection and NatsService

public static class AppBuilding
{
    public const string Publisher = "publisher";
    public const string Subscriber = "subscriber";
    public const string Requestor = "requestor";
    public const string Responder = "responder";

    public static IServiceCollection AddNatsServices(
        this IServiceCollection services,
        IConfiguration config)
    {
        services.Configure<NatsConfig>(opts =>
        {
            opts.AppType = config["app-type"] ?? "publisher";
            opts.NatsUrl = config["nats:url"] ?? "nats://localhost:4222";            
            opts.Subject = config["nats:subject"] ?? "some.subject";
            opts.QueueGroup = config["nats:group"];
        });
        services.TryAddSingleton<INatsConnection>(sp =>
        {
            var config = sp.GetRequiredService<IOptions<NatsConfig>>().Value;
            return new NatsConnection(new()
            {
                Url = config.NatsUrl,
                SerializerRegistry = NatsJsonSerializerRegistry.Default
            });
        });
        return services.AddHostedService<NatsService>();
    }
}

Program: Main

And finally, of course, our generic main to kick off the Host

using Microsoft.Extensions.Hosting;
using NATSUnleashed.CSharpIntro;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddNatsServices(builder.Configuration);

var app = builder.Build();

await app.RunAsync();

Excellent, at this stage our app is ready to build and then demo. Let’s kickoff the build quick

dotnet build -c Release

and we should see success similar to this

Restore complete (0.3s)
  NATSUnleashed.CSharpIntro succeeded (0.3s) → bin\Release\net9.0\NATSUnleashed.CSharpIntro.dll

Build succeeded in 1.0s

Running NATS Server with Docker

Before we demo, we need a NATS server running. Just as we did in Getting Started with NATS CLI we’ll start up NATS server using Docker. Docker provides a convenient way to do this. If you don't have Docker installed, you'll need to install it from the official Docker website.

From a new terminal execute the following command

docker run --rm --name my-nats -p "4222:4222" -p "8222:8222" nats -n nats1 -m 8222

This command is the same as before and breaks down into a few key pieces

1. We expose the default NATS port 4222 and the default monitoring port 8222

2. Name the NATS server instance with -n nats1

3. Enable NATS monitoring via -m 8222

You should then see an output similar to this

[1] [INF] Starting nats-server
[1] [INF]   Version:  2.11.1
[1] [INF]   Git:      [d78523b]
[1] [INF]   Name:     nats1
[1] [INF]   ID:       NBZG74EPNUPKYPVB7YFHOHABDAZ7EN3FFPHQYJUEAQODJYTZVNDMEH25
[1] [INF] Starting http monitor on *******:8222
[1] [INF] Listening for client connections on *******:4222
[1] [INF] Server is ready

Demo: Pub/Sub

Demo time! Open three new consoles to ./bin/Release/net9.0 and kickoff our app

Subscribers 2x

dotnet NATSUnleashed.CSharpIntro.dll --app-type subscriber

Publisher

dotnet NATSUnleashed.CSharpIntro.dll --app-type publisher

With those running, we should see messages published and delivered to each subscriber

Demo: Pub/Sub Queue Groups

Next, we’ll see Queue Groups in action. These allow you distribute messages randomly among subscribers. Stop the Publisher and both subscribers, then restart the subscribers with this new command

dotnet NATSUnleashed.CSharpIntro.dll --app-type subscriber --nats:group my-queue-group

And start the publisher again

dotnet NATSUnleashed.CSharpIntro.dll --app-type publisher

And observe the output, note each message is only delivered to a single subscriber

Demo: Request/Reply

Finally, for Request/Reply, shutdown the subscribers and publisher. Then kickoff one or more responder apps

dotnet NATSUnleashed.CSharpIntro.dll --app-type responder

Then kickoff the requestor app

dotnet NATSUnleashed.CSharpIntro.dll --app-type requestor

And the output should be similar to this, note that only one responder is received by the requestor per request

Wrap Up

Today, we built a C# dotnet console app that integrated with a NATS server via the NATS.Net client. Our app could run in either Pub/Sub or Request/Reply modes, and we demonstrated that functionality. This really lays the foundation for building some really cool stuff. With just some simple configuration and the clean API design of the client library we can get up and running quickly.

Until next time! All code is available at ConcurrentFlows GitHub

Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!

NATS is a location transparent messaging technology built around foundational Pub/Sub semantics. That means it doesn’t matter if a publisher is in one hemisphere, while a subscriber is in the opposite global hemisphere. As long as both the sending and receiving parties can connect to a NATS server, they can communicate with one another. NATS Subjects facilitate this communication, they are the mutual knowledge between publishers and consumers, i.e. a publisher publishes to a Subject, while a subscriber subscribes to a matching Subject.

NATS defines Subjects as a specifically formatted string consisting of Tokens separated by a . (dot), for instance

{token1}.{token2}.{tokenN}
market.data.stock.AAPL
market.data.forex.EURUSD
trade.order.submitted
trade.order.executed

These Subjects are just simple strings, and are by default ephemeral, but they create an expressive, powerful, addressing mechanism. You can think of Subjects as an address on an envelope, it captures in detail, the single destination the message is sent.

The Tokens form a hierarchy, category, and uniqueness of every publication destination, in the same way Country, State/Provence, City, Postal Code, uniquely define a postal destination. If you’re picking up on the foreshadowing here’s the reveal, the Subject doesn’t fully define the ultimate Delivery/Consumption of the message. In much the same way as a letter could be opened and read by one or many recipients, NATS messages published into a given Subject are delivered in a variety of ways. They can be transformed into entirely new subjects, delivered to specific subscribers in a Queue Group, or consumed by subscribers to a Wildcard Subject, i.e. market.data.stock.* Collectively this is the concept of Subject Awareness.

Subject Awareness

Subject Awareness refers to the way messages are routed and consumed based on the Subject of a message. Subject awareness allows some powerful expression. We can leverage the dynamic nature of Subjects to build relevant intelligence into applications based on the Subject’s content and structure. For instance:

1. Intelligent Consumption: Specifically subscribe to content of interest, i.e. market.data.stock.AAPL while ignoring market.data.forex.>, only messages related to APPL stock will be received, anything under the scope of forex is ignored

2. Contextual Communication: The Subject provides valuable context regarding the message content, purpose, etc. user.status.online indicates that a ‘User’ has become ‘Online’, sensor.humidity.kitchen immediately indicates the subscriber is receiving a ‘Humidity Sensor’ reading, from the ‘Kitchen’

3. Malleable Evolution: Unlike configuring some rigid topic/queue constructs upfront, Subjects are ephemeral by default and can be dynamically created, while Wildcard subscribers can be delivered messages from new unique Subjects without modification, i.e. subscribers to cluster.apps.*.status would receive ‘Status’ messages from all existing and new ‘Apps’ without modification

Publishing vs Subscribing

When a NATS Publisher sends a message, it assigns a Subject describing the context of the message. This context can specify content type or category, e.g. app.status, would be an App Status message. The Subject could contain the origin of the message, e.g. products.factory.east123, is a message from factory East 123. Even purpose can be defined, e.g. records.123.delete, Delete the entity with id: 123.

Subscribers on the other end express an interest in Subjects the same way a researcher might be interested in a given domain. That interest may be very specific, e.g. cluster.east.app.chat-app.user.user1, matching a specific user, of a specific app, in a specific cluster. Alternatively, it could be very broad, matching many individual Subjects, e.g. cluster.east.>, matching all Subjects within the scope of a specific cluster.

It’s important to make clear, a message will be published to a single specific Subject. This means a message published to market.data.stock.AAPL would not be received via a subscription to market.data.stock, or market.data.stock.APPL.average. Only a subscription pattern matching the published Subject will receive the message, e.g. market.data.stock.AAPL, market.data.stock.*, market.data.>, etc.

The Subject Hierarchy

Subjects in NATS are formed by a collection of dot separated Tokens. Each Token represents a level in a Subject hierarchy that collectively describes the Subject space of a particular domain.

Let’s explore an example to understand why hierarchy design can be important. If we wanted to describe an Order Service’s Subject space, we could first define the following Subject template, generally following the broad to specific rule of thumb:

• orders.status.{status}.order.{id}

This would form something like the following hierarchy

• orders.status.created.order.{id}

• orders.status.confirmed.order.{id}

• orders.status.shipped.order.{id}

• orders.status.delivered.order.{id}

This hierarchy effectively categorizes ‘Status’ by deliberately making that a constant level of the hierarchy. Also, the target entity Order is a static hierarchy level, the final one, with the specific Order Id following. Subscribers can express interest in a specific Status but all Order Id’s with the wildcard asterisk*, i.e. orders.status.created.order.* would receive all Created Orders. Alternatively, an interest in all Statuses can be expressed with an asterisk, i.e. orders.status.*.order.*. The contextual meaning of the asterisk remains clear with the context of the preceding Token, i.e. Status, or Order.

Further, if our Orders domain needs extension beyond Status and starts publishing a new category, we can add it without breaking existing subscriptions:

• orders.shipment.{stage}.order.{id}

Conceptually, Subjects in the new template exist somewhere between and among three higher-level Statuses

• orders.status.confirmed.order.{id}

• orders.status.shipped.order.{id}

• orders.status.delivered.order.{id}

But our earlier subscriptions, orders.status.*.order.*, etc. still only target the intended Status messages. To express interest in Shipment Stages, we can subscribe to orders.shipment.staged.order.*, orders.shipment.in-transit.order.*, orders.shipment.*.order.*, etc.

Subjects as Expressive Keys

Many messaging technologies make heavy use of Message Keys, these typically uniquely identify a particular categorization of the message or its content. A Topic representing car dealership inventory might use the Make as a message key, e.g. Ford, Toyota, Chevy, etc. It might not make much sense to use something like the Vin of each vehicle as a message key, given each is generally unique. The salient point is that the Cardinality of the key is a critical factor in message key selection.

Cardinality of the message key becomes so critical because it will typically drive message routing, its final partition and the ordering, even compaction, of related messages. If I needed all updates to my Ford inventory applied in the correct order, with something like Kafka, all updates would need to be in the same partition. Additionally, if my consuming process duration and message volume dictates the need for 100 consumers in a group, yet the cardinality of my message key is much less, the resulting partition distribution and consumer group balance is far from ideal.

NATS again takes a different approach. All messages are virtually keyless, consisting of only the destination Subject and payload, with optional headers and/or a reply subject. Subjects, as far as the NATS server is considered, are a single unit to which to deliver messages. It cannot, itself, be divided into partitions, only deeper layers of the hierarchy added to divide the Subject space.

Thanks to this, and the expressive hierarchy, a NATS message can uniquely identify many categorizations or identities of a message completely decoupled from consumption. The cardinality of the Subject has little role in subscriber distribution, or, to a degree, the proper ordering of related messages.

Where in the Kafka example my Key might be Ford, and that dictates its final partition and consequently its assigned consumer. The NATS message might have the Subject inventory.update.{make}.{model}.{year} and the subscribing group would express interest in inventory.update.Ford.> and within a Queue Group of 100 subscribers the message volume would be evenly distributed.

Consumption Superpower: Subjects vs Topics

The real expressive power of Subjects can be seen in the way we consume them and how that differs from something like a Kafka Topic. A Kafka Topic represents a single category of message(s) and their associated type(s). A Kafka Topic like factory1-machine-status would reasonably be expected to provide a stream of Machine Status updates from Factory 1. Likely we would key this Topic on something like MachineId. If we only care about a single Status, say Faulted, the client side consumer would need to do that filtering, still receiving all messages yet discarding most. Additionally, another Factory, say factory2, would be an entirely separate and discrete Topic.

The Subject in NATS however can represent and yield the entire Domain Structure however the engineer sees fit. With a Subject template like {factory}.{id}.{area}.{id}.{machine}.{id}.status.{status} we could express a subscription like the previous example with factory.*.area.*.machine.*.status.*. More importantly we can express the exact consumer interest that’s relevant. With a Topic of factory1-machine-status, both producer and consumers are constrained and coupled to all Statuses, all Machines, at Factory 1. The Subject hierarchy, however, decouples the intent of publishers from the interest of subscribers.

We could have a single publisher handling all Machines and a single Status: Nominal, Degraded, Faulted, etc. And on the other end have a single subscriber per Machine and all Statuses, or factory.1.area.1.machine.1.status.*, factory.1.area.1.machine.2.status.*, etc. Going a step further, we could also have a publisher sending general state updates for a Factory Area within the same Subject space, i.e. published to factory.1.area.{id}. These could be simply describing the details of the Factory Area, e.g. name, number of units, geofence, etc. What’s powerful is that this entire Subject space can then be ingested by a JetStream Stream and persisted as a single consistent unit. The Stream would subscribe to factory.*.area.*.machine.*.status.*, and consumers could view into any slice of this space necessary, e.g. factory.*.area.* - for general Area updates at all Factories, factory.1.area.*.machine.* - for updates from Factory 1, all Areas and all Machines, etc. The immense flexibility can be hard to get used to but with a proper hierarchy, filters and wildcards, you can express some very complex queries quite simply.

Wrap Up

A well formed Subject hierarchy is a complex topic with many areas for exploration. Hopefully, this at least gets you thinking in broader terms than the strict categorization of a Topic and excited to explore the malleable world of Subject Aware Messaging.

Recapping, we explored the concept of a NATS Subject, including publishing specificity and expressing subscriber interest. We reoriented message routing around addressing, categorizing, and keying messages into the malleable and descriptive nature of hierarchal Subjects. We also looked at the decoupling of publisher intent from subscriber interest, leveraging the Subject space to describe a broad domain while subscribers stay well scoped. It should be clear now, in the NATS world a well thought out and planned Subject hierarchy and template is a critical design decision but one that can evolve with your domain’s needs.

Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!

The technology landscape is full of message brokers and event/message buses: Kafka, Event Hub, Pulsar, SQS, Google Pub/Sub, and more in no particular order. In this crowded space, NATS stands out as a unique perspective of the connectivity problem space. Beyond simply passing messages around, NATS is oriented toward solving application connectivity from a first principles like foundation.

At first glance, NATS is a message bus, it facilitates location transparent Pub/Sub. But upon that, so much more can be built, and complex, nuanced patterns can be expressed. Basic Fan-Out/In Pub/Sub, Work Queue Distribution, Request/Reply semantics, Weighted & Mapped Routing, etc. Capabilities like this allow us a new way to approach the connectivity problem space. When we might traditionally reach for HTTP REST, or complex load balanced ingress, NATS provides an elegant alternative. And this is just with Core NATS before we dig into JetStream persistence, Key-Value Semantics, Object Storage, NEX Workloads, etc.

Where do we start?

In this introduction we’re going to accomplish three main objectives:

1. Install our tooling and stand up a NATS server

2. Pub/Sub messages using Subject based addressing

3. Execute Request/Reply built on top of Pub/Sub

Tooling

NATS CLI

First, install the NATS CLI tool, this is the unified tool to interact with the NATS server. From the CLI we can Pub/Sub, Request/Reply, manage Streams, Benchmark, etc. The tool can be installed a variety of ways as documented in the GitHub README and the binaries are available in GitHub Releases. For instance, you can leverage go and install the latest with

go install github.com/nats-io/natscli/nats@latest

Ensure NATS has been added to your $PATH and with the following you should see the latest version you have

nats --version
0.2.2

Conveniently, in addition to the typical -h, --help the CLI gives a selection of “cheat sheets” via nats cheat {cheat}. Looking through these will really help understanding how to work with the tool and what can be done with it.

nats cheat
Available Cheats:
    account
    auth
    bench
    consumer
    contexts
    errors
    events
    kv
    latency
    obj
    pub
    reply
    schemas
    server
    stream
    sub

NATS Server

Next, you’ll need the nats-server, this is the core of it all and a single feature packed binary. The binary itself can be downloaded via NATS.io, alternatively nats-server can be easily spun up in Docker/Podman. Here, I’ll use Docker and we’re going to configure a couple of items.

1. We’ll expose the default NATS port 4222 and the default monitoring port 8222

2. Name the NATS server instance with -n nats1

3. Enable NATS monitoring via -m 8222

docker run --rm --name my-nats -p "4222:4222" -p "8222:8222" nats -n nats1 -m 8222

[1] [INF] Starting nats-server
[1] [INF]   Version:  2.11.1
[1] [INF]   Git:      [d78523b]
[1] [INF]   Name:     nats1
[1] [INF]   ID:       ND6JF5FBLCHPSUBLSAJRLSHWANFQ2D6RJWSK3XJDFBIX5RTE3NQDH5ER
[1] [INF] Starting http monitor on *******:8222
[1] [INF] Listening for client connections on *******:4222
[1] [INF] Server is ready

Pub/Sub - NATS CLI

With our NATS server running, let’s begin with the simplest expression of NATS, Pub/Sub. Open three new terminals, two for subscription and one for publishing.

Next, separately begin two subscriptions with nats sub

nats sub some.subject

Executing this in both subscription terminals should yield some like this

Finally, let’s publish a message with nats pub and see it delivered to both subscribers.

From the NATS Pub terminal execute

nats pub some.subject "Hello World"

Altogether, we should see the message published and received by both subscribers

Adding Headers

Just for fun, let’s add a Header to our message, similar to HTTP Headers, these are Key/Value pairs that support multiple values. These can be used to pass metadata, or as we get more advanced can enable deduplication, optimistic concurrency, etc.

Execute the following nats pub command, passing a generic Content-Type header

nats pub some.subject --header "Content-Type:text/plain" "Hello World"

Altogether we should see the new output

Using Queue Groups

Previously, the published message was delivered to all subscribers, i.e. if we had 100 subscribers, every one would receive the message. However, sometimes we want to distribute messages to a group of subscribers. This is similar to the concept of a work queue, where each message is handled by one and only one subscriber.

To express this pattern in NATS we use a Queue Group. Defined on the subscription, all subscribers within the same group will share responsibility for a subject. NATS will deliver each message to one subscriber in via a nearly random selection that tends towards an even distribution.

Seeing it action, start two new subscriptions with the following command

nats sub some.subject --queue my_group

Then we can publish again

nats pub some.subject "Hello World"

And in result we should see that only one subscriber received the message

Note though, subscriptions outside the Queue Group and in other groups, are still delivered the message

Request/Reply - NATS CLI

Another common communication pattern is Request/Reply, where we expect a single reply for a given request. While this pattern is engrained within something like an HTTP call, NATS expresses this pattern from the basis of Pub/Sub. A Request is published while a temporary Inbox is formed and subscribed to by the “Requestor”, any subscriber can then Reply to the Request on the provided Inbox subject.

Let’s see it in action! First, open a new terminal and stand up a “Responder” to Requests

nats reply some.subject "Responded @ {{Time}}"

Then, from our NATS Pub terminal issue the Request

nats request some.subject "Hello World"

And we should see the Reply returned back to our requestor as expected

We’ll discuss this deeper when we code a Request/Reply pattern, but this illustrates the basic concept. Also note, that we published on the same Subject as before, our earlier Subscribers still received the new message, although they aren’t going to respond to our Request, still, interest in the Subject is respected and the message delivered.

Wrap Up

In this introduction we got familiar with standing up a NATS Server via Docker. Next, we got acquainted with the NATS CLI and its basic Pub/Sub, Request/Reply capabilities. With just a few commands, a single container, we’re already publishing and subscribing with location transparent and subject aware messaging. Our NATS Server, Publisher, and Subscribers can easily be globally remote from one another, and given access, could just as easily communicate. Clearly, we’re just beginning so stay tuned and we’ll keep exploring.

Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!

Previously, we've covered creating, authenticating, writing and testing against an Azure Service Bus. The next step on our journey is to read continuously using a ServiceBusProcessor. This component triggers event handlers and is itself very similar to a BackgroundService.

What is the ServiceBusProcessor?

The ServiceBusProcessor is a high-level abstraction around a set of ServiceBusReceivers that allows a consumer to continuously receive messages. The processor comes in two varieties: a standard processor for Queues/Topics, and a Session enabled processor that manages the locks around Service Bus Sessions [stay tuned 🤓].

The processor takes a pair of event handlers: one to handle the message, and another to handle any errors thrown during handling a message. With this model of processing, each message is managed through the event args passed to each delegate: ProcessMessageEventArgs and ProcessErrorEventArgs respectively. Through these arguments the handler can execute any of the typical Receiver operations, e.g. Abandon, Dead Letter, etc.

args.AbandonMessageAsync(args.Message);
args.DeadLetterMessageAsync(args.Message);

What We'll Build

We're going to build the core of what could become an Event-Driven microservice. This service will consume continuously from Azure Service Bus and delegate the messages to strongly typed handlers. The consuming process will be a Hosted Service derived from a BackgroundService.

Why a Background Service?

Why not just use a basicIHostedServiceimplementation?

Well, the key difference between an IHostedService and a BackgroundService lies in how the framework itself treats these components. A BackgroundService exposes a virtual nullable Task representing the execution of the Hosted Service. This is important because it gives the Host knowledge of your code's execution. Critically, the continuously running service is awaited and any exceptions are propagated out of the background to the Host.

Project Setup

First, dotnet new up a blank console application and install the following packages:

dotnet add package Azure.Identity
dotnet add package Azure.Messaging.ServiceBus
dotnet add package Microsoft.Extensions.Hosting
dotnet add package Bogus

The Queue Reader

To kickoff, the QueueReader will be a generic BackgroundService defined as below, including all usings:

using Azure.Messaging.ServiceBus;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using static System.Threading.CancellationTokenSource;

namespace ConcurrentFlows.AzureBusSeries.Part4.Reader;

public sealed class QueueReader<T>
    : BackgroundService,
    IAsyncDisposable
{
}

Next, we'll declare our dependencies and any necessary members:

public sealed class QueueReader<T>
    : BackgroundService,
    IAsyncDisposable
{
    private readonly ILogger<QueueReader<T>> logger;
    private readonly ServiceBusProcessor processor;
    private readonly IMessageHandler<T> handler;

    private CancellationTokenSource? stoppingCts;
}

We have a logger as a general practice, our key ServiceBusProcessor that will read from the queue, and a MessageHandler that will be passed any message received.

Next, our ServiceBusProcessor requires the definition of two event handlers: ProcessMessageAsync and ProcessErrorAsync. We'll keep these simple and compact. Process Message will delegate to the MessageHandler including cancelleation. Process Error will log the error details and move on. We don't need to directly Ack/Nack or Complete/Abandon/DLQ the message as we're using the Auto Complete feature of the processor by default.

private Task ProcessMessageAsync(ProcessMessageEventArgs args)
{
    var body = args.Message.Body;
    var obj = body.ToObjectFromJson<T>();
    var cts = CreateLinkedTokenSource(stoppingCts!.Token, args.CancellationToken);
    return handler.HandleAsync(obj, cts.Token);
}

private Task ProcessErrorAsync(ProcessErrorEventArgs args)
{
    logger.LogWarning("Error Processing {@Error}",
        new
        {
            args.Identifier,
            ErrorSource = $"{args.ErrorSource}",
            Exception = $"{args.Exception}"
        });
    return Task.CompletedTask;
}

Then, as a BackgroundService we need to implement the abstract ExecuteAsync method. The way we do this is key to interacting with our Host. A common pitfall is not recognizing that the Task returned from ExecuteAsync represents the lifetime of the BackgroundService.

The ServiceBusProcessor exposes two methods we're concerned with here: StartProcessingAsync and StopProcessingAsync. Each of these methods return a Task representing the operation itself, i.e. Starting or Stopping, but neither represents the ongoing processing. If we were to simply await StartProcessingAsync and leave it at that, our Host would consider the service complete, i.e. it's no longer running.

What we want is to Start Processing, and then, hold until the service is commanded to shut down, then we'll Stop Processing. To do this we'll create a helper extension method:

public static Task CompleteOnCancelAsync(
    this CancellationToken token)
{
    var tcs = new TaskCompletionSource();
    token.Register(t =>
    {
        if (t is TaskCompletionSource tcs)
            tcs.TrySetResult();
    }, tcs);
    return tcs.Task;
}

This converts the cancellation signal from a token into the completion of a Task.

With this in hand we can implement ExecuteAsync

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    stoppingCts = CreateLinkedTokenSource(stoppingToken);
    await processor.StartProcessingAsync(CancellationToken.None);

    await stoppingToken.CompleteOnCancelAsync();

    stoppingCts.Cancel();
    await processor.StopProcessingAsync(CancellationToken.None);
}

Within this method, we first create the linked token source that is later passed to our MessageHandler. Then we Start, await a shutdown signal, then cancel any outstanding processing, and finally Stop processing. The rest of the lifetime boiler plate is already within the BackgroundService.

Finally, we implement our constructor to inject dependencies and assign the event handling delegates. Also, we'll wrap everything up with async disposal.

public QueueReader(
    ILogger<QueueReader<T>> logger,
    ServiceBusProcessor processor,
    IMessageHandler<T> handler)
{
    this.logger = logger.ThrowIfNull();
    this.processor = processor.ThrowIfNull();
    this.handler = handler.ThrowIfNull();

    processor.ProcessMessageAsync += ProcessMessageAsync;
    processor.ProcessErrorAsync += ProcessErrorAsync;
}

public async ValueTask DisposeAsync()
{
    await processor.DisposeAsync();
    stoppingCts?.Dispose();
    base.Dispose();
}

All together now!

public sealed class QueueReader<T>
    : BackgroundService,
    IAsyncDisposable
{
    private readonly ILogger<QueueReader<T>> logger;
    private readonly ServiceBusProcessor processor;
    private readonly IMessageHandler<T> handler;

    private CancellationTokenSource? stoppingCts;

    public QueueReader(
        ILogger<QueueReader<T>> logger,
        ServiceBusProcessor processor,
        IMessageHandler<T> handler)
    {
        this.logger = logger.ThrowIfNull();
        this.processor = processor.ThrowIfNull();
        this.handler = handler.ThrowIfNull();

        processor.ProcessMessageAsync += ProcessMessageAsync;
        processor.ProcessErrorAsync += ProcessErrorAsync;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        stoppingCts = CreateLinkedTokenSource(stoppingToken);
        await processor.StartProcessingAsync(CancellationToken.None);

        await stoppingToken.CompleteOnCancelAsync();

        stoppingCts.Cancel();
        await processor.StopProcessingAsync(CancellationToken.None);
    }

    private Task ProcessMessageAsync(ProcessMessageEventArgs args)
    {
        var body = args.Message.Body;
        var obj = body.ToObjectFromJson<T>();
        var cts = CreateLinkedTokenSource(stoppingCts!.Token, args.CancellationToken);
        return handler.HandleAsync(obj, cts.Token);
    }

    private Task ProcessErrorAsync(ProcessErrorEventArgs args)
    {
        logger.LogWarning("Error Processing {@Error}",
            new
            {
                args.Identifier,
                ErrorSource = $"{args.ErrorSource}",
                Exception = $"{args.Exception}"
            });
        return Task.CompletedTask;
    }

    public async ValueTask DisposeAsync()
    {
        await processor.DisposeAsync();
        stoppingCts?.Dispose();
        base.Dispose();
    }
}

Using the Reader

To use the QueueReader we'll use our existing Service Bus Queue, create a message, a QueueWriter, a MessageHandler, and stand everything up in Host.

First, the message we'll send looks like this notification, just an id and string content:

public sealed record Notification(
    int Id,
    string Content);

Then the IMessageHandler implementation receives the Notification and logs it:

public sealed class NotificationHandler
    : IMessageHandler<Notification>
{
    private readonly ILogger<NotificationHandler> logger;

    public NotificationHandler(
        ILogger<NotificationHandler> logger)
        => this.logger = logger.ThrowIfNull();

    public Task HandleAsync(Notification message, CancellationToken cancelToken)
    {
        logger.LogInformation("Received Message:{NewLine}{Message}", 
            NewLine, message);
        return Task.CompletedTask;
    }
}

The next major component is the QueueWriter, which is very similar to the QueueSender we created before. The main difference being this one sends a configured batch of messages. Also, we throw in a little Bogus to get a fun, friendly, random name for the message.

public sealed class QueueWriter
    : BackgroundService,
    IAsyncDisposable
{
    private readonly ILogger<QueueWriter> logger;
    private readonly ServiceBusSender sender;
    private readonly int count;
    private readonly Faker faker = new();

    public QueueWriter(
        ILogger<QueueWriter> logger,
        ServiceBusSender sender,
        int count = 5)
    {
        this.logger = logger.ThrowIfNull();
        this.sender = sender.ThrowIfNull();
        this.count = count;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(5));
        using var cts = CreateLinkedTokenSource(stoppingToken);

        var messages = Enumerable.Range(1, count)
            .Select(id =>
            {
                var name = faker.Name.FirstName();
                var notification = new Notification(id, $"Hello from {name}");
                var body = JsonSerializer.Serialize(notification);
                var message = new ServiceBusMessage(body);
                return message;
            });

        await sender.SendMessagesAsync(messages, stoppingToken);
        logger.LogInformation("Finished");
    }

    public async ValueTask DisposeAsync()
    {
        await sender.DisposeAsync();
        base.Dispose();        
    }
}

Setting up the Host

First, we'll setup credentials the same as we configured them earlier in Part 2 - Default Credentials

public static class CredentialBuilder
{
    public static DefaultAzureCredential CreateDefaultCredential(
        this IConfiguration config)
    {
        var tenantId = config["TenantId"];
        return new(new DefaultAzureCredentialOptions()
        {
            TenantId = tenantId
        }.SetVisualStudioCredentialingOnly());
    }

    public static DefaultAzureCredentialOptions SetVisualStudioCredentialingOnly(
        this DefaultAzureCredentialOptions options)
    {
        options.ExcludeAzureCliCredential = true;
        options.ExcludeAzureDeveloperCliCredential = true;
        options.ExcludeAzurePowerShellCredential = true;
        options.ExcludeEnvironmentCredential = true;
        options.ExcludeInteractiveBrowserCredential = true;
        options.ExcludeVisualStudioCodeCredential = true;
        options.ExcludeWorkloadIdentityCredential = true;
        options.ExcludeManagedIdentityCredential = true;
        options.ExcludeSharedTokenCacheCredential = true;
        return options;
    }
}

Then we have the registrations for our Service Bus components

public static class Registrations
{
    private const string Identifier = "AzureBusSeries";

    public static IServiceCollection AddAzureBusComponents(
        this IServiceCollection services)
        => services
        .AddAzureBusClient()
        .AddAzureBusSender()
        .AddAzureBusProcessor();

    private static IServiceCollection AddAzureBusClient(
        this IServiceCollection services)
        => services.AddSingleton(sp =>
        {
            var config = sp.GetRequiredService<IConfiguration>();

            var credential = config.CreateDefaultCredential();
            var hostName = config["ServiceBusHost"];

            var options = new ServiceBusClientOptions()
            {
                TransportType = ServiceBusTransportType.AmqpWebSockets,
                Identifier = $"{Identifier}-Client"
            };

            return new ServiceBusClient(hostName, credential, options);
        });

    private static IServiceCollection AddAzureBusSender(
        this IServiceCollection services)
        => services.AddSingleton(sp =>
        {
            var config = sp.GetRequiredService<IConfiguration>();
            var client = sp.GetRequiredService<ServiceBusClient>();
            var queue = config["Queue"];

            var options = new ServiceBusSenderOptions()
            {
                Identifier = $"{Identifier}-Writer"
            };

            return client.CreateSender(queue, options);
        });

    private static IServiceCollection AddAzureBusProcessor(
        this IServiceCollection services)
        => services.AddSingleton(sp =>
        {
            var config = sp.GetRequiredService<IConfiguration>();
            var client = sp.GetRequiredService<ServiceBusClient>();
            var queue = config["Queue"];

            var options = new ServiceBusProcessorOptions()
            {
                Identifier = $"{Identifier}-Reader"
            };

            return client.CreateProcessor(queue, options);
        });
}

And finally, we pull it all together in our Program.cs

var builder = Host.CreateApplicationBuilder();

builder.Configuration.AddUserSecrets<Program>();

builder.Logging.AddConsole();

builder.Services
    .AddHostedService<QueueWriter>()
    .AddHostedService<QueueReader<Notification>>()
    .AddSingleton<IMessageHandler<Notification>, NotificationHandler>()
    .AddAzureBusComponents();

var app = builder.Build();

await app.RunAsync();

Execute!!

If you've got everything set up right and secrets configured, run the app and you'll see something like this logged to the console

info: ConcurrentFlows.AzureBusSeries.Part4.Writer.QueueWriter[0]
      Finished
info: ConcurrentFlows.AzureBusSeries.Part4.Handlers.NotificationHandler[0]
      Received Message:
      Notification { Id = 1, Content = Hello from Caden }
info: ConcurrentFlows.AzureBusSeries.Part4.Handlers.NotificationHandler[0]
      Received Message:
      Notification { Id = 2, Content = Hello from Rylan }
info: ConcurrentFlows.AzureBusSeries.Part4.Handlers.NotificationHandler[0]
      Received Message:
      Notification { Id = 3, Content = Hello from Rubie }
info: ConcurrentFlows.AzureBusSeries.Part4.Handlers.NotificationHandler[0]
      Received Message:
      Notification { Id = 4, Content = Hello from April }
info: ConcurrentFlows.AzureBusSeries.Part4.Handlers.NotificationHandler[0]
      Received Message:
      Notification { Id = 5, Content = Hello from Katelin }

Wrap Up & Repo

Here, we built a BackgroundService to host the ServiceBusProcessor. This kept the framework informed of our processing lifetime, delegated the message handling to a dependency and flowed all cancellation signals. Also, we built out the supporting components and gave our new QueueReader a test drive. Hopefully, this pattern is useful the next time you're creating a new event-driven dotnet app!

Happy Messaging!

Of course, all code is available on GitHub ⇒ ConcurrentFlows.AzureBusSeries

If there's anything specific you'd like covered regarding Service Bus, please drop an ask in the comments!

The previous post, Part 2: Authentication and Sending to a Queue, covered authenticating to and writing to a Service Bus Queue. To do this we used the QueueSender background service. While this worked, it begs the question; How do we test when integrating Azure Service Bus? That's exactly what we'll answer here!

QueueSender - Our SUT

Our System Under Test for this exercise is the previously created QueueSender. This background service starts up, sends a single message, and completes. The entire service, from Part 2, is given below:

public class QueueSender
    : BackgroundService
{
    private readonly ILogger<QueueSender> logger;
    private readonly ServiceBusSender sender;

    public QueueSender(
        ILogger<QueueSender> logger,
        ServiceBusSender sender)

    {
        this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
        this.sender = sender ?? throw new ArgumentNullException(nameof(sender));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(5));
        using var cts = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, timeout.Token);
        try
        {
            await sender.SendMessageAsync(new("Hello World"), cts.Token);

            logger.LogInformation("Sent Message!!!");
        }
        catch (OperationCanceledException ex) 
            when (timeout.IsCancellationRequested)
        {
            logger.LogWarning(ex, "Operation timed out");
            throw;
        }
        catch (OperationCanceledException ex) 
            when (stoppingToken.IsCancellationRequested)
        {
            logger.LogInformation(ex, "Shutdown early");
            throw;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Unhandled exception");
            throw;
        }
    }
}

What We'll Test

This simple service presents a couple of unique challenges: it's a background service, and we're using Service Bus as a message broker. Testing a background service can be difficult since the only public methods are StartAsync & StopAsync, this means we only have indirect access to the service. Testing with Service Bus is itself difficult since it cannot be ran locally and any test against it must treat it as the shared resource that it is.

With these considerations in mind, we'll focus our testing on integration and verify the following:

1. Does our system connect to our Service Bus?

2. Does the specified Queue exist?

3. Can we receive the message that we expect to be sent?

xUnit Project Setup

To start we'll create a new xUnit test project via the following:

dotnet new xunit -o Part3

Next, add a reference to the project with our SUT, in my case Part2.csproj

dotnet add reference Part2.csproj

Then, go ahead and delete UnitTest1.cs

Admin Client Creation

The key to integration testing an Azure Service Bus is the class ServiceBusAdministrationClient and the functionality it provides. From this client you can control, verify, and manipulate virtually anything about your Service Bus that you could via an IaC template or the Azure Portal itself.

Creating an Admin Client is as easy as creating a standard Service Bus Client. You only need the fullyqualifiedNamespace, a tokenCredential, and the default ServiceBusAdministrationClientOptions will work fine for our purpose.

new ServiceBusAdministrationClient(hostName, credentials, adminOptions);

Note though, the Identity and Credentials you use must have permissions to conduct the operations you require. The simple identity we set up previously will work for our tests.

Creating the Test Fixture

For each of our 3 tests outlined above, we'll leverage the same underlying Test Fixture. This fixture will provide access to a fresh IConfiguration, our ServiceBusAdminClient, and provide a ServiceBusReceiver.

public sealed class ServiceBusFixture
    : IAsyncLifetime
{
    private readonly ServiceBusClient client;

    public IConfiguration Config => BuildConfig();
    public ServiceBusAdministrationClient AdminClient { get; }

    public ServiceBusFixture()
    {
        var credentials = Config.CreateDefaultCredential();
        var hostName = Config["ServiceBusHost"];

        client = new ServiceBusClient(hostName, credentials, new()
        {
            TransportType = ServiceBusTransportType.AmqpWebSockets,
            Identifier = $"Test-Client"
        });

        AdminClient = new ServiceBusAdministrationClient(hostName, credentials, new());
    }

    public ServiceBusReceiver GetReceiver(string queue)
        => client.CreateReceiver(queue, options: new()
        {
            ReceiveMode = ServiceBusReceiveMode.ReceiveAndDelete,
            Identifier = $"Test-Receiver"
        });

    private static IConfiguration BuildConfig()
        => new ConfigurationBuilder()
        .AddUserSecrets<ServiceBusFixture>()
        .Build();

    public Task InitializeAsync()
        => Task.CompletedTask;

    public async Task DisposeAsync()
        => await (client?.DisposeAsync() ?? ValueTask.CompletedTask);
}

The Test Fixture also implements the IAsyncLifetime interface to properly dispose of the Service Bus Client after the tests complete.

Testing the Connection and Queue

Our first two tests can be combined;

1. Does our system connect to our Service Bus?

2. Does the specified Queue exist?

If we check for the existence of the specified Queue, we'll also ensure our connection to the Service Bus is working.

To test this, we'll create a new test file ServiceBusConnectionTests.cs and bring in our Test Fixture. From there we'll use the Admin Client to check for existence of the Queue within a given timeout since the operation is async and can fail.

public sealed class ServiceBusConnectionTests
    : IClassFixture<ServiceBusFixture>
{
    private readonly IConfiguration config;
    private readonly ServiceBusAdministrationClient adminClient;

    public ServiceBusConnectionTests(ServiceBusFixture fixture)
    {
        ArgumentNullException.ThrowIfNull(fixture);

        config = fixture.Config
            ?? throw new ArgumentNullException(nameof(fixture.Config));
        adminClient = fixture.AdminClient
            ?? throw new ArgumentNullException(nameof(fixture.AdminClient));
    }

    [Fact]
    public async Task Can_Connect_And_Queue_Exists()
    {
        using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(3));        
        var queueName = config["Queue"];

        var result = await adminClient.QueueExistsAsync(queueName, cts.Token);

        Assert.True(result);
    }
}

Testing Sending and Receiving

Our third test is a little more complicated:

3. Can we receive the message that we expect to be sent?

We'll actually be creating an instance of our SUT, the QueueSender, and ensuring the message that we expect to be sent can then be received. Our test follows essentially four steps:

1. Create a temporary test queue and point our config to it

2. Create and start the background serviceQueueSender

3. Receive any message from the test queue

4. Shutdown theQueueSenderand cleanup the test queue

To begin, create a new test class ServiceBusSenderTests.cs and inject the Test Fixture:

public sealed class ServiceBusSenderTests
    : IClassFixture<ServiceBusFixture>,
    IAsyncLifetime
{    
    private readonly ServiceBusAdministrationClient adminClient;
    private readonly IConfiguration config;
    private readonly ServiceBusReceiver receiver;
    private readonly string testQueue;
    private QueueProperties queueProps = default!;    

    public ServiceBusSenderTests(ServiceBusFixture fixture)
    {
        ArgumentNullException.ThrowIfNull(fixture);

        adminClient = fixture.AdminClient
            ?? throw new ArgumentNullException(nameof(fixture.AdminClient));
        config = fixture.Config
            ?? throw new ArgumentNullException(nameof(fixture.Config));

        testQueue = $"Test-{Guid.NewGuid()}";
        receiver = fixture.GetReceiver(testQueue);
    }
}

We generate a test queue name with the random UUID prefixed by 'Test' to identify it in case it isn't cleaned up. Then, we pull a new ServiceBusReceiver from our fixture pointed toward that test queue.

Then, we'll use the implementation of IAsyncLifetimeto get the existingQueuePropertiesduringInitializeAsync, this is an important step. We could just create any random new queue, however, for our test to really ensure our shared queue is setup correctly, we need to duplicate it with its existing properties, only renamed.

public async Task InitializeAsync()
{
    using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(3));
    var queueName = config["Queue"];
    var propsResponse = await adminClient.GetQueueAsync(queueName, cts.Token);
    queueProps = propsResponse.Value;
}

Then in our DisposeAsync we'll use the adminClient to clean up the test queue if it has been created.

public async Task DisposeAsync()
{
    if (adminClient is null)
        return;

    using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(3));        
    var queueExists = await adminClient.QueueExistsAsync(queue, cts.Token);
    if (queueExists)
        await adminClient.DeleteQueueAsync(queue, cts.Token);
}

Next, we have a helper method to build the QueueSender. This sets up a Service Collection to register all dependencies and then create our QueueSender based on the config instance we will point to our test queue.

private QueueSender BuildQueueSender(IConfiguration config)
    => new ServiceCollection()
    .AddSingleton(config)
    .AddSingleton<QueueSender>()
    .AddServiceBusForQueueSender()
    .AddLogging()
    .BuildServiceProvider()
    .GetRequiredService<QueueSender>();

Finally, in our test implementation we execute the following steps:

1. Update the config and create the test queue

2. Create and start the QueueSender

3. Receive and assert against any messages

4. Stop the QueueSender

We do this all within a 10 second timeout that ensures our test doesn't hang for too long and will move to clean up the test queue eventually.

[Fact]
public async Task Can_Send_And_Receive_Message()
{
    using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
    config["Queue"] = testQueue;        
    await adminClient.CreateQueueAsync(new CreateQueueOptions(queueProps)
    {
        Name = testQueue
    }, cts.Token);

    var sender = BuildQueueSender(config);
    await sender.StartAsync(cts.Token);

    var msg = await receiver.ReceiveMessageAsync(TimeSpan.FromSeconds(2), cts.Token);

    Assert.NotNull(msg);
    var expecetd = "Hello World";
    Assert.Equal(expecetd, $"{msg.Body}");

    await sender.StopAsync(cts.Token);
}

Wrap Up & Repo

This was solid introduction to the ServiceBusAdministrationClient, many users of Azure Service Bus never get introduced to this very powerful component. Here, we've leveraged these capabilities to integration test our connection with Service Bus, the existence of a Queue, and ultimately to verify the functionality of a background service. With this approach, multiple engineers and CI/CD pipelines can execute these tests against our shared Service Bus resource without conflicting and causing flaky test results.

Of course, all code is available on GitHub ⇒ ConcurrentFlows.AzureBusSeries

If there's anything specific you'd like covered regarding Service Bus, please drop an ask in the comments!

The previous post, Part 1: Creating an Azure Service Bus and Sending a Queue Message, covered setting up a basic Service Bus namespace, creating a simple queue and publishing a message via the Azure Portal. In this next part we'll begin interacting with the Azure Resources via code in C#, dotnet 8, setup auth with Azure Entra, and use an Azure Credential to access our namespace.

Authenticating

With Azure Service Bus there's two primary ways to authenticate a client app: Credentials w/ RBAC, and a Keyed Connection String. Standard best practice is to use a Credential via the DefaultAzureCredential This allows us to ultimately leverage a Managed Identity when our app is deployed to production, keeping our secrets out of configs, out of repos, and in the safest place they can be. That said, a simple connection string with a keyed access policy is an easy way to get started, but we won't cover that further here.

Understanding the Default Azure Credential

The Default Azure Credential is a composite of many dedicated credentialing options. Among them you have Visual Studio integration, CLI tools, and many more:

• EnvironmentCredential

• WorkloadIdentityCredential

• ManagedIdentityCredential

• SharedTokenCacheCredential

• VisualStudioCredential

• VisualStudioCodeCredential

• AzureCliCredential

• AzurePowerShellCredential

• AzureDeveloperCliCredential

• InteractiveBrowserCredential

[1] DefaultAzureCredential - Microsoft Learn

Wow, right!? What this allows us to do is setup our local dev environment using one Credential, while our deployed app leverages the Managed Identity. The Managed Identity itself does not exist beyond Azure. That's right, any attempt to authenticate via a Managed Identity outside of Azure results in failure to reach the protected 169.254.169.254:80 token host.

Configuring our Service Bus namespace

Today, we'll focus on getting a simple local app authenticated to our Service Bus using the Visual Studio integration, but similar steps would be taken with the other user level credentialing options. We're not deploying today, so we'll table the full-fledged Managed Identity for another time, our focus is Service Bus, not Entra fun.

For reference, this is also well covered via Microsoft Learn ⇒ Quickstart - Assign roles to your Microsoft Entra user, but we'll cover a little extra so ride with me here.

Our first step is to reach the Access Control section of Service Bus. This is done by clicking 'Access Control (IAM)' on the left ribbon. Then we'll 'Add role assignment' from the '+ Add' drop down.

With the selection of Roles in view, we have three built-in Roles to choose from

• Azure Service Bus Data Owner: Use this role to allow full access to Service Bus namespace and its entities (queues, topics, subscriptions, and filters)

• Azure Service Bus Data Sender: Use this role to allow sending messages to Service Bus queues and topics.

• Azure Service Bus Data Receiver: Use this role to allow receiving messages from Service Bus queues and subscriptions.

[2] Azure built-in roles for Azure Service Bus | Microsoft Learn

For our purposes, choose 'Azure Service Bus Data Owner' and select 'Next' at the bottom.

Now, we'll assign access to 'User, group, or service principal' by clicking '+ Select Members'. Next, if your identity is not already seen, use the search feature to find the correct Identity, select them and then we'll click 'Select' at the bottom of the dialog.

With the identity selected, click 'Review + assign' to move onto the final step and review your choices.

During the review, ensure the scope of privilege is correct and you've chosen the correct role to assign the desired Identity. Finally, click 'Review + assign' once more:

And that's it! The chosen Identity now has the Azure Service Bus Data Owner role, granting full access to the Service Bus namespace and its entities.

With that configured, we can now start building our first app, finally some code!

App Building Time!

We're going to create a simple hosted application that authenticates with our DefaultAzureCredential and sends a single 'Hello World' message to our existing Service Bus Queue.

Preparatory; Ensure you are signed into Visual Studio with the Identity you've granted Service Bus access to.

Project Stand Up

Moving right along, create an empty .Net 8 console app to work in, any way you like is fine:

dotnet new console --framework net8.0

Next up we're going to need some packages:

dotnet add package Azure.Identity
dotnet add package Azure.Messaging.ServiceBus
dotnet add package Microsoft.Extensions.Hosting

Default Credentialing

Diving right in now, create a new class to hold our extension methods:

public static class Extensions
{
}

First to go in here will be our Credential builder, the DefaultAzureCredential will of course attempt a variety of authenticating methods until one succeeds. Typically, you'd have a local Credential and an Azure deployed Credential, but here we are simply sticking with the VisualStudioCredential attached to your sign-in. So, we'll exclude all of the Credentials we don't need:

public static DefaultAzureCredentialOptions SetVisualStudioCredentialingOnly(this DefaultAzureCredentialOptions options)
{
    //Not strictly necessary but less is more
    options.ExcludeAzureCliCredential = true;
    options.ExcludeAzureDeveloperCliCredential = true;
    options.ExcludeAzurePowerShellCredential = true;
    options.ExcludeEnvironmentCredential = true;
    options.ExcludeInteractiveBrowserCredential = true;
    options.ExcludeVisualStudioCodeCredential = true;
    options.ExcludeWorkloadIdentityCredential = true;
    options.ExcludeManagedIdentityCredential = true;
    options.ExcludeSharedTokenCacheCredential = true;
    return options;
}

Next to note about our credentialing is the appropriate Tenant Id. This is a bit buried in the docs, but left to its default Tenant Id the VisualStudioCredential may result in something like this:

Azure.Identity.CredentialUnavailableException: Process "..\IDE\CommonExtensions\Microsoft\Asal\TokenService\Microsoft.Asal.TokenService.exe" has failed with unexpected error: TS003: Error, TS004: Unable to get access token. 'AADSTS50020: User account '{EUII Hidden}' from identity provider 'live.com'

does not exist in tenant 'Microsoft Services'

Clearly the default Tenant here, emphasis mine, is incorrect. Following the underlying error code you'll find this ⇒ Error AADSTS50020 - ... | Microsoft Learn which isn't all that helpful in this scenario. What needs to happen is our credentialing method needs the correct Tenant Id passed to it. You can find your Tenant Id with the helpful Find tenant ID | Microsoft Learn. Simply searching for Entra will get you to the 'Default Directory', or similar in your setup, where you'll find the Tenant Id you need:

You'll take that Tenant Id and either bring it in hardcoded or, as I've done, store it in local secrets.json via Secret Manager, [3] Safe storage of app secrets in development | Microsoft Learn

With that in config, we can finish setting up our Credential:

public static DefaultAzureCredential CreateDefaultCredential(this IConfiguration config)
{
    var tenantId = config["TenantId"];
    return new(new DefaultAzureCredentialOptions()
    {
        TenantId = tenantId
    }.SetVisualStudioCredentialingOnly());
}

Service Bus Container Registration

Next, we'll set up our Service Bus components in the IServiceCollection, again in our Extensions.cs

private const string Identifier = "AzureBusSeries";

public static IServiceCollection AddServiceBusForQueueSender(
    this IServiceCollection services)
    => services
    .AddSingleton(sp =>
    {
        //Resolve the formed Config
        var config = sp.GetRequiredService<IConfiguration>();
        //Create our Credentials using the previously created methods
        var credential = config.CreateDefaultCredential();        
        // {Your namespace}.servicebus.windows.net        
        var hostName = config["ServiceBusHost"];

        var options = new ServiceBusClientOptions()
        {
            //Standard
            TransportType = ServiceBusTransportType.AmqpWebSockets,
            //Identify this client on the bus - best for logging
            Identifier = $"{Identifier}-Client"
        };

        return new ServiceBusClient(hostName, credential, options);
    })
    .AddSingleton(sp =>
    {
        //Resolve the formed Config
        var config = sp.GetRequiredService<IConfiguration>();
        //Resolve the existing client
        var client = sp.GetRequiredService<ServiceBusClient>();
        //Pull the Queue name
        var queue = config["Queue"];

        var options = new ServiceBusSenderOptions()
        {
            //Identify this Sender to the bus - best for logging
            Identifier = $"{Identifier}-Sender"
        };

        return client.CreateSender(queue, options);
    });

The QueueSender

Our hosted app contains a single process, it will be kicked off on app start up, send its message, and finally the app will wait for us to exit. The single process here takes the form of a simple BackgroundService that receives our configured Logger<T> and the registered ServiceBusSender. This keeps the lifetime of both the client and sender controlled via the ServiceCollection/Provider. Never insatiate a Sender per message, you will exhaust your connections and enjoy the failure.

public sealed class QueueSender
    : BackgroundService
{
    private readonly ILogger<QueueSender> logger;
    private readonly ServiceBusSender sender;

    public QueueSender(
        ILogger<QueueSender> logger,
        ServiceBusSender sender)

    {
        this.logger = logger ?? throw new ArgumentNullException(nameof(logger));
        this.sender = sender ?? throw new ArgumentNullException(nameof(sender));
    }

    //Abstract class method impl, executes and ends    
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {        
        using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(5));
        using var cts = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, timeout.Token);
        try
        {
            await sender.SendMessageAsync(new("Hello World"), cts.Token);

            logger.LogInformation("Sent Message!!!");
        }
        catch (OperationCanceledException ex) 
            when (timeout.IsCancellationRequested)
        {
            logger.LogWarning(ex, "Operation timed out");
            throw;
        }
        catch (OperationCanceledException ex) 
            when (stoppingToken.IsCancellationRequested)
        {
            logger.LogInformation(ex, "Shutdown early");
            throw;
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Unhandled exception");
            throw;
        }
    }
}

Tying it Together - Program.cs

Finally, the app standup in our Program.cs is very minimal. We create the App Builder, add our config, setup logging, register services, build the app, and run.

var builder = Host.CreateApplicationBuilder();

builder.Configuration.AddUserSecrets<Program>();

builder.Logging.AddConsole();

builder.Services
    .AddHostedService<QueueSender>()
    .AddServiceBusForQueueSender();

var app = builder.Build();

await app.RunAsync();

Hello World!

If you've been following along, the app should kick off, and logs should roll in like the following:

With that, check your Queue and our greeting message should have arrived:

Wrap Up & Repo

We're getting a bit deeper now. We've successfully authenticated our local app with our Credential. We've setup that process in our Service Bus Access Control and properly provisioned access to our Identity. Finally, we laid a foundation for deeper and more complex interaction with our Service Bus namespace. The first question that comes to my mind; This is fine, but how would you test this? If only Service Bus offered a Docker container 🙃

Of course, all code is available on GitHub ⇒ ConcurrentFlows.AzureBusSeries

If there's anything specific you'd like covered regarding Service Bus, please drop an ask in the comments!

Welcome to my Azure Service Bus series! In this series we'll explore the widely used Azure Service Bus message broker, look at its features and capabilities, and how you can best leverage it in your projects.

My inspiration for this series came from the realization that while Service Bus is widely used in Azure/dotnet shops, there's not a lot of in-depth info and/or what's available is widely spread out and difficult to piece together. Many tutorials stop well short of the coolest features and capabilities that make Service Bus a great choice if you're working in Azure. Come on, why should Kafka get all the love?? 💘

Create the Service Bus

Within Azure everything is a Resource, and a 'Service Bus' takes the form of a Service Bus Namespace. This is essentially where you will send messages that ultimately get delivered to a Queue, Topic/Subscription.

To create the Service Bus Namespace from the portal, first log into your account and/or start your free trial here ⇒ Azure Free Account

Once you're into the portal, simply search for 'Service Bus' in the marketplace:

Next review the details and click 'Create':

Next, you'll follow the creation wizard. Start by entering you're Azure Subscription and the new/existing Resource Group the Namespace will live in.

Then, enter the Namespace name to use. This will become the name of the Service Bus and part of your connection string.

Also, select your local region to deploy the Service Bus Namespace.

Finally, select a pricing tier from the available options detailed here ⇒ Pricing Tiers - Service Bus

I've chosen 'Basic' to get us started, but feel free to go wild! Keep in mind 'Basic' will only let us create Queues, but we can upgrade later.

We'll skip across some nuance by clicking 'Review + Create'. This is where you'll see a summary of what you entered and what will be deployed. When you're satisfied with your selections, click 'Create':

Wait for the deployment to complete, once it's done, you can view the Resource Overview seeing the details of your shiny new Service Bus! I've removed some of my details 😉

And that's it, you now have a Service Bus Namespace!!!

Create a Queue

In Service Bus you have Queues and Topics. Queues are First-In-First-Out, FIFO, and each message will be delivered to one single consumer. A Topic though, has a set of Subscriptions and each Subscription acts as a FIFO Queue. This lets Topics implement Fan-Out or Pub/Sub. We'll get into more options, settings and nuance later but that's the basics.

To make our Namespace a bit more useful we'll start with a simple Queue. Select either '+ Queue' at the top or from the left ribbon:

A creation dialog will appear where you can configure your new Queue. Enter a Name, select the maximum queue size, TTL, Lock Duration, etc. You'll see below I've set the following:

• Max size to the minimum 1 GB

• Max delivery count of 3

• TTL reduced to 10 minutes

• Lock duration of a quick 1 minute

• And you can leave the final two check boxes unchecked

Click 'Create', and Boom! you now have a simple Queue!

First Message, From the Portal

For your first message, let's simply use the portal before we dive into code in the next post.

To send a message first open the details of your Queue by clicking on it from the Overview. From there, enter the 'Service Bus Explorer':

Once in the Explorer, click on 'Send messages'. From here you can send different content types, bodies, and set various properties. For the first one, let's send the classic 'Hello World' greeting:

Once you've clicked 'Send', set your Mode by clicking 'Peek Mode' and switch it to 'Receive Mode', then click on 'Receive messages'. Be sure to select 'ReceiveAndDelete' on the dialog that pops up.

Finally click 'Receive' and you'll be greeted by your message in the kindest way!

Don't you love that 'Paint' styling!?

Wrap Up

Quick and simple kickoff post; we created a Service Bus Namespace, created a Queue, finally sent and received a message on that Queue. Next we'll dive a layer deeper and start interacting with the Service Bus via a dotnet app.

If there's anything specific you'd like covered regarding Service Bus, please drop an ask in the comments!

Coupling comes in many variations and can sneak into a system if care is not taken. Within Software Engineering we can generally say that coupling is -

the degree of interdependence between software modules; a measure of how closely connected two routines or modules are [1]

To break things down in more detail we have -

Data Type Coupling

1. Content Coupling - Directly accessing and changing the data in another module

2. Common Coupling - Two modules sharing a global data structure

3. Data Coupling - A module passing data into another module through parameters or global data

Functional Type Coupling

1. Control Coupling - A module controlling the flow of execution in another module

2. Stamp/Temporal Coupling - Depending on the order of execution of other modules

3. Functional Coupling - A module's output becomes another module's input

Platform Type Coupling

1. Message Coupling - Communicating through a shared message-passing interface

2. Interop Coupling - Modules are dependent directly on a single platform's technology or protocols

But what does this look like as we take our level of abstraction higher?

Coupling Boundary

As we look at Distributed Software Systems we need a higher level of abstraction to examine the coupling between elements. This abstraction is known as the Architectural Quantum. An Architectural Quantum is defined from one perspective as an "independently deployable artifact, with high cohesion, high static coupling and synchronous dynamic coupling" [2]

The relationship to coupling is clear and we can define a Quantum as a bounded system exhibiting, high static coupling and synchronous dynamic coupling. These forms of coupling force the bounded system to be deployed as a unit that may eschew high cohesion.

Coupling, therefore, can cross even well-defined domain boundaries of microservices. Boundary crossing is what causes friction between teams and complex release strategies. Recognizing the underlying Quantum will enable us to eliminate unnecessary dependencies.

The Razor - Knowledge Discipline

Architecting a complex software system, specifically applying Continuous Architecture, is all about managing volatility and maximizing the Quality Attributes of the system. Minimizing and analyzing coupling comes down to Knowledge Discipline*.*

Knowledge Discipline - Knowledge, being both data & behavior, creates Architectural Quantum boundaries defined by singular ownership.

Singular ownership of Knowledge means that only one Quantum can be the source of truth for a domain's behavior or data. Let's look at a tightly coupled eCommerce example.

Knowledge Discipline - In Action

We'll look at the order placement system of, my favorite, Widgets Inc. Below it's clear the Order Service should not be orchestrating the downstream processes.

The Order Service has knowledge of the Inventory Domain/Service, also, it's aware of the Notification Service. Even worse, the Inventory Service is "validating" the Order for the Order Service based on its known inventory.

Due to the extraordinarily tight coupling among all three services, this system becomes a Distributed Monolith. Regardless of service breakdown, the tight coupling forms a single Quantum that must be deployed as a unit. Changes in one system ripple through others. If the individual services cannot be independently deployed, they're not "microservices" anymore. The monolith is simply further apart.

What a mess huh? Let's apply some Discipline between domains 🤓

Here, both the Order & Inventory Services listen for an Order Placed event. Independant of one another they execute their Domain Responsibility. The Order Service records the Order and the Inventory Service updates the Inventory. Neither knows anything about the other. The Notification Service recognizes the two resultant downstream events: Order Recorded & Inventory Updated. Independently of any source of these events, the Notification Service executes its responsibility.

We have, however, traded the Data & Control Coupling for Platform Coupling in the form of a Message Broker. This is a tradeoff we can live with and the reason why comes down to volatility.

The Message Broker we've introduced offers a distinct and high 9s SLA. Also, as infrastructure, it's subject to very little change. New and evolving Business Requirements have none to little effect on the Broker. This provides us the tradeoff of depending on a Message Broker and in return, our domains become independent from one another.

Wrapping Up

Applying Knowledge Discipline gives us a technique to determine if there's an undesired coupling between modules. By identifying knowledge that has multiple owners we can identify hot spots of coupling. Slicing our system in terms of knowledge ownership keeps bounded data & behavior independently deployable and executable.

[1] Wikipedia contributors. (2022, May 6). Coupling (computer programming). Wikipedia. https://en.wikipedia.org/wiki/Coupling_(computer_programming)

[2] N. Ford, M. Richards, P. Sadalage, Z. Dehghani, Software Architecture: The Hard Parts. Sebastopol, CA: O'Reilly, 2022

We're taught early and often that any I/O Bound operations should be done Asynchronously. A general rule of thumb is that anything that takes more than 50ms should be done async. While this is quite true in most cases, sometimes, just seeing an option to run a method async, we default to that implementation without much forethought.

Confluent Kafka Producer

Confluent offers a very nice Apache Kafka integration NuGet package Confluent.Kafka. Within this package you'll find nearly all the bits and pieces you'll need to connect with, consume and produce via Kafka. In particular, we'll look at the behavior of the default implementation of IProducer<TKey, TValue>

To first set the stage we'll pull in some Protobuf tooling via Google.Protobuf and Grpc.Tools NuGet packages. With those, we'll create a message as an instance of Message<TKey, TValue> with the following schema.

syntax = "proto3";

package ConcurrentFlows.KafkaProducer;

message WidgetEvent {
    string Greeting = 1;
}

Benchmark - Environment

We'll need a local Kafka cluster to run our benchmarks against and we can docker-compose.yml one up relatively easily via docker-compose up -d.

In addition to our cluster, we'll initialize two topics via topic-init.sh. One for a Sync Producer and one for an Async Producer.

Within our cluster, we've also stood up a Schema Registry and we'll use the associated Confluent NuGet package Confluent.SchemaRegistry.Serdes.Protobuf This package brings in the necessary bits and pieces to talk to the Schema Registry and serialize/deserialize, (Ser/Des), our Protobuf messages.

Finally, to easily create messages we'll leverage the Bogus NuGet package and create an instance of its Faker; Faker<WidgetEvent> = new();.

Benchmark - Setup

The complete benchmark setup can be found within ProduceBenchmarks.cs

To benchmark both Async and Sync Produce... methods we'll use BenchmarkDotNet and define a IProducer<TKey, TValue> for both Sync & Async. Each Producer will share the same config for both itself and its Registry.

Note: The default Acks setting is Acks = ALL. This means our Producers will wait for acknowledgment from all three replicas

First, the config

var producerConfig = new ProducerConfig()
{
    BootstrapServers = "localhost:9092,localhost:9093,localhost:9094",
    QueueBufferingMaxMessages = 500_000
};

var registryConfig = new SchemaRegistryConfig()
{
    Url = "localhost:8081",
};

Next, the Schema Registry

var registryClient = new CachedSchemaRegistryClient(registryConfig);

Then, our Sync Producer

syncProducer = new ProducerBuilder<string, WidgetEvent>(producerConfig)
    .SetValueSerializer(
        new ProtobufSerializer<WidgetEvent>(registryClient)
        .AsSyncOverAsync())
    .SetErrorHandler((p, e) 
        => Console.WriteLine(
            errorMessage, e.Code, e.Reason))
    .Build();

Finally, our Async Producer

asyncProducer = new ProducerBuilder<string, WidgetEvent>(producerConfig)
    .SetValueSerializer(
        new ProtobufSerializer<WidgetEvent>(registryClient))
    .SetErrorHandler((p, e) 
        => Console.WriteLine(
            errorMessage, e.Code, e.Reason))
    .Build();

Benchmarks - Methods

We'll measure the performance of two methods of interest

1. Produce("topic", TMessage, deliveryHandler)

2. ProduceAsync("topic", TMessage)

[Benchmark]
public void KafkaProducerSync()
{
    var msg = new Message<string, WidgetEvent>()
    {
        Key = $"{Guid.NewGuid()}",
        Value = faker.Generate()
    };
    syncProducer.Produce(sync_topic, msg,
        d =>
        {
            if (d.Error.IsError)
                throw new InvalidOperationException(
                    $"{d.Error.Code}:{d.Error.Reason}");
        });
}

[Benchmark]
public async Task KafkaProducerAsync()
{
    var msg = new Message<string, WidgetEvent>()
    {
        Key = $"{Guid.NewGuid()}",
        Value = faker.Generate()
    };
    await asyncProducer.ProduceAsync(async_topic, msg);
}

Benchmarks - Results

Running the benchmarks is as simple as kicking off our console project in Release

using BenchmarkDotNet.Running;
using ConcurrentFlows.KafkaProducer1;

Console.WriteLine("Starting Producer Benchmarks");
BenchmarkRunner.Run<ProducerBenchmarks>();

On the same platform, with the same previously stood-up cluster

Cluster -

• Network concurrentflows - Created

• Container zookeeper - Started

• Container broker3 - Started

• Container broker1 - Started

• Container broker2 - Started

• Container schema-registry - Started

• Container rest-proxy - Started

• Container topic-init - Started

Platform -

BenchmarkDotNet=v0.13.4, OS=Windows 11 (10.0.22621.1105)
12th Gen Intel Core i9-12900HK, 1 CPU, 20 logical and 14 physical cores
.NET SDK=7.0.100
[Host] : .NET 7.0.0 (7.0.22.51805), X64 RyuJIT AVX2
DefaultJob : .NET 7.0.0 (7.0.22.51805), X64 RyuJIT AVX2

We can see a significant difference between Produce() and ProduceAsync()

Key Takeaway -

Produce() is ~6,000 times faster than ProduceAsync() !!!

Why such a difference???

To begin with, note we had set QueueBufferingMaxMessages to 500,000. We did this because messages will stack up in the underlying Producer buffer as they work their way out to all replicas. Essentially, Produce() is "Producing" faster than we're "Delivering".

The big highlight is that -

Both Produce() and ProduceAsync() are Asynchronous 🤓

• ProduceAsync wraps the entire operation of "Producing" a message into a dotnet friendly Task<DeliveryResult<TKey, TValue>>

• Produce leverages a callback style future defined by the third optional parameter deliveryHandler

This async style difference lets us decouple "Producing" from "Delivering" by deferring the handling of our DeliveryReport<TKey, TValue> . This is as opposed to waiting for a DeliveryResult<TKey, TValue>> to be fully formed.

Wrap Up

Now don't go change all your code to Produce() , it's still an unnecessary blocking call, albeit with minimal impact, even when your cluster is not localhost . What's important is to understand how these methods leverage asynchronicity in different ways and the way this impacts: Acknowledgement, Error Management, etc.

Ideally, I'd propose an entire separation of concerns of hot path message production and Producing/Delivering the message to the wire. That may be the subject of a future post. 🙃

Finally, find all relevant code here
ConcurrentFlows.HashNode/ConcurrentFlows.KafkaProducer1

As a Consumer of your shiny new service, I want my answers Now. Real-time has become the de facto standard. It's no longer good enough to take our time digging through millions of possibilities and return ~500ms later. Don't believe, either, that this only applies to user interactivity.

Caching allows an application to retrieve required data from an in-memory data store. While quite faster than digging within a more persistent store, the in-memory call is not free.

Cache On-Demand

Typically, caches are remote with respect to the application itself. Even if local memory stores the cache, each call to the store increases the contention. The typical three-step process: first checks the cache and can return early, otherwise, the second step loads data out of the database, and third we cache the data for future use with a heuristic Time to Live (TTL). This On-Demand style is common among CRUD apps with a cache layer on top of a database-first mentality.

Cache On-Demand can encompass Cache Aside, Read/Write Through, and Write Behind, the patterns follow largely the same process. The owner of each step may vary or be done outside a hotter path. The key, though, is that each strategy requires Demand First and a Database Source of Truth (SoT).

Cache Ahead

Cache Ahead is all about Knowledge Management. Cache Ahead is more than simply Write Through. When we drive toward a realization of real-time, placing our slowest source of truth closest to our return is inefficient. In other words, even with a cache layer placed on top of a database, we've chosen the database as the true source of our data. We've relegated the cache to be little more than a band-aid on our architecture.

We need to take a better approach and address: Writing, Lifetime, and Eviction of cache data. Let's define this strategy by first dropping our CRUD preconceptions and viewing the problem through an Event-Driven lens.

Widgets Inc.

To stage an exploration of Cache Ahead we'll use the e-commerce unicorn Widgets Inc. The backend we're working with is composed of the Order Service, an Inventory Service, and Emal Service.

Specifically, let's explore the Order Service. Our service is such that it's both listening to Events and responding to HTTP Requests. I've placed the cache first in line with the database updated either via the App itself or in a Write-Through from our cache. We'll see Cache Ahead laid out as it would fulfill the following requirements of the Order Service

1. Validate an OrderRequest has Products in stock

2. Permit the ad-hoc querying of current Orders

3. Maintain the Customers' Order

Order Request - New Order

Keeping our Knowledge management perspective, a new Order must be validated and, if successful, persisted. This is the point in time when new knowledge enters our system.

We can assume that inventory has been maintained by our Inventory Service. We can confidently assume that, upon Order validation, we will hit the Inventory Service cache when checking stock levels. With a validated Order, we'll write to our cache.

This was relatively straightforward, much the same as a Write-Through wherein our database is backing up our cache. Note, the confident Inventory cache hit foreshadows how we'll Always have a Positive cache hit, i.e. if it's not in the cache it's not the database, no need to check, and the database is no longer an SoT.

Ad-Hoc Order Query

Permitting querying of any Order entails maintaining the Order during its active lifetime. Let's consider two external events that could occur affecting our Order: CustomerNotified and InventoryUpdated .

syntax = "proto3";

message CustomerNotified {
  string MessageId = 1;
  string CausationId = 2;
  string CustomerId = 3;
  Kind Kind = 4;

  enum Kind {
    Confirmation = 1;
    Cancellation = 2;
    Backordered = 3;
    Delivered = 4;
  }
}

message InventoryUpdated {
  string MessageId = 1;
  string CausationId = 2;
  string ProductId = 3;
  int32 Count = 4;
}

Each message causes a State Change to our Order, we maintain our Owned Domain perspective and emit that an Order has been updated via an OrderUpdated event. The Order Service will not drive downstream events but simply state its own actions. In this way we maintain the current state of the Order, keeping the cache always the SoT.

We could have seen a variety of scenarios:

1. Confirmation ⇒Order is confirmed

2. Cancelleation ⇒ Order was canceled

3. InventoryUpdated ⇒ Business logic could drive a Backordered

Each of these cases would/could cause an OrderUpdated event. For instance, if a product becomes Backordered the Customer should be notified resulting in a CustomerNotified | Backordered event.

Maintaining the Order - Eviction

So far, we haven't discussed anything about a Time to Live (TTL) or Eviction Policy. Keeping in mind that our objective of Cache Ahead is such that our cache is the primary Source of Truth we want to avoid any form of database fallback. The hard persistent store in this case is better suited as an Event Sourcing store and/or disaster backup, i.e. not an operational store.

The first scary thought is usually that this implies an unbound cache lifetime and bloated cache size, with the cloud spend associated. However, consider the On-Demand cache strategy against Cache Ahead in our Order Service context. If there's no or little demand for current Orders they're evicted from the cache but in the same breath ask; Does that make sense in the use case? In other words; Why are active Orders not interesting enough to remain cached, and if so can we be comfortable with little more than a best guess heuristic TTL?

Business Eviction

Cache Ahead is distinctly intended for an Event-Driven Architecture (EDA). Within our EDA and Order Service context; an Order has a real-world business defined Operational lifetime. That's what determines just how interesting our Order is and determines its lifetime within the cache.

Our particular Order Service context can easily highlight CustomerNotified | Delivered as an End of Life (EoL) of the Order. Surely, we could have a Delivery Service, or live beyond via a 7-Day Return Policy, however;

The key takeaway is that EoL is an event originating from the real life business context and our architecture reflects that reality.

That's a wrap

Cache Ahead is an Event-Driven caching strategy intended to meet the real-time needs of modern businesses. Oriented towards reflecting our real business operations and requirements. Databases don't go anywhere, but they begin to take a backseat operationally and serve analytics, backup, and recovery purposes.

What are your thoughts? What are the nuances I glossed over? Hit the comments and we'll chat!

Event-Driven Architecture is an evolution of a more generic Microservices Architecture. With Microservice Architecture, responsibility is bound within defined domain boundaries and each service takes full ownership of its domain. Ideally, no business operation would span across services and every task would fall neatly into a single microservices domain. Of course, the reality is more complicated and microservices must interact to accomplish any business operation. How they communicate becomes a paramount consideration and is exactly what Event-Driven Architecture addresses.

Working Example

Consider the simple e-commerce example; Our company Widgets Inc. has a backend composed of an Order Service, Inventory Service and Email Service. When a Customer places an Order the Order Service must record the details of the Order, the Inventory Service needs to reflect the change in inventory and the Email Service must notify the Customer of their Order status. Fundamentally, the operations necessary from each service are well defined by the business; it's the degree of coupling among them we can control.

Communicating

Great communication yields a resilient system and great communication is all about the degree of Coupling. When microservices work together they will need to communicate with one another. As microservices evolve independently from one another, communication between them requires deeper decoupling.

Tight and Dependant - Synchronous

In a synchronous approach, services directly call http endpoints in a request/response model. This tightly couples the system together and can lead to cascading failure. Further, this approach can yield a distributed monolith, brittle and difficult to operate, all the benefits of microservices are lost.

Ball Room Dancing - Orchestration

A slightly more asynchronous approach is to Orchestrate or Choreograph services within a workflow. In this approach, it's common to introduce an Orchestrator. An Orchestrator is responsible for coordinating the workflow and, if necessary, executing compensation upon failure. Here the services are only coupled to the Orchestrator while the Orchestrator is coupled to all participating services. This variation of tight coupling is due to treating this business operation as a Distributed Transaction.

I'll place Distributed Transactions, Orchestration vs Choreography out of the scope of this post but each yield varying tradeoffs.

Consider the Events

In both our Synchronous and Asynchronous variations, our Events can be viewed as commands with implicit knowledge of the Workflow and the Consumer of the command.

Bad Messages

For example, this might be our associated Message Schemas:

message PlaceOrder {
  string MessageId = 1;
  string ProductId = 2;
  int32 Count = 3;
}

message UpdateInventory {
  string MessageId = 1;
  string ProductId = 2;
  int32 Count = 3;
}

message NotifyCustomer {
  string MessageId = 1;
  string CustomerId = 2;
  string Status = 3;
}

Each Message is imperative and directs a Consumer to take action. Whether these messages are communicated Synchronously or Asynchronously is irrelevant. Within these messages lies the knowledge of What & When to execute particular functions. This becomes a brittle distributed mess because each service must be oriented to execute within a predefined workflow there's no room for changes or discrete failure.

Commands by their very nature mean the command source knows what must be done next. Whether it's an Orchestrator or the services are Choreographed, commanding one another spreads Knowledge where it shouldn't be. For example, within NotifyCustomer there's a string representing Status but exactly which domain understands Status?

Better Approach

What we must recognize is that Producers are discrete domains and we shouldn't muddy those separations. Events should reflect the perspective of the originating domain. This means that a single domain can only speak for itself. We won't let the Order Service command the Inventory Service. Neither will we need an Orchestrator to command our domains.

message PlaceOrder {
  string MessageId = 1;
  string ProductId = 2;
  int32 Count = 3;
}

message OrderPlaced {
  string MessageId = 1;
  string CausationId = 2;
  string ProductId = 3;
  int32 Count = 4;
}

message InventoryUpdated {
  string MessageId = 1;
  string CausationId = 2;
  string ProductId = 3;
  int32 Count = 4;
}

Now we speak in the past tense after the Order is placed. Each domain simply states its domain knowledge. The Order Service can only say the Order is placed, it cannot tell anyone to update their inventory. It's the responsibility of the Inventory Service to know what to do when an Order is placed. Finally, our Email Service knows that the Order is placed and inventory is updated indicating a predefined Customer notification.

More than simply a semantic argument, the underlying intention is to maintain domain integrity and service independence. The common contrary argument is that the state is more difficult to ascertain with this approach. Certainly a valid perspective, however, the state of any system is simply the aggregate of Events and can be known precisely via Message & Causation Ids.

TL;DR

Events should reflect the perspective of the Producer and not implicitly dictate the Events' Consumption.

With each latest iteration of C#, we see more and more pattern matching. Frankly, I'm deeply excited to see this evolution. C# of course can still be considered an Object Oriented language but MS has taken the Multi-modal moniker and then ran with it.

F# has been an incredible source of inspiration for the C# team, not to mention the Pythion influence when we left Program in the background. So long old friend 👋 We've seen a great deal of Functionally oriented features. From first-class tuples, immutable types, and of course pattern matching!

Pattern Power 💪

What is a Pattern again?

Simply put, a pattern is the shape of an input that you expect to see. You declare what to expect and allow our friend Roslyn to create the ultimate boolean expression to fulfill the pattern search.

List Patterns

The latest addition to the crowd is List Patterning We can take any sequence and describe what we'd like to switch on from a mix of syntax cues.

Patterns are described by

• Square brackets []

Within the brackets, we can match

• Any single element _ with the Discard Pattern

• Many contiguous elements .. with the Slice Pattern

We can even mix in existing patterns to further refine our Sequence Pattern

• Relational Patterns < , >= , etc.

• Logical Patterns and, or , not

And don't forget! Even with these complex List Patterns we still have a bit of a safety net to enforce fully closed pattern expressions as usual with switch {}

To Action - The basics

We have a basic Discard match

[Fact]
public void Match_Discard_Pattern()
{
    var seq = new[] { 1, 2 };
    var result = seq switch
    {
        [_, 2] => true,
        _ => false
    };
    result.Should().BeTrue();
}

Next, we'll Slice the sequence

[Fact]
public void Match_Slice_Pattern()
{
    var seq = new[] { 1, 2 };
    var result = seq switch
    {
        [..] => true,
        _ => false
    };
    result.Should().BeTrue();
}

We can use Relational Patterns

[Fact]
public void Match_Relational_Pattern()
{
    var seq = new[] { 1, 2 };
    var result = seq switch
    {
        [_, > 1] => true,
        _ => false
    };
    result.Should().BeTrue();
}

We can even mix in Logical Patterns

[Fact]
public void Match_Relational_Logical_Pattern()
{
    var seq = new[] { 1, 2 };
    var result = seq switch
    {
        [_, > 1 and < 3] => true,
        _ => false
    };
    result.Should().BeTrue();
}

Level Up Your Patterns

List Patterns are recursive and can contain any combination of nested Patterns. Combining all the available patterns, we can thoroughly inspect the content of a sequence and enforce positional relationships within the sequence. If we have a set of objects, say Person

public sealed record Person(string Name, int Age);

For us to express the idea that Josh must be second in our sequence we can write the following pattern

[Fact]
public void Match_Positional_Pattern()
{
    var seq = new Person[]
    {
    new("Mike", 23),
    new("Josh", 67)
    };
    var result = seq switch
    {
        [_, ("Josh", _)] => true,
        _ => false
    };
    result.Should().BeTrue();
}

We can even express a Property Pattern

[Fact]
public void Match_Property_Pattern()
{
    var seq = new Person[]
    {
    new("Mike", 23),
    new("Josh", 67)
    };
    var result = seq switch
    {
        [_, { Age: > 18 }] => true,
        _ => false
    };
    result.Should().BeTrue();
}

And don't forget we can capture sequence elements in our switch expression

[Fact]
public void Match_Var_Pattern()
{
    var seq = new Person[]
    {
    new("Mike", 23),
    new("Josh", 67)
    };
    var result = seq switch
    {
        [_, var person] => person,
        _ => null
    };
    result.Should().BeEquivalentTo(seq[1]);
}

Imagination is the limit!

Don't stop here! These new powerful patterns are just aching to be leveraged in new and brilliant ways. The C# team keeps delivering powerful advancements to an already incredible language.

Please share your creativity in the comments, what can you do with these patterns!?

So glad you made it back for Part 2, in Part 1: Broadcast Messaging - In Memory we created the implementation of something called Flow. Which is simply the broadcast of messages across many Consumers from many Originators leveraging TPL Dataflow internally.

Towards our AsyncMediator

Our next step is to introduce the concept of Message Completion. This means the Consumer indicates whether it has Completed processing or Failed to process the received message. We want to maintain the broadcast capabilities of the Flow and that feature will drive us away from our TPL Dataflow implementation.

What we'll gain is the ability to dynamically on board Consumers and have a greater fine-grained control of all Consumers. Additionally, this will permit the ability to support Queries and not limit us to Commands

Building a new flow

First, let's lay out some requirments of our new implemtntation:

• Must allow many Originators

• Must allow many Consumers

• Must allow Sinks to detach from Source

• Must only Complete an Envelope when all Consumers finish

Our Library Api

We want this new flow to be easy and simple to consume. Ideally we should have a simple registration and well known interfaces to consume. To start with our registration should look something like this, along with a sample Message.

public static IServiceCollection AddMsgChannel<TPayload>(this IServiceCollection services)
    where TPayload : notnull

public sealed record Message(int Id);

We intend to provide consumers of the library a simple interface to interact with. For example, the Originator of Messages would look something like this:

public sealed class Originator
{
    private readonly IChannelSource<Message> source;

    public Originator(IChannelSource<Message> source)
        => this.source = source;

    public async ValueTask ProduceManyAsync(int count, CancellationToken cancelToken)
    {
        var messages = Enumerable.Range(0, count)
            .Select(i => new Message(i).ToEnvelope());
        var sending = messages.Select(async msg => await source.SendAsync(msg));
        await Task.WhenAll(sending);
    }
}

And, a simple Consumer of Messages would look like this:

public sealed class Consumer
{
    private readonly IChannelSink<Message> sink;

    public Consumer(IChannelSink<Message> sink) 
        => this.sink = sink;

    public async Task<IEnumerable<Message>> CollectAllAsync(CancellationToken cancelToken)
    {
        var set = new List<Message>();
        try
        {

            await foreach (var envelope in sink.ConsumeAsync(cancelToken))
                set.Add(envelope.Payload);
        }
        catch (OperationCanceledException)
        { /*We're Done*/ }
        return set;
    }
}

Guiding Test

All that we need to expose is the simple diametrical interfaces; Sink & Source. With these and our requirements in mind, we can write a test and let that drive and affirm our implementation.

[Fact]
public async Task Source_BroadcastsTo_AllConsumers()
{
    var provider = new ServiceCollection()
        .AddMsgChannel<Message>()
        .AddTransient<Originator>()
        .AddTransient<Consumer>()
        .BuildServiceProvider();

    var originator1 = provider.GetRequiredService<Originator>();
    var originator2 = provider.GetRequiredService<Originator>();
    var consumer1 = provider.GetRequiredService<Consumer>();
    var consumer2 = provider.GetRequiredService<Consumer>();
    originator1.Should().NotBe(originator2);
    consumer1.Should().NotBe(consumer2);        

    var count = 100;
    using var cts = new CancellationTokenSource();
    var consume1 = consumer1.CollectAllAsync(cts.Token);
    var consume2 = consumer2.CollectAllAsync(cts.Token);
    var sending1 = originator1.ProduceManyAsync(count, cts.Token);
    var sending2 = originator2.ProduceManyAsync(count, cts.Token);

    await Task.WhenAll(sending1, sending2);
    cts.Cancel();
    var set1 = await consume1;
    var set2 = await consume2;

    set1.Should().HaveCount(count * 2);
    set2.Should().HaveCount(count * 2);
    set1.Should().BeEquivalentTo(set2);
}

Our test sets up the Dependency Registration, then creates two of each Producer and Consumers. We assert that each instance is unique and then kick off our action. We trigger the Consumers to start listening and trigger the Producers to start sending a preset count of Messages. Once our Producers complete, we trigger the CancellationToken to stop our Consumers. After awaiting the consumption Tasks we can assert that each Consumer has received all Messages, double the original count, since we have two Producers. Finally, we assert that both sets of Messages are equal, indicating that we did multicast from each Prodcer to every Consumer.

To beign our implementation we look to the humble Envelope. This will be the container and carrier of our Payload and serve as the conduit for various utilities.

The Envelope

The first piece we need to extend is the Envelope. The Envelope will be the item that allows us to communicate an Ack/Nack back to the Originator.

public abstract record Envelope
{
    public virtual string EnvelopeId => $"{GetHashCode()}";
}

This provides our base and gives us an overridable EnvelopeId defaulting to the hash code of the record itself.

public sealed record Envelope<TPayload>(
    TPayload Payload,
    TaskCompletionSource TaskSource,
    Task Execution)
    : Envelope
    where TPayload : notnull
{
    private TaskCompletionSource TaskSource { get; } = TaskSource;
    private Task Execution { get; } = Execution;

    public Exception? Failure { get; private set; }

    public void Complete()
        => TaskSource.TrySetResult();

    public void Fail(Exception exception)
    {
        TaskSource.TrySetException(exception);
        Failure = exception;
    }

    public TaskAwaiter GetAwaiter()
        => Execution.GetAwaiter();
}

The first derivation Envelope carries a Payload, that is, any object of type TPayload. It exposes two methods to Complete: ACK, or Fail: NACK. Next, it privately holds a TaskCompletionSource that is used to manage the completion of the Envelope. Finally, a private Task, whose Awaiter is exposed, this allows the Envelope to be awaited by an Originator.

Task & TaskCompletionSource

The execution Task represents the abstract idea of any work necessary to process the Envelope. Note that the work to process the Message is not tied to the Task directly. We're not awaiting any Consumer. Instead, we're maintaining an outstanding future that can be finalized by a Complete or Fail method.

The TaskCompletionSource is used to signal that the Task has finished. Conmplete sets the results as successful. Fail takes an Exception indicating that processing failed. However, the Exception won't be propagated via an await. We don't want the Originator, who may be awaiting the Envelope, to pop an Exception off the Task. Instead, we use the pattern of a nullable Exception to expose failure and encourage null propagation.

Packaging an Envelope

To pack this all up we first stuff in the Payload and create our TaskCompletionSource. We'll wrap this together with a timeout and async callbacks for onComplete and onFailure.

public static Envelope<TPayload> ToEnvelope<TPayload>(
    this TPayload payload,
    TimeSpan timeout,
    Func<Task> onCompleted,
    Func<Task> onFailure)
    where TPayload : notnull
    => new TaskCompletionSource()
        .CreateEnvelope(payload, timeout, onCompleted, onFailure);

Next, our TaskCompletionSource is used to form an ExecutionMontior. The job of the monitor is to maintain the timeout and execute either onCompleted or onFailure

private static Envelope<TPayload> CreateEnvelope<TPayload>(
    this TaskCompletionSource taskSource,
    TPayload payload,
    TimeSpan timeout,
    Func<Task> onCompleted,
    Func<Task> onFailure)
    where TPayload : notnull
    => new Envelope<TPayload>(
        Payload: payload,
        TaskSource: taskSource,
        Execution: taskSource.CreateExecutionMonitor(timeout, onCompleted, onFailure));

private static Task CreateExecutionMonitor(this TaskCompletionSource source,
    TimeSpan timeout,
    Func<Task> onCompleted,
    Func<Task> onFailure)
{
    return AsyncExecutionMonitor(source.Task, timeout, onCompleted, onFailure);

    async Task AsyncExecutionMonitor(
        Task completion,
        TimeSpan timeout,
        Func<Task> onCompleted,
        Func<Task> onFailure)
    {
        if (await completion.TryWaitAsync(timeout))
            await onCompleted();
        else
            await onFailure();
    }
}

public static async Task<bool> TryWaitAsync(this Task task, TimeSpan timeout)
{
    await Task.WhenAny(task, Task.Delay(timeout));
    return task.IsCompletedSuccessfully;
}

The async monitor applies a global timeout to the future we're maintaining for the processing work. We cannot Cancel the work when the timeout has expired. This is a direct consequence of not having direct control, coupling, to the work being performed. The best we can do in this position is to Fail on our side and move on.

Within TryWaitAsync we can see how the TrySetException from within the Envelope is rerouted into a bool consequence. It simply leverages WhenAny and only considers if the Task was successful. We know that any exception captured by the call to Fail(ex) sets the failure on the Envelope itself.

Completion

Now with Envelope<TPayload> in hand.

Think for a moment; How, within our broadcast framework, do we determine if a Message is Complete? We can't simply say the Originating message is Complete when only one Consumer finishes. And what happens if 1 out of 20 Consumers fail? A reasonable default would be to wait for all Consumers and assume failure if any single Message fails. This follows the pattern of at-most-once delivery, although, we could extend to dedicated retrying of failures and more complex strategies. Now, recall we have a dynamic Consumer Set that we don't have direct knowledge of, so how do we manage the broadcast?

The BroadcastBlock we used for our original implementation in Part 1 doesn't expose it's Consumers in any way. We don't see how many there are, nor even if all Consumers have been offered the Envelope. We'll need more control for this and that's where we'll leverage Channels

ChannelSource

To control our broadcast we need knowledge and control over "Subscribers"/Consumers of our Source. The first step is to create our own variant of the TPL Dataflow Link. This is a structure representing the connection of a Source and Sink.

public record struct SinkLink(Guid LinkId) : IDisposable
{
    private Action<Guid>? unlink = default;
    private bool disposed = false;

    public SinkLink(Guid linkId, Action<Guid> unlink)
        : this(linkId)
        => this.unlink = unlink;

    public void Dispose()
    {
        Dispose(disposing: true);
        GC.SuppressFinalize(this);
    }

    private void Dispose(bool disposing)
    {
        if (disposing && !disposed)
            unlink?.Invoke(LinkId);
        disposed = true;
        unlink = null;
    }
}

Our SinkLink is identifiable by its LinkId and it holds a reference that allows it to Unlink from the source via its LinkId.

Next, we'll capture the concept of many outgoing Messages in the form of an Outbox. This allows us to maintain record of all Messages broadcast and when they have Completed

public interface IOutbox<TPayload> where TPayload : notnull
{
    void Complete();
    Envelope<TPayload> GetEnvelope();
}

This Outbox is intended to be created, then Messages submitted and finally Completed.

internal sealed class EnvelopeOutbox<TPayload> : IOutbox<TPayload>
    where TPayload : notnull
{
    private readonly Envelope<TPayload> originator;
    private readonly Action<Guid> destructor;
    private readonly ConcurrentDictionary<Guid, Envelope<TPayload>> pool = new();
    private readonly ConcurrentBag<Exception> failures = new();

    private volatile int leaseCount = 0;
    private volatile bool complete = false;

    public EnvelopeOutbox(
        Envelope<TPayload> originator,
        Action<Guid> destructor)
    {
        this.originator = originator;
        this.destructor = destructor;
    }

    public void Complete()
        => complete = true;

    public Envelope<TPayload> GetEnvelope()
    {
        Interlocked.Increment(ref leaseCount);

        var id = Guid.NewGuid();
        var envelope = originator.Payload.ToEnvelope(
            TimeSpan.FromSeconds(30),
            () => EnvelopeDestructorAsync(id),
            () => EnvelopeDestructorAsync(id));

        pool[id] = envelope;
        return envelope;
    }

    private async Task EnvelopeDestructorAsync(Guid id)
    {
        Interlocked.Decrement(ref leaseCount);

        var envelope = pool[id];
        failures.MaybeAdd(envelope?.Failure);

        if (!ReadyForClosure) return;

        await CloseOutPool(id);
    }

    private async Task CloseOutPool(Guid id)
    {
        await Task.WhenAll(pool.Select(async e => await e.Value));

        if (failures.Any())
            originator.Fail(new AggregateException(failures));
        else
            originator.Complete();

        destructor(id);
    }

    private bool ReadyForClosure
        => leaseCount <= 0 && complete;
}

From this implementation we can see several things:

1. Creation is dependent on an Originating Message and a Destructor to handle the finalization of the Outbox

2. Each time a Message is requested we increment the internal leaseCount

3. The outgoing Messages are connected such that they are destructed, decrement the leaseCount and give us an opportunity to Close the Outbox via EnvelopeDestructorAsync

4. Once the Outbox is Complete and the leaseCount reaches zero we aggregate any Exceptions calling Complete or Fail on the Originator's Message

5. Finally, the Source provided Outbox Destructor can be executed.

All of this allows our Source to broadcast and maintain knowledge of all outgoing Messages. From the previous we can see we need to define two Factory Delegates

delegate IChannelSink<TPayload> SinkFactory<TPayload>()
    where TPayload : notnull;

delegate IOutbox<TPayload> OutboxFactory<TPayload>(
    Envelope<TPayload> originator,
    Action<Guid> destructor)
    where TPayload : notnull;

First, the SinkFactory will be responsible for creating a new ChannelSink<TPayload>. We can register this delegate and inject it if we ever find ourselves dynamically creating new Consumers. But for now, we'll implement it within our Source. Secondly, we have the OutboxFactory that creates an Outbox each time we need to broadcast a Message.

The advantage of having these delegates is that it allows us to place their implementation close to the source of their use. The SinkFactory will be held within the ChannelSource, allowing us to place the new Sink within the collection of targets held by the Source. The OutboxFactory, while not entirely necessary, is a simple abstraction of the Outbox constructor that allows us to create one on demand and mock its implementation for testing.

Diving in

Now let's dive right into the implementation of the Source, we have:

internal sealed class ChannelSource<TPayload>
    : IChannelSource<TPayload>,
    ILinkableSource<TPayload>
    where TPayload : notnull
{
    private readonly OutboxFactory<TPayload> factory;
    private readonly ConcurrentDictionary<Guid, ChannelWriter<Envelope<TPayload>>> channels = new();
    private readonly ConcurrentDictionary<Guid, IOutbox<TPayload>> outboundMsgs = new();

    public ChannelSource(OutboxFactory<TPayload> outboxFactory)
        => this.factory = outboxFactory;

    public async ValueTask<bool> SendAsync(Envelope<TPayload> envelope, CancellationToken cancelToken = default)
    {
        cancelToken.ThrowIfCancellationRequested();
        var outboxId = Guid.NewGuid();
        var outbox = factory(envelope, (id) => outboundMsgs.Remove(id, out _));
        outboundMsgs.TryAdd(outboxId, outbox);
        var writers = channels.Values;

        var writing = writers.Select(async writer =>
        {
            await Task.Yield();
            var outbound = outbox.GetEnvelope();
            return writer.TryWrite(outbound);
        });
        outbox.Complete();
        var results = await Task.WhenAll(writing);

        return results.All(s => s);
    }

    public SinkFactory<TPayload> SinkFactory
        => () =>
        {
            var linkId = Guid.NewGuid();
            var sinkLink = new SinkLink(linkId, id => channels.Remove(id, out _));
            var channel = Channel.CreateUnbounded<Envelope<TPayload>>();
            channels.TryAdd(linkId, channel);
            var channelSink = new ChannelSink<TPayload>(channel, sinkLink);
            return channelSink;
        };
}

Breaking this down we have two concurrent collections, each keyed with a UUID. The first, channels is dedicated to holding all the targets our Source is going to broadcast to. While outboundMsgs will maintain an Outbox for all Messages in flight

private readonly ConcurrentDictionary<Guid, ChannelWriter<Envelope<TPayload>>> channels = new();
private readonly ConcurrentDictionary<Guid, IOutbox<TPayload>> outboundMsgs = new();

Then we have an implementation of the factory delegate SinkFactory<TPayload>. This delegate implantation lives within the Source in order to link the new Sink to the Source via the SinkLink. Additionally, the destructor we pass enables removal of the target from the private set.

public SinkFactory<TPayload> SinkFactory
    => () =>
    {
        var linkId = Guid.NewGuid();
        var sinkLink = new SinkLink(linkId, id => channels.Remove(id, out _));
        var channel = Channel.CreateUnbounded<Envelope<TPayload>>();
        channels.TryAdd(linkId, channel);
        var channelSink = new ChannelSink<TPayload>(channel, sinkLink);
        return channelSink;
    };

And finally, the real work is done within SendAsync(..). First, we check the cancelToken. Then, we iterate through all targets. The syncronuous TryWrite is used since we know these targets are unbound Sinks and we've side stepped any idea of Channel completion. However, there's no need to wait for any single TryWrite so the operation yields to the iteration and we asyncronuously wait for all writes to complete. Finally, we return success or failure simply based on whether all writes were successful or not.

public async ValueTask<bool> SendAsync(
    Envelope<TPayload> envelope, 
    CancellationToken cancelToken = default)
{
    cancelToken.ThrowIfCancellationRequested();
    var outboxId = Guid.NewGuid();
    var outbox = factory(envelope, (id) => outboundMsgs.Remove(id, out _));
    outboundMsgs.TryAdd(outboxId, outbox);
    var writers = channels.Values;

    var writing = writers.Select(async writer =>
    {
        await Task.Yield();
        var outbound = outbox.GetEnvelope();
        return writer.TryWrite(outbound);
    });
    outbox.Complete();
    var results = await Task.WhenAll(writing);

    return results.All(s => s);
}

💥Well that was easy 🤣

ChannelSink

The last core piece of Part 2; the ChannelSink. A much simpler implementation, all we need to do is continuously read from the Channel and handle a standard call to Dispose.

internal sealed class ChannelSink<TPayload>
    : IChannelSink<TPayload>,
    IDisposable
    where TPayload : notnull
{
    private readonly ChannelReader<Envelope<TPayload>> reader;
    private readonly IDisposable sinkLink;
    private bool disposed = false;

    public ChannelSink(
        ChannelReader<Envelope<TPayload>> reader,
        IDisposable sinkLink)
    {
        this.reader = reader;
        this.sinkLink = sinkLink;
    }

    public IAsyncEnumerable<Envelope<TPayload>> ConsumeAsync(CancellationToken cancelToken = default)
        => reader.ReadAllAsync(cancelToken);

    public void Dispose()
    {
        Dispose(disposing: true);
        GC.SuppressFinalize(this);
    }

    private void Dispose(bool disposing)
    {
        if (!disposed && disposing)
            sinkLink.Dispose();
        disposed = true;
    }
}

Registration - I didn't forget

Of course we can't forget the full implementation of our IServiceCollection extension

public static IServiceCollection AddMsgChannel<TPayload>(this IServiceCollection services)
    where TPayload : notnull
    => services
        .SetSingleton<OutboxFactory<TPayload>>(_
        => (originator, destructor)
            => new EnvelopeOutbox<TPayload>(originator, destructor))
        .SetSingleton<SinkFactory<TPayload>>(sp =>
        {
            var source = sp.GetRequiredService<IChannelSource<TPayload>>();
            var linkable = source as ILinkableSource<TPayload>
                ?? throw new ArgumentException($"Source {source.GetType().Name} must be linkable", nameof(source));
            return linkable.SinkFactory;
        })
        .SetSingleton<IChannelSource<TPayload>, ChannelSource<TPayload>>()
        .AddTransient<IChannelSink<TPayload>>(sp => sp.GetRequiredService<SinkFactory<TPayload>>().Invoke());

Closing Up

Part 1 saw the most basic broadcast implementation. That got us on the right track splitting Origination and Consumption. Now, while maintaining separation, we've allowed the concept of Completion and Acknowledgement to be introduced. At the end of Part 2 we have succeeded in passing our test!. But more than that, we can now broadcast a message from multiple Producers to multiple Consumers and allow a positive or negative acknowledgement to flow back to the Producer via the Originating Message

🤯Soo much cooler visually! But seriously with this in place, the next step towards our Async Mediator is a small jump and we're well on the right path.

Tranistioning from Engineer to Engineering Leader is a process full of opportunity. Opportunity to learn, grow, and to expand your knowledge base into domains you only scratched the surface of. It's also a process ready and willing to teach a few hard lessons. One such lesson I learned was the application of Dependency Inversion as it pertains to leading a team of engineers. There are several flavors, variations and interpretations of the Dependency Inversion Principle (DIP). The one I find most applicable here is

• High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).
• Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions

How is that relevant?

Sounds quite technical, doesn't it? This is clearly only applicable with my head down jamming out code and drinking Monster, right? Nope turns out this might be one of the most important principles you can apply to leadership.

The whole point of this principle is to avoid a tight coupling between implementation details and the value they yield. Between the value forming business logic and underlying technical concerns.

Viewed from a leadership perspective, our value building business logic lies with the product and/or feature we're trying to deliver. That's the critical idea the business wants to bring into being. The driver of this idea is the Product Owner/Manager, while its creation lies with the Engineer.

Product Ownership

That nascent idea has intrinsic value outside of any technical implementation. It may well be implemented with an Archimedes' screw if that gets the job done.

Engineering Rigor

As Software Engineers we implement the business logic with rigor and pragmatism. Software Engineers are 100% coupled to the details of the implementation and know its every nuance.

Lesson Learned #207

And this is the lesson I learned: The responsibility of Engineering Leadership is providing that abstraction between two concerns. The deep in the weeds, detail oriented Engineering implementation and that value driving, idea realizing Product Ownership. Each side of the coin depends on an abstraction of the other.

Living this Dependency Inversion means the leader is simultaneously

1. Driven by the higher level Product Initiative
2. Provides the space and freedom to the engineering team to deliver

These complimentary perspectives yield the compound title Engineering Leader. Lesson learned, maybe next time it'll be Liskov substitution principle

Origination executes distinctly from Consumption

From the beginning 🤔

The idea of a Mediator is generally the go to when we want to decouple a Request from its Processing. There's already some well established and widely used mediator packages e.g. MediatR; J. Bogard, Mediator.NET; E. Ma, etc. These all quite well implement variations on the Mediator Pattern...👌. Largely we see these operating in process and not across the wire, with the exception of MassTransit - Mediator of course, but we won't go there right now.😎 Additionally we can see many of the fundamental ideas presented within the ...Query Side and ... Command Side of my architecture; S. Deursen, also of Simple Injector Rock Stardom 🎸

On the market today

Right up front; I really enjoy both these libraries but ultimately I'd like to go another direction 🤔

First, let's take a closer look; all these options are quite feature rich, and support similar architectural patterns/styles.

• CQRS Style Patterns
  • Query/QueryHandler, Request/Response
  • Command/CommandHandler
• Pub/Sub
  • Notifications
  • Publish Events
• Pipelines / Middleware / Behaviors
  • Pipelining & Behaviors
  • Pipeline Behaviors
  • Mediator.NET Pipelining
  • Mediator.NET Middleware

What about the Mediator?

Traditionally, a Mediator represents a focal point between an Originator and one or many Consumers. This allows the Originator to avoid explicitly referring to or knowing about the Consumers

The Consumers are typically conceptualized as Handlers of particular messages and can form Pipelines or chains. Each Handler playing a specific role in the choreography. Additional and complimentary concepts can be added with associated Middleware or Behaviors, etc. promoting some aspects of loose coupling. But the call from Originator to Handler is still synchronous with regard to its Registration. The Registration couples, albeit in the background, the Originator and the Consumer and both processes are tightly working together via the Mediator.

For instance MediatR holds a reference to all of its Handlers and dispatches associated Requests and/or Notifications. Mediator.Net has a nice abstraction in place; Pipes form its central core and you can even stream responses back via the receive pipeline. 💘IAsyncEnumerable. However, all Handlers are registered and well known. Both of these methodologies yield a static topology.

These structures have the advantage when you want a response back from a particular request. But I'm not interested in call and response; receiving a message should yield another subsequent message and we'll leave any correlleation back to the origination to simply be implicit.

Schema Driven Mediator

Conceptually

Conceptually what we're after is much the same as you'd find in a distributed messaging broker: Kafka, Pulsar, RabbitMQ, etc. With this in mind we'll split our message Origination entirely from it's Consumption;

This is the key I want changed; Origination executes distinctly from Consumption

Architectural Objectives

Primary Goals

Fundamentally we want an entirely decoupled architecture; the only shared knowledge should be the message Schema.

1. Decoupling between processing units
   • Schema Driven communication
   • The Originating Process should remain separate from the Consuming Process
2. Consumer Declared Consumption - this is where we decouple
   • No concept of registering Handlers
   • Consumers and their associated process may come and go
3. Broadcast & Multicast Origination
   • One or Many Originators can produce a message
   • One or Many Consumers can listen

Above, we can see the type of outbound Broadcast/Multicast style brokerage we're after. A decoupled, fanout of Messages, is relatively easy to achieve.

Putting it together

Foundations

Our first step is to split the Originator from Consumer and further avoid any implicit coupling. All Requests, Commands, Notifications, Responses, etc. will travel asynchronously. The building blocks of this system will be the abstract concepts of Schema, Message, and ultimately Flow.

• Schema - Defines the shape of data/payload/behavior to be implemented by a Message
• Message - Takes the form defined by the Schema and represents an immutable and unique representation of that Schema at a point in time
• Flow - Is the unidirectional transmission of Messages. A single Flow only permits a single Schema

Async Message Fanout

Abstractions

Producing a Flow are our two fundamental building blocks: FlowSource & FlowSink. These complimentary components define either end of a Flow constrained to a single Schema. The Messages will be transmitted in the background of our application via a Source/Sink set of abstractions. Each is defined below:

public interface IFlowSource<TSchema>
    : IAsyncDisposable, IDisposable
    where TSchema : Envelope
{
    ValueTask<bool> EmitAsync(TSchema message, CancellationToken cancelToken = default);
}

public interface IFlowSink<TSchema>
    : IAsyncDisposable, IDisposable
    where TSchema : Envelope
{
    IAsyncEnumerable<TSchema> ConsumeAsync(CancellationToken cancelToken = default);
}

Note, both ends of our unidirectional Flow are (Async)Disposable. The unidirectional nature of the Flow leads directly to two implications.

1. Disposing a Sink, disposes only that recipient endpoint
2. Disposing a Source, however, closes the entirety of the Flow

Here we've also defined a constraint of Envelope, as seen, this constraint, i.e our base Schema, ensures our Messages always have a CurrentId and a CausationId. All this means is that we know where we are now and where we came from.

public abstract record Envelope(
    string CurrentId,
    string CausationId);

Simple Message passing via the twin concepts of Source/Sink allow the Origination and Consumption to be decoupled. Our Source, via an Originator, produces messages at one end and the Sink receives them into a Consumer. This takes advantage of much the same concepts of any Message Bus but applies an asynchronous distributed behavior within a single application.

Implementations

Our implementations are built upon the now standard System...Dataflow namespace containing a wide variety of Dataflow primitives. This library is quite powerful in terms of message passing and parallel processing; complimenting the typical async/await paradigm. Other implementations are of course possible;

• A little RX - System.Reactive, et. al.
• Rolling a bit with System.Threading.Channels

Each has tradeoffs of course, the main driver toward Dataflow was the simplicity of separating Message Passing and Message Consuming. I don't want to force any Consumer into participating in any semantic Observable Pipeline, especially if it is instead better represented as a discrete step. Flow remains opt-in and can be as simple as injection of the Sink and reading from it. Channels of course could underpin Dataflow but natively Channels are not intended to support any kind of Broadcast/Multicast.

FlowSink

To build our Flow first we'll look at the Sink. The purpose and intention of this component is to be injected via dependency injection into any Consumer that has an interest in the defined Schema. That is; any defined Consumer that must take action upon receiving a Message with the defined Schema. I've collapsed most of the disposal for brevity; the key piece is disposing of the link back to the Source.

internal sealed class FlowSink<TSchema>
    : IFlowSink<TSchema>
    where TSchema : Envelope
{
    private BufferBlock<TSchema>? Buffer { get; set; }

    private readonly IDisposable link;
    private volatile bool isDisposed;


    public FlowSink(ILinkableSource<TSchema> source)
    {
        Buffer = new(new()
        {
            EnsureOrdered = true,
            BoundedCapacity = DataflowBlockOptions.Unbounded
        });
        link = source.LinkTo(Buffer);
    }

    public IAsyncEnumerable<TSchema> ConsumeAsync(CancellationToken cancelToken = default) 
        => Buffer.ThrowIfDisposed(isDisposed)
            .EnumerateSource(cancelToken)
            .Attempt(onError: ex =>
            {
                this.Dispose();
                return AsyncEnumerable.Empty<TSchema>();
            });

    public void Dispose() {/*...*/}
    public ValueTask DisposeAsync() {/*...*/}
    private void DisposeCore()
    {
        isDisposed = true;
        link?.Dispose();
        Buffer = null;
        GC.SuppressFinalize(this);
    }
}

Our FlowSink is built on top of an internal BufferBlock bound to our Schema. We maintain a concrete implementation of BufferBlock with the intention of later using its Count but this could well be represented as a IPropagatorBlock until specificities are necessary. Next, the only item we're dependent on for construction is an ILinkableSource<TItem> defined as follows

LinkableSource

internal interface ILinkableSource<TSchema>
    where TSchema : Envelope
{
    IDisposable LinkTo(ITargetBlock<TSchema> sink);
}

Keeping this interface internal allows us to keep this concept scoped within our package. This narrowly exposes the internal mechanism of Linking two Dataflow Blocks together. Once linked, each block will process messages as defined by its implementation and operate independently of any other within the bounds set by the creation and linking.

Lastly, we can see that the Buffer is consumed via a Disposal and Exception protected cancellable extensions to IAsyncEnumerable. While not necessary for this, we've decorated the CancellationToken as the target of [EnumeratorCancellation] for broader use cases. To keep things brief I'll leave the full implementations for review via GitHub

[return: NotNull]
public static T ThrowIfDisposed<T>(
        this T? target,
        bool isDisposed)

internal static Func<IAsyncEnumerable<TSchema>> EnumerateSource<TSchema>(
        this ISourceBlock<TSchema> source,
        CancellationToken cancelToken)

public static IAsyncEnumerable<T> Attempt<T>(
        this Func<IAsyncEnumerable<T>> iterator,
        Func<Exception, IAsyncEnumerable<T>> onError,
        Func<Exception, bool>? canHandle = default)
        where T : class

These are relatively simple extensions, although Attempt has quite the signature 🤯 But each do just what's on the tin; ThrowIfDisposed, EnumerateSource; i.e. Orchestrate the Enumerator, and finally let Attempt manage the execution.

FlowSource

Next is our Broadcast/Multicast enabled Source. This is accomplished by exposing the capabilities of a BroadcastBlock. This block clones, in our case - returns, each message received and Offers it to each Linked block. The importance of Offer is such that if a Linked block cannot take the Message; that Message is then dropped, i.e. lost forever and for good. This leads to Backpressure, another high point for choosing Dataflow yet out of scope here, but we set all Blocks with an UnboundedCapacity for simplicity to begin. So Source can be implemented as such; again with collapsed disposal, we're both Completing and then awaiting Completion of the Source during cleanup.

internal sealed class FlowSource<TSchema>
    : IFlowSource<TSchema>,
    ILinkableSource<TSchema>
    where TSchema : Envelope
{
    private BroadcastBlock<TSchema>? Source { get; set; }

    private volatile bool isDisposed;    

    public FlowSource()
        => Source = new(msg => msg,
            new()
            {
                EnsureOrdered = true,
                BoundedCapacity = DataflowBlockOptions.Unbounded
            });

    public ValueTask<bool> EmitAsync(TSchema message, CancellationToken cancelToken = default)
        => Source.ThrowIfDisposed(isDisposed)
            .OfferAsync(message, TimeSpan.FromMilliseconds(300), cancelToken)
            .Attempt(onError: ex => ValueTask.FromResult(false));



    IDisposable ILinkableSource<TSchema>.LinkTo(ITargetBlock<TSchema> sink)
        => Source.ThrowIfDisposed(isDisposed)
            .LinkTo(sink, new()
            {
                PropagateCompletion = true,
            });

    public void Dispose() { /*...*/}
    public async ValueTask DisposeAsync() { /*...*/}
    private async ValueTask DisposeAsyncCore()
    {
        isDisposed = true;
        Source?.Complete();
        await (Source?.Completion ?? Task.CompletedTask);
        Source = null;
        GC.SuppressFinalize(this);
    }
}

This implementation exposes EmitAsync and transforms the standard Task<bool> of the Block into a disposed protected and orchestrated Attempt yielding a ValueTask<bool> via simple extensions. Additionally, we separately implement the internal ILinkableSource<TItem> interface to connect any downstream Sinks. This exposes a disposed protected call into .LinkTo ensuring that Completion is propagated. With this configuration set; if the Source is disposed and thus Completed this information will flow down to all Sinks which will then exit any ongoing or new iteration upon the Sink.

Registration

With just these two components we can achieve the first two Primary Goals, albeit with a little DI trickery

1. Decoupling between processing units
2. Consumer Declared Consumption

public static IServiceCollection AddFlow<TSchema>(this IServiceCollection services)
    where TSchema : Envelope
{
    services.TryAddSingleton<IFlowSource<TSchema>, FlowSource<TSchema>>();
    services.TryAddTransient<IFlowSink<TSchema>>(sp =>
    {
        var source = sp.GetRequiredService<IFlowSource<TSchema>>();
        var linkable = source as ILinkableSource<TSchema>
            ?? throw new ArgumentException(
                $"Invalid FlowSource Registration. Source Type, {source.GetType().Name}, is not linkable");
        return new FlowSink<TSchema>(linkable);
    });
    return services;
}

Broadcast/Multicast

Registering the FlowSource as a singleton ensures any consumer of this interface is Emitting to the same BroadcastBlock. This allows one to many Originators to enter Messages into the flow.

Declared Consumption

Thanks to the leniency of the default Dependency Injection of dotnet, i.e. IServiceCollection/IServiceProvider; we can register our Sink as a Transient. Doing so may yield a captured dependency if the consuming service has a longer lifetime than our Sink. However, the advantage is that we ensure each Consumer receives a unique and linked instance of our Sink. In this variation we're leveraging the DI container to dynamically construct our topology. Assuming, 🤞, our Consumers dispose of the Sink properly it will be appropriately removed from the Flow topology.

Closure

With just these two primary components, FlowSource & FlowSink, we have achieved a basic decoupling and fanout of Messages. We really don't even need a specific Mediator to do this.

The next evolution is going to be tying things together via the Envelope. This will allow us to put backpressure on a Source while maintaining independence. Additionally, it will provide a vector to ack/nack our Source.

Of course, all the code for this post can be found on GitHub ConcurrentFlows.AsyncMediator

When we first encounter Domain Modeling we might find ourselves surrounded by examples of people, animals or cars. These concepts are given properties and we start to combine and collect them. We quickly learn the nature of inheritance by learning about specialization of generic concepts; a dog vs. an animal, etc. Then we discuss the behavior of these concepts and have to ask does a Dog: bark, speak or just make noise? So composition is then introduced as an answer to all the noises to be made. At this point the concept of Object Oriented design is starting to take shape. We learn that in these constraints an Object, as it was presented, defines its shape and its behavior.

With that, we may then be thrust into Domain Driven Design wherein we learn about the Legos of our Domain Model; the aggregate, entity, value object and so on. Concepts well in hand we construct Rich Domain Models ensuring their invariants and described via our ubiquitous language. We stand back with pride at our carefully choreographed model as it expresses our Domain Experts understanding of the Domain Space.

The Test of Time

All is well and good until the underpinnings of our Rich Domain Model are redefined, shuffled, evolved and put through the meat grinder that is time. Our model was perfect; Time made it wither.

Business demands innovation and the expert knowledge of today is the antiquated ramblings of tomorrow. So how do we model a Domain built on sand? I'd argue the typical Domain Drive Design approach is to lay a bedrock of Domain Expert driven ubiquitous language, from this bedrock comes the choreography. Confident in our assumptions the Rich Domain Model is viewed as malleable to whatever the business demands. But it was the expertise of the Domain Expert that shifted. New ideas, optimizations, approaches and evolving definitions all crash though our model tearing down the house of cards. So what then? We build it again? Based off our latest snapshot of a Domain Expert? How can something so malleable become so rigidly orchestrated?

Well maybe we just missed something.

Reflecting

Our Domain Model was built as an expression of an instant of time. That is the key failure; we represented the Domain Space in a snapshot. Just as a picture is out of date as soon as it's taken, our model is growing stale even as it's implemented. That's is the salient point; our model ignored time.

As engineers we have to recognize that our Domain Experts don't deal in code, they don't get heated about anemic vs. rich models, they don't run their own K8's cluster at home. A Domain Expert squeezes the most defined Success possible out of a given Domain. And every noun in the preceding sentence is subject to evolution.

What we typically resort to is a Domain Model, but this is a singular noun describing an instant in time. As engineers our expression medium, code, allows us to transit time effortlessly. Therefore our model has to be evolved to capture the business value we're after. In order to capture the true business value of the Domain Expert we need to capture the Domain Space and the Evolutionary Operator on that Domain Space.

Redefining

To make progress, as engineers, we stand on our definitions and define our key concepts.

• Domain Space: A vector field defined by the degrees of freedom of its constituent nouns and verbs
• Evolutionary Operator: A function that evaluates a point within the Domain Space
• Domain Expert: An Evolutionary Operator on a Domain Space.
• Domain Model: A point within a Domain Space.

These were likely not the definitions you had in mind! But through the lens of these definitions we see our Domain Model as woefully inadequate to describe what the business is trying to accomplish. Representing a single point within our Domain Space is certain to fall victim to the fate we've described above.

Taking The Plunge

Preamble

You won't see code examples here. Code is the hammer we forge our iron concepts with. Whatever your language, whatever your technique, whatever your paradigm; the concepts are what gets forged into business value.

Domain Space

You might ask, "I've made it this far, now what the hell are you talking about?"

A vector field is is the assignment of a vector to every point within a set whose structure is defined by its constituents. These constituents are defined by their degrees of freedom; the axis along which they're allowed to change.

By simply defining the nouns and verbs within our Domain Space we start to imply constraints, i.e thing X can change along axis Y. We define nouns in terms of change vectors. Our verbs interact with the nouns and link them in nearly infinite combinations forming networks. Doing this exercise brings our Domain Space alive and allows us to see what Domain Models can be produced. The possibilities become nearly infinite! Time and all other axes are integrated into one holistic picture.

Evolutionary Operator

We define an Evolutionary Operator that works against our defined Domain Space with the critical guidance of our Domain Expert. Remember that the Domain Expert is there to squeeze out success but their expertise is at a point in time. Instead of capturing this expertise in a static Domain Model, we dig deeper. We conceptualize the Domain Expert's expertise as a function evaluated against our Domain Space that yields a Domain Model.

Each evolution of the Domain Expert yields a different Evolutionary Operator and when applied to our Domain Space it yields a new Domain Model.

What Have We Gained

In this exercise we have stressed the importance of defining a Domain Space. A broad malleable space that represents the context of your business is key. Remember when defining this, your not capturing the state of the Domain Experts view yet, your capturing the real world physical variability and time dependent evolutionary vectors of each noun and verb within your Domain.

Finally

By modeling a Domain Space you create a nearly infinite amount of Domain Models. Each point in a Domain Experts evolution can yield an executable Domain Model derived from the Domain Space.

It's this evolutionary robustness the differentiates a Domain Model from a Domain Space.

Just food for thought we you design your next project.

Imagine, if you will, that in the course of designing a new api you have the requirement to implement a complex process. The approach presented here will produce a completely message driven architecture that can be implemented and integrated with any messaging infrastructure that you choose. This post will cover one way in which you decouple your complex process into a sequence of phases decoupled from the calling code by a messaging system. This system is similar to a pervious post about a simple actor system. We're also going to extend the concept presented in that post and marry the concepts with ideas presented in the post message routing.

Challenge

The challenge here is to derive a generic Process Management System. This system should be capable of driving a user defined process and do so in completely decoupled message driven way.

Solution

Infrastructure

First we'll cover the infrastructure of our solution. These components will lay the ground work for our sample.

The Messages

This implementation is going to be completely message driven and independent of the request context. The Process Handlers will live as BackgroundServicesand always be alive and ready to process messages. For this system to work first we need to define some base Message records.

public record ProcessStartMessage<TInput>(TInput Input);

public record ProcessStartedMessage(Guid ProcessId);

public record ProcessEndedMessage(Guid ProcessId, ProcessPhase Phase);

public record ProcessActivity;

public record ProcessMessage<TInput>(Guid ProcessId, ProcessPhase Phase, TInput Input);

public record ProcessMessage<TInput, TActivity>(Guid ProcessId, ProcessPhase Phase, TInput Input, TActivity Activity)
    : ProcessMessage<TInput>(ProcessId, Phase, Input)
    where TActivity : ProcessActivity
    where TInput : ProcessInput;

public record ProcessInput;

public record ProcessPhase(string Phase)
{
    public static ProcessPhase Completed = new ProcessPhase(nameof(Completed));
    public static ProcessPhase Failed = new ProcessPhase(nameof(Failed));
}

Let's go through each message and base type.

• ProcessStartMessage: This message signals the ProcessStartHandler to begin the defined process, this message is used as a type argument constarint and should be the base class of a derived process specific message as we'll see in amoment.
• ProcessStartedMessage: This is the message that will be published when the process begins and provides the id assigned to the process, so that calling code can use the process id to query information about the state of the process. In this post we'll leave most of the calling code out and focus on the Process Management System
• ProcessEndedMessage: This message signals the calling code that the process has ended, it provides the process id and the phase of the process when the process stopped. This incorporation of the Phase can tell the client whether the process aborted or completed successfully.
• ProcessMessage<TInput>: This is base of all process messages, this is what we'll use to constrain our PhaseTransitionHandler such that it has access to the properties we need.
• ProcessMessage<TInput, TActivity>: This message along with an Activity type will be used throughout the process lifetime. The Activity parameter is associated with the ProcessMessage with a particular activity that's being carried out. This message also carries along with it the full input object, the current phase of process and the id of the process to which it's associated.
• ProcessActivity and ProcessInput: are just constraint records to be derived from for an Activity or Input respectively.
• ProcessPhase: Process Phase defines the high level general state of the process. For example a derived ProcessPhase could be: Validation, StageRollback, StageInsert. We can derive and define our process in terms of phases. Each phase may generate N messages with N activities. In the base class we define two general Phases: Completed and Failed these two phases are made common to all derived phase sets. I've chose this style of defining phases instead of something like enumeration because personally I don't enums and this record gives us a better constraint since we cannot derive from a base enum.

A process is a set of messages that are initiated by phase transition. A high level view of a process can be seen in this figure

As you can see each phase emits N messages to be collected by the Process Phase Transition Handler. This handler then determines if a phase transition is necessary and if so generates the next phase of messages. If the Process Phase Transition Handler determines the process is Complete a completed message is emitted and the process is over.

The Message System

Let's now look at the communication stream. This stream abstracts our underlying messaging system into a simple interface with just the methods we need. We could implement this interface with: Channels, Observables, Kafka, Azure Servicebus, SignalR, Azure Pub/Sub or any other messaging system we choose so long as this interface is met.

public interface IMessageSystemReader<TMessage>
{
    public ValueTask<bool> MessageReady(CancellationToken token = default);
    public IAsyncEnumerable<TMessage> ReadAllAsync(CancellationToken token = default);
    public Task Completion { get; }
}

public interface IMessageSystemWriter<TMessage>
{
    public ValueTask WriteAsync(TMessage message, CancellationToken token = default);
    public Task Shutdown();
}

Using the Interface Segregation principle we break this interface down into two, one reader and one writer. These interfaces also define a way to shutdown the stream and and notify the client that the stream is now closed.

In this post we'll use channels as our underlying transport.

public class MessageSystem<TMessage>
    : IMessageSystemReader<TMessage>,
    IMessageSystemWriter<TMessage>
{
    private ChannelReader<TMessage> reader;
    private ChannelWriter<TMessage> writer;

    public MessageSystem()
    {
        var channel = Channel.CreateUnbounded<TMessage>();
        reader = channel.Reader;
        writer = channel.Writer;
    }

    public Task Completion => reader.Completion;

    public ValueTask<bool> MessageReady(CancellationToken token = default)
        => reader.WaitToReadAsync(token);

    public IAsyncEnumerable<TMessage> ReadAllAsync(CancellationToken token = default)
        => reader.ReadAllAsync(token);

    public ValueTask WriteAsync(TMessage message, CancellationToken token = default)
        => writer.WriteAsync(message, token);

    public Task Shutdown()
    {
        writer.Complete();
        return reader.Completion;
    }
}

Message System Extensions

As you can see the implementation is just a simple wrapper around a pair of Channel reader/writers. Next let's add some helper methods to this interface such that all we have to worry about in our Handler code is the logic necessary to handle our defined process.

public static class MessagingExtensions
{
    public static async IAsyncEnumerable<TMessage> ContinuousWaitAndReadAllAsync<TMessage>(
        this IMessageSystemReader<TMessage> reader,
        [EnumeratorCancellation] CancellationToken token = default)
    {
        while (!reader.Completion.IsCompleted && !token.IsCancellationRequested)
        {
            var readerReady = await reader.MessageReady(token);
            if (readerReady)
                await foreach (var message in reader.ReadAllAsync(token))
                    yield return message;
        }
    }

    public static async ValueTask RouteMessageByTypeAsync(this IWriterProvider provider, object message)
    {
        var writer = provider.RequestWriter(message);
        await writer.WriteAsync((dynamic)message);
    }

    public static async ValueTask<bool> SendIfEnding(this ProcessPhase phase, IWriterProvider provider, Func<ProcessEndedMessage> endedMessageFatory)
    {
        if (phase == ProcessPhase.Completed || phase == ProcessPhase.Failed)
        {
            await provider.RouteMessageByTypeAsync(endedMessageFatory());
            return true;
        }
        return false;
    }

    public static bool ContainsActivity<TActivity>(this IEnumerable<dynamic> messages)
        where TActivity : ProcessActivity 
        => messages.Any(msg => msg.Activity.GetType() == typeof(TActivity));
}

We have one method that wraps the reader waiting and receiving into one easy async enumerator. Then we have the writer method that dispatches a message by it's type. And we have one extension to determine if the process has completed or failed and send out the ended message. And we also have a method to determine what activies exist in the current set of messages. This begs the question; Ok, well what's the WriterProvider?

The Writer Provider

public class WriterProvider : IWriterProvider
{
    private readonly IServiceProvider serviceProvider;

    public WriterProvider(IServiceProvider serviceProvider)
    {
        this.serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
    }

    public dynamic RequestWriter(object message)
    {
        var writerType = typeof(IMessageSystemWriter<>).MakeGenericType(message.GetType());
        dynamic writer = serviceProvider.GetService(writerType);
        if (writer is null)
            throw new ArgumentException($"Message Writer not registered for {message.GetType().Name}");
        return writer;
    }
}

You may ask, why do we not just use a TMessage type argument instead of message.GetType(), well this is done because TMessage could be a base type and we want the Writer we get to be strongly typed to the actual message that's being sent. Our WriterProvider simply hides the IServiceProvider and uses it to retrieve the correct writer. Remember now that the writer type can be implemented with nearly any transport that can at least satisfy the basic interface. So if we change transports all our code doesn't need to change only the implementation of IMessageSystem<TMessage> needs to change.

Dynamic is chosen for the return type because, remember, we used message.GetType() for the generic type we got from the IServiceProvider. We want to preserve that type and not constrain it to a TMessage which may be a base type.

Process Start

Next let's see how we start a process. This is an abstract class that implements the vast majority of functionality any derived handler would need. The responsibilities of this handler are quite simple. It lives as a BackgorundService and listens for a particularly defined TStartMessage. Once a message is received it responds with a defined overridden StartedMessageFactory result. Then it emits the first Phase messages down the line to the next part of the system.

public abstract class ProcessStartHandler<TStartMessage, TInput>
    : BackgroundService
    where TStartMessage : ProcessStartMessage<TInput>
    where TInput : ProcessInput
{
    private readonly IMessageSystemReader<TStartMessage> startReader;
    private readonly IWriterProvider writerProvider;
    private readonly IMessageFactory<TInput> messageFactory;
    private readonly ProcessPhase startPhase;

    public ProcessStartHandler(
        IWriterProvider writerProvider,
        IMessageFactory<TInput> messageFactory,
        IMessageSystemReader<TStartMessage> startReader,
        ProcessPhase startPhase)
    {
        this.startReader = startReader ?? throw new ArgumentNullException(nameof(startReader));
        this.writerProvider = writerProvider ?? throw new ArgumentNullException(nameof(writerProvider));
        this.messageFactory = messageFactory ?? throw new ArgumentNullException(nameof(messageFactory));
        this.startPhase = startPhase ?? throw new ArgumentNullException(nameof(startPhase));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var message in startReader.ContinuousWaitAndReadAllAsync(stoppingToken))
        {
            var processId = Guid.NewGuid();
            await SendStartedMessage(processId);
            await SendFirstPhaseMessages(processId, message);
        }
    }

    private async ValueTask SendStartedMessage(Guid processId)
    {

        var message = StartedMessageFactory(processId);
        await writerProvider.RouteMessageByTypeAsync(message);
    }

    private async ValueTask SendFirstPhaseMessages(Guid processId, TStartMessage message)
    {
        await foreach (var newMessage in messageFactory[startPhase](processId, startPhase, message.Input, Enumerable.Empty<object>()))
        {
            await writerProvider.RouteMessageByTypeAsync(newMessage);
        }

    }

    protected abstract ProcessStartedMessage StartedMessageFactory(Guid processId);
}

With our handy extension methods in use for both reading and writing our code in here is straight to the point: Listen for start, Emit started with ProcessId & Emit first phase messages. Now you may be asking; What's the message factory and why is it needed? Well we don't want our handlers to have too much responsibility. The start handler already covers a lot of functionality and it shouldn't be responsible for knowing how to format/create/generate the first phase messages. If it were to be responsible for that we could end up with a huge list of dependencies and even new type parameters to deal with. This would be unsatisfactory, so we delegate to a single purpose object: the message factory.

The Message Factory

public interface IMessageFactory<TInput> 
    : IReadOnlyDictionary<ProcessPhase, Func<ProcessPhase, ProcessInput, IEnumerable<object>, IAsyncEnumerable<object>>>
    where TInput : ProcessInput
{

}

public abstract class MessageFactory<TInput>
    : Dictionary<ProcessPhase, Func<Guid, ProcessPhase, TInput, IEnumerable<object>, IAsyncEnumerable<object>>>,
    IProcessMessageFactory<TInput>
    where TInput : ProcessInput
{

}

As you can see this is a generic interface that simply needs the type of TInput and the rest is quite simple. Based on a ProcessPhase key we get a Func that takes the current phase, the ProcessInput and a collection that is describing the currently received messages for this process. With these inputs it generates a batch of messages to be sent out. When this interface is implemented each phase of the implementing process must be accounted for.

We also have a handy abstract class for implementations to inherit from such that all they are concerned with is defining what messages come from specific keys.

Phase Transitioning

Next, we're going to look at the next abstract BackgroundService, this is the only other operating service in this system. This Handler listens for any of the predefined TMessage, collects them and uses both the MessageFactory we've seen and the upcoming PhaseTransitions factory.

public abstract class PhaseTransitionHandler<TMessage, TInput>
    : BackgroundService
    where TMessage : ProcessMessage<TInput>
    where TInput : ProcessInput
{
    private readonly IWriterProvider writerProvider;
    private readonly IPhaseTransitions<TInput> phaseTransitions;
    private readonly IMessageFactory<TInput> messageFactory;
    private readonly IMessageSystemReader<TMessage> reader;

    private ConcurrentDictionary<Guid, ICollection<object>> MessagesReceived = new ConcurrentDictionary<Guid, ICollection<object>>();

    public PhaseTransitionHandler(
        IWriterProvider writerProvider,
        IPhaseTransitions<TInput> phaseTransitions,
        IMessageFactory<TInput> messageFactory,
        IMessageSystemReader<TMessage> reader)
    {
        this.writerProvider = writerProvider ?? throw new ArgumentNullException(nameof(writerProvider));
        this.phaseTransitions = phaseTransitions ?? throw new ArgumentNullException(nameof(phaseTransitions));
        this.messageFactory = messageFactory ?? throw new ArgumentNullException(nameof(messageFactory));
        this.reader = reader ?? throw new ArgumentNullException(nameof(reader));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var message in reader.ContinuousWaitAndReadAllAsync(stoppingToken))
        {
            if (MessagesReceived.TryGetValue(message.ProcessId, out var messages))
                messages.Add(message);
            else
                MessagesReceived[message.ProcessId] = new List<object>() { message };

            var nextPhase = phaseTransitions[message.Phase](message.Phase, message.Input, MessagesReceived[message.ProcessId]);
            if (await nextPhase.SendIfEnding(writerProvider, () => EndedMessageFactory(message.ProcessId, nextPhase)))
            {
                MessagesReceived.Remove(message.ProcessId, out var _);
                await CompletedEvent(nextPhase, message);
            }
            else if (ShouldTransition(message.Phase, nextPhase))
            {
                await PhaseTransitionEvent(nextPhase, message);
                await foreach (var newMessage in messageFactory[nextPhase](message.ProcessId, nextPhase, message.Input, MessagesReceived[message.ProcessId]))
                {
                    await writerProvider.RouteMessageByTypeAsync(newMessage);
                }
            }
        }
    }

    protected abstract ProcessEndedMessage EndedMessageFactory(Guid processId, ProcessPhase phase);

    protected virtual ValueTask CompletedEvent(ProcessPhase phase, TMessage message)
        => ValueTask.CompletedTask;

    protected virtual ValueTask PhaseTransitionEvent(ProcessPhase phase, TMessage message) 
        => ValueTask.CompletedTask;

    private bool ShouldTransition(ProcessPhase currentPhase, ProcessPhase newPhase)
        => !currentPhase.Equals(newPhase);
}

This handler listens for any of defined TMessage types and collects them by ProcessId. Each time a message is received and stored the PhaseTranstions component is used to check whether or not to transition to a new phase. If a Phase transition is required the handler then uses the MessageFactory to produce and then send the next batch of messages. This handler also provides hooks into the process for both the CompletedEvent and the PhaseTransitionEvent such that inheriters can perform any extra logic they might want on those two events.

The Phase Transition Dictionary

Similar to the MessageFactory our PhaseTransition dictionary is keyed by ProcessPhase and produces a function that takes: the current phase, the original input, a collection of returned message and produces a ProcessPhase if the phase produced is different from the current phase, the PhaseTransitionHandler responds appropriately by moving on to the next batch of messages.

public interface IPhaseTransitions<TInput>
    : IReadOnlyDictionary<ProcessPhase, Func<ProcessPhase, TInput, IEnumerable<object>, ProcessPhase>>
    where TInput : ProcessInput
{

}

public abstract class PhaseTransitions<TInput>
    : Dictionary<ProcessPhase, Func<ProcessPhase, TInput, IEnumerable<object>, ProcessPhase>>,
    IPhaseTransitions<TInput>
    where TInput : ProcessInput
{

}

Again we have a handy abstract base class that inherits a Dictionary so all the implementation has to do is fill itself with the correct Phase Transitions.

The Worker

So now we have a whole system intended to produce messages based on the phase of a process. But we still need something to process those messages and do the actual work those messages entail. Enter the Command Handler, we could call this an Actor but to fit the terminology used here we'll just refer to it as a Handler. I already posted about Query Actors which take a query and then return a response. Since our process is completely async we won't be using the request/response architecture described in that post. Instead our command handler will process the incoming message or command and send out a response.

public abstract class CommandHandler<TMesssage> : BackgroundService
{
    protected readonly IMessageSystemReader<TMesssage> reader;

    public CommandHandler(IMessageSystemReader<TMesssage> reader)
    {
        this.reader = reader ?? throw new ArgumentNullException(nameof(reader));
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var command in reader.ContinuousWaitAndReadAllAsync(stoppingToken))
        {
            await HandleAsync(command, stoppingToken);
        }
    }

    public abstract ValueTask HandleAsync(TMesssage command, CancellationToken stoppingToken);
}

The Registrations

Finally we need to register all the components with our DI. This is done through a set of extension methods

public static class RegistrationExtensions
{
    public static IServiceCollection AddMessageSystem<TMessage>(this IServiceCollection services)
    {
        var messageSystem = new MessageSystem<TMessage>();
        services.AddSingleton<IMessageSystemWriter<TMessage>>(messageSystem);
        services.AddSingleton<IMessageSystemReader<TMessage>>(messageSystem);
        return services;
    }

    public static IServiceCollection AddCommandHandler<TActor, TCommand>(this IServiceCollection services)
        where TActor : CommandHandler<TCommand> 
        => services
            .AddHostedService<TActor>()
            .AddMessageSystem<TCommand>();

    public static IServiceCollection AddWriterProvider(this IServiceCollection services)
        => services.AddSingleton<IWriterProvider, WriterProvider>();

    public static IServiceCollection AddProcessStartHandler<THandler, TMessageFactory, TStartMessage, TEndMessage, TInput>(this IServiceCollection services)
        where THandler : ProcessStartHandler<TStartMessage, TInput>
        where TMessageFactory : MessageFactory<TInput>
        where TStartMessage : ProcessStartMessage<TInput>
        where TInput : ProcessInput
        => services
            .AddHostedService<THandler>()
            .AddSingleton<IMessageFactory<TInput>, TMessageFactory>()
            .AddMessageSystem<TStartMessage>()
            .AddMessageSystem<TEndMessage>();

    public static IServiceCollection AddPhaseTransitionHandler<THandler, TPhaseTransitions, TMessage, TInput>(this IServiceCollection services)
        where THandler : PhaseTransitionHandler<TMessage, TInput>
        where TPhaseTransitions : PhaseTransitions<TInput>
        where TMessage : ProcessMessage<TInput>
        where TInput : ProcessInput 
        => services
            .AddHostedService<THandler>()
            .AddSingleton<IPhaseTransitions<TInput>, TPhaseTransitions>()
            .AddMessageSystem<TMessage>();
}

These extension methods are all we need to register our entire Process Management System. The first registers the implementation of our Message System based on a defined message type parameter. The next, registers our Command Handler and it's associated Message System The next just registers our Writer Provider. Then we register the Process Start Handler along with the Message Factory and the appropriate input and output channels. And, finally we register the Phase Transition Handler, the Phase Transitions Dictionary and it's input channels.

Sample Demo

Overview

Now that we have all the infrastructure in place it's time to make use of it. For this example we'll be doing a basic "Hello World" type process. We'll define all our messages, phases, and handlers. The goal is to have process that has two steps: Validate the input, Respond with a message.

The Messages

The messages for our sample define the ways in which downstream consumers will respond and how we expect them to act. We define our process as a series of messages. This sample process will first validate our input with some simple validation, then it will write out the message to consumers. This process extends the ProcessPhase by defining two additional phases of our process Validation and SayHello.

public record SayHelloInput(string Name)
    : ProcessInput;

public record SayHelloProcessStartMessage(SayHelloInput Input)
    : ProcessStartMessage<SayHelloInput>(Input);

public record SayHelloProcessStartedMessage(Guid ProcessId)
    : ProcessStartedMessage(ProcessId);

public record SayHelloEndedMessage(Guid ProcessId, ProcessPhase Phase)
        : ProcessEndedMessage(ProcessId, Phase);

public record SayHelloProcessMessage<TActivity>(Guid ProcessId, ProcessPhase Phase, SayHelloInput Input, TActivity Activity)
    : ProcessMessage<SayHelloInput, TActivity>(ProcessId, Phase, Input, Activity)
    where TActivity : ProcessActivity;

public record SayHelloValidation(SayHelloInput Input)
        : ProcessActivity;

public record SayHelloValidationSuccess
    : ProcessActivity;

public record SayHelloValidationFailure
    : ProcessActivity;

public record SayHelloResponseActivity(string Input)
    : ProcessActivity;

public record SayHelloCompletedActivity
    : ProcessActivity;

public record SayHelloResponseMessage(string message);

public record SayHelloPhases(string Phase)
        : ProcessPhase(Phase)
{
    public static ProcessPhase Validation = new ProcessPhase(nameof(Validation));
    public static ProcessPhase SayHello = new ProcessPhase(nameof(SayHello));
}

These messages, activities and inputs define our Say Hello process. We start with the start message that carries the user input into our process stream. We have a derived started message to signal the client that the process has started and a derived ended message to signal that the process has ended in a particular phase. Next we have a derived process message that will be used to carry out our process doing specified activities. We have defined four activities for this process SayHelloValidation SayHelloValidationSuccess, SayHelloValidationFailure and SayHelloRepsonse. These activities will be coordinated by our PhaseTransition handler and our eventual message factory we'll see in a moment. And finally we defined the phases of our process as Validation and SayHello, remember our base record ProcessPhase already defines Completed and Failed for us so all we need to do is define the working phases of our process.

The Process Handlers

Next we define our derived handlers for our specific process, first the Start Handler

public class SayHelloStartHandler : ProcessStartHandler<SayHelloProcessStartMessage, SayHelloInput>
{
    public SayHelloStartHandler(
        IWriterProvider writerProvider,
        IMessageFactory<SayHelloInput> messageFactory,
        IMessageSystemReader<SayHelloProcessStartMessage> startReader)
        : base(writerProvider, messageFactory, startReader, SayHelloPhases.Validation)
    {
    }

    protected override ProcessStartedMessage StartedMessageFactory(Guid processId)
        => new SayHelloProcessStartedMessage(processId);
}

Since we don't need any additional functionality, all we do is specify the working types and implement the message factory and constructor and we're done with this! Then for our concreate Message Factory

public class SayHelloMessageFactory : MessageFactory<SayHelloInput>
{
    public SayHelloMessageFactory()
    {
        this[SayHelloPhases.Validation] = (processId, phase, input, messages) => ValidationPhaseMessages(processId, phase, input, messages);
        this[SayHelloPhases.SayHello] = (processId, phase, input, messages) => SayHelloPhaseMessages(processId, phase, input, messages);
    }

    public IAsyncEnumerable<object> ValidationPhaseMessages(Guid processId, ProcessPhase phase, SayHelloInput input, IEnumerable<dynamic> currentMessages)
        => new[]
        {
            new SayHelloProcessMessage<SayHelloValidation>(processId, phase, input, new SayHelloValidation(input))
        }.ToAsyncEnumerable();

    public IAsyncEnumerable<object> SayHelloPhaseMessages(Guid processId, ProcessPhase phase, SayHelloInput input, IEnumerable<dynamic> currentMessages)
        => new[]
        {
            new SayHelloProcessMessage<SayHelloResponseActivity>(processId, phase, input, new SayHelloResponseActivity(input.Name))
        }.ToAsyncEnumerable();
}

Since we only have two phases and each phase only produces one message our Message Factory is quite simple in this case. Next we define our Phase Transition Handler

public class SayHelloPhaseTransistionsHandler : PhaseTransitionHandler<SayHelloProcessMessage<ProcessActivity>, SayHelloInput>
{
    public SayHelloPhaseTransistionsHandler(
        IWriterProvider writerProvider,
        IPhaseTransitions<SayHelloInput> phaseTransitions,
        IMessageFactory<SayHelloInput> messageFactory,
        IMessageSystemReader<SayHelloProcessMessage<ProcessActivity>> reader)
        : base(writerProvider, phaseTransitions, messageFactory, reader)
    {
    }

    protected override ProcessEndedMessage EndedMessageFactory(Guid processId, ProcessPhase phase)
        => new SayHelloEndedMessage(processId, phase);
}

And again no special behavior is needed so all we're doing is setting type arguments, and implementing the constructor and ended message factory. Finally, our Phase Transitions gets defined.

public class SayHelloPhaseTransitions : PhaseTransitions<SayHelloInput>
{
    public SayHelloPhaseTransitions()
    {
        this[SayHelloPhases.Validation] = (phase, input, messages) => ValidationPhaseTransition(phase, input, messages);
        this[SayHelloPhases.SayHello] = (phase, input, messages) => SayHelloPhaseTransition(phase, input, messages);
    }

    public ProcessPhase ValidationPhaseTransition(ProcessPhase phase, SayHelloInput input, IEnumerable<dynamic> currentMessages)
    {
        if (currentMessages.ContainsActivity<SayHelloValidationSuccess>())
            return SayHelloPhases.SayHello;
        else if (currentMessages.ContainsActivity<SayHelloValidationFailure>())
            return SayHelloPhases.Failed;
        return phase;
    }

    public ProcessPhase SayHelloPhaseTransition(ProcessPhase phase, SayHelloInput input, IEnumerable<dynamic> currentMessages)
    {
        if (currentMessages.ContainsActivity<SayHelloCompletedActivity>())
            return SayHelloPhases.Completed;
        return phase;
    }
}

Note how we use the state provided to compute the next phase of the process, if validation fails we move directly to failed, if validation succeeds we continue on to the next phase. Once the SayHelloCompletedActivity has been emitted we mark the process as Completed.

The Workers

First we implement our validation handler. The validation we're doing here is to simply check that the string is not too long.

public class SayHelloValidationHandler
    : CommandHandler<SayHelloProcessMessage<SayHelloValidation>>
{
    private readonly IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> successWriter;
    private readonly IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> failureWriter;

    public SayHelloValidationHandler(
        IMessageSystemReader<SayHelloProcessMessage<SayHelloValidation>> reader,
        IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> successWriter,
        IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> failureWriter)
        : base(reader)
    {
        this.successWriter = successWriter ?? throw new ArgumentNullException(nameof(successWriter));
        this.failureWriter = failureWriter ?? throw new ArgumentNullException(nameof(failureWriter));
    }

    public override async ValueTask HandleAsync(SayHelloProcessMessage<SayHelloValidation> command, CancellationToken stoppingToken)
    {
        var input = command.Input.Name;
        if (input.Length > 10)
            await failureWriter.WriteAsync(
                new SayHelloProcessMessage<ProcessActivity>(command.ProcessId, command.Phase, command.Input, new SayHelloValidationFailure()));
        else
            await successWriter.WriteAsync(
                new SayHelloProcessMessage<ProcessActivity>(command.ProcessId, command.Phase, command.Input, new SayHelloValidationSuccess()));
    }
}

Notice how we only need to implement the HandleAsync method and are provided with read command from our Message System. If validation passes we emit a successful activity, if validation fails we emit a failed activity and the Phase Transition Dictionary and Handler will take it from there. Next we define our Say Hello Handler

public class SayHelloResponseHandler
    : CommandHandler<SayHelloProcessMessage<SayHelloResponseActivity>>
{
    private readonly IMessageSystemWriter<SayHelloResponseMessage> responseWriter;
    private readonly IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> completedWriter;

    public SayHelloResponseHandler(
        IMessageSystemReader<SayHelloProcessMessage<SayHelloResponseActivity>> reader,
        IMessageSystemWriter<SayHelloResponseMessage> responseWriter,
        IMessageSystemWriter<SayHelloProcessMessage<ProcessActivity>> completedWriter)
        : base(reader)
    {
        this.responseWriter = responseWriter ?? throw new ArgumentNullException(nameof(responseWriter));
        this.completedWriter = completedWriter ?? throw new ArgumentNullException(nameof(completedWriter));
    }

    public override async ValueTask HandleAsync(SayHelloProcessMessage<SayHelloResponseActivity> command, CancellationToken stoppingToken)
    {
        var message = $"Hello there, {command.Input.Name}";
        await responseWriter.WriteAsync(new SayHelloResponseMessage(message));
        await completedWriter.WriteAsync(new SayHelloProcessMessage<ProcessActivity>(command.ProcessId, command.Phase, command.Input, new SayHelloCompletedActivity()));
    }
}

Again all we're doing is implementing the HandleAsync method with our logic which is to simply emit a response message and emit a completed message back to our infrastructure.

The Registrations

Next we register all our process's components with simple extension method.

public static class RegistrationExtensions
{
    public static IServiceCollection AddSayHelloProcess(this IServiceCollection services)
        => services
        .AddWriterProvider()
        .AddProcessStartHandler<SayHelloStartHandler, SayHelloMessageFactory, SayHelloProcessStartMessage, SayHelloEndedMessage, SayHelloInput>()
        .AddPhaseTransitionHandler<SayHelloPhaseTransistionsHandler, SayHelloPhaseTransitions, SayHelloProcessMessage<ProcessActivity>, SayHelloInput>()
        .AddCommandHandler<SayHelloValidationHandler, SayHelloProcessMessage<SayHelloValidation>>()
        .AddCommandHandler<SayHelloResponseHandler, SayHelloProcessMessage<SayHelloResponseActivity>>()
        .AddMessageSystem<SayHelloResponseMessage>()
        .AddMessageSystem<SayHelloProcessStartedMessage>();
}

Note how we add dedicated Message Systems for our Started Message and our ResponseMessage

The Client Code

We'll keep our client code simple and not respond to any failure and have it simply timeout if a successful message is not received, in this post we're not interested in elaborate clients. We're simply proving the orchestration of handlers works.

[ApiController]
[Route("[controller]")]
public class SampleController : ControllerBase
{
    private readonly IMessageSystemWriter<SayHelloProcessStartMessage> startWriter;
    private readonly IMessageSystemReader<SayHelloResponseMessage> responseReader;

    public SampleController(
        IMessageSystemWriter<SayHelloProcessStartMessage> startWriter,
        IMessageSystemReader<SayHelloResponseMessage> responseReader)
    {
        this.startWriter = startWriter ?? throw new ArgumentNullException(nameof(startWriter));
        this.responseReader = responseReader ?? throw new ArgumentNullException(nameof(responseReader));
    }

    [HttpGet]
    public async ValueTask<SayHelloResponseMessage> Get(string name)
    {
        var startMessage = new SayHelloProcessStartMessage(new SayHelloInput(name));
        await startWriter.WriteAsync(startMessage);
        using var readTokenSource = new CancellationTokenSource();
        readTokenSource.CancelAfter(TimeSpan.FromSeconds(1));
        return await responseReader.ContinuousWaitAndReadAllAsync(readTokenSource.Token).FirstAsync();
    }
}

Here we take a writer for the Start Message and pass our input. We also take a reader for the output and provide a CancellationToken to timeout the request if anything fails. We also don't distinguish between Response Messages for individual requests, we could identify them by Process Id but we'll leave that for next time.

And finally we can try everything out and receive a response!

Summary

Here we've defined a generic infrastructure to orchestrate complex processes based on two handlers, two dictionaries and a set of messages. The beautiful thing about this system is that it can fundamentally be used with any underlying transport. Complex process are broken down into simple phases, messages and transitions.

Well I hoped you enjoyed my latest. As always the full code can be found at ConcurrentFlows.HashNode GitHub

Take an AspNet Core Api, for example, we have chosen for this; a pattern of Queries/Commands and Handlers. We're going to explore this pattern and see what we can add to the body of knowledge. When implementing the Command/Query/Handler pattern we often end up with many small classes and specialized handlers. This is great for testing and can lead to excellent code organization and separation of concerns. But when integrating with Controllers we find the many handlers need to be injected and we tend towards the mediator pattern. One great implementation of this is MediatR, or you can go a similar route taken by one of the great maintainers of SimpleInjector, who proposes a simple query dispatcher in the great post Meanwhile... on the query side of my architecture. While that post is dated all the way back to 2011, it's still a fantastic reference for what I refer to as Application Scale CQRS.

CQRS At Different Levels

The Command Query Responsibility Segregation principles can be applied at various levels of your architecture. At the Systems Level we see Event Sourcing and a separation of read/write data models. But at the Application Scale we see a breakdown of classes. From monolithic Services and Repositories to single purpose Handlers. This provides a great separation of concerns and extensibility while keeping code cohesive within a single domain.

The Dispatch Issue

One common problem we find in the Mediator Pattern is the need to dynamically invoke un-instantiated handlers at runtime. Even with the super fast SimpleInjector this can take time especially if we're invoking a complex web of handlers to accomplish a given task. So the question becomes how can we retain the benefits of Application Scale CQRS while avoiding the runtime costs of instantiation of handlers.

The Actor Pattern

The Actor Pattern represents a shift in thinking. Instead of request scoped handlers I propose a pool of interconnected communicating handlers. While Matt Ferderer defines the properties of an actor to include:

• Store Data
• Receive messages from other actors
• Pass messages to other actors
• Create additional child actors

I'm going to take a different angle.

Our Goals: A simple hosted actor system

I'm going to redefine an Actor a little a differently for our api specific use case, an Actor should:

• Live outside the request stream
• Be able to receive messages asynchronously
• Return a result asynchronously
• Communicate with other actors in the system without instantiating them
• Finally, be decoupled from the calling code

In order to achieve these goals we're going to combine elements of the Mediator Pattern and the Actor Pattern. And just for examples sake we're going to focus on the Query side of CQRS, the Command side is much simpler and maybe the topic of a future post.

Our Solution

The Basic Infrastrucuture

First we start with the basic infrastructure that will make up our system. You surely have seen many iterations of IQuery, IQueryHandler , etc. However, we're going to change them slightly to fit with our Actor approach. For starters we define an ActorQuery as such:

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Infrastructure
{
    public record ActorQuery<TPayload, TAnswer>(TPayload Payload);
}

Note in this record we define the payload that we will submit to the actor system and we define the type of answer we expect back. We intentionally use the term Answer to distinguish it form a typical Task Result. This aligns more so with the concept of; Every query has an answer. Next we define our base hosted QueryActor this actor will be resoponsible for reciveing the input payload and emitting the answer to the query

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Infrastructure
{
    public abstract class QueryActor<TQuery>
        : BackgroundService
    {
        protected readonly ChannelReader<KeyValuePair<Guid, TQuery>> queryReader;
        protected readonly ChannelWriter<KeyValuePair<Guid, dynamic>> answerWriter;

        public QueryActor(
            ChannelReader<KeyValuePair<Guid, TQuery>> queryReader, 
            ChannelWriter<KeyValuePair<Guid, dynamic>> answerWriter)
        {
            this.queryReader = queryReader ?? throw new ArgumentNullException(nameof(queryReader));
            this.answerWriter = answerWriter ?? throw new ArgumentNullException(nameof(answerWriter));
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested && !queryReader.Completion.IsCompleted)
            {
                var messageReady = await queryReader.WaitToReadAsync(stoppingToken);
                if (messageReady)
                {
                    await foreach (var query in queryReader.ReadAllAsync(stoppingToken))
                    {
                        await HandleAsync(query, stoppingToken);
                    }
                }
            }
        }

        public abstract Task HandleAsync(KeyValuePair<Guid, TQuery> query, CancellationToken stoppingToken);
    }
}

Our base QueryActor takes a ChannelReader that accepts the input query keyed with a unique identifier. This unique identifier will be used to connect Answers with Queries. We also make use of the dynamic type and an unconstrained query type parameter to allow this base Actor to handle any query and any answer. This approach is shown now in the what connects our Actor System together: the AnswerStream.

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Infrastructure
{
    public interface IAnswerStream : IHostedService
    {
        ValueTask<dynamic> SubmitQuery<TQuery>(TQuery query);
    }
}

First we've defined a simple interface to interact with our Actor System. This interface allows us to submit a query and await a result from Actor System. Notice also that we e're extending the IHostedService interface to allow this stream to live outside the request scope and serve it's primary purpose of matching Answers to Queries

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Infrastructure
{
    public class AnswerStream : BackgroundService, IAnswerStream
    {
        private readonly IServiceProvider serviceProvider;
        private readonly ChannelReader<KeyValuePair<Guid, dynamic>> answerReader;

        public AnswerStream(
            IServiceProvider serviceProvider, 
            ChannelReader<KeyValuePair<Guid, dynamic>> answerReader)
        {
            this.serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider));
            this.answerReader = answerReader ?? throw new ArgumentNullException(nameof(answerReader));
            QueryResults = new ConcurrentDictionary<Guid, TaskCompletionSource<dynamic>>();
        }

        private ConcurrentDictionary<Guid, TaskCompletionSource<dynamic>> QueryResults { get; }

        public async ValueTask<dynamic> SubmitQuery<TQuery>(TQuery query)
        {
            var writer = serviceProvider.GetRequiredService<ChannelWriter<KeyValuePair<Guid, TQuery>>>();
            var queryId = Guid.NewGuid();
            var resultSource = new TaskCompletionSource<dynamic>();
            QueryResults.TryAdd(queryId, resultSource);
            await writer.WriteAsync(new KeyValuePair<Guid, TQuery>(queryId, query));
            return await resultSource.Task;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            while (!stoppingToken.IsCancellationRequested && !answerReader.Completion.IsCompleted)
            {
                var resultsReady = await answerReader.WaitToReadAsync(stoppingToken);
                if (resultsReady)
                    await foreach (var result in answerReader.ReadAllAsync(stoppingToken))
                        if (QueryResults.TryRemove(result.Key, out var resultSource))
                            resultSource.TrySetResult(result.Value);
            }
        }
    }
}

We see in this implementation the only use of runtime resolution is the instantiation of a singleton ChannelWriter to input the supplied query into our Actor System. This is a low cost operation since the ChannelWriter will be held and made available as a singleton in our ServiceProvider. We see also, that when a query is submitted it is asigned an unquie identifer that will later be used to match it up with its results, it is also associated with a TaskCompletionSource<TAnswer> and this associated Task is returned to the caller to await the results from our AnswerStream. The ExecuteAsync method continuously waits for results to match up with assigned queries and manages the ConcurrentDictionary that stores the associated TaskCompletionSources. And again, we use dynamic for the answer type to allow this AnswerStream to handle any query and any Answer. More could be done here to better manage the ConcurrentDictionary but this is just basic working sample.

The Last piece of infrastructure we need is way to register our actor system with the ServiceProvider. To accomplish that we create an extension method to add all the necessary bit and pieces to our DI.

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Infrastructure
{
    public static class RegistrationExtensions
    {
        public static IServiceCollection AddActor<TActor, TQuery>(this IServiceCollection services)
            where TActor : QueryActor<TQuery>
        {            
            services.AddHostedService<TActor>();
            var inputChannel = Channel.CreateUnbounded<KeyValuePair<Guid, TQuery>>();
            services.AddSingleton(inputChannel.Writer);
            services.AddSingleton(inputChannel.Reader);            
            return services;
        }

        public static IServiceCollection AddAnswerStream(this IServiceCollection services)
        {
            services.AddSingleton<IAnswerStream, AnswerStream>();
            services.AddHostedService(sp => sp.GetRequiredService<IAnswerStream>());
            var answerChannel = Channel.CreateUnbounded<KeyValuePair<Guid, dynamic>>();
            services.AddSingleton(answerChannel.Writer);
            services.AddSingleton(answerChannel.Reader);
            return services;
        }
    }
}

This extension registers:

• Our AnswerStream
• Our input Channels
• Our answer Channels
• And of course our Actor implementation.

Sample Demo

The Input Side

For demonstration of this system we are going to query for the factorial of a given number. And as an example of utilizing multiple types we'll return an input message. We start with the api queries that is received through our RESTish interface.

namespace ConcurrentFlows.HostedActorSystem.Queries.Api
{
    public record GetFactorialApiQuery([FromQuery] int Message);
    public record GetMessageApiQuery([FromQuery]string Message);
}

Notice how we're not only making use of the new Record types here but also Record model binding in our controller.

namespace ConcurrentFlows.HostedActorSystem.Controllers
{
    [ApiController]
    [Route("[controller]")]
    public class SampleController : ControllerBase
    {
        private readonly IAnswerStream answerStream;

        public SampleController(IAnswerStream answerStream)
            => this.answerStream = answerStream ?? throw new ArgumentNullException(nameof(answerStream));

        [HttpGet]
        public async Task<IActionResult> GetFactorial([FromQuery] GetFactorialApiQuery query)
        {
            var actorQuery = new GetFactorialActorQuery(query.Message);
            var result = await answerStream.SubmitQuery(actorQuery);
            return Ok(result);
        }

        [HttpGet("message")]
        public async Task<IActionResult> ReturnMessage([FromQuery] GetMessageApiQuery query)
        {
            var actorQuery = new GetMessageActorQuery(query.Message);
            var result = await answerStream.SubmitQuery(actorQuery);
            return Ok(result);
        }
    }
}

Our controller takes the input api query, translates it to an ActorQuery and submits it to our system. It also gets injected with an AnswerStream that will provide access to the Actor System and the result from the query.

Interconnected Actors

The factorial query in particular is anActorQuery that is a system of two interconnected actors that are invoked with these two queries.

public record GetFactorialActorQuery(int Payload) : ActorQuery<int, int>(Payload);
public record GetReverseRangeActorQuery(int Payload) : ActorQuery<int, IAsyncEnumerable<int>>(Payload);

These queries invoke two interconnected actors which cooperate to produce a result. The deepest Actor is the ReverseRangeQueryActor. This Actor has the responsibility of producing a range of int's in reverse order.

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Actors
{
    public class GetReverseRangeQueryActor : QueryActor<GetReverseRangeActorQuery>
    {
        public GetReverseRangeQueryActor(
            ChannelReader<KeyValuePair<Guid, GetReverseRangeActorQuery>> queryReader,
            ChannelWriter<KeyValuePair<Guid, dynamic>> answerWriter)
            : base(queryReader, answerWriter)
        {

        }

        public override async Task HandleAsync(KeyValuePair<Guid, GetReverseRangeActorQuery> query, CancellationToken stoppingToken)
        {
            var range = Enumerable.Range(1, query.Value.Payload).Reverse().ToAsyncEnumerable();
            var answer = new KeyValuePair<Guid, dynamic>(query.Key, range);
            await answerWriter.WriteAsync(answer);
        }
    }
}

Notice how this Actor keys the results with the same key provided in the input.

The next Actor up is the FactorialQueryActor. This Actor takes the input value from the query, the range provided by the ReverseRangeQueryActor and computes the factorial.

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Actors
{
    public class GetFactorialQueryActor : QueryActor<GetFactorialActorQuery>
    {
        private readonly IAnswerStream actorStream;

        public GetFactorialQueryActor(
            ChannelReader<KeyValuePair<Guid, GetFactorialActorQuery>> queryReader,
            ChannelWriter<KeyValuePair<Guid, dynamic>> answerWriter,
            IAnswerStream actorStream)
            : base(queryReader, answerWriter)
        {
            this.actorStream = actorStream ?? throw new ArgumentNullException(nameof(actorStream));
        }

        public override async Task HandleAsync(KeyValuePair<Guid, GetFactorialActorQuery> query, CancellationToken stoppingToken)
        {
            var rangeQuery = new GetReverseRangeActorQuery(query.Value.Payload - 1);
            IAsyncEnumerable<int> result = await actorStream.SubmitQuery(rangeQuery);
            var factorial = await result.AggregateAsync(query.Value.Payload, (x, y) => x * y);
            var answer = new KeyValuePair<Guid, dynamic>(query.Key, factorial);
            await answerWriter.WriteAsync(answer);
        }
    }
}

Notice how these Actors are connected using the very same AnswerStream setup that's used to communicate from the controller to the Actor System. The AnswerStream integrates the Actor System so that all Actors can communicate asynchronously without any knowledge of each other except for the defined Query types, just like our standard Application Level CQRS. This leaves us completely decoupled by the queries.

Our Other Actor

The GetMessageActorQuery simply returns the input message.

namespace ConcurrentFlows.HostedActorSystem.ActorSystem.Actors
{
    public class GetMessageQueryActor : QueryActor<GetMessageActorQuery>
    {
        public GetMessageQueryActor(
            ChannelReader<KeyValuePair<Guid, GetMessageActorQuery>> queryReader,
            ChannelWriter<KeyValuePair<Guid, dynamic>> answerWriter)
            : base(queryReader, answerWriter)
        {

        }

        public override async Task HandleAsync(KeyValuePair<Guid, GetMessageActorQuery> query, CancellationToken stoppingToken)
        {
            await answerWriter.WriteAsync(new KeyValuePair<Guid, dynamic>(query.Key, query.Value.Payload));
        }
    }
}

ConfigureServices: The last piece of the puzzle

The last final piece to show is the simple registration of all our actors.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddSwaggerGen(c =>
    {
        c.SwaggerDoc("v1", new OpenApiInfo { Title = "ConcurrentFlows.HostedActorSystem", Version = "v1" });
    });
    services.AddAnswerStream();
    services.AddActor<GetFactorialQueryActor, GetFactorialActorQuery>();
    services.AddActor<GetReverseRangeQueryActor, GetReverseRangeActorQuery>();
    services.AddActor<GetMessageQueryActor, GetMessageActorQuery>();
}

And now with everything in place we can test our simple Hosted Actor System and get a result!

Summary

How cool was that! I don't know about you but that simple answer is pretty satisfying! We've created a simple Hosted Actor System to replace our request scoped Handlers. Our Actors communicate completely asynchronously and without knowledge of one another through our AnswerStream. These Actors live outside the request scope along with our AnswerStream. Always ready and able to process incoming requests in an asynchronous decoupled manner. Kinda beautiful in a way! Now I'm not saying go out and replace your architecture with this style but only to consider this simple Hosted Actor pattern when the problem space warrants it.

You can find all code GitHub ConcurrentFlows.HashNode