What is a NATS Subject?
NATS is a location transparent messaging technology built around foundational Pub/Sub semantics. That means it doesn’t matter if a publisher is in one hemisphere, while a subscriber is in the opposite global hemisphere. As long as both the sending and receiving parties can connect to a NATS server, they can communicate with one another. NATS Subjects facilitate this communication, they are the mutual knowledge between publishers and consumers, i.e. a publisher publishes to a Subject, while a subscriber subscribes to a matching Subject.
NATS defines Subjects as a specifically formatted string consisting of Tokens separated by a . (dot), for instance
{token1}.{token2}.{tokenN}
market.data.stock.AAPL
market.data.forex.EURUSD
trade.order.submitted
trade.order.executed
These Subjects are just simple strings, and are by default ephemeral, but they create an expressive, powerful, addressing mechanism. You can think of Subjects as an address on an envelope, it captures in detail, the single destination the message is sent.
The Tokens form a hierarchy, category, and uniqueness of every publication destination, in the same way Country, State/Provence, City, Postal Code, uniquely define a postal destination. If you’re picking up on the foreshadowing here’s the reveal, the Subject doesn’t fully define the ultimate Delivery/Consumption of the message. In much the same way as a letter could be opened and read by one or many recipients, NATS messages published into a given Subject are delivered in a variety of ways. They can be transformed into entirely new subjects, delivered to specific subscribers in a Queue Group, or consumed by subscribers to a Wildcard Subject, i.e. market.data.stock.* Collectively this is the concept of Subject Awareness.
Subject Awareness
Subject Awareness refers to the way messages are routed and consumed based on the Subject of a message. Subject awareness allows some powerful expression. We can leverage the dynamic nature of Subjects to build relevant intelligence into applications based on the Subject’s content and structure. For instance:
Intelligent Consumption: Specifically subscribe to content of interest, i.e. market.data.stock.AAPL while ignoring market.data.forex.>, only messages related to APPL stock will be received, anything under the scope of forex is ignored
Contextual Communication: The Subject provides valuable context regarding the message content, purpose, etc. user.status.online indicates that a ‘User’ has become ‘Online’, sensor.humidity.kitchen immediately indicates the subscriber is receiving a ‘Humidity Sensor’ reading, from the ‘Kitchen’
Malleable Evolution: Unlike configuring some rigid topic/queue constructs upfront, Subjects are ephemeral by default and can be dynamically created, while Wildcard subscribers can be delivered messages from new unique Subjects without modification, i.e. subscribers to cluster.apps.*.status would receive ‘Status’ messages from all existing and new ‘Apps’ without modification
Publishing vs Subscribing
When a NATS Publisher sends a message, it assigns a Subject describing the context of the message. This context can specify content type or category, e.g. app.status, would be an App Status message. The Subject could contain the origin of the message, e.g. products.factory.east123, is a message from factory East 123. Even purpose can be defined, e.g. records.123.delete, Delete the entity with id: 123.
Subscribers on the other end express an interest in Subjects the same way a researcher might be interested in a given domain. That interest may be very specific, e.g. cluster.east.app.chat-app.user.user1, matching a specific user, of a specific app, in a specific cluster. Alternatively, it could be very broad, matching many individual Subjects, e.g. cluster.east.>, matching all Subjects within the scope of a specific cluster.
It’s important to make clear, a message will be published to a single specific Subject. This means a message published to market.data.stock.AAPL would not be received via a subscription to market.data.stock, or market.data.stock.APPL.average. Only a subscription pattern matching the published Subject will receive the message, e.g. market.data.stock.AAPL, market.data.stock.*, market.data.>, etc.
The Subject Hierarchy
Subjects in NATS are formed by a collection of dot separated Tokens. Each Token represents a level in a Subject hierarchy that collectively describes the Subject space of a particular domain.
Let’s explore an example to understand why hierarchy design can be important. If we wanted to describe an Order Service’s Subject space, we could first define the following Subject template, generally following the broad to specific rule of thumb:
orders.status.{status}.order.{id}
This would form something like the following hierarchy
orders.status.created.order.{id}
orders.status.confirmed.order.{id}
orders.status.shipped.order.{id}
orders.status.delivered.order.{id}
This hierarchy effectively categorizes ‘Status’ by deliberately making that a constant level of the hierarchy. Also, the target entity Order is a static hierarchy level, the final one, with the specific Order Id following. Subscribers can express interest in a specific Status but all Order Id’s with the wildcard asterisk*, i.e. orders.status.created.order.* would receive all Created Orders. Alternatively, an interest in all Statuses can be expressed with an asterisk, i.e. orders.status.*.order.*. The contextual meaning of the asterisk remains clear with the context of the preceding Token, i.e. Status, or Order.
Further, if our Orders domain needs extension beyond Status and starts publishing a new category, we can add it without breaking existing subscriptions:
orders.shipment.{stage}.order.{id}
Conceptually, Subjects in the new template exist somewhere between and among three higher-level Statuses
orders.status.confirmed.order.{id}
orders.status.shipped.order.{id}
orders.status.delivered.order.{id}
But our earlier subscriptions, orders.status.*.order.*, etc. still only target the intended Status messages. To express interest in Shipment Stages, we can subscribe to orders.shipment.staged.order.*, orders.shipment.in-transit.order.*, orders.shipment.*.order.*, etc.
Subjects as Expressive Keys
Many messaging technologies make heavy use of Message Keys, these typically uniquely identify a particular categorization of the message or its content. A Topic representing car dealership inventory might use the Make as a message key, e.g. Ford, Toyota, Chevy, etc. It might not make much sense to use something like the Vin of each vehicle as a message key, given each is generally unique. The salient point is that the Cardinality of the key is a critical factor in message key selection.
Cardinality of the message key becomes so critical because it will typically drive message routing, its final partition and the ordering, even compaction, of related messages. If I needed all updates to my Ford inventory applied in the correct order, with something like Kafka, all updates would need to be in the same partition. Additionally, if my consuming process duration and message volume dictates the need for 100 consumers in a group, yet the cardinality of my message key is much less, the resulting partition distribution and consumer group balance is far from ideal.
NATS again takes a different approach. All messages are virtually keyless, consisting of only the destination Subject and payload, with optional headers and/or a reply subject. Subjects, as far as the NATS server is considered, are a single unit to which to deliver messages. It cannot, itself, be divided into partitions, only deeper layers of the hierarchy added to divide the Subject space.
Thanks to this, and the expressive hierarchy, a NATS message can uniquely identify many categorizations or identities of a message completely decoupled from consumption. The cardinality of the Subject has little role in subscriber distribution, or, to a degree, the proper ordering of related messages.
Where in the Kafka example my Key might be Ford, and that dictates its final partition and consequently its assigned consumer. The NATS message might have the Subject inventory.update.{make}.{model}.{year} and the subscribing group would express interest in inventory.update.Ford.> and within a Queue Group of 100 subscribers the message volume would be evenly distributed.
Consumption Superpower: Subjects vs Topics
The real expressive power of Subjects can be seen in the way we consume them and how that differs from something like a Kafka Topic. A Kafka Topic represents a single category of message(s) and their associated type(s). A Kafka Topic like factory1-machine-status would reasonably be expected to provide a stream of Machine Status updates from Factory 1. Likely we would key this Topic on something like MachineId. If we only care about a single Status, say Faulted, the client side consumer would need to do that filtering, still receiving all messages yet discarding most. Additionally, another Factory, say factory2, would be an entirely separate and discrete Topic.
The Subject in NATS however can represent and yield the entire Domain Structure however the engineer sees fit. With a Subject template like {factory}.{id}.{area}.{id}.{machine}.{id}.status.{status} we could express a subscription like the previous example with factory.*.area.*.machine.*.status.*. More importantly we can express the exact consumer interest that’s relevant. With a Topic of factory1-machine-status, both producer and consumers are constrained and coupled to all Statuses, all Machines, at Factory 1. The Subject hierarchy, however, decouples the intent of publishers from the interest of subscribers.
We could have a single publisher handling all Machines and a single Status: Nominal, Degraded, Faulted, etc. And on the other end have a single subscriber per Machine and all Statuses, or factory.1.area.1.machine.1.status.*, factory.1.area.1.machine.2.status.*, etc. Going a step further, we could also have a publisher sending general state updates for a Factory Area within the same Subject space, i.e. published to factory.1.area.{id}. These could be simply describing the details of the Factory Area, e.g. name, number of units, geofence, etc. What’s powerful is that this entire Subject space can then be ingested by a JetStream Stream and persisted as a single consistent unit. The Stream would subscribe to factory.*.area.*.machine.*.status.*, and consumers could view into any slice of this space necessary, e.g. factory.*.area.* - for general Area updates at all Factories, factory.1.area.*.machine.* - for updates from Factory 1, all Areas and all Machines, etc. The immense flexibility can be hard to get used to but with a proper hierarchy, filters and wildcards, you can express some very complex queries quite simply.
Wrap Up
A well formed Subject hierarchy is a complex topic with many areas for exploration. Hopefully, this at least gets you thinking in broader terms than the strict categorization of a Topic and excited to explore the malleable world of Subject Aware Messaging.
Recapping, we explored the concept of a NATS Subject, including publishing specificity and expressing subscriber interest. We reoriented message routing around addressing, categorizing, and keying messages into the malleable and descriptive nature of hierarchal Subjects. We also looked at the decoupling of publisher intent from subscriber interest, leveraging the Subject space to describe a broad domain while subscribers stay well scoped. It should be clear now, in the NATS world a well thought out and planned Subject hierarchy and template is a critical design decision but one that can evolve with your domain’s needs.
Have a specific question about NATS? Want a specific topic covered? Drop it in the comments!
