NATS Weekly #36

Week of July 18 - 24, 2022

🗞 Announcements, writings, and projects

A short list of  announcements, blog posts, projects updates and other news.

️ 💁 News

News or announcements that don't fit into the other categories.

• An architecture decision record (ADR) has been written up on Subject Transform (such as subject mapping) which documents the rules that apply. There is already a great table of examples.

⚡Releases

Official releases from NATS repos and others in the ecosystem.

• k8s/nats - v0.17.2

⚙️ Projects️

Open Source and commercial projects (or products) that use or extends NATS.

• wasmflow - Recently announced and open sourced noting usage of NATS

🎬 Media

Audio or video recordings about or referencing NATS.

• MCIoTCoE Panel – Where the paths meet: Explore the intersection of Public Cloud, Edge, and networks - featuring Todd Beets, Synadia
• Rethink Connectivity: Episode 3 - Building an Interactive Whiteboard with NATS Websockets - by Jeremy Saenz, Synadia

📖 Articles

Blog posts, tutorials, or any other text-based content about NATS.

• The Next Generation of Queuing: JetStream for OpenFaaS - by Han Verstraete and Alex Ellis, OpenFaas

💬 Discussions

Github Discussions from various NATS repositories.

• nats.net - TLS: Using CA and Credentials files

🧑‍🎓 Examples

New or updated examples on NATS by Example.

• JSON for Message Payloads (Rust)
• Concurrent Message Processing (Rust)
• Limits-based Streams (Go)

💡 Recently asked questions

Questions sourced from Slack, Twitter, or individuals. Responses and examples are in my own words, unless otherwise noted.

When should I use a push consumer vs. a pull-based one?

This is a pretty common question and I touched on this in the Grokking NATS Consumers series. Pulling out some quotes (no pun intended):

[for pull consumers] the control over when messages are requested is more predictable (and comforting), and the implicit ability to scale is wonderful to have

And..

[...] a push consumer as a queue group with explicit ack is functionally the same as a pull consumer, however the control flow is different.

In general, default to pull consumers except when the following two conditions are true:

• Ordered consumption of messages is required
• The subscriber can handle the push rate sufficiently

Regarding the use case, this ordered push consumer is ideal for replay or replication of state (or state transitions modeled as events). Either events happen before other events or versions/changes in state happens in a certain order.

For example, if the data in a stream were to be ingested into a database, a common approach is to use an OrderedConsumer (which is a convenience option for an ephemeral consumer + a few configuration options). The ingestion process would startup and define the last sequence that was consumed (defaulting to zero). On each transactional write into the database, the sequence of the last message consumed would be written to the database in the same transaction. This way, if the ingest process crashes, it can be recreated with an offset of the last sequence present in the database.

The ordered consumer is particularly optimized for this purpose, but a standard durable consumer would work as well.

How can I be sure my stream is highly available?

When getting started with JetStream, it is common to start with a stream configured with one replica. You may deploy this into production and then quickly realize you need to restart the NATS server or the underlying node (whether it is hardware, a VM, or a container).

What happens to publishers or consumers of this stream during that period of time? They get a response that the stream is not available.

To solve this, you need to take advantage of stream replication. When configuring a stream, the replicas can be specified which then ensures at least majority of the copies are written to prior to acknowledging the publisher.

As of the 2.8.x release of NATS server, stream replicas can be dynamically changed to increase or decrease the size. Of course this requires a cluster so that each copy can be placed on a separate node, but this will provide both data redundancy and high availability for clients if one of the copies is offline (for replication of three called R3) or two copies in the case of R5.

Are messages tagged with the user/client that published it?

The short is no, not implicitly. However, the sender could be included as a header by the application code easily. Of course the chosen identity of the user/client is up to the application.

// Initialize a new nats.Msg value with the header map defined.
msg := nats.NewMsg("foo.bar")
msg.Data = []byte("...")
msg.Header.Set("Myapp-User-Id", "...")
// set more headers 

// Publish the constructed message.
nc.PublishMsg(msg)