During [RethinkConn 2022](https://synadia.com/events/rethink/2022), [Jean-Noel Moyne](https://www.linkedin.com/in/jean-noel-moyne-87b95/), Synadia’s Field CTO, presented a lightning talk  [Comparing & Contrasting NATS to Kafka](https://www.youtube.com/watch?v=F2wutVP733c&list=PLgqCaaYodvKY6xRbvB6ffON0_ahyhDsoM&index=13). This has been the most popular video on the event’s YouTube playlist and led to a growing number of requests from our community for more content on this topic.

The consistent feedback we receive from customers and the community is the appreciation for NATS overall simplicity. This manifests in the ease and power of developing NATS applications as well as the operability of a NATS deployment.

One of the most repeated requests we receive has been for a total cost of ownership (TCO) comparison against several real-world, and comparable workloads. Users want to know if the “simplicity” is a trade-off vs functionality or performance.

Synadia engaged and sponsored the [Mcknight Consulting Group](https://mcknightcg.com/) to conduct an independent evaluation of this comparison.

![](https://a.storyblok.com/f/166429/498x204/3ebd91e1e2/nats-tco.png)

We were delighted that the conclusions from the TCO analysis validate that NATS gives you both simplicity and performance… and it will cost you less 🙂. Check out the [full report](https://synadia.com/lp/nats-tco-comparison-kafka) for the details of testing methodology, benchmarks, and analysis.

That said, the NATS maintainers' stance has always been to evaluate the performance for your specific use case and requirements. This was the motivation for building a robust [benchmarking](https://github.com/nats-io/natscli#benchmarking-and-latency-testing) capability directly into the NATS CLI to make it simple to model and benchmark workloads against your real world environment.

As highlighted in the presentation, there are fundamental architectural differences between the two platforms. A white paper discussing the functional and architectural differences between NATS and Kafka is going to be released in a subsequent post.

If you are just getting started with NATS, take advantage of the various resources that are available:

- Documentation - <https://docs.nats.io>
- Examples - <https://natsbyexample.com/>
- Screencasts - <https://rethink.synadia.com/>
- Newsletter -<https://synadia.com/newsletter>
- Slack - <https://slack.nats.io>

NATS.io TCO Comparison with Kafka

NATS.io Total Cost of Ownership (TCO) Comparison with Kafka

This is part two of the Distributed SCADA with [NATS.io](http://NATS.io) series of blog posts. In the [previous post,](/blog/connecting-distributed-scada-with-natsio-part-one) we discussed the basics of SCADA and proposed a distributed SCADA model that meets modern demands head on.

## NATS.io to the Rescue

The wider the SCADA system is, the more distributed it becomes, the harder it is to securely connect and operate. A closely bound system might be able to operate with centralised decision making and state-machine style mimics, but a widely distributed system with many components becomes a matter of heuristics and distributed state-machine process handling. To word it a different way, we move from a synchronous, tightly time bound system, to one that's event-driven, asynchronous and with headless anomaly detection and operational control. Due to tight control requirements of any automated process, the real-time control points like PLCs must remain in close proximity to the process hardware they interact with and so it must be rethought how to connect to, pull information from and control these islands of automation by Level 2 and above of the classic SCADA architecture.

**Noteworthy points**

- Wide Area Network (WAN) links are expensive and the internet is cheap
- The internet is unreliable and considered hostile
- Extending enterprise IP networks is complex and increases security risks
- PLC interfaces are arcane, insecure to obscure controls and are resource constrained
- PLCs are not general units of compute, but fixed, real-time systems without virtual memory *(as always, it depends on manufacturer)*, meaning they're not flexible like Linux
- Systems that interact with PLCs ideally must be in close proximity, are carefully gated and all systems are configured based on Zero Trust principles

Let's start by rethinking the SCADA diagram. We want a secure connective substrate between the Level 1 and Level 2 systems. This substrate trusts nothing by default and requires each system that accesses it to have an authentication and authorisation token.

![](https://a.storyblok.com/f/166429/1024x698/68205fb4d9/nats_scada_fabric.png)

The diagram above shows two component types of Level 1.5. Both Client and Node components can communicate with PLC systems in an abstracted fashion, removing the headache of handling different data units and the address based locations of data. Both clients also initiate communications to the PLCs and to NATS, easing communications through firewalls and removing location headaches and the reliance on privately or publicly routable IP addresses.

**Data Fabric Client (C)**

Has relevant configuration to subscribe to meaningful command and control conversations, as well as publish data that is of interest for Level 2 and above systems. It will transmit state changes and measurements of interest. This component is useful in small remote deployments.

**Data Fabric Node (N)**

Same as the client above, with the addition of a local NATS server running as a leaf node. This leaf node can provide continued NATS functionality in the event of a WAN disconnect event and also house local streams, sourced from data published to NATS subjects.

Both of these components inherit zero trust principles and require both authentication and authorisation to the NATS network and they can provide both rate-limiting and permit-list handling of control messages, further protecting the Level 1 devices.

Core NATS in this scenario replaces MQTT and can provide high performance subject based messaging for request/reply and publish/subscribe semantics, meaning it's great as a communications broker that transports messages between connected components. Because NATS has an MQTT integration, any Level 1 components that are enabled with MQTT can also connect to NATS without requiring any special treatment.

When NATS is configured with JetStream its persistence layer, in this scenario NATS replaces Kafka. It acts as a streaming system and stores messages with a sequence number and when you use durable consumers, a feature of NATS, if your consuming software disconnects, it can carry on where it left off. Filtering subscriptions and streams is easy too because NATS is based on subject based naming, meaning you can subscribe granularly to subject names, like `foo.bar.baz` or just `foo.*`.

Regarding network connectivity, few to no network modifications need to be made:

- NATS is based on unicast IPv4 or IPv6. No multicast, IGMP or weird networking requirements
- NATS clients initiate a connection to the server
- NATS servers when clustered initiate full mesh unicast connections to each other
- Islands of NATS clusters connect to each other through gateways, in which one gateway connection per cluster is made
- NATS leaf nodes initiate a connection to a server (just like NATS client code)
- NATS has TLS capabilities built right in, so NATS can work without encrypted VPN tunnels

When distributed authentication and authorisation is configured, each connection above can be authenticated with JWTs, a PKI system called `nkey`, or your own in-house system. Authorisation is also possible through the use of JWTs. NATS in this mode is called decentralised auth, in which each application or process will be mapped to a NATS account and each connection is mapped to a NATS user.

Below shows how a real-world scenario might look based on this model.

![](https://a.storyblok.com/f/166429/1024x910/b0fd3a86d5/enterprise_nats_scada.png)

Classic Level 2 to Level 4 can be assumed to be on an enterprise network and extending NATS between on-premises data centers to public cloud or to the edge is as easy as it looks above.

## Zero Trust Data Fabric

Multiple aspects exist when discussing zero trust for NATS. A NATS cluster can authenticate at the connection level and will transparently bridge communications between different accounts and user connections on the same system. NATS for these scenarios is split into four sections.

**NATS Cluster Connectivity**

Clusters are authenticated at the connection level and communicate bidirectionally.

**NATS Operator**

This is the configuration at an operational level for the whole NATS cluster. A NATS operator mints account tokens and has credentials to monitor and operate the entire cluster.

**NATS Accounts**

These are minted by the operator system and accounts are analogous to applications. Each SCADA application could have an account and accounts can share data between each other by explicit import and export configuration. Accounts create user tokens.

**NATS Users**

These are analogous to application connections. Each connection therefore has an authentication token and is authorised to engage on specific actions and subjects. A Level 1.5 client will be configured with a NATS user token, which grants it access. When each client connects to NATS, it only needs to know about one or two connections and by the magic of the Gossip protocol, it becomes aware of all of the available points of connectivity.

## Zero Trust Computer Networks

For completeness, another important aspect to consider is a prescriptive network configuration for every point of connectivity. Each device, whether PLC, supervisory machine, HMI and anything else connecting to an Ethernet and IP network has a MAC address. Network access should be limited to known MAC addresses and the network should report any unknown MAC addresses attempting to connect before shutting down physical ports that are disconnected or are in error state. Measures should be deployed to administratively shut down Ethernet ports should they be disconnected out of a maintenance window and bubble up alerts to operations engineers who can manage situations accordingly. Protocols like 802.1x help with network access control, but few industrial devices seem to support it.

## Summary

In this post, we’ve covered a data fabric architecture with NATS and the network requirements of this ubiquitous data fabric solution. We also covered some high-level thoughts on zero-trust networking.

Part three, the final post of this mini-series will discuss the real-world implications of a Zero Trust Level 1.5 Data Fabric Layer.

Connecting Distributed SCADA with NATS.io (Part Two)

Connecting Distributed SCADA with NATSio (Part Two)

It's impossible to have an industrial control conversation without talking about Supervisory Control & Data Acquisition (SCADA). It was around in the 1970s and despite a handful of technological advancements, it's still familiar and recognisable to anyone who stopped paying attention. SCADA is applicable to industrial process automation patterns and the list is huge, but examples include manufacturing lines, road junction turnings, power distribution and railways. At the heart of these systems are Programmable Logic Controllers (PLCs), which can take readings from physical sensors, make decisions and manipulate outputs to control connected components. PLCs evolved from racks of electrical components like relays, circuit breakers and contactors that electricians designed and maintained with wire links. They were not easy to change and required some skill.

SCADA is an architecture and pattern split into levels:

- Level 0 is where you will find sensors and actuators, like temperature and flow sensors, motors and solenoids
- Level 1 is about the first point of distributed control in the form of PLCs
- Level 2 contains supervisory computers which provides information for supervisor screens
- Level 3 contains production control systems like monitoring production targets
- Level 4 is focused on production scheduling

![](https://a.storyblok.com/f/166429/1024x642/86e715acb2/classic_scada.png)*Classic SCADA layers*

In summary, SCADA can be represented as software running on computers and servers, human interface systems, PLCs, inputs (sensors) and outputs (actuators).

A little like IT automation workflow engines, the whole point of PLCs is to distribute control of a process away from a central computer so real time decisions and adjustments can be made to the process the PLC is controlling. The process could be a beer bottling facility, power distribution or a railway junction. Logic dictates the need to monitor, observe and track processes across an organisation's estate is of paramount importance for operations, safety, reliability, product output, optimisation, billing and troubleshooting reasons. The saying *"It's software, no one died"*, isn't applicable in this field.

![](https://a.storyblok.com/f/166429/1024x365/29a7416366/s7-1500.jpg)*The Siemens S7-1500 family PLC, Copyright* [*Siemens*](https://new.siemens.com/global/en/products/automation/systems/industrial/plc/simatic-s7-1500.html)

PLCs enabled an army of electro-mechanical engineers to build complex systems and and have stood their ground as an economically priced control point. They have gained performance capabilities as CPU, memory and storage continue to drop in price and It's no surprise then that companies utilising this technology like utility, manufacturing and transport to name but a few, have islands of automated control systems strewn across counties, states, countries and continents. To make matters more interesting, these devices vary in age, manufacturer and capabilities.

## Programming & Operations

These are some of the original requirements of the PLC laid out by the Hydra-Matic division of General Motors back in 1968:

- A solid-state system that was flexible like a computer, but priced competitively with relay logic system
- Easily maintained and programmed inline with the already accepted relay ladder logic way of doing things
- It had to work in an industrial environment with dirt, moisture, electromagnetism and vibration
- It had to be modular in form to allow for easy exchange of components and expandability

![](https://a.storyblok.com/f/166429/1017x449/2a0eadf404/iec_61131_3_languages.png)*PLC programming languages; source IEC-61131_3*

PLC programming started out with ladder logic because that’s the way engineers at the time implemented industrial process control. Power flowed from the left, passing through relay style contacts and switches, which then terminated on outputs called coils. This is very similar to working with logic gates in digital electronics.

Modern PLC programming languages include Ladder logic (LAD), functional block diagrams (FBD), instruction lists (IL) and structured text (ST) methods. To summarise this paragraph, you can approach PLC programming from an electrical background or from traditional imperative programming. Neat. If you’re curious what the programming specification looks like, it’s in a document called [IEC-61131-3](https://webstore.iec.ch/publication/4552)

Regarding operations, each vendor supports network level protocols over Ethernet & IP and these include [Modbus](https://modbus.org/), [OPC-UA](https://opcfoundation.org/about/opc-technologies/opc-ua/), [DNP3](https://www.dnp.org/About/Overview-of-DNP3-Protocol) and vendor specific protocols like Siemens [S7](http://gmiru.com/article/s7comm/#:\~:text=The%20S7%20protocol%20is%20wrapped,presented%20in%20the%20figure%20below.). If you’re used to working with REST heavy web programming, you’re in for a ride. Some of these protocols are arcane and are not security oriented, leading to poor implementation or avoidance altogether. State level threat actors in 2023 are all too real to ignore and so connecting these systems securely and easily is key.

Note that PLC software isn’t compiled and you cannot easily run a suite of unit tests. Modifying PLC programs is done by directly downloading to the PLC device and any integrations like a data fabric must take into account changes simultaneously.

SCADA offers a pattern of connectivity for components from the hierarchical diagram, but as deployments and device counts increase, having one SCADA system that's centrally deployed and managed becomes difficult to maintain. There is a device count explosion at the edge and each industry sector has their own version of that problem. In power distribution there are millions of smart consumer meters, in manufacturing, countless numbers of sensors. Process control systems need access to this device data to make better decisions and in 2023, the industrial sector agrees that the classic model is upside down. Devices should present their data to distributed decision making logic, easily and securely without arcane protocols that offer little to no support and lack of widespread industry expertise. There is a burning need for control and analytical software to subscribe to Level 1 PLC data without exposing security sensitive devices directly to an enterprise network. Enterprise style networking topologies for this use case demand a large budget and present huge complexity and security nightmares.

Solving this requires a data fabric that:

- Publishes data in a push manner
- Offers command and control capabilities like request-response semantics
- Is highly performant
- Offers large scale fan-in and fan-out patterns
- Transforms point-to-point IP communications to named conversations
- Does not rely on DNS for end-to-end connectivity

You’re in luck. [NATS.io](http://NATS.io) has you covered.

## Connecting a PLC to NATS

The YouTube video below demonstrates a contrived use-case in which NATS is connected to a PLC with a simple Modbus client. This client in turn authenticates to the NATS cluster and collects information from the PLC in a closed-loop transmitting the data to a NATS stream. This client is configuration driven and publishes named data that other systems can subscribe to. It also does not rely on direct protocol support from the PLC other than the simple Modbus protocol. Newer and more powerful PLCs support a wider range of protocols and building a pluggable adaptor system for your data fabric is achievable, but beyond the scope of this demo. A normalised way to think about this is presenting inputs, outputs, specific memory and tags to NATS subjects, serving connected systems through pub-sub and request-reply semantics.

The core thesis presented in this post is not to access the PLC directly from Level 2 of the SCADA stack, but to gate Level 1 components and insulate them from issues of connectivity, manageability and security. A software client accesses the Level 1 components and transmits data to NATS over a cheap cellular connection and with the persistence layer of a local TLS enabled NATS Leaf Node, it can buffer collected data and transmit it at a later time in the event of a cellular network issue.

[  
![](https://a.storyblok.com/f/166429/1024x573/72e43db757/youtube_scada_01_embed.png)](https://www.youtube.com/watch?v=n2n3nDxZrUc)*Clicking on the image will take you to* [*YouTube*](https://www.youtube.com/watch?v=n2n3nDxZrUc)

Should you immediately rip out MQTT and replace it with NATS? It doesn’t make much sense to rip out one pub/sub system for another, and that’s not where NATS shines.

Much like how IT has migrated to the cloud and multi-cloud, it should be clear at this point, SCADA systems face the same headache of connectivity. The PLCs themselves will always have to execute in close proximity to whatever process they’re running, but the state data and computed state in the form of streamed events, needs to be available enterprise wide for analytical, monitoring and process mimic systems to make sense of what’s going on. Let’s talk about a concrete use case in power distribution automation.

## The Upside Down SCADA Problem

In power distribution, substations and micro-energy grids reside in many locations and each system manages the state of electrical switch-gear and multitudes of sensors. As the number of these locations increases, so do the autonomy requirements resulting in decisions being made close to the sensor source and switch-gear control point. For example, imagine a housing estate having an electrical surge issue or tree strike on power lines, a decision to burn the tree off of the lines or break the connection may need to be made. That decision might be made based on multiple sensory inputs like available capacity in the area, with the point being, a wide set of varied data is required to make that decision.

Distributed but horizontally communicating decision making systems that still fit in the hierarchical abstracted SCADA model acting on the inputs, but having an enterprise network spanning thousands of locations, all taking part in a single SCADA design, is asking for trouble. One compromised system results in chaos and the operational costs would be astronomical.

The outcome of a decision is to do nothing, or turn some switch-gear on or off. Therefore, it makes sense for the raw data to be dealt with in a distributed manner directly by PLCs and to publish the state change event to any other system that can make use of the decision and input data. A lot of other data could be shared in a local neighbourhood of substations, enabling cluster like behaviour. Let’s re-imagine SCADA with NATS!

![](https://a.storyblok.com/f/166429/1024x698/68205fb4d9/nats_scada_fabric.png)

If a connected neighbourhood of substations subscribed to the very end-points they feed through smart meter technology, decision making logic can be distributed and closer to the sources of input data, enabling hive based optimisation and efficiencies. Real world numbers of smart meters are in the millions and a centralised system would be overwhelmed with this level of information. An enterprise network solution to connect these components would not provide cost effective connectivity nor the security profile required to scale. What is needed is a cheap connectivity medium with resiliency built into the system that communicates over it.

A solution to this is to have a constrained and authenticated data collection layer that securely transmits named and type specific data to a flexible messaging system. This messaging system should be able to handle slow and fast data producers and consumers, and offer high fan-in and fan-out patterns. If you’re not familiar with NATS, all the boxes are ticked so far. With a NATS solution, one lightweight and highly performant component is deployed multiple times. It can deliver publish-subscribe, request-reply, message streaming, key-value and object store semantics and it’s one system to operate and understand, placing minimal load on operations teams. Its communication requirements are simple with connected clients and leaf nodes initiating encrypted, authenticated and authorised connections to the cluster. It’s the NATS advantage.

By separating decision making code from data collection, non-real time systems like billing and prediction can live in public clouds where software developers can move rapidly. Real-time control loops stay close to the field devices and a data fabric exists to extract metrics and provide them to the layers above, without placing additional load on field devices. Extra intelligence could be built into data fabric gateways that understand the devices they connect to, which would solve unit conversion like millivolts to volts and it would also remove low level configuration requirements from the data fabric components.

The cellular industry is going through a transition of this nature in 4G and 5G systems with a split between real-time and non-real-time systems and the location of the components that interact. Some components are required to live close to a radio tower to cope with activities like radio frequency management and others like user management can live elsewhere.

Below shows a distributed SCADA system, powered by a NATS data fabric. One fabric with many conversations powers the whole solution. NATS handles the relationships between components that show interest in each other and enables ease of communication.

![](https://a.storyblok.com/f/166429/1024x681/d6918200c1/rethinking_scada.png)

## Summary

With NATS, we can transform the relationship between physical location and break the concrete SCADA model. By creating a data fabric that sits between Level 1 and Level 2, data can be published organisation wide without losing the ability to respond to solicited queries. Real-time decision making logic in PLCs remains close to the equipment they’re connected to and higher level process oriented logic like process optimisations, historians, operational systems that deal with alarms and alerts, billing and planning systems can be elsewhere. In the use case of distribution automation, local power microgrids can make use of consumer data and make optimisation decisions autonomously. Safety and operational system integrity isn’t lost either, because real-time data streams and disk or memory persisted streams can co-exist.

Thanks for reading part one and if you’re done here, that’s absolutely fine! In part two, we explore enhancing SCADA with NATS.

Connecting Distributed SCADA with NATS.io (Part One)

Connecting Distributed SCADA with NATSio (Part One)

## **FileWave Device Management System**

[Filewave ](https://www.filewave.com)is a cross-platform/multi-platform Device Management System that helps big organizations to manage their devices from a single interface.​

We are able to manage devices of different platforms:

- ​Windows
- macOS
- iOS
- Apple TV
- Android
- Chromebook

​and we're able to:

- Get inventory data (CPU, RAM, free disk space, ....)
- Install software and updates
- Send specific commands (reboot, lock , ....) to devices​

## **FileWave Before NATS**

​Before we switched to [NATS](https://nats.io), the internal messaging was implemented using [ZeroMQ.](https://zeromq.org/)

ZeroMQ is a library that offers several mechanisms and patterns to implement a lot of different communication strategies. It works well but has several limitations (for our use case):

- It's just a library: so we had to implement a "central" server
- A single socket can use just a single pattern; if you need a PUB/SUB and a REQ/REPLY you have to open two sockets
- Not supported in web applications; the messaging with the WEB interfaces was implemented by "converting" ZeroMQ messages and sending them to the UI using SSE (Server Sent Event)
- It does not support out-of-the-box TLS
- It does not support use cases that require persistence

## **Why we needed to migrate**

​The main drivers for migrations were:​

- We have strict requirements for having TLS on all communication channels
- We need support for web applications
- Message persistence and replay are something we'll need to use soon​

![](https://a.storyblok.com/f/166429/651x567/efc14a2079/filewave_sketch-01.png)

## **Why NATS?**

​We did some research based on our needs:

- ​Client libraries for (at least) C/C++, Python, Web Browsers
- Cloud-ready/friendly
- mTLS support
- Easy to install/configure and monitor
- With good documentation and community

And we found a few candidates:

- RabbitMQ
- Kafka
- NATS

​We decided then to do some parallel investigations and, based on our specific use cases - we decided that NATS was the best choice for us.   
  
The main drivers for picking NATS were:

- ​Easy to integrate into our installers (thanks to Go)
- By far the simplest to install and configure
- Very small install size and memory footprint (thanks to the Go engine)
- Pre-built binaries for all platforms we need to support
- Wide-range of authentication systems (for instance we use mTLS for devices and server components and JWT for authenticating users on User Interfaces)

FileWave: Why we moved from ZeroMQ to NATS

FileWave: Why we moved from ZeroMQ to NATS, and picked it over Kafka and RabbitMQ

It's great to see the continued innovation from the [NATS](http://nats.io) community & ecosystem. At our core, we at Synadia believe that every enterprise in the world needs to be a globally distributed data company to deliver better products, a superior customer experience, and find new markets.  

  
So when [HarperDB](https://harperdb.io) approached us with the challenge to help them build a global data platform with extreme low latency as part of their v4 release we jumped at the opportunity. There was great alignment in the vision for where customers are struggling with the [skyrocketing costs](https://a16z.com/2021/05/27/cost-of-cloud-paradox-market-cap-cloud-lifecycle-scale-growth-repatriation-optimization/) of the “public cloud” lock-in, siloed applications & microservices, and the challenges in delivering real-time data to diverse edge endpoints. 

We look forward to collaborating with technology innovators like HarperDB and accelerating our joint customers' journey to the Edge.

Read the full [HarperDB 4.0 release blog ](https://harperdb.io/blog/planet-scale-applications-for-everyone/)for all of the amazing new features and how NATS is key to achieving the success of HarperDB 4.0.

NATS powers the new HarperDB 4.0 platform

Synadia has been on a roll lately.

We recently were awarded the [Stratus Award](https://www.bintelligence.com/blog/2022/9/26/59-leading-cloud-computing-companies-products-and-services) for [Cloud Disruptor](https://www.einpresswire.com/article/592479318/synadia-named-a-global-leader-in-cloud-computing) of the year. What we love about this award is that it is not for a single cloud service, rather it recognizes the truly game changing capabilities provided by Synadia and the [NATS.io](http://NATS.io) platform for enabling our customers to deliver globally distributed applications across any cloud, geography or device.

On the product front, the NATS Server [version 2.9 release](https://nats.io/blog/nats-server-29-release/) was a major milestone which was a result of the combined efforts from over 40 community contributors and the Synadia team. The 2.9 NATS server release provides both significant new capabilities as well as platform stability and resiliency. One of the repeated requests from our users has been around simplifying the challenge of delivering consistent data to highly distributed edge environments, reliably and with very low latency. Not an easy thing to accomplish on a global basis to edge devices such as autos, wind turbines, or satellites. NATS Server v2.9 provides great functionality to enable this data locality use-case both simply and securely.

Our ecosystem also continues to expand with amazing partnerships and joint solutions. Here are just a couple of the recent highlights:

- [**Cosmonic** ](http://cosmonic.com)\- WebAssembly is arguably one of the hottest technology trends for building globally distributed multi-cloud applications. Learn how NATS is powering the Cosmonic wasmCloud platform - <https://synadia.com/blog/wasmcloud>
- [**HarperDB**](https://harperdb.io) - Listen to our own Byron Ruth discuss with Margo McCabe how NATS is powering a globally distributed data platform - <https://rss.com/podcasts/harperdb-select-star/635752/>
- [**ControlPlane**](http://controlplane.com) - When SAFE Health Systems needed a platform to securely deliver healthcare data on a global basis without managing cloud infrastructure, they selected ControlPlane & Synadia -   <https://synadia.com/articles/synadia-control-plane>
- Synadia has partnered with the [5G Innovation Lab](https://5goilab.com/) to accelerate a reference architecture for cross industry cloud native & data streaming applications deployments. We are currently engaged with several very interesting discussions about the integration of NATS with 5G. Stay tuned.
- The world’s largest [**IoT Community**](https://iotpractitioner.com/) invited Synadia to join their mission of educating & fostering global collaboration on best practices and business value. We are pleased to be featured in the latest whitepaper covering innovations in [distributed computing & cloud native applications](https://iotpractitioner.com/intelligence-report-volume-16-cloud-native-applications/).

Given all of the progress and activity we thought it was time to get our global community together to more formally update everyone. Our upcoming event will be a ½ day virtual event November 10th, full of product updates, demos and use-cases. We are calling it [**RethinkConn 2022**](https://synadia.com/events/rethink/2022) to highlight how Synadia is fundamentally transforming the distributed computing and connectivity market. Please register for this free event <https://synadia.com/events/rethink/2022>.

We are also looking forward to seeing everyone at Kubecon + CloudNativeCon in Detroit next week. Join Tomasz Pietrek, Jeremy Saenz & Todd Beets from Synadia for their NATS Project Maintainer talk “Connect All the Things! Using NATS to Simplify Communication Everywhere”, Thursday October 27, 2022 2:30pm - 3:05pm ET. [Add it to your Kubecon schedule today](https://kccncna2022.sched.com/event/182OD)!

Synadia is on a roll!

Synadia on a Roll

## Matrix Dendrite

[Matrix](https://matrix.org/) is a federated protocol for real-time communications built on top of distributed data structures and [Dendrite](https://github.com/matrix-org/dendrite) is an implementation of a Matrix homeserver built following a microservice architecture. It was originally intended to help us to overcome some load issues that were present in earlier monolithic implementations at the time. As a result, flexibility, performance, and scalability have been three critical areas of focus in our development.

## In the Beginning with Kafka

When we first started the Dendrite project, we were using Kafka as a means to distribute events and asynchronous tasks between microservice components. In addition to being able to scale up and down as needed, we also needed durability in the message queue to ensure that if parts of the service were restarted, no tasks or events were lost. Given our need for deployment flexibility, it’s equally important for all of the microservices to run under a single process as it is to be able to split them out and run them under separate processes - or even on different machines. We wanted to be able to target embedded and small platforms, as well as the datacentre.

While Kafka allowed us to get earlier versions of Dendrite working quickly, we eventually found it to be relatively heavy when used in a full polylith microservice deployment and didn’t scale down as well as we had hoped for smaller deployments. To successfully run under a single process without a standalone server handling the events and tasks, we were also forced to mock up our own lightweight Kafka-ish queue implementation, which carried with it additional maintenance burden. We weren't satisfied with the performance of either mode of operation and found that data retention was especially difficult to manage.

---

> ### "NATS gave us everything we used Kafka for, but had a number of additional benefits."
>
> \-- Neil Alexander, software developer, Matrix
>
> ---

## Why [NATS.io](http://NATS.io) is a Better Fit for Dendrite

We evaluated a number of different alternatives and eventually landed on [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream), which in effect gave us everything we used Kafka for but had a number of additional benefits. For a start, it was considerably easier to get running. The [NATS Server ](https://github.com/nats-io/nats-server)consumed far less system resources. Being written in Go, we were able to embed NATS into our own monolith-style binaries so we didn't even need a separate server instance in those smaller deployments. We contributed some patches to make it possible to communicate with the NATS Server in-process without using external sockets; this made it possible to both easily target mobile devices, and venture into the web browser under WebAssembly too.

---

**NATS Benefits vs. Kafka:**

- **Consumes less system resources**  -- Less infrastructure spend than Kafka
- **Written in Go** -- Easy to embed, no need for separate server instances like in Kafka
- **Simpler data retention** -- Using JetStream, easier to manage and configure retention than Kafka
- **Reduced management overhead** -- Easier for developers to manage and maintain than Kafka
- **Reduced codebase complexity** -- simpler code stack due to elimination of complicated persistent storage requirements

---

For us, switching to JetStream was transformational. Performance is excellent overall. We make heavy use of the interest-based retention policies to ensure that work is cleaned up when it is complete. We take advantage of being able to manually acknowledge items to ensure they were processed successfully before cleaning up. We've been able to successfully manage pools of workers from NATS streams and considerably reduce the amount of complexity in our codebase in a number of places which no longer need their own persistent storage. We've even had it successfully running in application-embedded scenarios on iOS and Android as a part of our peer-to-peer demos.

In summary, our team is always on the lookout for the right tools and architectures for the performance and flexibility our users require. As part of our evolution we made the shift from Kafka to NATS, and have not looked back. Stay tuned to see what interesting updates lay ahead next year for Dendrite!

## About the Author

Neil Alexander is a software developer working on [Matrix (and Dendrite)](https://matrix.org) at [Element](https://element.io). He is interested in security, privacy, decentralised systems, and computer networking. Neil also has experience in end-user compute and enterprise-scale systems architecture.

Dendrite Project moves from Kafka to NATS.io

Dendrite Project moves from Kafka to NATS

We proudly announced last week the[ first release](https://github.com/nats-io/nats-server/releases/v2.9.0) of the 2.9.x series! 🚀 This is a major release for us in ways I’ll go over in this post.

First thing’s first. Thank you to the **48 contributors** ❤️ through GitHub issues, discussion posts, and pull requests. Beyond those direct contributors, hundreds of folks in our Slack community helped out with their questions, use cases and testing of early versions.

The 2.9 release focuses on improving the [JetStream](https://docs.nats.io/nats-concepts/jetstream) persistence layer in multiple ways including:

- ⚖️[ Increased stability and resilience](https://nats.io/blog/nats-server-29-release/#stability-and-resilience)
- 🌏[ Greater scale and mobility](https://nats.io/blog/nats-server-29-release/#scale-and-mobility)
- 🔧[ Improved operability and security](https://nats.io/blog/nats-server-29-release/#operability-and-security)

# JetStream Elevating NATS to a True Global Utility for Communications

Announced in March 2021, JetStream extends NATS to provide *at-least-once* and *exactly-once* delivery guarantees as a first-class subsystem of the NATS server. The community quickly embraced JetStream and we have seen a lot of novel use cases that require enhanced scaling and performance. JetStream contributes to the big vision for NATS of providing a **connective technology** for all communications use cases between applications in the cloud, on the edge, and even in space.

JetSteam truly takes NATS to the next level, not only by adding persistence and the ability to better handle IoT / edge applications that suffer from intermittent connectivity but also with entirely new capabilities like an integrated [key-value](https://docs.nats.io/using-nats/developer/develop_jetstream/kv) and [object store](https://docs.nats.io/using-nats/developer/develop_jetstream/object) (beta). With JetStream, NATS becomes a true utility tool – building on the vision of NATS creator Derek Collison, to make connectivity a global utility supporting numerous use cases with a single tech stack. Like the rest of NATS, all of this comes in a fully [open sourced](https://github.com/nats-io/nats-server) package, wrapped into a small,[ static binary](https://docs.nats.io/nats-server/installation), with no (as in zero) dependencies. 🎉

# **Taking JetStream to the Next Level**

The 2.9 release takes all of this goodness to the next level. Some of the specific benefits the 2.9 brings to users include:

- Reduced time and bandwidth for replication catch-up
- Improved message distribution
- Better handling of memory usage for containerized environments
- Replica and mirror-based direct gets
- Stream and key-value message republishing
- Consumer replica, storage, and filter subject change
- Server tags and portable streams

NATS 2.9 also adds key new capabilities in security and operability including:

- AES-GCM cipher for JetStream file-based encryption
- Account purge operation
- Templates for scoping signing keys
- New and enhanced subject mapping functions.

A special thanks to the core NATS contributor team for working so hard over the last few weeks to get this release right. Please read the full release [blog post](https://nats.io/blog/nats-server-29-release/#stability-and-resilience) on the [NATS.io](http://NATS.io)[ project blog](https://nats.io/blog/). You’ll find code examples and much greater detail.

Please [download NATS 2.9](https://nats.io/download). Take it for a spin. Tell us what you think on [Slack](https://slack.nats.io) or[ GitHub](https://github.com/nats-io). And if you are interested in [contributing to ongoing NATS development](https://nats.io/contributing/), we’d love to have you.

Credit: Stephen Leonardi / Unsplash

NATS Server 2.9 Release: More Stability, Greater Scale, Better Security

WebAssembly aka Wasm was first published in 2017, allowing developers to bring their code to the browser to execute applications in high performance. In 2019, Mozilla introduced WASI - Web Assembly System Interface. This allowed Wasm apps to access resources beyond the browser, and run far more powerful systems.

As a result, WebAssembly (Wasm) has become a very hot topic with distributed application developers & architects. It is driving a major revolution in application platform design, even moving beyond the public cloud as a destination - and to the rapidly growing Edge ecosystem. 

We see a natural affinity for the integration of Wasm and NATS, and have seen numerous developers implementing the two together. This is both for large enterprises trying to re-architect brittle cloud environments designed for vendor lock-in but also for emerging startups & SaaS disruptors that are building new global data businesses without any legacy technical debt.

Services like <a href="https://wasmcloud.com/" target="_blank">wasmCloud</a> are creating a whole new paradigm for Cloud-Native developers by eliminating entire classes of problems such as security, portability and device independence.  

If you want to see more of this in action, this recent article from Brooks Townsend, a maintainer on the wasmCloud team, is an excellent overview of the possibilities with Wasm and NATS.

Explore the article <a href="https://wasmcloud.com/blog/globally_distributed_webassembly_applications_with_wasmcloud_and_nats/" target="_blank">Globally Distributed WebAssembly Applications with wasmCloud and NATS</a>.

WebAssembly Applications with wasmCloud and NATS

wasmCloud

## The evolving and demanding nature of edge systems requires a new set of characteristics. 

In the [last post](../blog/adaptive_edge_1), we talked about the requirements we see emerging for applications as we move from a Cloud Economy to a Connection Economy where smart connectivity is the primary challenge rather than compute and storage. To handle the requirements of modern distributed applications, Adaptive Edge Systems should offer the following capabilities:

## 1. Performant
Adaptive Edge systems should be at least as performant as incumbent systems and ideally considerably more performant on a per compute basis. This can be measured in various metrics including latency, message or stream throughput, or  compute required per million messages, to name a few. This also usually indicates lower energy consumption, a critical component of IoT and remote environment requirements. 

## 2. Easy to deploy and maintain
One of the key problems with incumbent messaging and streaming platforms is the challenges in deployment and the heavy infrastructure requirements. Because Adaptive Edge systems must be dynamic and will likely encourage experimentation, these systems must be simple and fast to deploy and easy to maintain. A good rule of thumb is that, if a team or dedicated engineer is required to maintain a solution, then it is not suitable for Adaptive Edge use cases.

## 3. Lightweight and works in any environment
Incumbent messaging and streaming systems generally require considerable compute capacity as well as supporting systems. For example, Apache Kafka requires not only a large compute instance but also a JVM; this usually means Kafka installations require multiple instances and are not suitable to install in resource constrained environments common in IoT or other Adaptive Edge applications. For an Adaptive Edge connection fabric to be effective, it must be able to run on all types of edge devices from small sensors and RaspberryPIs to standard physical servers and cloud infrastructure. The connection fabric must also maintain all features at any scale; the fabric must be environment agnostic.

## 4. Resilient and self-healing
While this may be a requirement for any enterprise software system, connection fabrics are the backbone of SLAs and the glue that binds multi-cloud applications to reduce downtime and improve latency. For this reason, Adaptive Edge connection solutions must boast higher-than average reliability and durability — more akin to what is expected with telecommunications systems and other regulated technologies judged to be mission critical to society. If portions of a connection fabric crash or slow down, then other nodes in the cluster should instantly absorb any traffic and workloads.

## 5. Client / language agnostic
Modern applications may be composed of dozens or hundreds of smaller applications and microservices. Those smaller applications and microservices may use different languages. It is not uncommon to see a half-dozen languages represented in different functions of a distributed application. Adaptive Edge connection solutions must support a wide range of languages and also have a straightforward integration schema using a well-structured API.

## 6. Multi-cloud and cloud-agnostic
Most Adaptive Edge applications do need to connect to the cloud. To future proof those applications and enable better SLAs for their services as well as reducing single point of failure, connection fabrics should be able to bridge clouds and provide multi-cloud capability. This will include moving workloads from one cloud to another and load balancing across clouds. The same capability is also useful within clouds.

## 7. Location independent
This describes both geography and application or system topology. Admins and users should be able to add and remove clients on Adaptive Technology connection fabrics without requiring configuration changes. Similarly, clients should be recognized by the fabric no matter their location.

## 8. Tolerates connectivity disruptions
The adaptive part of Adaptive Edge includes the ability to handle disruptions in connectivity and then resume connections while sharing data asynchronously and when possible with centralized application servers. This also implies that Adaptive Edge systems must be able to deliver both batch and stream processing, and switch between the two depending on circumstances and conditions.

## 9. Persists data intelligently
Data persistence is essential for giving Adaptive Edge applications the ability to run backend processes or make business logic decisions without having to call back to a cloud data center. More advanced connection fabrics integrate a key/value and object store with data persistence, to simplify the application architecture and remove the need to deploy a distributed database alongside the connection fabric.

## 10. Aggregates and distributes data intelligently
Adaptive Edge systems must support data aggregation at the edge. Data that is salient to the cloud gets to the cloud. Data that is only salient at the edge stays at the edge. This minimizes unnecessary data tromboning, saving on bandwidth and infrastructure costs and allowing application architects to better optimize data handling and business logic.

## 11. Secure by default
Adaptive Edge applications are dynamic in ways that break old security paradigms tied to IP addresses. Securing more complex topologies with dynamic client and node structures requires security to be a part of  the connection fabric and not applied as a secondary or tertiary layer, or as a mix of different integrated technologies. One solution is to assign clients to accounts and limit accounts to namespaces. This makes segregation default and delivers true multi-tenancy as part of the architecture. Of critical importance - the connection fabric never stores or accesses private keys.Security is bifurcated from connectivity.

## 12. Supports multi-tenancy
Related to security, multi-tenancy for connection fabrics enables the use of the fabric as a global utility traversing applications or clouds. For example, a true Adaptive Edge solution can silo message and streaming data for a microservice limiting reception or subscription to a specific namespace, even as other microservices in the same fabric are limited to communicating with a different namespace.

## 13. Scales quickly, horizontally and vertically
Part of any connection fabric’s ability to adapt must be rapid scalability to handle bursts in demand or topology shifts when part of a cluster shuts down or moves traffic from one fabric server to another. Quick scaling should translate into end users never noticing that their connection fabric has had to scale up or add capacity. This scaling should be equally viable horizontally (adding more servers) or vertically (adding capacity to existing servers).

## 14. Client simplicity and naivete
Adaptive Edge connection fabric clients should remain simple and stateless, focused on connecting, publishing or receiving data. Because no server or state data is maintained, new clients can be added to a connection fabric without topology changes. This future proofs the connection fabric and allows for easy adds and modifications to client configurations.

## Conclusion: A New Age of Adaptive Edge
From autonomous vehicles to the new energy grid to Industry 4.0, applications increasingly require proximity to activity. Next generation distributed, multi-cloud and edge-autonomous systems need more than what legacy message brokers and streaming data systems can offer. This is growing the demand for adaptive connection fabrics that are lightweight, secure, distributed and environment-agnostic. These fabrics can fit multiple deployment paradigms and adapt and grow with edge applications, often replacing both messaging and streaming systems. These systems save infrastructure costs while increasing security and simplicity. With the adaptive edge, apps become the center of the universe rather than the clouds that or servers that run them. By applying the 14 factors, architects can ensure the messaging and streaming capabilities of their infrastructure meets the needs of adaptive edge applications, today and tomorrow. 


14 Factors of Adaptive Edge: Pt. 2

Adaptive Edge 2

## How the next generation of distributed, multi-cloud and edge-autonomous systems are forcing the evolution of message brokers and streaming data systems into adaptive connection fabrics that are lightweight, secure, distributed and environment-agnostic. 

We have entered a new era of distributed applications and infrastructure driven by performance, security and resilience on the Edge. Makers of distributed applications require new capabilities that allow applications to continue functioning effectively, even in conditions of poor or intermittent connectivity. As performance expectations and service level agreements approach near real-time for a growing array of functionalities, applications require greater persistence and autonomy on far more widely distributed arrays of nodes; the Edge becomes the compute fabric, the memory and the application tier, all rolled into one. The Adaptive Edge is the new construct that fulfills these new and more exacting requirements. Just like web applications forced a rapid evolution for technology architectures in the early Cloud Era, Edge Applications are forcing a similarly rapid evolution in the Edge Era. This paper attempts to define a set of design patterns and principles for the applications and architectures on the Adaptive Edge.

## 14 Factors

Message brokers and event streaming software are critical infrastructure for modern software applications and technology infrastructure. These systems make it possible for applications, systems, and services to exchange information, even if they are not written in the same software language or are deployed on different platforms or environments. Message brokers emerged as software and hardware became more distributed and composed of discrete services. Event streaming software — or event buses — emerged from the message broker world to handle massive volumes of streaming information and ensure that data could be found by users that need it. 

Message brokers and event streaming both relied on one-to-many architectures. Over time, as systems grew still more distributed, the publish-subscribe (pub/sub) model for message brokers and event streaming became more common, allowing end users or services to subscribe to and store streams of data or messages that they needed to consume. 

While the core functionality provided by legacy designs for message brokers and event buses remains crucial, current systems can no longer handle the requirements of modern distributed applications. These applications are rapidly moving to the “Edge” and are requiring local and adaptive processing and persistence capabilities. These capabilities are either not readily available or are complicated and expensive to provide with current message broker or streaming data technologies. This is largely because the paradigm for moving and exchanging data and messages has shifted from a one-to-many to many-to-many. 

Because legacy message and eventing systems are not designed to live on the edge, they often suffer from high-latency and inconsistent delivery of key data to or between edge clients. Because the edge is where data is generated by distributed applications interacting with users and sensors, the question becomes whether to move that data back to the cloud for processing or to process it on the edge with distributed business logic. By extension, this raises the requirement of managing and scaling data aggregation on the edge in a manner that allows for distributed applications to make decisions based on the latest information and not be forced to “trombone” large volumes of data back to cloud data centers.

Further, the requirements of many-to-many architectures require the ability to continuously shape and reshape the topography, taking into account intermittent connectivity, concerns about data transit and storage costs, and the limitations of SLAs and capabilities of cloud computing providers. Enterprises not only want the ability to run applications on the edge but also to allow multiple clients on the edge to act in concert until connectivity is re-established. 

Applications such as IoT or mobile applications today need the ability to run either as a full messaging and connection server on a small form-factor or to function merely as a simple client, bridging the nodes of the application back to more centralized compute and storage. Even architects of more traditional applications that are not required to run on the edge are seeking out adaptive capabilities that require a more flexible connection fabric to handle messaging and streaming; for example, companies are looking to stand up the same application in multiple clouds while maintaining centralized awareness and management of processes and data. These cloud-agnostic designs require a unified service that can provide messaging and eventing within one cloud or across multiple clouds without a dependence on a hosted service in any single cloud. 

Another emergent use case is building an internal SaaS capability that extends an external SaaS service into an organization but allows for an on-premise version that mirrors all capabilities and benefits from product and feature improvements of the external service without exposure to the public Internet. (This configuration is increasingly popular in regulated industries or in at-scale organizations seeking to reduce costs). 

All of these emerging use cases are versions of a similar architecture — what we are calling the Adaptive Edge. The Adaptive Edge Architecture is a flexible deployment topology overlaid atop of legacy architectures based on physical data centers and cloud computing. Adaptive Edge can either integrate with these constructs or function as an independent construct.  that supports one-to-many, many-to-many, and combinations of the two along with sharding of the connection fabric into local clusters that can continue to deliver back-end processing and application functionality even with eventual and discontinued connectivity. 

The Adaptive Edge also requires not only persistence but a lightweight attached data store (such as a key/value or object store)  that can make persistence intelligent to reduce data roundtrips and better enable edge application functionality.  A crucial lynchpin in this Adaptive Edge Architecture is a multi-tenant security model that makes it easier to create tightly partitioned namespaces and sub-groups for data connections, messaging and eventing and other actions. These capabilities make the Adaptive Edge a more flexible paradigm that becomes a general purpose massively distributed control plane. This plane is needed for the next generation of multi-cloud and edge-native applications.

## Designing Systems for the Adaptive Edge
A common pattern among distributed systems has emerged with a central group of applications servicing and receiving data from edge nodes. Usually there is telemetry coming from the edge. Sometimes edge nodes have their own services around command and control and access to local data. In essence, this is a federated edge, loosely coupled and malleable for supporting existing and creating new services. Often identified as an IoT artifact, this pattern also works quite well for other use cases that benefit from an adaptive edge layer and flexible service and message delivery.

Enterprises are executing this architecture by integrating multiple technology systems and hacking them into the Adaptive Edge pattern. These designs require large amounts of resources, staff time, and infrastructure in order to become operational. Yet, in the end, hacked-together Adaptive Edge architectures generate systems that are fragile and break easily, difficult to monitor, and challenging to secure.
  
To build successfully for the Adaptive Edge, a different approach is required, one that looks at creating a federation of equally important or capable nodes or servers, which can reside in any form factor and are just as comfortable in a small Windows PC as in a cloud server. For a truly Adaptive Edge,  communications systems must support and even improve on incumbent paradigms for both one-to-one and one-to-many communications models. But Adaptive Edge systems must also function as a superset of both one-to-one and one-to-many, behaving more like a fabric or mesh that enables almost any shape of communications network topology. These communication fabrics behave more like a graph, allowing connections to form, self-organize and exist as part of the greater whole but remain as independent subsets or connect back to the communication fabric opportunistically and when possible.

Discover more in [Part 2](../blog/adaptive_edge_2)...

14 Factors of Adaptive Edge: Pt. 1

Adaptive Edge 1

As innovation takes off at the Edge, one aspect we’re all trying to avoid is complexity. There is simply no room at the Edge for complexity— or much else for that matter. Simple, fast, robust, small yet powerful and dynamic is the name of the game. These demands are high. Enter ‘cloud native’ tech with its lack of assumptions, microservices architectures, container-based applications and dynamic orchestration of compute, storage, and networking resources and it’s quickly becoming apparent how fit for purpose cloud native is for the Edge.

The team of engineers at [Synadia](../../index.jsx), a SUSE One partner, in conjunction with Community Maintainers, develop and maintain the open source <a href="https://nats.io" target="_blank">NATS project</a>, part of the <a href="https://cncf.io" target="_blank">Cloud Native Computing Foundation</a>. The NATS communications system is used for all kinds of interesting implementations but we’ve invited Synadia to share two compelling Edge use cases from the energy sector demonstrating the power of cloud native solutions at the Edge. ~Bret

> _Bret Dayley has spent 30+ years in high tech roles with various software companies. A fan of open source and all things good, Bret enjoys making things happen and having a good time while doing it. While fascinated with technology, on the weekends you’re likely to find Bret enjoying the great outdoors on a dual sport moto… with some gadgets along for the ride._

## SUSE guest blog authored by Brian Flannery, VP Global Sales, Synadia

As workloads increasingly run at the edge, there is a massive paradigm shift underway to allow location dependent and independent applications to run on any device – securely and efficiently.

Complexity is not something developers want to introduce into edge architectures.  Moreover, edge workloads need to be able to instantly work with cloud, hybrid, and on-premises applicationswithout any re-architecting.  This is where <a href="https://www.suse.com/products/k3s/" target="_blank">K3s</a> and <a href="https://nats.io" target="_blank">NATS</a> work superbly together.

![K3s and NATS](../blog/posts/k3s_nats.png)

NATS is an extremely lightweight, performant, and secure (Zero Trust design) messaging utility. The two together offer edge device and application orchestration (K3s) and communication and data ingestion (NATS), solving two critical design requirements for any developer.

Two interesting, recent real-world examples come from the Energy industry.

## How Duke Energy uses K3s and NATS at the Edge

<a href="https://www.duke-energy.com/our-company/about-us" target="_blank">Duke Energy</a> and its Emerging Technology Office (ETO) are committed to innovation in renewable energy resources, energy storage, distributed intelligent grid architectures, machine to machine interoperability through standards like OpenFMB, and best practice Zero Trust cybersecurity. Consequently, ETO has built a proof-of-concept system named ZTAG, “Zero Trust Applications for the Grid”.

ZTAG represents a multi-year effort to develop a best-in-class architecture for the Distribution Grid. The system has now been deployed at the ETO Mount Holly Microgrid using OpenFMB. Each OpenFMB node is a field device with compute resources like an industrial PC or gateway. The system provisions and deploys nodes, updates node applications, and automates key renewal through distributed PKI. The architecture leverages Docker for application containerization, Kubernetes for container management, SPIFFE/SPIRE with Trusted Platform Module (TPM) identity management for workload attestation, and NATS for secure publish and subscribe communications. The system is scalable and secure from the data center to field devices. The goal is for all field devices to adopt node attributes.

SUSE’s K3s is a critical component in the ZTAG architecture. Multiple distributed OpenFMB nodes form logical clusters that work in small groups to implement distributed intelligent (DI) applications at the electric grid edge. These small “edge” clusters are grouped together in a hierarchy of clusters to perform higher level grid functionality. K3s enables quick, secure workload/application updates providing an environment for the development of secure DI applications.

![Duke Energy](../blog/posts/duke_energy.png)

The ZTAG team also works closely with Synadia’s team, as NATS is also a key component of ZTAG. NATS provides low-latency, multi-path, asymmetric, and event driven data flows from device-to-device, device-to-application, and application-to-application throughout the ZTAG architecture. NATS also provides distributed security allowing actors to only communicate with other authorized actors. These features enable the electric grid to no longer be bound by only north-south data flows from devices in the field to so-called “head end” systems in the back office. This new east-west, peer-to-peer messaging architecture enabled by NATS is foundational for distributed applications in the future electric grid.

## How PowerFlex uses K3s, Rancher, and NATS at the Edge

<a href="http://www.powerflex.com/" target="_blank">PowerFlex</a> is a leading national provider of intelligent onsite energy solutions that support carbon-free electrification and transportation. As a single full-service provider, PowerFlex customizes clean technology solutions to help clients achieve their energy and sustainability goals. Through the comprehensive <a href="https://www.powerflex.com/products/powerflex-x/" target="_blank">PowerFlex X platform</a>, PowerFlex leverages patented smart software to control, monitor, and optimize a client’s distributed energy resources to reduce cost and maximize return on investment.

SUSE’s K3s  is a critical part of how PowerFlex builds and manages highly distributed bare-metal systems on the edge. When PowerFlex added Rancher to the mix, it provided an extra layer of observability while keeping hardware and software secure. PowerFlex developers can hit the ground running with K3S on hardware or in VMs, and they value the access that Rancher gives them to pre-production environments. The PowerFlex Sight Reliability Engineering team has confidence in their ability to address issues in production thanks to the rapid and secure access Rancher provides.

Ted Lee, Principal Software Engineer at PowerFlex, and Robbie Hughes, Full Stack Software Developer on the PowerFlex Cloud X team, are replacing legacy REST services with an event-based architecture with NATS at the center. Ted said, “I’m excited that NATS provides a secure connection between our cloud and edge services where messages are just there.” Robbie echoes his excitement and adds that “the leaf node topology is amazing because it allows our sites to operate independently even when networks go offline.”

![PowerFlex](../blog/posts/powerflex_energy.png)

Interested in getting started at the Edge with NATS? Contact [info@synadia.com](info@synadia.com).

## Additional resources on using K3s and NATS:

 * <a href="https://www.youtube.com/watch?v=zbDjr8rcwZs" target="_blank">NATS & K3S | Extend Your Apps to the Edge in Under 60 Seconds</a>
 * SUSECON 2022 session: <a href="https://susecon.com/sessions.html?search=nats#/session/1649088878519001ApvY" target="_blank">Running a Distributed K3s Control Plane at the Edge with NATS JetStream [TUT-1217]</a>

 ## About the Author

 Brian Flannery is VP Global Sales at Synadia, and has experience scaling the business at multiple technology companies during the blast-off phase of growth. Prior to Synadia he was the CRO for a Salesforce development services partner, and also spent time at Open Source Software providers such as Eucalyptus and rPath prior to their acquisitions.

K3s and NATS

NGS Guides: Battery Spa

## How to Use NATS.io to Set Up a SOX Compliance System

An enterprise customer of ours recently asked for guidance around how their NATS.io enabled system could help them achieve SOX compliance. [Sarbanes Oxley Compliance](https://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act) encompasses a very broad set of practices around financial record keeping and reporting, touching nearly every aspect of running a business. While ensuring financial accuracy, undergoing compliance is like going to the dentist - no-one looks forward to doing it but it has to be done.

NATS provides several fundamental features that can help businesses to more easily achieve SOX compliance when properly used. This is most applicable to *Sarbanes–Oxley Section 404: Assessment of internal control*.

These features are centered around NATS advisories and System Events.  **Caveat:** In corner cases, system events and advisories may not be generated, published, or saved in the event of hardware failure, unexpected server shutdown, and other error conditions.

[Exabeam](https://www.exabeam.com) provides a very nice SOX compliance [checklist](https://www.exabeam.com/information-security/sox-compliance-requirements-and-checklist/). There is a lot to SOX compliance; this article is by no means exhaustive but we can run through some of the features of NATS as a good starting place.

## Prevent data tampering

*Implement systems that track logins and detect suspicious login attempts to systems used for financial data.*

This is fairly straightforward to implement for NATS. NATS generates [system events](https://docs.nats.io/running-a-nats-service/configuration/sys_accounts) that can be used to track these and persist them in the NATS [JetStream](https://docs.nats.io/nats-concepts/jetstream) subsystem. [Create a stream](https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/streams#creating) and specfify the subjects you'll want to track to store these events.

The system event subjects you’ll want your stream(s) to include are:

- $SYS.ACCOUNT.*.CONNECT (client connects)
- $SYS.ACCOUNT.*.DISCONNECT (client disconnects)
- $SYS.SERVER.*.CLIENT.AUTH.ERR (authentication error)

If you are using leaf nodes, then include these subject:
- $SYS.ACCOUNT.*.LEAFNODE.CONNECT (leaf node connects)
- $SYS.ACCOUNT.*.LEAFNODE.DISCONNECT (leaf node disconnects)

You can then develop an application report and inspect these events to ensure no suspicious users connect, track down the source of authentication errors, and the frequency, time, and duration of connections are in line with normal parameters of your system. The definition of *suspicious* is left to the implementer.

Note that JetStream streams are tamper-proof, with the exception of erasing a message for GDPR purposes (this action can be locked down and audited).

## Record timelines for key activities

*Implement systems that can apply timestamps to all financial or other data relevant to SOX provisions. Store such data at a remote, secure location and encrypt it to prevent tampering.*

You’re covered by adding a bit of process and would want to ensure relevant financial transactions are stored in the NATS JetStream subsystem. JetStream supports [encryption at rest](https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest) to protect your data. Each message stored in JetStream automatically has a timestamp associated with it, set at the time the message was first stored.  This applies to your event and NATS systems events (if stored in a stream).

```text
$ nats s get transactions 2
Item: transaction#2 received 2022-05-03 17:38:18.114054892 +0000 UTC on Subject payable
```

Data in streams can be [mirrored](https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/replication#mirrors) or [sourced](https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/replication#sources) to a remote location and/or [backed up](https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/disaster_recovery#backup) to files for remote storage.

## Build verifiable controls to track access

*Implement systems that can receive data from practically any organizational source, including files, FTP, and databases, and track who accessed or modified the data.*

NATS has various connecting technologies, but let's focus on tracking of NATS assets.  In a secure NATS system, each connection has a user associated with it.  NATS provides rich monitoring tools to identify which users are accessing the system at any given time. Furthermore, each time there is a CRUD operation on a JetStream asset an [advisory](https://docs.nats.io/running-a-nats-service/nats_admin/monitoring/monitoring_jetstream#advisories) can be sent which includes the identity of the user.  These advisories can be stored in a stream or read by a process to store externally.

## Test, verify and disclose safeguards to auditors

*Implement systems that can report daily to selected officials in the organization that all SOX control measures are working properly. Systems should provide access to auditors using permissions, allowing them to view reports and data without making any changes.*

This is wide ranging.  NATS provides [monitoring](https://docs.nats.io/running-a-nats-service/nats_admin/monitoring#monitoring-nats) and logging in to ensure the system is up, and streams containing metadata of advisories and system events can be replayed anytime to generate a report. Given proper permissions, data stored in NATS can be accessed to generate a report anytime.
	
## Report on the effectiveness of safeguards

*Implement systems that generate reports on data that have streamed through the system, critical messages and alerts, security incidents that occurred and how they were handled.*

Earlier we discussed persisting events correlated to these items to streams. If a user has permissions they can replay the stream containing system events and jetstream audit events to generate a report.


## Detect and disclose security breaches

*Implement security systems that can analyze data, identify signs of a security breach and generate meaningful alerts, automatically updating an incident management system.*

*Implement systems that log security breaches and also allow security staff to record their resolution of each incident. Enable auditors to view reports showing which security incidents occurred, which were successfully mitigated and which were not.*

This would fall on the NATS operators to develop a system that meets the needs of the business. Though the system events generated, NATS provides plenty of data points for an enterprise to quickly build a custom system that does exactly this.

## Summary

SOX compliance is a very broad and loosely defined initiative; this article does not cover every aspect, but can serve as a starting point for enterprises that need to be compliant and are using NATS. We on the NATS team would be certainly open to suggestions as to how to make this easier; if you have ideas or would like to offer contributions in this area please reach out.

## About the Author

Colin Sullivan is VP of Product [Synadia Communications](https://synadia.com). If your enterprise needs assistance with SOX Compliance Synadia, or NATS usage in general Synadia can provide guidance.


Blog