# Synadia + Control Plane

Multi-Cloud is a buzzword today but one with real bite. In real conversations with real [NATS.io](http://NATS.io) users and engineering teams, we hear it over and over again. These teams want this to move away from dependency on a single cloud ASAP. The reasons why are clear:

- Eliminate cloud lock-in
- Improve resiliency and availability
- Gain more control over performance and SLA fulfillment
- Optimize pricing and enable smart arbitrage across zones, geographies and instance types or cloud services

This is why we are so excited about our partnership with [Control Plane](https://controlplane.com/). Control Plane is a platform that abstracts away all of the complexities of building multi-cloud applications. On Control Plane, microservices can run simultaneously on any combination of cloud compute and consume any combination of cloud services without embedded credentials. The platform handles identity conveyance and authorization uniformly, utilizing best-practices/least privilege principles consistently and securely.

Control Plane now integrates [Synadia’s NGS managed NATS-as-a-Service offering](https://synadia.com/ngs) to provide multi-cloud messaging, data streaming, eventing, and persistence capabilities to applications developers on a global-scale.

With NGS, Control Plane customers gain a unified connectivity utility and data mesh that removes the need to manage and provision different services on individual clouds. With simple script and easy configs, they can replace virtually any heavyweight messaging, data streaming, and pub/sub platforms.  NGS also includes JetStream, a highly configurable persistence layer included in [NATS.io](http://NATS.io). JetStream allows you to set detailed parameters and persistence policies and includes an inline key-value store for caching (replacing Redis, in many instances) and an object store.

“Our customers love NGS because it gives them a virtual ‘nervous system’ that can move data and information in any pattern they want, in any cloud environment,” says Doron Grinstein, founder and CEO of Control Plane.

“NATS and NGS are incredibly light, very fast, efficient and versatile. It was designed for fanning out to customers. Now, with JetStream, it can also handle pushing information down for eventing and durability,” says Dimy Jannot, CTO of SAFE Health, which uses Synadia NGS and Control Plane for granular policy application and certificate management across clouds. “The integration with Control Plane allows us to leverage NGS in all of our environments without managing credentials. We can manage developer and workload permissions at a granular level that aligns with our zero-trust model. NGS and NATS are lower cost, easier to manage and far more flexible than Kafka. I am pushing it everywhere.”

“Synadia’s customers demand a simple and secure multi-cloud experience. Integrating with Control Planes’ Universal Cloud Identity model extends our zerotrust implementation and accelerates developer productivity ” says Synadia CEO and founder Derek Collison, the creator of [NATS.io](http://NATS.io).

Control Plane gives any organization Big-Tech cloud portability with a few lines of code. On Control Plane, your workload runs as-if-native on all clouds. Control Plane routes to containerized workloads in a geo-optimized manner, delivering ultra-low latency. A single workload can consume endpoints from multiple cloud accounts across different clouds, using any cloud-specific service. Workloads securely and efficiently call any TCP/UDP endpoints behind a firewall, such as in a private data center or inside a VPC.

For Control Plane users, Synadia NGS offers a connectivity Swiss Army Knife that is easier, faster, and less costly than RabbitMQ, MQTT, Kafka, SMS/SQS, and most gRPC-based messaging infrastructure. Written in Go, [NATS.io](http://NATS.io) offers lightning-fast performance and low latency. Used with Control Plane, NGS can handle a wide variety of use common cases including:

- Multi-Cloud / Multi-Geo messaging
- Zero Trust policy and authentication simplification
- IoT and edge applications
- User management of distributed applications
- Coordination of active/active architectures and near-instant failover

Control Plane and Synadia are making multi-cloud a whole lot easier. Here are some additional resources:

- Read the [documentation here](https://docs.controlplane.com/release-notes#cli-0.0.59)
- Reusable [code examples](https://natsbyexample.com/)
- Sign up for a [free NGS account](https://synadia.com/ngs)
- We can’t wait to see what you think or if you are interested in a demo please send us an email to [info@synadia.com](mailto:info@synadia.com) or stop by the Control Plane booth #SU93 at KubeCon + CloudNativeCon this October in Detroit.

Creating a Global Virtual Healthcare Cloud with Control Plane & Synadia NGS

Synadia & Control Plane

# [Sote](https://sote.com/)

- Industry: Logistics forwarding serving Africa
- Location: Nairobi, Kenyan
- Employees: 50+
- Tech Stack:AWS, PostgreSQL, Golang (back-end services), React (front-end), [Synadia NGS](https://synadia.com/ngs)
- Language: Go

## **Summary**

Sote provides digital logistics shipping products internationally to, from, and within Africa. The company operates an ERP system designed specifically for customs and shipping logistics in that continent. Sote CTO and co-founder Scott Yacko uses Synadia's NGS globally managed [NATS.io](http://NATS.io) service as the primary data and control plane for all the various back-end business services required to deliver application functionality. Sote initially built customer HTTP connections for service management but needed a messaging and connectivity platform with lower latency and higher performance. Yacko deployed NGS quickly, reducing latency by more than 2x, simplifying his code and cloud infrastructure, and future-proofing Sote for additional product requirements down the road.

## **Introduction**

Sote provides a digital logistics support platform for shipping products internationally. International shipments require detailed understanding of customs laws and practices and specific tax requirements for different shipment types in different countries. A SaaS solution, Sote has created and operates an ERP system designed specifically for customs and logistics. The company focuses on the African market and aims to be the logistics gateway for the continent. Hosted in an AWS datacenter in Ireland, Sote’s architecture runs on Containers, with hundreds of containers running concurrently during peak times. Back-end services are written in Go connecting to a React front-end, with Postgres as its primary data store.

To ensure complete decoupling of all services down to the most atomic level possible, CTO and co-founder Scott Yacko created a system of “business services” that developers can compose into compound applications. Unlike microservices, where in a single organization, developer teams might create similar services, in Sote’s architecture there is only one business service for each specific use case. This creates better efficiency by preventing engineers from rewriting microservices and enforcing shared resources and consistency. As a former AWS director, Yacko wanted to minimize divergence and design a system that was fast, resilient, persistent, and efficient, reusing the same code resources and services as widely as possible.

## Challenge: A Lightweight, Non-Blocking Communications Back-End

To compose applications from business services, “You can create what we refer to as a gatherer service, and its only purpose is to manage the interaction between those other two business services,” says Yacko. In this architecture, fast and flexible service-to-service communications is essential. Yacko has experience building out these back-end communications systems with JavaBeans and Kafka but wanted something more flexible and lightweight.

The Sote team initially built the communications mesh using HTTP. After considerable efforts to reduce latencies, Sote was still seeing wait times of two seconds or longer for some requests. “It was going through a ton of hops, and we tried everything to reduce the slow response. But it wasn’t meeting our needs for a responsive system,” says Yacko.

The effort to tune the bespoke communications service was taking up significant engineering resources and was requiring a larger infrastructure spend than Yacko had hoped. “We are a small company,” explained Yacko, “We can’t afford a lot of servers running in a lot of data centers.”  Because Sote was delivering a critical, time sensitive service, it needed guaranteed availability and data persistence. “We can’t just tell our clients as they are waiting to clear customs, ‘Sorry, we had to restart. We’ll get you your status soon.’”

---

**Business Benefits**

- Improved customer experience
- Ability to more easily extend and expand services
- Reductions in engineering time and server costs

**Technology Benefits**

- 2-3x reduction in communications latency
- Simplification of tech stack
- Reducing tech debt
- Future-proofing app for IoT and scale-out operations

---

## Solution: [NATS.io](http://NATS.io) and NGS Managed by Synadia

Yacko researched alternatives to Kafka and came across [NATS.io](http://NATS.io). He asked his network of contacts if they had heard of NATS. Several told him they had used it and found it lightning-fast and easy to use. Yacko started prototyping and began asking questions in the [NATS.io](http://NATS.io)[ Slack](https://slack.nats.io). Synadia engineers guided him through a test deployment and gave him early access to NGS, a managed version of NATS operated by Synadia engineers. NGS offers the type of communication infrastructure that Sote needed as an on-demand, global utility optimized for easy scalability and minimal latency. As part of NATS and NGS, Sote also could use JetStream, a flexible but powerful persistence layer incorporated in the NATS server software. Sote was an early JetStream user.

---

> ### 
>
> “We started using NGS in production even prior to the GA,” says Yacko. “It’s been extremely high quality and extremely stable."
>
> *Scott Yacko  -- CTO and co-founder, Sote*

---

## **Results**

It took Sote only 48 hours to set up a POC and test-drive NGS. “I loved what I saw and decided to use it in production,” says Yacko. With NGS, Sote enjoys request latencies of 500 to 700 milliseconds, which is 4-to-6 times faster than the previous system. “We started using NGS in production even prior to the GA,” says Yacko. “It’s been extremely high quality and extremely stable.

Yacko also viewed NATS and NGS to future-proof the Sote stack. “Unlike Kafka, I didn't have any pre-prescribed constraints. NATS is basically a scratch pad. I can do whatever I want to do in the header or in the body. I can parse text, binary or JSON, even in the same message.” When Sote eventually starts tracking shipments using IoT devices, NGS will help the company scale to a higher volume and also provide a scale-out communications backbone with global coverage.

NATS and NGS allowed the Sote back-end team to stop worrying about managing a mesh of HTTP connections and let NGS handle the connection fabric of business service communications. This included user management, security, gathering services into apps, observability, and all other core functions. “The backend team was ecstatic. We were taking exorbitant amounts of time tuning it and it just wasn't going to scale,” explains Yacko. “We were going to have to abandon the entire architecture. We went through a three-month effort to replace the HTTP with NATS while adding new features.”

Over time, Sote has continued to work closely with the Synadia engineering team to keep Sote’s applications running smoothly. “We have never had to wait more than 48 hours for Synadia to turn around an issue,” says Yacko. “You can’t ask for more than that as a startup company.”

Sote Slashes Latency More Than 2x Running Synadia’s  NGS Utility

Case Study: Sote Cuts Latency 3x, Improve Resilience with NGS

We have entered a new era of distributed applications and infrastructure driven by the unique performance, security and resilience required on the Edge. Developers of distributed applications need new capabilities that allow applications to continue functioning effectively, even in conditions of poor or intermittent connectivity. As performance expectations and service level agreements approach near real-time for a growing array of functionalities, applications require greater persistence and autonomy on far more widely distributed arrays of nodes; the Edge becomes the compute fabric, the memory and the application tier, all rolled into one.

As innovation takes off at the Edge, one aspect we’re all trying to avoid is complexity. The Synadia Adaptive Edge vision encompasses the delivery of real-time microservices & applications across any cloud, any geography, and to any device. As developers & architects start with our platform they quickly discover how they can dramatically reduce their costs & simplify the software stack by removing components like:

- Load Balancers
- Firewalls
- VPN/VPC
- Traffic Shapers
- External Latency Tracking
- Service Mesh
- Cross cloud SecOps

We are thrilled to partner with the [IoT Community](https://iotpractitioner.com/) & [Valuer.AI](http://Valuer.AI)[ ](https://www.valuer.ai/)on this report.  Explore the full article - [IoT Community Cloud-Native Applications.](https://a.storyblok.com/f/166429/x/254b41d620/iot_cloud-native-applications.pdf)

Are you working on IoT & Edge application initiatives? Let Synadia help, contact us - [info@synadia.com](mailto:info@synadia.com).

NATS for the Adaptive Edge - A cloud native platform for IoT developers

IoT Portal Platform

- Location: California, U.S.A.
- Employees: \~25
- Industry: Software, Consumer Technology, Media
- Tech Stack: Kubernetes, NATS+Jetstream, NGS, MongoDB, Postgres, Elastic, TensorFlow, Istio, VueJS, AWS

## **Summary**

[Personal.ai](http://Personal.ai) is a revolutionary artificial intelligence platform that's trained on the memories provided by conversations, social media interactions and other forms of communication. The system builds an AI model for each user, with the mission of empowering every individual to own their intelligence and be their own thought leader. The AI is self-trained on data an individual creates and captures as life happens. [Personal.ai](http://Personal.ai) automatically categorizes all data and creates a structured view of an individual’s life that enables many use cases including time capsules, AI-powered mentorship, always-on AMA, and interactive NFTs.

[Personal.ai](http://Personal.ai) uses [NGS](https://synadia.com/ngs) to connect its various microservices, reduce latency for job completion, and reduce infrastructure costs.  The backend of the application runs in Kubernetes clusters in Amazon Web Services (AWS) with Synadia’s NGS utility providing connectivity for streaming, messaging and data persistence with JetStream.

## Background

[Personal.ai](http://Personal.ai) is attempting something that has never been done before — to literally create a personal artificial intelligence agent for creators and people to give them creative and intellectual superpowers. [Personal.ai](http://Personal.ai) will also serve as a living memory construct that can either augment a user’s own memory or talk to other people with conversations and responses that interpret how the user themself would respond and think. “I was having a conversation with my dad a while back. Basically, he was saying, "Hey, I never see you. I don't really talk to you very much. Do you think there will be a day that I can talk to you like I talk to Alexa?," explains [Personal.ai](http://Personal.ai) CTO Sharon Zhang. “That sounded like a really good problem to solve.”

[Personal.ai](http://Personal.ai) leverages various technologies, such as automated speech recognition, natural language processing, and content collection to capture all media, content and spoken words generated by a user and turns it into structured data. Unlike most AI companies which rely on larger generalized data models, [Personal.ai](http://Personal.ai) constructs a model for each user in about 20 minutes. The model self-trains and improves over time based on the content it ingests and processes. The user controls their personal data and content; all processing takes place in a cloud environment leveraging the Oasis blockchain technology to track changes and maintain a chain of custody. “It's really thinking about AI and how we can use blockchain to preserve privacy while solving a core human problem,” says Zhang.

## Application Architecture

The [Personal.ai](http://Personal.ai) application incorporates many thousands (and eventually millions) of parallel streams of data at the same time. A user might require multiple streams simultaneously to capture voice data, online or web activity and content from other sources. Each message is critical because it consists of unique data. A persistence layer preserves stream data (audio, text, etc) until a worker is available for processing. [Personal.ai](http://Personal.ai) opted for a unidirectional pub/sub model that pushed data labeled by subject to workers before sending it after processing into databases for storage. [Personal.ai](http://Personal.ai) uses WebSockets to collect data from traditional HTTP sources and uses a VueJS front-end for audio capture. Audio streams are processed by Speechmatics, with provides automated speech recognition as a service. The data storage layer includes a Postgres database for storing relational information about users, a Mongo object store as intermediate storage for fast writes, and an Elastic storage layer for longer-term storage and quick queries. [Personal.ai](http://Personal.ai) also stores some data in the AWS S3 Object Store.

[Personal.ai](http://Personal.ai) needed a messaging and data streaming system that provided:

- Fast speeds with low resource requirements
- A flexible persistence layer
- A broad array of client languages
- Rapid scalability
- Low management overhead
- Flexibility to follow streaming or messaging conventions
- Always-on reliability

## Challenges: Resiliency, Reliability, Cost, Extensibility and Management Overhead

This unique application architecture, the strict performance requirements, and the large number of components presented many DevOps challenges.“We need to be able to manage these parallel streams at the same time" says Bala Sista, VP of Engineering at [Personal.ai](http://Personal.ai). “We have thousands upon thousands of streams going on at the same time, each going to their own unique data model.” Additionally, [Personal.ai](http://Personal.ai) has multiple microservices for other middleware and backend functions as well as external services for automated speech recognition. [Personal.ai](http://Personal.ai) uses a hybrid on-chain/off-chain instance of the Oasis blockchain to enforce data permissions and verify and publish data changes to the distributed ledger. ”There are a lot of moving parts in the system,” says Sista.

Because the volume, number, and type of data streams are unpredictable, [Personal.ai](http://Personal.ai) required a fast-scaling system and an always-on messaging and streaming service. An individual user might have multiple streams running simultaneously, either for multiple audio streams or data ingested from social media and community forums, to name two examples. With highly sensitive data flowing through its system, [Personal.ai](http://Personal.ai) wanted robust user segregation. The individual data models required to create each user’s AI meant that [Personal.ai](http://Personal.ai) would need to manage a large number of subjects through its messaging and streaming infrastructure. Sista wanted fast messaging with a small footprint to save costs.

Because [Personal.ai](http://Personal.ai) used a wide variety of software languages and technologies, it required ready-made clients for easy integration into the messaging and streaming infrastructure. To hold onto user data until it was sure all necessary processing had taken place, [Personal.ai](http://Personal.ai) required a flexible persistence and caching layer. The persistence layer would ideally be integrated with the message and streaming infrastructure. Lastly, the messaging and streaming system had to be reliable with high availability, and able to operate in lower-resource environments like individual smartphones.

Sista initially built [Personal.ai](http://Personal.ai) using Amazon Lambda serverless functions with a Simple Queuing Service (SQS) and Simple Messaging Service (SMS) to call up the workers and manage the pub/sub infrastructure. [Personal.ai](http://Personal.ai) deployed a Redis key-value store for persistence. A bug in the [Personal.ai](http://Personal.ai) health checks resulted in a massive AWS bill. Other concerns emerged when bottlenecks appeared in data pipelines and management of expensive GPU instances used for specific processes was imprecise. “We knew then we needed to move to something else and Kubernetes was the best option,” says Sista.

In this new approach, each user and their associated streams and data became a Kubernetes micro-pod in Amazon’s managed EKS Kubernetes service. Sista also wanted to move away from the SMS/SQS pairing to a more robust, reliable, flexible and easier to manage messaging and streaming infrastructure. [Personal.ai](http://Personal.ai) ruled out Kafka from the start due to its heavy infrastructure requirements, scaling challenges, and high management overhead.

Other messaging systems, such as RabbitMQ and MQTT, were also ruled out due to the inability to easily run active-active configurations at scale or to handle streaming data plus considerable overhead and resource requirements.

## Why [Personal.ai](http://Personal.ai) Chose NGS, NATS and JetStream

Sista had previously heard of NATS, a well-known and widely used open source connection fabric hosted by the Cloud Native Computing Foundation ([CNCF](https://www.cncf.io/)). NATS provided both data streaming and traditional messaging capabilities. NATS had also just added a [new persistence layer called JetStream](https://docs.nats.io/nats-concepts/jetstream) which included a [key value store](https://docs.nats.io/using-nats/developer/develop_jetstream/kv) for caching and materialized views. Synadia, the corporate steward of NATS, also offered NGS, an as-a-service offering of NATS running in multiple geos and multiple clouds.

---

Technology Benefits

- Low latency, high performance
- Short learning curve
- Small footprint and resource requirements
- 45+ clients and pre-baked connectors
- Combines data streaming with persistence layer and KV store
- Can scale horizontally or vertically in seconds
- Works equally well as a data plane and as a management plane
- Extensible to SaaS++ model with Leaf Nodes

Business Benefits

- Low cost with high performance
- Open source and hosted by CNCF
- Simple and robust data security
- Reliable — never goes down and doesn’t lose data
- Minimal management overhead — less than one FTE
- Covers use cases of multiple other solutions (messaging, streaming, caching, KV store)

---

NATS proved to be fast, flexible and easy to manage. NATS also has a extensibility feature called Leaf Nodes, which allows users to connect NGS to on-device or on-premise NATS instances. With Leaf Nodes, NATS and NGS can behave like a single distributed system that would enable the AI and data processing to run directly on edge devices like smartphones without requiring connectivity back to the cloud. The edge caching would also enable eventual data consistency and ensure that captured data could persist until all processing was finished - on cloud-based GPUs, for example, With 45 language clients and counting, Sista found that the NATS ecosystem covered everything

“We compared all the messaging and streaming technologies and it was clear right away that NATS checked all the boxes,” says Sista. [Personal.ai](http://Personal.ai) was able to get their first NATS test environment up in a matter of minutes. “The YAML manifests are very clear and easy to use. The way NATS is designed makes startup quick and painless,” says Sista, who notes that the community and the Synadia team were extremely helpful and responsive. [Personal.ai](http://Personal.ai) found that NATS installed on small public cloud instances was extremely cost effective compared to other messaging and streaming options like Kafka and RabbitMQ. Despite the smaller compute capacity of these instances, NATS was still handling high volumes of streams and messages without any performance lags. “I can run NATS on my laptop and it always connects and works well,” explains Zhang. “I could never do that with the other messaging and streaming technologies.”

The wide number of clients mean Sista could connect, observe and monitor all internal services using NATS and NGS. NGS, in particular, eliminated all overhead from managing and maintaining the NATS software itself, while also delivering even faster response times for streams and messages. With JetStream, [Personal.ai](http://Personal.ai) was able to eliminate a separate key value store and simplify its architecture. Sista also configured NATS to manage all of [Personal.ai](http://Personal.ai)’s microservices and external services for ASR. Most importantly, NGS and NATS proved highly reliable and available, “It just doesn’t go down,” says Zhang, ”We can always count on NATS.”

Streaming, Messaging and Persistence for Personal.ai

Personal.ai

# <a href="https://intelecy.com" target="\_blank">Intelecy</a>

- Industry: Industrial
- Location: Oslo, Norway
- Customers: TINE, Heidelberg Cement, IFF, AM Nutrition, Eidsiva Bioenergi, Glencore Nikkelverk
- Tech Stack: Kubernetes, Nomad, ClickHouse, gRPC, NATS, Synadia NGS, Kafka, Microsoft Azure, Microsoft Windows, Linux
- Application Language: Go

Intelecy is a No-Code AI platform that helps industrial companies optimize factory and plant processes with real-time machine learning insights. These insights allow process engineers to manually or automatically improve outcomes, quality and sustainability without requiring significant custom code or hardware. Designed to help factories move towards greater automation and true “lights out” operational capabilities, Intelecy securely collects data from tens of thousands of sensors in a factory and pushes that data into the cloud using NATS.io and Synadia’s NGS service. NGS is a global NATS utility running in multiple clouds and multiple geos. Intelecy uses NATS clients to create an edge-aware application with intelligence and persistence running on all of its connected systems including low-resource processors running on factory machinery and sensors.

## How It Works: A Unique Bi-Directional Data Pipe for ML

Intelecy provides a bi-directional data pipeline to give industrial companies cutting-edge machine learning for on-the-fly insights to optimize operations. Users, including process engineers and continuous improvement teams, can either view insights and data directly on Intelecy’s front-end or integrate the data and machine learning insights into other systems of record or presentation layers and analytics engines.
Intelecy’s bidirectional capability is unique. Unlike other industrial process analytics systems, Intelecy can write data back to the control units in the factories. This enables automation and closed-loop optimization based on rules created by engineering teams or machine learning models. “For example, a dairy can use a machine learning forecast model to set temperature control systems using the real-time predicted state of the cheese production process,” explains Jonathan Camp, Intelecy’s Chief Security Officer. “The process engineering team can use Intelecy insights to identify trends and then automate temperature adjustments on a vat of yogurt to ensure quality and output are not compromised.”

## What Intelecy Needed for Messaging and Streaming

Low Latency: Less than two seconds round-trip time for processing streamed data and returning ML-driven insights to customers
Segmentation and Security: Secure and simple customer segmentation of data streaming in a multi-tenant environment
Portability: A cloud-native and edge-friendly solution that was portable and cloud agnostic
Intelecy looked at several streaming and messaging applications, including some using MQTT, but found them to be less flexible and requiring more resources for stand up and maintenance. Intelecy briefly looked at Apache Kafka but decided against using it extensively for data streaming. “Kafka has requirements that it puts on the client,” says Camp. “The client has to be aware of a lot of things. I wanted the client to be really simple.”

## Opting for NATS + Synadia NGS

Intelecy opted to replace its polling-based pub-sub data fabric with Synadia’s NGS. “The hosted service closed the deal,” says Camp. “We are a small startup. We don’t have time to manage everything.” The Intelecy data plane consists of a fabric of NATS services running on NGS. For Intelecy, the gateway controller connects to the gateway, which in turn is linked to the factory or plant sensor web and controllers. These endpoints often number in the tens of thousands. The Intelecy gate fetches data in real-time from those sensors in one of the data formats common to industrial processes and writes that process data back to NGS to a specific topic. Although originally written with a BoltDB caching layer, Intelecy added persistence when Synadia added the JetStream persistence and durability layer to NGS. JetStream made NGS far more useful to Intelecy.

## NGS + NATS Technology Benefits and Results for Intelecy

- 15-30x reduction in roundtrip latency
- Cloud and infrastructure agnostic - no lock-in
- Requires zero maintenance
- Improved resiliency and higher availability
- Improved security for multi-tenancy
- Eliminated need for complex proxy or service mesh
- Enables combination of data streaming and batch
- Handles large message sizes
- Intelecy Business Benefits
- Onboard and manage customers easily
- Improved security
- Less technology complexity
- Reduced operational overhead
- Easy scalability

Download the entire [Intelecy case study](../synadia-case-study-intelecy.pdf) to see the full indepth technical detail.

How Intelecy Optimizes Factory Processes with  NATS, NGS and JetStream

Intelecy

# [Form3](https://www.form3.tech/)

* Industry: FinTech
* Location: UK Remote workforce
* Employees: 256
* Customers: Square, Mastercard, Lloyds, Barclays 
* Tech Stack: Logz, Linkerd, Docker, NATS, Kubernetes, K3S, HashiCorp Vault, AWS, CockroachDB, PostGres, Linux
* Application Language: Go

Founded In 2016, Form3 is revolutionizing the world of payment processing with an always on, cloud-native, Payments-as-a-Service platform. Today, Form3 is trusted by some of  the UK and Europe’s biggest Tier1 banks and fastest-growing FinTech’s to handle their critical payments architecture. Back-end and inter-bank payment processing is a highly regulated business that is expensive, infrastructure-intensive, and suffers from a slow pace of innovation. Form3 sought to radically improve on the old payments processing technology model by moving the entire operation into the cloud and designing a cloud-native application structure with microservices. “When we are talking about banks, it's not the most modern stack,” explains Form3 evangelist Adelina Simion. “To have a platform fully hosted in the cloud, that is cloud native, is a huge step forward. We don’t have actual hosted servers running our services.”


## How it Works: A Single Payments API to Remove Complexity

Form3’s service sits in between banks and the external payments processing systems such as Swift, BACS Faster Payments and SEPA. Each of these systems has their own schema for formatting messages and retries, and has different security requirements. The multiple standards and requirements generate complexity and require considerable maintenance and troubleshooting. Form3 provides a single payment processing API and abstract away the complexity, allowing banks to consume payment processing as a service. “We take away the burden of multiple integrations and we can do this at scale and in the cloud,” explains Simion.

Architecturally, putting payments processing into the cloud makes it much simpler for Form3 to stand up and expand a truly distributed application infrastructure. This would push them closer to end users, reduce latency, and improve availability. Form3 deploys a cloud-native architecture running on Kubernetes, with loosely coupled microservices.


## The Form3 Application Architecture

The application is written in Go and composed of multiple services running on Elastic Kubernetes Service (EKS) with the Linkerd service mesh handling microservice routing, observability, and performance guarantees. Customer financial services institutions and fintechs send a payment processing request to the Form3 API, which is hosted in AWS. Form3 translates the request into their internal request format and uses AWS’ Simple Notification Service (SNS) and SQS to create a paired pub-sub event bus for processing messages. In this system, payment requests “fan out” to one of the four supported external payment processing systems (SWIFT, BACS, SEPA, Faster Payments). uses the AWS Simple Queuing Systems as a pub sub system to define rules for processing the request, including recipient, retries, and more. For SWIFT, BACS and SEPA. Form3 validates each payment request and provides a secure gate connecting to those services.

For Faster Payments, the regulations require a separate physical instance. To process Faster Payments requests, which have a guaranteed SLA of two-hours transaction time, Form3 is required to lease physical lands and hardware. The Faster Payments processing cannot communicate with the cloud except using these dedicated circuits and dedicated hardware. “We previously had a third-party doing that for us but as we were growing, it was not good enough so we made our own physical data centers with leased lines in it.”  Both in AWS and the FPS-dedicated data center, the Form3 application runs on Kubernetes.


## What Form3 Needed: A MultiCloud, Lightweight Lighting-Fast Event Bus

The company guarantees it will complete a payment request in no more than 500 milliseconds. The AWS overall uptime SLA is 99.9%. This was not sufficient for critical payments infrastructure. Says Simion, “Some of our customers were asking what would happen if AWS goes down? We can’t lose our processing capability.” Creating a higher availability service would require creating a multi-cloud capability with instant switching from one cloud to another in an outage. To summarize, Form3 needed an event bus that was faster to reduce latency and lighter weight to run in physical data centers. To back up tight 500 millisecond customer SLAs, Form3 needed to run replicas of its applications running in multiple public clouds in an active-active **immediately consistent** configuration. For compliance, Form3 needed to run its full application in low-resource physical data centers.


## NATS + JetStream Benefits and Results for Form3

Form3 chose to replace its legacy pub-sub event bus and message services with [NATS](https://nats.io/) and the [JetStream](https://docs.nats.io/nats-concepts/jetstream) persistence layer, which is part of the NATS architecture. Because NATS is so simple to set up, has such low latency and high capacity per server, NATS works equally well in almost any environment and in any cloud.

Because it is a complete pub-sub system, NATS can replace both SQS and SNS with a single service. NATS also includes NATS Streaming, which can enable NATS to act as a data streaming service with configurable message persistence. This is important for mission critical message streams — such as payment status — that require “at-least once” delivery rather than NATS Core’s “at-most once” delivery. However, NATS Streaming is being replaced by JetStream, a more advanced, more flexible and easier to deploy persistence layer that also includes a key/value database, data-at-rest encryption by default, and horizontal scalability.


## Form3 Benefits and Results

The Form3 engineering team tested NATS and NATS Streaming as a replacement event bus for their SQS and SNS pairing. The Form3 team quickly determined that NATS met their expectations in terms of latency, reliability and resilience and compute requirements. Form3 worked closely with the Synadia team to ensure that the planned architecture would meet the sub-500ms latency requirements. After this, Form3 made the architectural switch, replacing SQS and SNS with NATS. The top latency for messages immediately dropped to less than 50 milliseconds, a more than 6x improvement that offered Form3 strong assurances it could meet SLA requirements for transaction processing, even under heavy loads. 

<figure>
    <img src="/articles/news/form3_nats_prod.png"
         alt="NATS in Production at Form3">
    <figcaption><i>NATS in Production at Form3</i></figcaption>
</figure>

What also impressed the Form3 engineering team was how far smaller instances of NATS in terms of CPU and RAM could process even more transactions than the company's previous architecture. For the Faster Payments physical data center and leased lines, Form3 also has deployed NATS JetStream because it needed to apply more configurable persistence. NATS fit into the limited footprint environment of the data center without difficulty and actually enabled Form3 to handle a greater volume of payments than the previous architecture, something that will simplify and reduce future scaling costs in this more expensive environment. 

Form3 is in the process of deploying NATS to create a cloud-agnostic architecture leveraging JetStream and its highly configurable persistence layer. Workloads could easily shift from one cloud to the next, using NATS JetStream and Leaf Nodes, a method that NATS uses to easily  bridge NATS servers. “It lets us break free from the SLAs of any one cloud and create our own,” says Simion. “None of this would happen if we couldn’t have a proper event bus across the clouds.”

* 6x decline in average latency, from 300ms to 50ms
* Higher messaging throughput on smaller cloud footprint
* High reliability — never requires restarts
* Lightweight - no JVM, no Zookeeper
* MultiCloud ready in active/active with immediate consistency
* Flexibility to handle streaming or message bus
* Highly configurable persistence layer
* Runs in low resource environments
* Supports active-active architectures with minimal configuration

Download the entire [Form3 case study](../synadia-case-study-form3.pdf) to see the full indepth technical detail.


Articles & Case Studies

Creating a Global Virtual Healthcare Cloud with Control Plane & Synadia NGS

Sote Slashes Latency More Than 2x Running Synadia’s NGS Utility

NATS for the Adaptive Edge - A cloud native platform for IoT developers

Streaming, Messaging and Persistence for Personal.ai

How Intelecy Optimizes Factory Processes with NATS, NGS and JetStream

How Form3 Built a Multi-Cloud Low-Latency Payments Service with NATS.io + JetStream