5 CNCF projects for streaming and messaging

Event-based architecture is becoming more common in cloud-native ecosystems. By focusing on events, services can trigger additional functionality automatically, reducing manual steps and changing the traditional request-response communication paradigm. In such streaming environments, services “subscribe” to an event source or “publish” events for other services to listen to. Often they carry out a combination of both actions. Connecting services in this way can eliminate over-recovery issues.

Streaming and event management platforms, such as Kafka, RabbitMQ, or NATS, add a layer to a system of services, focusing explicitly on passing messages between services. This helps applications stay decoupled from each other while still allowing communication in a common format. Below, I’ll review CNCF projects related to cloud-native streaming and messaging. These five open-source packages are useful utilities to help you implement a publish-subscribe model in your service ecosystem.

1. Cloud Events

A specification for describing event data in a common way

Git Hub | Website

A disadvantage of event proliferation is that each event editor system describes events differently. When connecting a large number of services, you will likely encounter various protocols and event sources, requiring conversions. CloudEvents aims to solve these event management problems. This is a specification for describing events in a consistent way, which could facilitate the integration of event sources and improve interoperability in the field.

The CloudEvents specification dictates the formatting of necessary information, such as id, type, source and specversion, as well as for other optional values. Here is an example CloudEvent represented in JSON according to the specification:

    "specversion" : "1.0",
    "type" : "com.github.pull_request.opened",
    "source" : "https://github.com/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : ""

CloudEvents is part of the CNCF Serverless working group. As of this writing, the CloudEvents project is in incubation with the CNCF.


A high-performance server for NATS.io, the cloud-native and edge-native messaging system

Git Hub | Website

NATS is a high performance and portable communication layer well suited for distributed systems. The NATS server is a publish-subscribe server that uses JetStream for real-time data streaming. The community supports over 40 different client libraries in different languages, making it easy to communicate with a NATS server whether the client is mobile, edge, desktop, web, on-premises or in the cloud. The project was originally created by Derek Collison and is supported by Synadia. At the time of writing these lines, NATS is an incubator project within the CNCF.

3. Pravega

Storage for a streaming world

Git Hub | Website

Another cloud-native streaming architecture to consider is Pravega, a CNCF sandbox project. Building on the success of streaming platforms such as Pulsar and Kafka, Pravega goes even further to support a wider range of applications and features. Some of Pravega’s unique capabilities include long-term retention, large data ingestion, and consistent state replication. In-stream autoscaling is a unique feature of Pravega, which could help respond to changes in ingest rates for certain partitions. Some potential use cases for Pravega include predictive maintenance for IoT, real-time billing, and cyber threat detection.

4. Strimzi

Apache Kafka running on Kubernetes

Git Hub | Website

Strimzi offers an easy way to get Kafka, the popular message bus, up and running on Kubernetes with just a few commands. Strimzi provides container images and operators that simplify the process of deploying and running Kafka clusters and components, configuring access to Kafka, and securing access to Kafka, among other features. Strimzi employs four operators to accomplish these tasks: the cluster operator, entity operator, subject operator, and user operator. Strimzi is compatible with various Kubernetes deployment configurations. To get started with Strimzi, check out the latest Quick Start Guide.

5. Tremor

An early stage event processing system for unstructured data

Git Hub | Website

Tremor is an event processing system intended to aid platform engineers working in high-density throughput situations where constant availability is critical. Tremor is packed with features and could replace several other data processing and distribution tools. The tool was designed at Wayfair and has been in production since 2018, and it continues to evolve with additional use cases. But since it was designed with e-commerce in mind, engineers supporting similar situations may find it particularly useful. To get started with Tremor, you can extract the Docker image here.

In our hyper-connected digital economy, a robust connective tissue is needed to connect services reliably and securely. Platform middleware like the ones above can help enable a publish-subscribe model, a model that has become necessary to enable decoupled architecture and real-time streaming. These packages and projects all contribute in different ways to integrating events into the cloud-native ecosystem.

Comments are closed.