Waterstream Deployment Overview

Typical deployment of the Waterstream looks like this:

Waterstream deployment

Multiple Waterstream nodes (typically running as Docker containers, JAR may be provided on demand) are running behind the non-sticky load balancer. MQTT can connect via the load balancer to any Waterstream node, and in case of the connection disruption - re-connect at a later point to any other node while maintaining its session state (in-flight messages, subscriptions etc.). All Waterstream data is persisted to the Kafka or other compatible streaming platform - no persistent volumes needed for Waterstream nodes, any node can come and go at any time together with its disk content.

Waterstream nodes expose metrics which can be collected by Prometheus and displayed in Grafana as a dashboard.

Waterstream needs to check a license when it starts. There are 2 options here - either a license file or AWS Marketplace subscription. You can request the development license at https://waterstream.io/try-waterstream/ or sign up for AWS Marketplace subscription at https://aws.amazon.com/marketplace/pp/prodview-x4fnyp54socdw .

You can create Waterstream deployment scripts from scratch (see Waterstream Configuration for the complete list of the configuration parameters) or start from some of the readily available and adapt them for your needs - we’ll make a brief overview here.

AWS CloudFormation

This is most straightforward way to start with Waterstream on AWS: https://github.com/simplematter/waterstream-aws-ecs-cloudformation

  • No need for any locally installed tools - the only pre-requisite is the AWS account and the DockerHub account. The scripts can run in AWS Console.

  • (Optional) Kafka setup with AWS MSK - can be skipped if you already have Kafka cluster

  • Uses AWS ECS Fargate to deploy the Waterstream Docker containers

  • Include Grafana + Prometheus

  • Include a “testbox” - small VM which includes command-line MQTT clients properly configured for connecting to the Waterstream

  • Support SSL for the MQTT connection

AWS Terraform with ConfluentCloud

If you prefer Terraform rather than CloudFormation you can use these scripts to deploy Waterstream on AWS: https://github.com/simplematter/waterstream-aws-ecs-ccloud-terraform

  • Relies on Confluent Cloud <https://confluent.cloud/> as a Kafka cluster provider

  • Uses AWS ECS Fargate to deploy the Waterstream Docker containers

  • Includes Grafana + Prometheus

  • Include a “testbox” - small VM which includes command-line MQTT clients properly configured for connecting to the Waterstream

  • Support SSL for the MQTT connection

GCP Terraform

Default approach for deploying Waterstream on GCP: https://github.com/simplematter/waterstream-gcp-terraform

  • Relies on Confluent Cloud <https://confluent.cloud/> as a Kafka cluster provider

  • Deploys directly on compute instance rahter than through k8s - this approach can handle more TCP connections, but somewhat sacrifices observability

  • Includes Grafana + Prometheus

Kubernetes Terraform

The most versatile approach - runs on any generic Kubernetes cluster: https://github.com/simplematter/waterstream-k8s-terraform

  • Relies on Confluent Cloud <https://confluent.cloud/> as a Kafka cluster provider

  • Includes scripts for starting Kubernetes clusters on AWS, GCP and Azure - in case you don’t have K8s yet

  • Includes Grafana + Prometheus

Local playground - docker-compose

To test the Waterstream locally with the minimal effort you can use https://github.com/simplematter/waterstream-playground

  • Includes ZooKeeper, Kafka, Waterstream, Grafana and Prometheus

  • Uses docker-compose to run all the needed components