Waterstream Quick Start

To try out Waterstream on your computer please follow the next steps

Prerequisites

If you don’t have Confluent Platform and Confluent CLI on your machine, the Docker Compose quick start is an easier alternative that requires only Docker. For Azure Event Hubs deployments, use Quick Start with Azure Event Hubs.

Start dependencies

This guide assumes that you have Zookeeper and Kafka running locally on default ports (2181 and 9092 respectively). You can use Confluent CLI to start them all together:

bin/confluent local start kafka

or individually:

bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
bin/kafka-server-start ./etc/kafka/server.properties

See Confluent Platform quickstart for more details.

Create Kafka topics

You can create the necessary Kafka topics using Confluent CLI:

kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_sessions --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_connections --partitions 5 --replication-factor 1 \
  --config cleanup.policy=delete --config retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_messages --partitions 5 --replication-factor 1 \
  --config retention.ms=86400000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \
  --config retention.ms=300000

If you don’t have Confluent CLI on your machine you can use Docker image to do the same thing:

docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_sessions \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_retained_messages \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_connections \
  --partitions 5 --replication-factor 1 --config cleanup.policy=delete \
  --config retention.ms=600000
docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_messages \
  --partitions 5 --replication-factor 1 --config retention.ms=86400000
docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic __waterstream_heartbeat \
  --partitions 5 --replication-factor 1 --config retention.ms=300000

Start Waterstream

Assuming that Waterstream license file is in current directory and named waterstream.license run following command to start Waterstream:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e KAFKA_SESSIONS_TOPIC=mqtt_sessions \
  -e KAFKA_RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e KAFKA_CONNECTIONS_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka waterstreamio/waterstream-kafka:1.6.0

Logs will be displayed in the console. If you want to run in the background instead - replace -it with -d. If you want to keep the container after termination - remove --rm flag.

More configuration parameters documentation available here

Note

Starting from Waterstream 1.6.0 the waterstreamio/waterstream-kafka image is a multi-arch manifest list that works on both amd64 and arm64. Docker automatically selects the right binary for the host platform, so no architecture-specific image name is needed. If you were previously pulling waterstreamio/waterstream-kafka-arm64v8 on ARM64 hosts, switch to waterstreamio/waterstream-kafka instead.