Waterstream Quick Start

To try out Waterstream on your computer please follow the next steps

Prerequisites

If you don’t have Confluent Platform and Confluent CLI on your machine you can use docker-compose as an alternative - this will be described later on.

Start dependencies

This guide assumes that you have Zookeeper and Kafka running locally on default ports (2181 and 9092 respectively). You can use Confluent CLI to start them all together:

bin/confluent local start kafka

or individually:

bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
bin/kafka-server-start ./etc/kafka/server.properties

See Confluent Platform quickstart for more details.

Create Kafka topics

You can create the necessary Kafka topics using Confluent CLI:

kafka-topics --zookeeper localhost:2181 --create \
  --topic mqtt_sessions --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --zookeeper localhost:2181--create \
  --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --zookeeper localhost:2181 --create \
  --topic mqtt_connections --partitions 5 --replication-factor 1 \
  --config cleanup.policy=delete --config retention.ms=600000
kafka-topics --zookeeper localhost:2181 --create \
  --topic mqtt_messages --partitions 5 --replication-factor 1 \
  --config retention.ms=86400000
kafka-topics --zookeeper localhost:2181 --create \
  --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \
  --config retention.ms=300000

If you don’t have Confluent CLI on your machine you can use Docker image to do the same thing:

KAFKA_TOPICS_COMMAND="docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics"
$KAFKA_TOPICS_COMMAND --zookeeper localhost:2181--create --topic mqtt_sessions \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
$KAFKA_TOPICS_COMMAND --zookeeper localhost:2181--create --topic mqtt_retained_messages \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
$KAFKA_TOPICS_COMMAND --zookeeper localhost:2181 --create --topic mqtt_connections \
  --partitions 5 --replication-factor 1 --config cleanup.policy=delete \
  --config retention.ms=600000
$KAFKA_TOPICS_COMMAND --zookeeper localhost:2181 --create --topic mqtt_messages \
  --partitions 5 --replication-factor 1 --config retention.ms=86400000
$KAFKA_TOPICS_COMMAND --zookeeper localhost:2181 --create --topic __waterstream_heartbeat \
  --partitions 5 --replication-factor 1 --config retention.ms=300000

Start Waterstream

Assuming that Waterstream license file is in current directory and named waterstream.license run following command to start Waterstream:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka:1.4.29-SNAPSHOT

Logs will be displayed in the console. If you want to run in the background instead - replace -it with -d. If you want to keep the container after termination - remove --rm flag.

More configuration parameters documentation available here

Start Waterstream on ARM64 platform

Everything remains the same, except of Docker image - use simplematter/waterstream-kafka-arm64v8. The command to start Waterstream will look like this:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka-arm64v8:1.4.29-SNAPSHOT