Waterstream Quick Start¶

To try out Waterstream on your computer please follow the next steps

Prerequisites¶

Confluent Platform 4.0.1 or newer.
Confluent CLI
Docker
License file for the Waterstream - contact us if you don’t have one yet. You can get a license for development or small-scale personal use for free.

If you don’t have Confluent Platform and Confluent CLI on your machine you can use docker-compose as an alternative - this will be described later on.

Start dependencies¶

This guide assumes that you have Zookeeper and Kafka running locally on default ports (2181 and 9092 respectively). You can use Confluent CLI to start them all together:

bin/confluent local start kafka

or individually:

bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
bin/kafka-server-start ./etc/kafka/server.properties

See Confluent Platform quickstart for more details.

Create Kafka topics¶

You can create the necessary Kafka topics using Confluent CLI:

kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_sessions --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092--create \
  --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_connections --partitions 5 --replication-factor 1 \
  --config cleanup.policy=delete --config retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_messages --partitions 5 --replication-factor 1 \
  --config retention.ms=86400000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \
  --config retention.ms=300000

If you don’t have Confluent CLI on your machine you can use Docker image to do the same thing:

KAFKA_TOPICS_COMMAND="docker run -ti --network=host confluentinc/cp-kafka:4.0.1 /usr/bin/kafka-topics"
$KAFKA_TOPICS_COMMAND --bootstrap-server localhost:9092--create --topic mqtt_sessions \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
$KAFKA_TOPICS_COMMAND --bootstrap-server localhost:9092--create --topic mqtt_retained_messages \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
$KAFKA_TOPICS_COMMAND --bootstrap-server localhost:9092 --create --topic mqtt_connections \
  --partitions 5 --replication-factor 1 --config cleanup.policy=delete \
  --config retention.ms=600000
$KAFKA_TOPICS_COMMAND --bootstrap-server localhost:9092 --create --topic mqtt_messages \
  --partitions 5 --replication-factor 1 --config retention.ms=86400000
$KAFKA_TOPICS_COMMAND --bootstrap-server localhost:9092 --create --topic __waterstream_heartbeat \
  --partitions 5 --replication-factor 1 --config retention.ms=300000

Start Waterstream¶

Assuming that Waterstream license file is in current directory and named waterstream.license run following command to start Waterstream:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka:1.4.29-SNAPSHOT

Logs will be displayed in the console. If you want to run in the background instead - replace -it with -d. If you want to keep the container after termination - remove --rm flag.

More configuration parameters documentation available here

Start Waterstream on ARM64 platform¶

Everything remains the same, except of Docker image - use simplematter/waterstream-kafka-arm64v8. The command to start Waterstream will look like this:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka-arm64v8:1.4.29-SNAPSHOT