Waterstream Quick Start with the Confluent Platform¶

To set up Waterstream in an on-premises environment with the Confluent Platform and Docker, follow the steps below.

Prerequisites¶

Confluent Platform
Docker
License file for the Waterstream - contact us if you don’t have one yet. You can get a license for development or small-scale personal use for free.
(Optional) Confluent CLI

Start dependencies¶

This guide includes the steps taken from Quick Start for Confluent Platform.

Install the Confluent Platform by downloading the Docker Compose file

wget https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.4.0-post/cp-all-in-one-kraft/docker-compose.yml

Start the Confluent Platform in detached mode:

docker-compose up -d

See Quick Start for Confluent Platform for more details. For manual setup other than Docker, check Install and Upgrade Confluent Platform.

Create Kafka topics¶

You can create the necessary Kafka topics using Confluent Control Center. Please note that the proposed settings are not suitable for production environments, and multiple Kafka topics can be used to store MQTT messages. Some settings can be customized in the Expert Configuration’s Customize Settings section. Refer to the image below for guidance:

Using the Confluent Control Center, create the following list of topics

Topic Details¶
Topic Name	Partitions	Replication Factor	Cleanup Policy	Min Compaction Lag (ms)	Delete Retention (ms)
mqtt_sessions	5	1	compact	60000	600000
mqtt_retained_messages	5	1	compact	60000	600000
mqtt_connections	5	1	delete	(keep default)	600000
mqtt_messages	5	1	(keep default)	(keep default)	86400000
__waterstream_heartbeat	5	1	(use default)	(use default)	300000

After creating all the topics, the topic list should resemble the example shown below:

Alternatively, you can use the following script to create the required topic.

docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_sessions \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_retained_messages \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_connections \
  --partitions 5 --replication-factor 1 --config cleanup.policy=delete \
  --config retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_messages \
  --partitions 5 --replication-factor 1 --config retention.ms=86400000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic __waterstream_heartbeat \
  --partitions 5 --replication-factor 1 --config retention.ms=300000

If Confluent CLI tools are installed, you can use the following commands as an alternative to using commands with Docker.

kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_sessions --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092--create \
  --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_connections --partitions 5 --replication-factor 1 \
  --config cleanup.policy=delete --config retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_messages --partitions 5 --replication-factor 1 \
  --config retention.ms=86400000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \
  --config retention.ms=300000

Start Waterstream¶

To start Waterstream, assuming that the Waterstream license file is in the current directory and named waterstream.license, execute the following command:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka:1.4.29-SNAPSHOT

The console will display the logs. If you prefer to run Waterstream in the background, replace -it with -d. If you want to keep the container after termination, remove the --rm flag.

For more information on configuration parameters, refer to the available documentation

Start Waterstream on ARM64 platform¶

All steps remain the same except for the Docker image; - use simplematter/waterstream-kafka-arm64v8. The command to start Waterstream on an ARM64 platform will look as follows:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka-arm64v8:1.4.29-SNAPSHOT