Waterstream Quick Start with the Confluent Platform

To set up Waterstream in an on-premises environment with the Confluent Platform and Docker, follow the steps below.

Prerequisites

Start dependencies

This guide includes the steps taken from Quick Start for Confluent Platform.

  • Install the Confluent Platform by downloading the Docker Compose file

wget https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.4.0-post/cp-all-in-one-kraft/docker-compose.yml
  • Start the Confluent Platform in detached mode:

docker-compose up -d

See Quick Start for Confluent Platform for more details. For manual setup other than Docker, check Install and Upgrade Confluent Platform.

Create Kafka topics

You can create the necessary Kafka topics using Confluent Control Center. Please note that the proposed settings are not suitable for production environments, and multiple Kafka topics can be used to store MQTT messages. Some settings can be customized in the Expert Configuration’s Customize Settings section. Refer to the image below for guidance:

Editing the mqtt_sessions topic

Using the Confluent Control Center, create the following list of topics

Topic Details

Topic Name

Partitions

Replication Factor

Cleanup Policy

Min Compaction Lag (ms)

Delete Retention (ms)

mqtt_sessions

5

1

compact

60000

600000

mqtt_retained_messages

5

1

compact

60000

600000

mqtt_connections

5

1

delete

(keep default)

600000

mqtt_messages

5

1

(keep default)

(keep default)

86400000

__waterstream_heartbeat

5

1

(use default)

(use default)

300000

After creating all the topics, the topic list should resemble the example shown below:

Editing the mqtt_sessions topic

Alternatively, you can use the following script to create the required topic.

docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_sessions \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_retained_messages \
  --partitions 5 --replication-factor 1 --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_connections \
  --partitions 5 --replication-factor 1 --config cleanup.policy=delete \
  --config retention.ms=600000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_messages \
  --partitions 5 --replication-factor 1 --config retention.ms=86400000
docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic __waterstream_heartbeat \
  --partitions 5 --replication-factor 1 --config retention.ms=300000

If Confluent CLI tools are installed, you can use the following commands as an alternative to using commands with Docker.

kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_sessions --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092--create \
  --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \
  --config cleanup.policy=compact \
  --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_connections --partitions 5 --replication-factor 1 \
  --config cleanup.policy=delete --config retention.ms=600000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic mqtt_messages --partitions 5 --replication-factor 1 \
  --config retention.ms=86400000
kafka-topics --bootstrap-server localhost:9092 --create \
  --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \
  --config retention.ms=300000

Start Waterstream

To start Waterstream, assuming that the Waterstream license file is in the current directory and named waterstream.license, execute the following command:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka:1.4.24

The console will display the logs. If you prefer to run Waterstream in the background, replace -it with -d. If you want to keep the container after termination, remove the --rm flag.

For more information on configuration parameters, refer to the available documentation

Start Waterstream on ARM64 platform

All steps remain the same except for the Docker image; - use simplematter/waterstream-kafka-arm64v8. The command to start Waterstream on an ARM64 platform will look as follows:

docker run -it --rm \
  -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \
  -e MQTT_PORT=1883 \
  -e SESSION_TOPIC=mqtt_sessions \
  -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \
  -e CONNECTION_TOPIC=mqtt_connections \
  -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \
  -v `pwd`/waterstream.license:/etc/waterstream.license:ro \
  --network host \
  --name waterstream-kafka simplematter/waterstream-kafka-arm64v8:1.4.24