Waterstream Quick Start with the Confluent Platform =================================================== To set up Waterstream in an on-premises environment with the Confluent Platform and Docker, follow the steps below. Prerequisites ------------- - `Confluent Platform `_ - `Docker `_ - License file for the Waterstream - `contact us `_ if you don't have one yet. You can get a license for development or small-scale personal use for free. - (Optional) `Confluent CLI `_ Start dependencies ------------------ This guide includes the steps taken from Quick Start for Confluent Platform. - Install the Confluent Platform by downloading the Docker Compose file .. code-block:: bash wget https://raw.githubusercontent.com/confluentinc/cp-all-in-one/7.4.0-post/cp-all-in-one-kraft/docker-compose.yml - Start the Confluent Platform in detached mode: .. code-block:: bash docker-compose up -d See `Quick Start for Confluent Platform `_ for more details. For manual setup other than Docker, check `Install and Upgrade Confluent Platform `_. Create Kafka topics ------------------- You can create the necessary Kafka topics using Confluent Control Center. Please note that the proposed settings are not suitable for production environments, and multiple Kafka topics can be used to store MQTT messages. Some settings can be customized in the Expert Configuration's Customize Settings section. Refer to the image below for guidance: .. figure:: img/cc-topic-edit.png :alt: Editing the mqtt_sessions topic :scale: 50 :align: center Using the Confluent Control Center, create the following list of topics .. list-table:: Topic Details :widths: 25 15 20 20 25 25 :header-rows: 1 * - Topic Name - Partitions - Replication Factor - Cleanup Policy - Min Compaction Lag (ms) - Delete Retention (ms) * - mqtt_sessions - 5 - 1 - compact - 60000 - 600000 * - mqtt_retained_messages - 5 - 1 - compact - 60000 - 600000 * - mqtt_connections - 5 - 1 - delete - (keep default) - 600000 * - mqtt_messages - 5 - 1 - (keep default) - (keep default) - 86400000 * - __waterstream_heartbeat - 5 - 1 - (use default) - (use default) - 300000 After creating all the topics, the topic list should resemble the example shown below: .. figure:: img/cc-topic-list-after-creation.png :alt: Editing the mqtt_sessions topic :scale: 50 :align: center Alternatively, you can use the following script to create the required topic. .. code-block:: bash docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_sessions \ --partitions 5 --replication-factor 1 --config cleanup.policy=compact \ --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000 docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_retained_messages \ --partitions 5 --replication-factor 1 --config cleanup.policy=compact \ --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000 docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_connections \ --partitions 5 --replication-factor 1 --config cleanup.policy=delete \ --config retention.ms=600000 docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic mqtt_messages \ --partitions 5 --replication-factor 1 --config retention.ms=86400000 docker exec -ti broker /usr/bin/kafka-topics --bootstrap-server localhost:9092 --create --topic __waterstream_heartbeat \ --partitions 5 --replication-factor 1 --config retention.ms=300000 If Confluent CLI tools are installed, you can use the following commands as an alternative to using commands with Docker. .. code-block:: bash kafka-topics --bootstrap-server localhost:9092 --create \ --topic mqtt_sessions --partitions 5 --replication-factor 1 \ --config cleanup.policy=compact \ --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000 kafka-topics --bootstrap-server localhost:9092--create \ --topic mqtt_retained_messages --partitions 5 --replication-factor 1 \ --config cleanup.policy=compact \ --config min.compaction.lag.ms=60000 --config delete.retention.ms=600000 kafka-topics --bootstrap-server localhost:9092 --create \ --topic mqtt_connections --partitions 5 --replication-factor 1 \ --config cleanup.policy=delete --config retention.ms=600000 kafka-topics --bootstrap-server localhost:9092 --create \ --topic mqtt_messages --partitions 5 --replication-factor 1 \ --config retention.ms=86400000 kafka-topics --bootstrap-server localhost:9092 --create \ --topic __waterstream_heartbeat --partitions 5 --replication-factor 1 \ --config retention.ms=300000 Start Waterstream ----------------- To start Waterstream, assuming that the Waterstream license file is in the current directory and named waterstream.license, execute the following command: .. code-block:: bash :substitutions: docker run -it --rm \ -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \ -e MQTT_PORT=1883 \ -e SESSION_TOPIC=mqtt_sessions \ -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \ -e CONNECTION_TOPIC=mqtt_connections \ -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \ -v `pwd`/waterstream.license:/etc/waterstream.license:ro \ --network host \ --name waterstream-kafka simplematter/waterstream-kafka:|release| The console will display the logs. If you prefer to run Waterstream in the background, replace ``-it`` with ``-d``. If you want to keep the container after termination, remove the ``--rm`` flag. For more information on configuration parameters, refer to the :doc:`available documentation ` Start Waterstream on ARM64 platform ----------------------------------- All steps remain the same except for the Docker image; - use ``simplematter/waterstream-kafka-arm64v8``. The command to start Waterstream on an ARM64 platform will look as follows: .. code-block:: bash :substitutions: docker run -it --rm \ -e KAFKA_BOOTSTRAP_SERVERS=PLAINTEXT://localhost:9092 \ -e MQTT_PORT=1883 \ -e SESSION_TOPIC=mqtt_sessions \ -e RETAINED_MESSAGES_TOPIC=mqtt_retained_messages \ -e CONNECTION_TOPIC=mqtt_connections \ -e KAFKA_MESSAGES_DEFAULT_TOPIC=mqtt_messages \ -v `pwd`/waterstream.license:/etc/waterstream.license:ro \ --network host \ --name waterstream-kafka simplematter/waterstream-kafka-arm64v8:|release|