Waterstream Features

Waterstream offers a few features beyond the standard MQTT specification requirements.

Some of them are experimental, which means that we’re still collecting feedback on them and future development may break compatibility. Give us your feedback on such features on the support forum to set their future: https://discuss.waterstream.io/

(Experimental) Start subscription from the specified timestamp

From Waterstream version 1.4.10. Requires MQTT v.5

Typically, when client subscribes to MQTT topic it doesn’t receive the messages that were published before (except of one retained message per topic). If you want to read the older messages, you may specify user property kafkaTopicStartTimestamp which contains comma-separated list of pairs: kafkaTopic: startTimestamp, where startTimestamp is a Unix timestamp in milliseconds (i.e. milliseconds from the start of the epoque).

For example, using Mosquitto client you could do something like this:

#Get timestamp in milliseconds
$ date +%s"000"
1668412470000
#Send some messages before subscribing
$ mosquitto_pub -h localhost -p 1883 -t foo -m "before sub1"
$ mosquitto_pub -h localhost -p 1883 -t foo -m "before sub2"
#Subscribe with the custom start timestamp
$ mosquitto_sub -h localhost -p 1883 -t "#" -V 5 -D SUBSCRIBE user-property kafkaTopicStartTimestamp 'mqtt_messages:1668412470000' -v
foo before sub1
foo before sub2

Environment variable REWIND_MAX_DEPTH_SECONDS limits the oldest point in time to which MQTT clients can rewind, in order to avoid excessive consumption by misbehaving clients. The default is 5184000 (60 days).

Limitations: - if this client is already reading from the specified Kafka topic (that is, the client has subscribed to the MQTT topic mapped to the Kafka topic) then such parameter has no effect - new MQTT topic subscriptions will start from now. Otherwise, it would be hard to avoid the duplicate messages delivery.