Routing

Overview

The ingestion chain can be configured to allow telemetries to be sent to zero, one, or multiple routes based on their keys by using the routing feature. Routing rules are associated with a device type so they will be applied to all devices that have this device type.

Each key of a telemetry payload can be independently sent to a specific route.

Example

Let's take an example. The EB Rent company rents electric bikes, and uses Kamea to monitor the bikes and track their position. They have configured Kamea with the following routes: - influxdb: used to send telemetries to InfluxDB, a time-series database. This will allow them to make statistics about the usage of the bikes. - geofencing: used to send GPS telemetries to an internally developed micro-services that raises an alarm if the bike are going too far from the city they are attached to. - real_time_data: used to send data to a websocket, to make them available in real time to the users. In this example, EB Rent wants to track the bikes in real-time on a map.

EB Rent have created an Electric bike device type, and configured its routing rules like this:

EB Rent also has configured the fallback to be the route influxdb, and configured the route real_time_data to not trigger the fallback.

The device periodically sends JSON data, always with the same structure. The device type is configured to use the built-in JSON codec. Here is an example of one of those payloads:

{
    "battery_percentage": 76,  
    "current_speed_kmh": 12,
    "trip": {
        "total_distance_km": 1,
        "total_remaining_km": 3,
    },    
    "position": {
        "latitude": "45°46'03.9\"N",
        "longitude":"4°48'48.6\"E"
    }
}

Routing rules

There are several things to note from the previous example: - Some routing keys target nested properties. For example position/latitude is nested. - # has been used to target multiple keys. This is a wildcard. - The key trip/total_remaining_km is sent by the bike, but is not present in the routing rules. It does not necessarily mean that it will be lost. It can still be used if a fallback has been configured.

Nested properties

As shown in the previous example, values can be nested. It would also be possible with another format and codec, for example with Protobuf. Nesting is not exclusive to JSON. But no matter the format, the concept stays the same. The slash character (/) is used to indicate the nesting in the routing rules.

{
    "a": {
        "b/c": 10
    }
}

Routing rules must target keys that have a raw value, and not an object. In the EB Rent example, those rules are valid, and will provide the expected output:

• position/latitude
• position/longitude

But the rule position will not automatically forward all the nested values. In order to do that, it is required to use a multi-level wildcard: position/#.

Wildcards

Wildcards are special characters that can be used to target multiple keys. It can be useful to catch nested values for example, or when the payloads have undetermined keys.

Two wildcards are available: - #: multi-level wildcard - +: single-level wildcard

Multi-level wildcard

The # wildcard can be replaced by any sequence of keys. # is a valid rule by itself: it means that all the keys of the payload will be routed to the associated route.

In many cases, it will avoid having to write a lot of rules to target all the payload keys. For example, if a device sends this payload:

{
    "temperature": {
        "sensor1": 15,
        "sensor2": 8,
        "sensor3": 19,
        "sensor...": "many other sensor values",
    }
}

It will be much more convenient to use temperature/# rather than temperature/sensor1, temperature/sensor2, etc... It would also be convenient if we did not know in advance how many sensors are present.

# will catch nested properties, no matter the depth. For example, a/# will cath a/b/c/d along with a/e.

Single-level wildcard

The + wildcard replaces one and only one segment in the routing rule. Contrary to the multi-level wildcard, it does not need to be at the end of the rule.

For example, a/#/c can catch a/b1/c, a/b2/c, etc..

It is also a valid rule by itself when used alone, and will catch all keys at the root level of the payload, as long as they are raw values and not nested objects. Consequently, in the EB Rent example, using + as a routing rule will catch those values: - battery_percentage - current_speed_kmh

Fallback

Configuring routing rules presents a risk: it is possible that some keys sent from the devices remain unmatched after having applied all the routing rules. To avoid those keys to be definitely lost, the routing can be configured with an additional parameter: the fallback route.

When a fallback route has been configured for a device type, all unmatched data will be sent to the fallback. It is possible to configure some routes to prevent them from being considered during the fallback evaluation, see the fallback trigger section.

In the EB Rent example, the key trip/total_remaining_km is not matched by any route. But the company has configured the device type to fallback to the route influxdb. Consequently, the data trip/total_remaining_km will not be lost, and will be saved to InfluxDB.

The fallback is optional, it does not have to be set. In this case, if it was not set, the data trip/total_remaining_km would have been lost, without any way to ever retrieve it.

Routes

From a technical perspective, a route is a data output on the message bus used internally by Kamea. Each route has a dedicated subscriber, which will typically store the telemetry in a database. Other usages are possible. For instance, Kamea uses a route to send data to the websocket that handles the real-time telemetries.

Routes are relatively simple objects, they contain: - A name, used only for users to identify them. It does not have any technical impact. - A route type used to determine which subscriber will handle the data. - The fallback trigger setting. - A configuration in JSON format used by the subscriber.

Route type

The route type is only used to determine which subscriber must handle the data. It is an arbitrary value with no technical impact. By convention, using the name of the final destination of the data makes it more explicit. For example, out of the box, when using InfluxDB, Kamea will be configured with the route type influxdb.

Fallback trigger

The main goal of the fallback is to avoid data loss. In almost all cases, we want all telemetries to be saved in at least one database. However, some routes, like the real-time data one, will not store the data anywhere. Consequently, considering them when evaluating whether or not a data should be sent to the fallback could be an issue. We might want to consider that data should be sent to the fallback if it is not matched by any route leading to data persistence.

In order to do that, routes can be configured with a boolean setting: triggersFallback. When set to false, a route will not be taken into account for the fallback evaluation. It is recommended that it is configured to be true for all routes except the real-time data one.

Configuration

The route mechanism makes Kamea extensible by design, because it is possible to add a new route type, create a route associated to it, and add a custom subscriber to the route. For example, a customer could send data to their home-made data-lake by creating a .NET agent that would listen to the Kamea message bus on the topic associated to their dedicated route type. But this customer might also want to dynamically configure their subscriber. For example, they might want to update the data time-to-live depending on the key. Kamea offers a way to do it: the route configuration.

It is possible to configure a route with an arbitrary JSON object. The configuration will be stored by the ingestion chain, and will be made available to the new subscriber. Since Kamea cannot know in advance the expected format, it does not force any structure for the configuration. It is up to the user to submit a configuration that is understandable by their subscriber.

Two routes with the same route type can have different configurations. They will be handled by the same subscriber, but the subscriber will be provided a different configuration for each case.