Orchestration and Services

In the world of IoT devices being deployed everywhere, it sometimes become difficult to keep track of your fleet logs and status. Fluent-Bit is a lightweight and high performance telemetry agent for logs, device metric or traces.

It allows:

• Definition of Input, which correspond to the data you want to collect

• Usage of Parsers to transform any unstructured log entry and give them a structure that makes easier it processing and further filtering

• Filters pass to only select a specific subset of your logged data to be forwarded

• Flexible Output definition which allows to easily route the processed data to any location

Here is a simple example to show how to collect RAM usage of a specific application called my-amazing-app:

[INPUT]
    Name          proc
    Proc_Name     my-amazing-app
    Interval_Sec  1
    Interval_NSec 0
    Fd            true
    Mem           true

[OUTPUT]
    Name   stdout
    Match  *

[OUTPUT]
    Name  http
    Match *
    Host  192.168.2.3
    Port  80
    URI   /something

This will monitor the memory usage of the application my-amazing-app every second. The two OUTPUT are defined to prints the logs on stdout and send them on a specific URI at the IP 192.168.2.3.

Fluent-Bit defines a lot of input type to gather system metrics. It is also possible to use custom commands to logs more specific data. For example, let's say we want to get the disk usage of my main disk. The df command gives this output:

$ df -k
Filesystem     1K-blocks      Used Available Use% Mounted on
tmpfs            3246320      3500   3242820   1% /run
efivarfs             438       309       125  72% /sys/firmware/efi/efivars
/dev/nvme0n1p3 975262680 806211136 119437160  88% /
tmpfs           16231600    414416  15817184   3% /dev/shm
tmpfs               5120         4      5116   1% /run/lock
tmpfs           16231600         0  16231600   0% /run/qemu
/dev/nvme0n1p1    243852      6272    237580   3% /boot/efi
tmpfs            3246320      5936   3240384   1% /run/user/1001

Knowing that the disk is the /dev/nvme0n1p3, we can then use a combination of grep and jq to gather the information and output them in a JSON format:

$ df -k | grep nvme0n1p3 | jq -R -c -s 'gsub(" +"; " ") | split(" ") | { "disk_total": .[1], "disk_used":  .[2], "disk_avail":  .[3]}'
{"disk_total":"975262680","disk_used":"806195688","disk_avail":"119452608"}

Next we must fill the Fluent-Bit configuration file with the new Input, using the Exec plugin:

[INPUT]
    Name          exec
    Tag           disksize
    Command       df -k | grep nvme0n1p3 | jq -R -c -s 'gsub(" +"; " ") | split(" ") | { "disk_total": .[1], "disk_used":  .[2], "disk_avail":  .[3]}'
    Parser        json
    Interval_Sec  3600

[FILTER]
    Name       nest
    Match      disksize
    Operation  nest
    Wildcard   *
    Nest_under custom

[OUTPUT]
    Name  http
    Match *
    Host  192.168.2.3
    Port  80
    URI   /something

This will gather the disk size using our custom command every hour. The Nest FILTER allow to nest all the data gathered from the JSON under a custom key:

[0] disksize: [[1737017868.008708965, {}], {"custom"=>{"disk_total"=>"975262680", "disk_used"=>"806364640", "disk_avail"=>"119283656"}}]

Useful links:

• Fluent-Bit documentation: https://docs.fluentbit.io/manual

• Torizon OS example: https://developer.toradex.com/torizon/torizon-platform/device-monitoring-in-torizoncore/