Orchestration and Services
In the world of IoT devices being deployed everywhere, it sometimes become difficult to keep track of your fleet logs and status. Fluent-Bit is a lightweight and high performance telemetry agent for logs, device metric or traces.
It allows:
• Definition of Input, which correspond to the data you want to collect
• Usage of Parsers to transform any unstructured log entry and give them a structure that makes easier it processing and further filtering
• Filters pass to only select a specific subset of your logged data to be forwarded
• Flexible Output definition which allows to easily route the processed data to any location
Here is a simple example to show how to collect RAM usage of a specific application called my-amazing-app:
[INPUT]
Name proc
Proc_Name my-amazing-app
Interval_Sec 1
Interval_NSec 0
Fd true
Mem true
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name http
Match *
Host 192.168.2.3
Port 80
URI /something
This will monitor the memory usage of the application my-amazing-app every second. The two OUTPUT are defined to prints the logs on stdout and send them on a specific URI at the IP 192.168.2.3.
Fluent-Bit defines a lot of input type to gather system metrics. It is also possible to use custom commands to logs more specific data. For example, let's say we want to get the disk usage of my main disk. The df command gives this output:
$ df -k
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 3246320 3500 3242820 1% /run
efivarfs 438 309 125 72% /sys/firmware/efi/efivars
/dev/nvme0n1p3 975262680 806211136 119437160 88% /
tmpfs 16231600 414416 15817184 3% /dev/shm
tmpfs 5120 4 5116 1% /run/lock
tmpfs 16231600 0 16231600 0% /run/qemu
/dev/nvme0n1p1 243852 6272 237580 3% /boot/efi
tmpfs 3246320 5936 3240384 1% /run/user/1001
Knowing that the disk is the /dev/nvme0n1p3, we can then use a combination of grep and jq to gather the information and output them in a JSON format:
$ df -k | grep nvme0n1p3 | jq -R -c -s 'gsub(" +"; " ") | split(" ") | { "disk_total": .[1], "disk_used": .[2], "disk_avail": .[3]}'
{"disk_total":"975262680","disk_used":"806195688","disk_avail":"119452608"}
Next we must fill the Fluent-Bit configuration file with the new Input, using the Exec plugin:
[INPUT]
Name exec
Tag disksize
Command df -k | grep nvme0n1p3 | jq -R -c -s 'gsub(" +"; " ") | split(" ") | { "disk_total": .[1], "disk_used": .[2], "disk_avail": .[3]}'
Parser json
Interval_Sec 3600
[FILTER]
Name nest
Match disksize
Operation nest
Wildcard *
Nest_under custom
[OUTPUT]
Name http
Match *
Host 192.168.2.3
Port 80
URI /something
This will gather the disk size using our custom command every hour. The Nest FILTER allow to nest all the data gathered from the JSON under a custom key:
[0] disksize: [[1737017868.008708965, {}], {"custom"=>{"disk_total"=>"975262680", "disk_used"=>"806364640", "disk_avail"=>"119283656"}}]
Useful links:
• Fluent-Bit documentation: https://docs.fluentbit.io/manual
• Torizon OS example: https://developer.toradex.com/torizon/torizon-platform/device-monitoring-in-torizoncore/