The Raw Export plugin implments a new storage type for Torrus. It allows exporting the raw collected data in plain files. Currently the counter values are exported without changes, and in the future there will be support for exporting counter increments.
The data is exported in text form, one datasource per line.
Each line contains three or four fields, separated by the characters defined
by raw-field-separator parameter:
The timestamp of the collected data, in format defined
by raw-timestamp-format parameter.
A text string that identifies the datasource. It is defined by raw-rowid
parameter in the datasource leaf.
If raw-counter-base is not defined, the third and the last field contains
the raw value that was collected. It may be a very large integer
in case if the datasource is a 64-bit counter.
The value is set to U in the event of a counter reset, such as
when a monitored device is rebooted.
If raw-counter-base is set to 32 or 64, the third field contains the
counter increment, and the fourth field contains the time interval for it.
The time interval in seconds, if the datasource is exported as a counter.
The raw export files are written by a background thread, leaving the main
thread more time to perform the collecting work. The torrus schedulerinfio
reports the size of the raw update queue in runtime statistics.
Follow the Torrus installation guide for the main software installation. This plugin requires the multithreading support in Torrus. It would not function in a non-multithreaded environment.
Unpack the plugin distribution package in some directory:
gzip -dc tp-rawexport-1.X.tar.gz | tar xvf -
Then run the Torrus plugin installation utility:
torrus install_plugin tp-rawexport-1.X
Restart Torrus collector processes.
In order to get use of Raw export, the storage-type parameter should
contain raw as one of the comma-separated values. Usually
storage-type is set to the value raw,rrd.
The following parameters define the raw storage operation:
raw-datadir
Mandatory, the directory where the raw export files are stored.
raw-file
Mandatory, the export file name. It can contain any strftime(3) conversion
characters, but the percent sign (%) should be replaced with hash
sign (#). Also ^X^ defines a period of X seconds, and this string is
replaced with the number of seconds from today's start of the day in local time
zone, rounded to the specified period. For examnple, ^600^ defines a period
of 10 minutes, and this will cause a new file to be created every 10 minutes.
raw-field-separator
Mandatory, the field separator inside the export line.
raw-timestamp-format
Mandatory, timestamp format in strftime(3) notation, with the percent sign
(%) replaced with hash sign (#).
raw-rowid
Mandatory, a string that uniquely identifies the datasource.
raw-counter-base
Optional. Valid values: 32 or 64.
When defined, it declares the given datasource as a 32- or 64-bit counter.
The stored value is then recorded as a pair (increment, interval) with a
defined field separator.
raw-counter-maxrate
Optional. If it is defined, it should contain the maximum rate at which
the counter can increment, in units per second. It is used only when
raw-counter-base is defined. A counter may accidentally increase to huge
values because of a remote device reset, and this option prevents such
bursts from recording.
The following XML template is installed as examples/rawexport-example.xml
by the plugin installer. It provides basic functionality for exporting of
IF-MIB counters. The example refers to a parameter raw-file-suffix which
is not part of this plugin, and is given only as an example. The
discovery instruction (DDX) file listed below defines this parameter at
the host level.
Note that your templates should also declare the parameters as expandable, as given in the eample below.
It is important that raw-file depends on collector-instance. It
ensures that several collector instances for a given tree would never write
to the same file concurrently.
This example stores the data in files that depend on the calendar date. This makes them convenient to grab for daily processing. For example, at 3:00 every morning you can move all the files that are more than 2 hours old for further processing. Also their alphabetical sorting would list them in temporal order.
File: examples/rawexport-example.xml
<configuration>
<param-properties>
<prop param="raw-file" prop="expand" value="1"/>
<prop param="raw-rowid" prop="expand" value="1"/>
</param-properties>
<datasources>
<template name="raw-defaults-ifoctets-examle">
<param name="raw-datadir" value="/srv/torrus/raw" />
<param name="raw-file"
value="#Y-#m-#d.%collector-instance%.%raw-file-suffix%" />
<param name="raw-field-separator" value="," />
<param name="raw-timestamp-format" value="#d#b#Y:#H:#M:#S" />
<param name="raw-rowid" value="%system-id%,%interface-name%,%rrd-ds%" />
</template>
</datasources>
</configuration>
The following is a fragment of a Devdiscover instructions file (DDX) which sets up the raw export for input and output byte counters, for all interfaces whose names start with 'eth':
<host>
<param name="snmp-host" value="mylinuxhost"/>
<param name="include-files" value="examples/rawexport-example.xml"/>
<param name="custom-host-templates" value="raw-defaults-ifoctets-examle"/>
<param name="host-copy-params" value="raw-file-suffix"/>
<param name="raw-file-suffix" value="rawexport"/>
<param name="selectors" value="raw"/>
<param name="raw-selector-type" value="RFC2863_IF_MIB"/>
<param name="raw-selector-expr" value="{ifSubtreeName}"/>
<param name="raw-ifSubtreeName" value="^eth"/>
<param name="raw-selector-actions"
value="InBytesParameters,OutBytesParameters"/>
<param name="raw-InBytesParameters-arg">
storage-type=raw,rrd;
raw-counter-base=64;
raw-counter-maxrate=1e15
</param>
<param name="raw-OutBytesParameters-arg">
storage-type=raw,rrd;
raw-counter-base=64;
raw-counter-maxrate=1e15
</param>
</host>