Data Collection Method (1) FILE/SFTP
- Daniel Kim (Unlicensed)
Analytics
FILE Method
When the file mode input is set, the collector reads and processes the file of the running server.
Additional Option Configuration
In file mode, data is read from localhost, so you only need to check the pathname and file read permissions of that file.
Option Name | Description | Remarks |
|---|---|---|
| LOG_SOURCE | Path where log file is located | Must be written in absolute path. |
Example (1) Checking Log File
The following example is the file mode input method that collects data from the "/var/log/syslog" file and inputs it to the Machbase server.
First, you need to make sure that the input file can be read by the collector process.
[mach@localhost ~]$ head /var/log/syslog head: cannot open '/var/log/syslog' for reading: Permission denied
The above results show that there is no read access to the input file.
You need to grant read permission to the input file (/var/log/syslog) to the user running collector so that the collector process can read the input file.
The following example shows the process.
[mach@localhost ~]$ su # chmod 744 /var/log/syslog # exit [mach@localhost ~]$
Again, if you check whether you can read the input file using the head command, you get the following result.
[mach@localhost ~]$ head /var/log/syslog Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [mach@localhost ~]$
Example (2) Creating Regular Expression File
After verifying the permissions on the input file, you need to create a regular expression file to parse the data.
You can use the machregex tool provided by Machbase to check that your regular expression matches the data format.
The machregex tool sets two regular expressions as parameters and enters data.
- The first regular expression (REGEX) is used to parse the input data.
- The second regular expression (END_REGEX) is used to separate input data one by one.
The following example shows how to run machregex using regular expressions and data files.
[mach@localhost ~]$ head /var/log/syslog > syslog [mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$) ============================================================================ SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ) ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ] 0 (0:15) => [Jun 20 04:31:43] 1 (0:3) => [Jun] 2 (4:6) => [20] 3 (7:15) => [04:31:43] 4 (16:25) => [localhost] 5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started. ]
The output value of the parsing result is represented by six tokens. According to this file, Machbase uses 0, 4, and 5 tokens and discards the remaining tokens.
- 0: Time representation string. To convert this string to datetime, you must specify format.
- 4: Host name.
- 5: Log data message.
If the data you want to analyze is processed without any problems according to the above parsing rules, you can use the same syslog.rgx file provided by Machbase.
(This file is in the $MACHBASE_HOME/collector/regex/ folder.)
If you use the regular expression file in this folder, you can set only the file name without using the file path in the REGEX_PATH variable of the template file .
Example (3) Creating Template File
Below is an example of a syslog.tpl template file.
############################################################################### # Copyright of this product 2013-2023, # Machbase Inc. or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=FILE LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "file_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append. # 1: Truncate. # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create. CREATE_TABLE_MODE=2
In collector settings, files that are not the default file path must specify an absolute path (a path beginning with '/') and a file name.
The file name to be read is specified in the LOG_SOURCE variable, and the regular expression file for parsing the data must also be set.
Example (4) Running Collector
Set the information for connecting to the Machbase server and the table generation method, and when the template file is set, execute the collector as follows.
Mach> create collector localhost.file_syslog from "/home/machbase_home/collector/syslog.tpl"; Created successfully. Mach> ALTER COLLECTOR localhost.file_syslog START; Altered successfully.
If the collector is successfully created and executed (and only if there is no table), the table will be created and data will begin to be input.
To check if the data is being input normally, you can check it by executing a SELECT query on the generated table.
Mach> SELECT * FROM file_syslogtable ORDER BY _arrival_time asc LIMIT 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ FILE 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start FILE 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected.
SFTP Method
You can use SFTP mode to collect data from remote files.
Remote files must be accessible via SFTP. It is similar to FILE mode, but SFTP related variables should be set because the file is accessed through SFTP.
Additional Option Configuration
To collect data in SFTP mode, the following variables must be set.
Option Name | Description | Remarks |
|---|---|---|
| LOG_SOURCE | Data file path and file name of remote location | Must be written in absolute path. |
| SFTP_HOST | SFTP server IP address | |
| SFTP_PORT | SFTP server port number | Is set to 22 by default if not port number is set. |
| SFTP_USER | SFTP username | |
| SFTP_PASS | SFTP password |
If you can not read the login and collection files with SFTP, you should try the following.
- Troubleshoot SFTP sign-in issues
- Troubleshoot file permissions (see FILE method)
Example (2-3) Creating Regular Expression/Template File
Once this is done, create a regular expression file and a template file.
For the description of this operation, refer to FTP method above.
############################################################################### # Copyright of this product 2013-2023, # Machbase Corporation (Incorporation) or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=SFTP SFTP_HOST=127.0.0.1 SFTP_PORT=22 SFTP_USER=mach SFTP_PASS=mach LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "sftp_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append # 1: Truncate # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create CREATE_TABLE_MODE=2
Example (4) Running Collector
The following example shows how to create a collector using SFTP with the above template file.
Mach> create collector localhost.sftp_syslog from "/home/mach/mach_collector_home/collector/sftp_syslog.tpl"; Created successfully. Mach> alter collector localhost.sftp_syslog start; Altered successfully.
If you have successfully created and started the Collector, you can see the data collected by the Collector as follows:
Mach> select * from sftp_syslogtable order by _arrival_time asc limit 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ SFTP 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start SFTP 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected.