List of Contents
마크베이스 collector가 지원하는 다양한 데이터 수집 모드와 이를 적용하는 방법을 기술한다. Collector가 지원하는 데이터 수집 모드는 FILE, SFTP, SOCKET, ODBC이다.
아래 설명은 collector와 마크베이스 서버가 동일한 장비에 설치되어 있고, 서버의 hostname이 "localhost"이며, colletor는 127.0.0.1:9999에서 동작하는 것을 가정한다.
Setup Data Collection Method
데이터 입력 모드는 COLLECT_TYPE변수를 조정하면 바꿀 수 있다. 현재는 FILE과 SFTP, SOCKET, ODBC등을 설정할 수 있다. 각 모드에 대해서 추가 변수를 설정해아 한다.
FILE Method
파일 모드 입력을 설정하면 collector가 실행중인 서버의 파일을 읽어서 처리한다.
Setup Additional Values
파일 모드에서는 데이터를 localhost에서 읽기 때문에, 그 파일의 경로명과 파일 읽기 권한만 체크하면 된다.
| Name of value | Description | Others |
|---|---|---|
| LOG_SOURCE | 로그 파일이 위치한 경로 | 디렉토리명을 항상 full name으로 실행해야 한다. |
Example
아래의 예제는 파일 모드 입력 방법으로 "/var/log/syslog" 파일의 데이터를 수집하여 마크베이스 서버에 입력하도록 하는 것이다. 먼저 입력 파일을 collector 프로세스가 읽을 수 있는지 확인해야 한다.
[mach@localhost ~]$ head /var/log/syslog head: cannot open '/var/log/syslog' for reading: Permission denied
위 결과를 보면 입력 파일에 대해서 읽기 권한이 없는 것을 알 수 있다. 입력 파일(/var/log/syslog)에 읽기 권한을 collector를 실행하는 사용자에게 부여하여 collector프로세스가 입력 파일을 읽을 수 있도록 해야 한다. 아래의 예제는 그 과정을 보여준다.
[mach@localhost ~]$ su # chmod 744 /var/log/syslog # exit [mach@localhost ~]$
head 명령을 이용하여 입력 파일을 읽을 수 있는지 검사하면 다음의 결과를 얻는다.
[mach@localhost ~]$ head /var/log/syslog Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [mach@localhost ~]$
입력 파일에 대한 권한을 확인하고 나서, 데이터를 파싱하기 위한 정규표현식 파일을 작성해야 한다. 작성한 정규표현식이 데이터 포멧과 맞는지를 확인하기 위해서 마크베이스가 제공하는 machregex툴을 이용할 수 있다.
machregex툴은 두개의 정규표현식을 매개변수로 설정하고 데이터를 입력한다. 첫번째 정규표현식(REGEX)은 입력데이터를 파싱하기 위해서 사용되고, 두번째 정규표현식(END_REGEX)는 입력 데이터를 한건씩 분리하기 위해서 사용된다. 아래의 예제는 machregex를 정규표현식과 데이터 파일을 이용하여 실행해 본 것이다.
[mach@localhost ~]$ head /var/log/syslog > syslog [mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$) ============================================================================ SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ) ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ] 0 (0:15) => [Jun 20 04:31:43] 1 (0:3) => [Jun] 2 (4:6) => [20] 3 (7:15) => [04:31:43] 4 (16:25) => [localhost] 5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started. ]
It displays a log data in five tokens. In this case, Machbase basically uses token number 0, 4 and 5 and the rest of the tokens will be discarded.
Tokens:
0: When timestamp values of syslog are inserted, it needs to specify datetime format for analysis.
4: It is a name of a host.
5: It is a message of log data.
If syslog data are properly processed with the parsing rule above, the user can use syslog.rgx file without having to create own rgx file. Machbase basically provides syslog.rgx file and the rgx files can be found in the "$MACHBASE_HOME/collector/regex" folder. The user can use these files by only inputting file names in REGEX_PATH in a template file without specifying paths. It sets up a template file in order to execute the collector after completing the configuration of regular expression files. Simply change the setting of $MACHBASE_HOME/collector/syslog.tpl which is provided for syslog.
############################################################################### # Copyright of this product 2013-2023, # Machbase Inc. or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=FILE LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "file_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append. # 1: Truncate. # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create. CREATE_TABLE_MODE=2
When setting up the collectors, it is required to specify all the files to the absolute path except the ones located in the default path. If syslog files are located in other paths, the user needs to modify the path of LOG_SOURCE. When a new file is created, REGEX_PATH must be modified as each regular expression has its own rules. Use the absolute path for rgx files, but for the files located in "$MACHBASE_HOME/collector/regex" folder, only entering a name of file would do the work. It inputs Machbase related information into Output Setting. With the current setting, data are inserted in file_syslogtable of Machbase database which uses the port 5656 installed in the local host. Then, specify a path of tpl file and create a collector after completing tpl and rgx settings, and then run the collector as below.
[mach@localhost ~]$ machsql
=================================================================
Machbase Client Query Utility
Release Version 3.5.0.b0fb4e1.official
Copyright 2015, Machbase Inc. or its subsidiaries.
All Rights Reserved.
=================================================================
Machbase server address (Default:127.0.0.1):
Machbase user ID (Default:SYS)
Machbase user password:
MACHBASE_CONNECT_MODE=INET, PORT=5656
Mach> create collector localhost.file_syslog from "/home/machbase_home/collector/syslog.tpl";
Created successfully.
Elapsed Time : 0.037
Mach>
Mach> ALTER COLLECTOR localhost.file_syslog START;
Altered successfully.
When the collector is successfully executed, it collects data and a table will be created automatically if there is none. In order to check whether the data are properly inserted with the HEAD command, display 10 lines from the oldest data.
Mach> SELECT * FROM file_syslogtable ORDER BY _arrival_time asc LIMIT 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ FILE 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start FILE 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected.
The syslog above shows that the data are properly collected.
SFTP Method
It collects log data from a remote host. Therefore, it is required to have the information about the remote host. It is very similar to FILE method except the fact that data files are located in a remote host.
Setup Additional Values
It requires the information about the remote host and needs to enter a path of file located in the remote host.
| Name of variable | Description | Other Information |
|---|---|---|
| LOG_SOURCE | The path of a file in a remote host. | Full names of file and directory should be used. |
| FTP_HOST | Address of a remote host | |
| SFTP_PORT | Port number of a remote host | If port number is not specified, port 22 will be used as default. |
| SFTP_USER | Login ID to access a remote host | If ID is not specified, login with a anonymously. |
| SFTP_PASS | Port number of a remote host | If PWD is not specified, login with a anonymously. |
Example
It describes how to set and execute SFTP method. For the sake of explanation, it sets the address of a remote host to 127.0.0.1 (local host). Then, it collects syslog data from "/var/log/syslog" and store them in the Machbase. Now, you need to check whether you have proper authentication to access the file by printing "/var/log/syslog" file with "head" command.
[mach@localhost ~]$ head /var/log/syslog head: cannot open '/var/log/syslog' for reading: Permission denied
Permission denied as shown above. Thus, you need to get the authentication of the file first. For /var/log/syslog, the root user has the authentication about syslog file so that the user needs to log in with the root user account and grant read-only permission to other users. Proceed the following and necessary works after login with mach account.
In this example, with the super-user authentication, it grants read-only permission to other users.
[mach@localhost ~]$ su # chmod 744 /var/log/syslog # exit [mach@localhost ~]$ Then, log out from the super-user account and log in with the user's own account again to display "/var/log/syslog" file using HEAD command.
Then, log out from the super-user account and log in with the user's own account again to display "/var/log/syslog" file using HEAD command.
[mach@localhost ~]$ head /var/log/syslog Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [mach@localhost ~]$
Once it is accessed to log files, it needs to create regular expression file to analyze the log files. In order to create regular expression file, it needs END_REGEX and REGEX. Machbase provides a tool to verify regular expression, machregex. For machregex, two arguments and a file are used. The first argument is REGEX to parse data records. The second argument is END_REGEX that can distinguish a record from multiple lines of log data. Type in < and a path of a file, then it will display results by using REGEX and END_REGEX after parsing. If it outputs too many lines, it would be difficult to check. Thus, extract only 10 lines of syslog from the top and evaluate them. Note that the same example which is described above is used again.
[mach@localhost ~]$ head /var/log/syslog > syslog [mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$) ============================================================================ SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ) ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ] 0 (0:15) => [Jun 20 04:31:43] 1 (0:3) => [Jun] 2 (4:6) => [20] 3 (7:15) => [04:31:43] 4 (16:25) => [localhost] 5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started. ]
It displays a log data in five tokens. In this case, Machbase basically uses token number 0, 4 and 5 and the rest of the tokens will be discarded.
Tokens:
0: When timestamp values of syslog are inserted, it needs to specify datetime format for analysis.
4: It is a name of a host.
5: It is a message of log data.
If syslog data are properly processed with the parsing rule above, the user can use syslog.rgx file without having to create own rgx file. Machbase provides syslog.rgx file and the rgx files can be found in the "$MACHBASE_HOME/collector/regex" folder. The user can use these files by only entering file names in REGEX_PATH of a template file without specifying paths.
It sets a template file in order to operate the collector after completing the configuration of regular expression files. Simply change the setting of $MACHBASE_HOME/collector/syslog.tpl for setting up syslog.
############################################################################### # Copyright of this product 2013-2023, # Machbase Corporation (Incorporation) or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=SFTP SFTP_HOST=127.0.0.1 SFTP_PORT=22 SFTP_USER=mach SFTP_PASS=mach LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "sftp_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append # 1: Truncate # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create CREATE_TABLE_MODE=2
When setting up the collectors, it is required to specify all the files to the absolute path except the ones located in the default path. If syslog files are located in other paths, the user must change the path of LOG_SOURCE. When accessing a remote host, it needs the related information such as the port address (SFTP_HOST), port number (SFTP_PORT), USER ID (SFTP_USER) and password (SFTP_PASS). When SFTP_HOST was not inserted, it doesn't work properly. However, the port number 22 will be used if SFTP_PORT is not specified. Without a user ID and password, it will log in anonymously. When a new file is created, REGEX_PATH must be modified as each regular expression has its own rules. Use the absolute path for rgx file, but for the files located in "$MACHBASE_HOME/collector/regex" folder, entering a name of file would do the work. It inputs Machbase related information. With the current setting, it collects data into sftp_syslogtable of Machbase which uses the port 5656 installed in the local host. Then, specify a path of tpl file and create a collector after completing tpl and rgx settings, and then run the collector.
[mach@localhost ~]$ machsql
=================================================================
Machbase Client Query Utility
Release Version
3.5.0.b0fb4e1.official
Copyright 2015, Machbase Inc. or its subsidiaries.
All Rights Reserved.
=================================================================
Machbase server address (Default:127.0.0.1):
Machbase user ID (Default:SYS)
Machbase user password:
MACH_CONNECT_MODE=INET, PORT=5656
mach> create collector localhost.sftp_syslog from "/home/mach/mach_collector_home/collector/sftp_syslog.tpl";
Created successfully.
Elapsed Time : 0.034
mach> alter collector localhost.sftp_syslog start;
Altered successfully.
After START command is successfully operated, the collector gathers data, and a table will be created automatically based on options if there is none. In order to check whether the data are properly inserted by using HEAD command, output only 10 lines from the oldest data.
mach> select * from sftp_syslogtable order by _arrival_time asc limit 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ SFTP 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start SFTP 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected.
The example shows that data are safely collected.