List of Contents
마크베이스 collector가 지원하는 다양한 데이터 수집 모드와 이를 적용하는 방법을 기술한다. Collector가 지원하는 데이터 수집 모드는 FILE, SFTP, SOCKET, ODBC이다.
아래 설명은 collector와 마크베이스 서버가 동일한 장비에 설치되어 있고, 서버의 hostname이 "localhost"이며, colletor는 127.0.0.1:9999에서 동작하는 것을 가정한다.
데이터 입력 모드는 COLLECT_TYPE변수를 조정하면 바꿀 수 있다. 현재는 FILE과 SFTP, SOCKET, ODBC등을 설정할 수 있다. 각 모드에 대해서 추가 변수를 설정해아 한다.
파일 모드 입력을 설정하면 collector가 실행중인 서버의 파일을 읽어서 처리한다.
파일 모드에서는 데이터를 localhost에서 읽기 때문에, 그 파일의 경로명과 파일 읽기 권한만 체크하면 된다.
Name of value | Description | Others |
---|---|---|
LOG_SOURCE | 로그 파일이 위치한 경로 | 디렉토리명을 항상 full name으로 실행해야 한다. |
Example
아래의 예제는 파일 모드 입력 방법으로 "/var/log/syslog" 파일의 데이터를 수집하여 마크베이스 서버에 입력하도록 하는 것이다. 먼저 입력 파일을 collector 프로세스가 읽을 수 있는지 확인해야 한다.
[mach@localhost ~]$ head /var/log/syslog head: cannot open '/var/log/syslog' for reading: Permission denied |
위 결과를 보면 입력 파일에 대해서 읽기 권한이 없는 것을 알 수 있다. 입력 파일(/var/log/syslog)에 읽기 권한을 collector를 실행하는 사용자에게 부여하여 collector프로세스가 입력 파일을 읽을 수 있도록 해야 한다. 아래의 예제는 그 과정을 보여준다.
[mach@localhost ~]$ su # chmod 744 /var/log/syslog # exit [mach@localhost ~]$ |
head 명령을 이용하여 입력 파일을 읽을 수 있는지 검사하면 다음의 결과를 얻는다.
[mach@localhost ~]$ head /var/log/syslog Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [mach@localhost ~]$ |
입력 파일에 대한 권한을 확인하고 나서, 데이터를 파싱하기 위한 정규표현식 파일을 작성해야 한다. 작성한 정규표현식이 데이터 포멧과 맞는지를 확인하기 위해서 마크베이스가 제공하는 machregex툴을 이용할 수 있다.
machregex툴은 두개의 정규표현식을 매개변수로 설정하고 데이터를 입력한다. 첫번째 정규표현식(REGEX)은 입력데이터를 파싱하기 위해서 사용되고, 두번째 정규표현식(END_REGEX)는 입력 데이터를 한건씩 분리하기 위해서 사용된다. 아래의 예제는 machregex를 정규표현식과 데이터 파일을 이용하여 실행해 본 것이다.
[mach@localhost ~]$ head /var/log/syslog > syslog [mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$) ============================================================================ SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ) ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ] 0 (0:15) => [Jun 20 04:31:43] 1 (0:3) => [Jun] 2 (4:6) => [20] 3 (7:15) => [04:31:43] 4 (16:25) => [localhost] 5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started. ] |
파싱 결과로 출력되는 값은 6개의 토큰으로 표시된다. 마크베이스는 0, 4, 5의 토큰을 이용하고 나머지 토큰들은 버린다.
Tokens:
0: 시간 표현 문자열 이 문자열을 datetime으로 변환하려면 format을 지정해야 한다.
4: 호스트 명칭
5: 로그 데이터 메시지.
분석하려는 데이터가 위의 파싱 룰로 문제없이 처리된다면, 마크베이스가 제공하는 syslog.rgx파일을 이용하면 된다. 이 파일은 $MACHBASE_HOME/collector/regex/ 폴더에 있다. 이 폴더에 있는 정규표현식 파일을 이용할 경우, 탬플릿 파일의 REGEX_PATH 변수에 파일경로를 쓰지 않고 파일명 만을 설정해도 된다.
아래에 syslog.tpl 탬플릿 파일의 예를 기술한다.
############################################################################### # Copyright of this product 2013-2023, # Machbase Inc. or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=FILE LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "file_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append. # 1: Truncate. # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create. CREATE_TABLE_MODE=2 |
콜렉터 설정에서 기본 파일 경로가 아닌 파일들은 반드시 절대 경로('/'로 시작하는 경로)와 파일명을 명시해야 한다. 읽어들일 파일명은 LOG_SOURCE변수에 지정하고, 그 데이터를 파싱하기 위한 정규표현식 파일도 설정해야 한다. 마크베이스 서버에 접속하기 위한 정보와 테이블 생성 방식을 설정하여 탬플릿 파일 설정이 끝나면 다음과 같이 collector를 실행한다.
[mach@localhost ~]$ machsql ================================================================= Machbase Client Query Utility Release Version 3.5.0.b0fb4e1.official Copyright 2015, Machbase Inc. or its subsidiaries. All Rights Reserved. ================================================================= Machbase server address (Default:127.0.0.1): Machbase user ID (Default:SYS) Machbase user password: MACHBASE_CONNECT_MODE=INET, PORT=5656 Mach> create collector localhost.file_syslog from "/home/machbase_home/collector/syslog.tpl"; Created successfully. Elapsed Time : 0.037 Mach> Mach> ALTER COLLECTOR localhost.file_syslog START; Altered successfully. |
collector의 생성 및 실행이 성공하였다면, 필요시 테이블이 생성되고 데이터가 입력되기 시작한다. 데이터가 정상적으로 입력된 것인지를 확인하려면 생성된 테이블에 질의를 실행하여 확인할 수 있다.
Mach> SELECT * FROM file_syslogtable ORDER BY _arrival_time asc LIMIT 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ FILE 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start FILE 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. FILE 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) FILE 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) FILE 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected. |
원격 파일에서 데이터를 수집하기 위해서 SFTP모드를 사용할 수 있다. 원격 파일을 SFTP를 통해서 집근할 수 있어야 한다. FILE 모드와 유사하나 파일을 SFTP를 통하여 접근하므로 SFTP관련 변수를 설정하여야 한다.
SFTP모드로 데이터를 수집하려면 다음의 변수들을 추가 설정하여야 한다.
Name of variable | Description | Other Information |
---|---|---|
LOG_SOURCE | 원격 위치의 데이터 파일 경로및 파일명 | 절대경로명이 필요함 |
FTP_HOST | SFTP 서버의 IP 주소 | |
SFTP_PORT | SFTP서버의 포트 번호 | 포트번호를 설정하지 않으면 22번이 기본으로 사용됨 |
SFTP_USER | SFTP 사용자명 | |
SFTP_PASS | SFTP 패스트워드 |
Example
It describes how to set and execute SFTP method. For the sake of explanation, it sets the address of a remote host to 127.0.0.1 (local host). Then, it collects syslog data from "/var/log/syslog" and store them in the Machbase. Now, you need to check whether you have proper authentication to access the file by printing "/var/log/syslog" file with "head" command.
[mach@localhost ~]$ head /var/log/syslog head: cannot open '/var/log/syslog' for reading: Permission denied |
Permission denied as shown above. Thus, you need to get the authentication of the file first. For /var/log/syslog, the root user has the authentication about syslog file so that the user needs to log in with the root user account and grant read-only permission to other users. Proceed the following and necessary works after login with mach account.
In this example, with the super-user authentication, it grants read-only permission to other users.
[mach@localhost ~]$ su # chmod 744 /var/log/syslog # exit [mach@localhost ~]$ Then, log out from the super-user account and log in with the user's own account again to display "/var/log/syslog" file using HEAD command. |
Then, log out from the super-user account and log in with the user's own account again to display "/var/log/syslog" file using HEAD command.
[mach@localhost ~]$ head /var/log/syslog Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:31:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="http://www.rsyslog.com"] start Jun 20 04:31:46 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. Jun 20 04:35:01 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:40:01 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:45:01 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) Jun 20 04:50:01 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) Jun 20 04:55:01 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [mach@localhost ~]$ |
Once it is accessed to log files, it needs to create regular expression file to analyze the log files. In order to create regular expression file, it needs END_REGEX and REGEX. Machbase provides a tool to verify regular expression, machregex. For machregex, two arguments and a file are used. The first argument is REGEX to parse data records. The second argument is END_REGEX that can distinguish a record from multiple lines of log data. Type in < and a path of a file, then it will display results by using REGEX and END_REGEX after parsing. If it outputs too many lines, it would be difficult to check. Thus, extract only 10 lines of syslog from the top and evaluate them. Note that the same example which is described above is used again.
[mach@localhost ~]$ head /var/log/syslog > syslog [mach@localhost ~]$ machregex "^(([a-zA-Z]+)\\s+([0-9]+)\\s+([0-9:]*))\\s(\\S*)\\s+((?:[^\\0])*)$" ".*" < syslog Pattern => (^(([a-zA-Z]+)\s+([0-9]+)\s+([0-9:]*))\s(\S*)\s+((?:[^\0])*)$) ============================================================================ SUCCESS[2] (rc=7)(Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ) ALL (0:82) => [Jun 20 04:31:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. ] 0 (0:15) => [Jun 20 04:31:43] 1 (0:3) => [Jun] 2 (4:6) => [20] 3 (7:15) => [04:31:43] 4 (16:25) => [localhost] 5 (26:82) => [kernel: imklog 5.8.10, log source = /proc/kmsg started. ] |
It displays a log data in five tokens. In this case, Machbase basically uses token number 0, 4 and 5 and the rest of the tokens will be discarded.
Tokens:
0: When timestamp values of syslog are inserted, it needs to specify datetime format for analysis.
4: It is a name of a host.
5: It is a message of log data.
If syslog data are properly processed with the parsing rule above, the user can use syslog.rgx file without having to create own rgx file. Machbase provides syslog.rgx file and the rgx files can be found in the "$MACHBASE_HOME/collector/regex" folder. The user can use these files by only entering file names in REGEX_PATH of a template file without specifying paths.
It sets a template file in order to operate the collector after completing the configuration of regular expression files. Simply change the setting of $MACHBASE_HOME/collector/syslog.tpl for setting up syslog.
############################################################################### # Copyright of this product 2013-2023, # Machbase Corporation (Incorporation) or its subsidiaries. # All Rights reserved ############################################################################### # # This file is for Machbase collector template file. # ################################################################### # Collect setting ################################################################### COLLECT_TYPE=SFTP SFTP_HOST=127.0.0.1 SFTP_PORT=22 SFTP_USER=mach SFTP_PASS=mach LOG_SOURCE=/var/log/syslog ################################################################### # Process setting ################################################################### REGEX_PATH=syslog.rgx ################################################################### # Output setting ################################################################### DB_TABLE_NAME = "sftp_syslogtable" DB_ADDR = "127.0.0.1" DB_PORT = 5656 DB_USER = "SYS" DB_PASS = "MANAGER" # 0: Direct insert # 1: Prepared insert # 2: Append APPEND_MODE=2 # 0: None, just append # 1: Truncate # 2: Try to create table. If table already exists, warn it and proceed. # 3: Drop and create CREATE_TABLE_MODE=2 |
When setting up the collectors, it is required to specify all the files to the absolute path except the ones located in the default path. If syslog files are located in other paths, the user must change the path of LOG_SOURCE. When accessing a remote host, it needs the related information such as the port address (SFTP_HOST), port number (SFTP_PORT), USER ID (SFTP_USER) and password (SFTP_PASS). When SFTP_HOST was not inserted, it doesn't work properly. However, the port number 22 will be used if SFTP_PORT is not specified. Without a user ID and password, it will log in anonymously. When a new file is created, REGEX_PATH must be modified as each regular expression has its own rules. Use the absolute path for rgx file, but for the files located in "$MACHBASE_HOME/collector/regex" folder, entering a name of file would do the work. It inputs Machbase related information. With the current setting, it collects data into sftp_syslogtable of Machbase which uses the port 5656 installed in the local host. Then, specify a path of tpl file and create a collector after completing tpl and rgx settings, and then run the collector.
[mach@localhost ~]$ machsql ================================================================= Machbase Client Query Utility Release Version 3.5.0.b0fb4e1.official Copyright 2015, Machbase Inc. or its subsidiaries. All Rights Reserved. ================================================================= Machbase server address (Default:127.0.0.1): Machbase user ID (Default:SYS) Machbase user password: MACH_CONNECT_MODE=INET, PORT=5656 mach> create collector localhost.sftp_syslog from "/home/mach/mach_collector_home/collector/sftp_syslog.tpl"; Created successfully. Elapsed Time : 0.034 mach> alter collector localhost.sftp_syslog start; Altered successfully. |
After START command is successfully operated, the collector gathers data, and a table will be created automatically based on options if there is none. In order to check whether the data are properly inserted by using HEAD command, output only 10 lines from the oldest data.
mach> select * from sftp_syslogtable order by _arrival_time asc limit 10; COLLECTOR_TYPE COLLECTOR_ADDR ----------------------------------------------------- COLLECTOR_ORIGIN COLLECTOR_OFFSET ---------------------------------------------------------------------------------------------------------- TM HOST --------------------------------------------------------------------------------------------------------------------- MSG ------------------------------------------------------------------------------------ SFTP 127.0.0.1 /var/log/syslog 81 2016-06-20 04:31:43 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 217 2016-06-20 04:31:43 000:000:000 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="15062" x-info="h ttp://www.rsyslog.com"] start SFTP 127.0.0.1 /var/log/syslog 256 2016-06-20 04:31:46 000:000:000 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started. SFTP 127.0.0.1 /var/log/syslog 431 2016-06-20 04:35:01 000:000:000 localhost CROND[15111]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 606 2016-06-20 04:40:01 000:000:000 localhost CROND[15188]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 681 2016-06-20 04:40:01 000:000:000 localhost CROND[15187]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 856 2016-06-20 04:45:01 000:000:000 localhost CROND[15265]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1031 2016-06-20 04:50:01 000:000:000 localhost CROND[15341]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) SFTP 127.0.0.1 /var/log/syslog 1106 2016-06-20 04:50:01 000:000:000 localhost CROND[15342]: (root) CMD (/usr/lib64/sa/sa1 1 1) SFTP 127.0.0.1 /var/log/syslog 1281 2016-06-20 04:55:01 000:000:000 localhost CROND[15419]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --loc k-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok) [10] row(s) selected. |
The example shows that data are safely collected.