Linux Collector
The Zebrium Fluentd output plugin, ze-fluentd-plugin, sends the logs you collect with Fluentd on Linux to Zebrium for automated anomaly detection. You can access the plugin at the Zebrium GitHub repository at https://github.com/zebrium/ze-fluentd-plugin.
For instructions on deploying the Zebrium Fluentd collector for Docker environments, see the instructions in Docker Container Log Collectors.
System Requirements
The following Linux operating system distributions are supported:
- Ubuntu 16.04, 18.04, or 20.04
- CentOS or Red Hat Enterprise Linux 7 or 8
- Amazon Linux 2
Installing the Collector
-
If the environment uses a proxy server, see Operating with a Proxy Server, below.
-
Determine which deployment name to use for the <YOUR_SERVICE_GROUP> value, below.
-
If your account has multiple deployments, go to the Zebrium user interface, click the Deployment drop-down in the top-right navigation bar, and switch to the deployment you want to use to collect Windows logs.
-
In the Zebrium user interface, go to the Integrations & Collectors page (Settings () Integrations & Collectors).
-
Click the Log Collectors and copy the command from the Linux Log Collector dialog. This command includes the Zebrium API server URL (<ZAPI_URL>) and authentication token (<AUTH_TOKEN>) values.
button under -
Update the command from step 4 with the relevant values and run it in a shell on the host. The command uses the following format:
curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<YOUR_SERVICE_GROUP>" /bin/bash
The default system log file paths are defined by the ZE_LOG_PATHS environment variable. The default value is:
"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"
You can use the ZE_USER_LOG_PATHS environment variable to add more user-specific log file paths. For example, to add app log files at /app1/log/app1.log and /app2/log/\*.log, you can set ZE_USER_LOG_PATHS to:
"/app1/log/app1.log,/app2/log/*.log"
Upgrading the Collector
The upgrade command is similar to the installation command:
curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<deployment_name>" OVERWRITE_CONFIG=1 /bin/bash
Please note that setting OVERWRITE_CONFIG to 1 will cause /etc/td-agent/td-agent.conf to be upgraded to the latest version.
Uninstalling the Collector
To uninstall:
curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_OP=uninstall /bin/bash
Installing on Hosts with Existing td-agent Configuration
You can add the Zebrium output plugin on a host with existing td-agent configuration without running the Zebrium log collector installer.
-
Download the Zebrium output plugin from https://github.com/zebrium/ze-fluentd-plugin/releases/download/1.37.2/fluent-plugin-zebrium_output-1.37.2.gem.
-
Run the following command in the same directory where fluent-plugin-zebrium_output-1.37.2.gem is saved:
sudo td-agent-gem install fluent-plugin-zebrium_output
-
Add Zebrium output configuration to the /etc/td-agent/td-agent.conf file.
The following is an example configuration that duplicates log messages and sends one copy to Zebrium:
<match **> @type copy # Zebrium log collector <store> @type zebrium ze_log_collector_url "ZE_LOG_COLLECTOR_URL" ze_log_collector_token "ZE_LOG_COLLECTOR_TOKEN" ze_host_tags "ze_deployment_name=#{Socket.gethostname},myapp=test2" @log_level "info" <buffer tag> @type file path "/var/td-agent/zebrium" flush_mode "interval" flush_interval "60s" </buffer> </store> <store> @type OTHER_OUTPUT_PLUGIN ... </store> </match>
Configuration for td-agent
The configuration file for td-agent is at /etc/td-agent/td-agent.conf. The following parameters must be configured for your instance:
Parameter | Description | Note |
---|---|---|
ze_log_collector_url | Zebrium log host URL | Provided by Zebrium after your account has been created. |
ze_log_collector_token | Authentication token | Provided by Zebrium after your account has been created. |
path | Log files to read | Both files and file patterns are allowed. Files should be separated by comma. The default value is '"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"' |
ze_host_tags | Host meta data | This parameter is optional. You can pass meta data in key-value pairs, the format is: "key1=value1,key2=value2". We suggest at least set one tag for deployment name: "ze_deployment_name=<your_deployment_name>" |
ze_host_in_logpath | Log path component for remote host name | This parameter is optional. For situations where a remote host name is embedded in the log file directory path structure, e.g. "/var/log/remote/<host>/...", this can be used as the originating host for the log by setting this parameter to the path component to be used for the hostname. The value should be an integer, 1-based. In this example the configuration would be "ze_host_in_logpath=4". |
ze_forward_tag | Tag to specify log-forwarded sources | This parameter is optional. It can be used to indicate sources that are being used for remote log forwarding, by specifying a specific fluentd "tag" to one or more sources. The default tag value is "ze_forwarded_logs". |
ze_path_map_file | Path mapping file | This parameter is optional. It allows embedded semantic data (ids, tags,configs) in logfile paths to be parsed and added to Zebrium log data. Set to the full path of a JSON file containing mapping information. Default is empty string. See Log Path Mapping, below. |
User Log Paths
User log paths can be configured via /etc/td-agent/log-file-map.conf. During log collector service startup, if /etc/td-agent/log-file-map.conf exists, log collector service script writes log paths defined in /etc/td-agent/log-file-map.conf to /etc/td-agent/conf.d/user.conf. Please note any user log paths configured at installation time via ZE_USER_LOG_PATHS must be added to /etc/td-agent/log-file-map.conf to avoid being overwritten.
{ "mappings": [ { "file": "/app1/log/error.log", "alias": "app1_error" }, { "file": "/app2/log/error.log", "alias": "app2_error" }, { "file": "/var/log/*.log", "exclude": "/var/log/my_debug.log,/var/log/my_test.log" } ] }
Filtering Specific Log Events
To exclude certain sensitive or noisy events from being sent to Zebrium, you can filter them at the source collection point:
-
Add the following in /etc/td-agent/td-agent.conf after other @include:
@include conf.d/log_msg_filters.conf
-
Create a config file /etc/td-agent/conf.d/log_msg_filters.conf that contains the following:
<filter TAG_FOR_LOG_FILE> @type grep <exclude> key message pattern /<PATTERN_FOR_LOG_MESSAGES>/ </exclude> </filter>
-
Restart the td-agent with the following command:
sudo systemctl restart td-agent
Example
Below is an example log_msg_filters.conf file for filtering out specific messages from a Vertica log file at /fast1/vertica_catalog/zdb/v_zdb_node0001_catalog/vertica.log.
In this example, the Fluentd tag for file is node.logs.<FILE_NAME_REPLACE_/_WITH_DOT> (replace all slashes with dots in the file path):
<filter node.logs.fast1.vertica_catalog.zdb.v_zdb_node0001_catalog.vertica.log> @type grep <exclude> key message pattern /^[^2]|^.[^0]|TM Merge|Authenticat|[Ll]oad *[Bb]alanc[ei]|\[Session\] <INFO>|\[Catalog\] <INFO>|\[Txn\] <INFO>|Init Session.*<LOG>/ </exclude> </filter>
Log Path Mapping
Log path mapping allows semantic information (like "tags", "ids", and "configs") to be extracted from log paths and passed to the Zebrium backend. For example, this can include log-file specific host information or business-related tags that are embedded in the path of the log file can be extracted.
You can configure log path mapping using a JSON file, with the following format:
{ "mappings": { "patterns": [ "regex1", ... ], "tags": [ "tag_name", ... ], "ids": [ "id_name",... ], "configs": [ "config_name",... ] } }
Set "patterns" to regular expressions to match the log file path. Each regex-named capture in a matching regular expression will be compared to the "tags", "ids", and "configs" sections and added to the corresponding record section(s). Use the ze_path_map_file configuration parameter to specify the path to the JSON file.
Configuring Multiple Zebrium Service Groups Within a Single Collector
You can use a single td-agent to send log files to multiple Zebrium service groups. You should be familiar with advanced fluentd configuration for this feature. We recommended that you review the official documentation at https://docs.fluentd.org/configuration/config-file.
The following settings are required:
- Each service group needs to have its own source block and match block definitions.
- In each source block, the path should be as specific as possible.
- The paths in source blocks should not overlap.
- Each source block needs a unique pos_file (td-agent will create the file if it does not exist).
- Each source block should include a unique tag to specify which match block or service group should pick up the log events.
- Each match block should match on the tag in its corresponding source block.
- ze_log_collector_url, ze_log_collector_token, and ze_log_collector_type will probably be the same in all match blocks.
- ze_host_tags specifies the service group name with "ze_deployment_name=".
- each match block requires a unique buffer path, which will be created if the specified path does not exist.
The following example shows how this could be configured in /etc/td-agent/td-agent.conf:
<source> @type tail path "/var/log/auth.log" format none path_key tailed_path pos_file /var/log/td-agent/position_file_1.pos tag seamus1 read_from_head true </source> <source> @type tail path "/var/log/syslog" format none path_key tailed_path pos_file /var/log/td-agent/position_file_2.pos tag seamus2 read_from_head true </source> @include conf.d/user.conf @include conf.d/containers.conf @include conf.d/systemd.conf <match seamus1> @type zebrium ze_log_collector_url "https://trial.zebrium.com" ze_log_collector_token "<your token here>" ze_log_collector_type "linux" ze_host_tags "ze_deployment_name=seamusfirstservicegroup" <buffer tag> @type file path /var/log/td-agent/buffer1/out_zebrium.*.buffer chunk_limit_size "1MB" chunk_limit_records "4096" flush_mode "interval" flush_interval "60s" </buffer> </match> <match seamus2> @type zebrium ze_log_collector_url "https://trial.zebrium.com" ze_log_collector_token "<your token here, should be the same as above>" ze_log_collector_type "linux" ze_host_tags "ze_deployment_name=seamussecondservicegroup" <buffer tag> @type file path /var/log/td-agent/buffer2/out_zebrium.*.buffer chunk_limit_size "1MB" chunk_limit_records "4096" flush_mode "interval" flush_interval "60s" </buffer> </match>
You should set "patterns" to regular expressions to match the log file path. Each regex named captured in a matching regular expression will be compared to the "tags", "ids", and "configs" sections and added to the corresponding record sections. Use the ze_path_map_file configuration parameter to specify the path to the JSON file.
Usage
Start and Stop Fluentd
You can start or stop the Fluentd agent with the following command:
sudo systemctl <start | stop> td-agent
Testing Your Installation
After the collector has been deployed in your environment, your logs and anomaly detection will be available in the Zebrium user interface.
Troubleshooting
In the event that Zebrium requires the collector logs for troubleshooting, the logs are located in the following locations:
- Collector installation log: /tmp/zlog-collector-install.log.*
- Collector runtime log: /var/log/td-agent/td-agent.log
In case of an HTTP connection error, check the spelling of the Zebrium host URL. Also check that any network proxy servers are configured appropriately.
Contact Zebrium at support@zebrium.com if you need any assistance.
Environment Variables
If the environment is using a proxy server to access the Internet then standard variables, such as http_proxy, should be configured prior to installation. For more information, see Operating with a Proxy Server.
Operating with a Proxy Server
If the agent environment requires a non-transparent proxy server to be configured, you should do this at two points:
- The standard http_proxy and https_proxy environment variables must be set in the local environment when the installer is run. This allows the installer to access the Internet to download necessary components.
- After installation is run, the system service also needs to have the same environment variables available. This allows the Zebrium agent to communicate with the log host to send logs.
Setting the Proxy Server in a systemd Environment
If the agent service is run from systemd and a proxy server is in use, the service needs to have the appropriate proxy configuration added to systemd. This may not be needed if your system is already configured, so that all systemd services globally use a proxy.
To do this, after the installation is performed, edit the file /etc/systemd/service/td-agent.service.d/override.conf to add environment configuration lines for the proxy server. For example:
Environment=http_proxy=myproxy.example.com:8080
After this is done, run the following commands to reload the systemd daemon and start the service:
sudo systemctl daemon-reload
sudo systemctl restart td-agent