Linux Collector

The Fluentd output plugin, ze-fluentd-plugin, sends the logs you collect with Fluentd on Linux to Skylar Automated RCA for automated anomaly detection. You can access the plugin at the GitHub repository at https://github.com/zebrium/ze-fluentd-plugin.

For instructions on deploying the Skylar Automated RCA Fluentd collector for Docker environments, see the instructions in Docker Container Log Collectors.

System Requirements

The following Linux operating system distributions are supported:

• Ubuntu 16.04, 18.04, or 20.04
• CentOS or Red Hat Enterprise Linux 7 or 8
• Amazon Linux 2

Installing the Collector

1. If the environment uses a proxy server, see Operating with a Proxy Server, below.

2. Determine which deployment name to use for the <YOUR_SERVICE_GROUP> value, below.

3. If your account has multiple deployments, go to the Skylar Automated RCA user interface, click the Deployment drop-down in the top-right navigation bar, and switch to the deployment you want to use to collect Windows logs.

4. In the Skylar Automated RCA user interface, go to the Integrations & Collectors page (Settings () Integrations & Collectors).

5. Click the Linux button under Log Collectors and copy the command from the Linux Log Collector dialog. This command includes the Skylar Automated RCA API server URL (<ZAPI_URL>) and authentication token (<AUTH_TOKEN>) values.

6. Update the command from step 4 with the relevant values and run it in a shell on the host. The command uses the following format:

   curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<YOUR_SERVICE_GROUP>" /bin/bash

The default system log file paths are defined by the ZE_LOG_PATHS environment variable. The default value is:

"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"

You can use the ZE_USER_LOG_PATHS environment variable to add more user-specific log file paths. For example, to add app log files at /app1/log/app1.log and /app2/log/\*.log, you can set ZE_USER_LOG_PATHS to:

"/app1/log/app1.log,/app2/log/*.log"

Upgrading the Collector

The upgrade command is similar to the installation command:

curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_LOG_COLLECTOR_URL=<ZAPI_URL> ZE_LOG_COLLECTOR_TOKEN=<AUTH_TOKEN> ZE_HOST_TAGS="ze_deployment_name=<deployment_name>" OVERWRITE_CONFIG=1 /bin/bash

Please note that setting OVERWRITE_CONFIG to 1 will cause /etc/td-agent/td-agent.conf to be upgraded to the latest version.

Uninstalling the Collector

To uninstall:

curl https://raw.githubusercontent.com/zebrium/ze-fluentd-plugin/master/install_collector.sh | ZE_OP=uninstall /bin/bash

Installing on Hosts with Existing td-agent Configuration

You can add the Skylar Automated RCA output plugin on a host with existing td-agent configuration without running the Skylar Automated RCA log collector installer.

1. Download the Skylar Automated RCA output plugin from https://github.com/zebrium/ze-fluentd-plugin/releases/download/1.37.2/fluent-plugin-zebrium_output-1.37.2.gem.

2. Run the following command in the same directory where fluent-plugin-zebrium_output-1.37.2.gem is saved:

   sudo td-agent-gem install fluent-plugin-zebrium_output

3. Add Skylar Automated RCA output configuration to the /etc/td-agent/td-agent.conf file.

   The following is an example configuration that duplicates log messages and sends one copy to Skylar Automated RCA:

   <match **>
     @type copy
     # Skylar log collector
     <store>
       @type zebrium
       ze_log_collector_url "ZE_LOG_COLLECTOR_URL"
       ze_log_collector_token "ZE_LOG_COLLECTOR_TOKEN"
       ze_host_tags "ze_deployment_name=#{Socket.gethostname},myapp=test2"
       @log_level "info"
       <buffer tag>
         @type file
         path "/var/td-agent/zebrium"
         flush_mode "interval"
         flush_interval "60s"
       </buffer>
     </store>
     <store>
         @type OTHER_OUTPUT_PLUGIN
         ...
     </store>
   </match>

Configuration for td-agent

The configuration file for td-agent is at /etc/td-agent/td-agent.conf. The following parameters must be configured for your instance:

Parameter	Description	Note
ze_log_collector_url	Skylar Automated RCA log host URL	Provided by Skylar Automated RCA after your account has been created.
ze_log_collector_token	Authentication token	Provided by Skylar Automated RCA after your account has been created.
path	Log files to read	Both files and file patterns are allowed. Files should be separated by comma. The default value is '"/var/log/*.log,/var/log/syslog,/var/log/messages,/var/log/secure"'
ze_host_tags	Host meta data	This parameter is optional. You can pass meta data in key-value pairs, the format is: "key1=value1,key2=value2". We suggest at least set one tag for deployment name: "ze_deployment_name=<your_deployment_name>"
ze_host_in_logpath	Log path component for remote host name	This parameter is optional. For situations where a remote host name is embedded in the log file directory path structure, e.g. "/var/log/remote/<host>/...", this can be used as the originating host for the log by setting this parameter to the path component to be used for the hostname. The value should be an integer, 1-based. In this example the configuration would be "ze_host_in_logpath=4".
ze_forward_tag	Tag to specify log-forwarded sources	This parameter is optional. It can be used to indicate sources that are being used for remote log forwarding, by specifying a specific fluentd "tag" to one or more sources. The default tag value is "ze_forwarded_logs".
ze_path_map_file	Path mapping file	This parameter is optional. It allows embedded semantic data (ids, tags,configs) in logfile paths to be parsed and added to Skylar Automated RCA log data. Set to the full path of a JSON file containing mapping information. Default is empty string. See Log Path Mapping, below.

User Log Paths

User log paths can be configured via /etc/td-agent/log-file-map.conf. During log collector service startup, if /etc/td-agent/log-file-map.conf exists, log collector service script writes log paths defined in /etc/td-agent/log-file-map.conf to /etc/td-agent/conf.d/user.conf. Please note any user log paths configured at installation time via ZE_USER_LOG_PATHS must be added to /etc/td-agent/log-file-map.conf to avoid being overwritten.

{
  "mappings": [
    {
      "file": "/app1/log/error.log",
      "alias": "app1_error"
    },
    {
      "file": "/app2/log/error.log",
      "alias": "app2_error"
    },
    {
      "file": "/var/log/*.log",
      "exclude": "/var/log/my_debug.log,/var/log/my_test.log"
    }
  ]
}

Filtering Specific Log Events

To exclude certain sensitive or noisy events from being sent to Skylar Automated RCA, you can filter them at the source collection point:

1. Add the following in /etc/td-agent/td-agent.conf after other @include:

   @include conf.d/log_msg_filters.conf

2. Create a config file /etc/td-agent/conf.d/log_msg_filters.conf that contains the following:

   <filter TAG_FOR_LOG_FILE>
     @type grep
     <exclude>
       key message
       pattern /<PATTERN_FOR_LOG_MESSAGES>/
   </exclude>
   </filter>

3. Restart the td-agent with the following command:

   sudo systemctl restart td-agent

Example

Below is an example log_msg_filters.conf file for filtering out specific messages from a Vertica log file at /fast1/vertica_catalog/zdb/v_zdb_node0001_catalog/vertica.log.

In this example, the Fluentd tag for file is node.logs.<FILE_NAME_REPLACE_/_WITH_DOT> (replace all slashes with dots in the file path):

<filter node.logs.fast1.vertica_catalog.zdb.v_zdb_node0001_catalog.vertica.log>
  @type grep
  <exclude>
    key message
    pattern /^[^2]|^.[^0]|TM Merge|Authenticat|[Ll]oad *[Bb]alanc[ei]|\[Session\] 
<INFO>|\[Catalog\] <INFO>|\[Txn\] <INFO>|Init Session.*<LOG>/
  </exclude>
</filter>

Log Path Mapping

Log path mapping allows semantic information (like "tags", "ids", and "configs") to be extracted from log paths and passed to the Skylar Automated RCA backend. For example, this can include log-file specific host information or business-related tags that are embedded in the path of the log file can be extracted.

You can configure log path mapping using a JSON file, with the following format:

{
  "mappings": {
    "patterns": [
      "regex1", ...
    ],
    "tags": [
      "tag_name", ...
    ],
    "ids": [
      "id_name",...
    ],
    "configs": [
       "config_name",...
    ]
  }
}

Set "patterns" to regular expressions to match the log file path. Each regex-named capture in a matching regular expression will be compared to the "tags", "ids", and "configs" sections and added to the corresponding record section(s). Use the ze_path_map_file configuration parameter to specify the path to the JSON file.

Configuring Multiple Skylar Automated RCA Service Groups Within a Single Collector

You can use a single td-agent to send log files to multiple Skylar Automated RCA service groups. You should be familiar with advanced fluentd configuration for this feature. We recommended that you review the official documentation at https://docs.fluentd.org/configuration/config-file.

The following settings are required:

• Each service group needs to have its own source block and match block definitions.
• In each source block, the path should be as specific as possible.
• The paths in source blocks should not overlap.
• Each source block needs a unique pos_file (td-agent will create the file if it does not exist).
• Each source block should include a unique tag to specify which match block or service group should pick up the log events.
• Each match block should match on the tag in its corresponding source block.
• ze_log_collector_url, ze_log_collector_token, and ze_log_collector_type will probably be the same in all match blocks.
• ze_host_tags specifies the service group name with "ze_deployment_name=".
• each match block requires a unique buffer path, which will be created if the specified path does not exist.

The following example shows how this could be configured in /etc/td-agent/td-agent.conf:

<source>
  @type tail
  path "/var/log/auth.log"
  format none
  path_key tailed_path
  pos_file /var/log/td-agent/position_file_1.pos
  tag seamus1
  read_from_head true
</source>

<source>
  @type tail
  path "/var/log/syslog"
  format none
  path_key tailed_path
  pos_file /var/log/td-agent/position_file_2.pos
  tag seamus2
  read_from_head true
</source>

@include conf.d/user.conf
@include conf.d/containers.conf
@include conf.d/systemd.conf

<match seamus1>
  @type zebrium
  ze_log_collector_url "https://trial.zebrium.com"
  ze_log_collector_token "<your token here>"
  ze_log_collector_type "linux"
  ze_host_tags "ze_deployment_name=seamusfirstservicegroup"
  <buffer tag>
    @type file
    path /var/log/td-agent/buffer1/out_zebrium.*.buffer
    chunk_limit_size "1MB"
    chunk_limit_records "4096"
    flush_mode "interval"
    flush_interval "60s"
  </buffer>
</match>

<match seamus2>
  @type zebrium
  ze_log_collector_url "https://trial.zebrium.com"
  ze_log_collector_token "<your token here, should be the same as above>"
  ze_log_collector_type "linux"
  ze_host_tags "ze_deployment_name=seamussecondservicegroup"
  <buffer tag>
    @type file
    path /var/log/td-agent/buffer2/out_zebrium.*.buffer
    chunk_limit_size "1MB"
    chunk_limit_records "4096"
    flush_mode "interval"
    flush_interval "60s"
  </buffer>
</match>

You should set "patterns" to regular expressions to match the log file path. Each regex named captured in a matching regular expression will be compared to the "tags", "ids", and "configs" sections and added to the corresponding record sections. Use the ze_path_map_file configuration parameter to specify the path to the JSON file.

Usage

Start and Stop Fluentd

You can start or stop the Fluentd agent with the following command:

sudo systemctl <start | stop> td-agent

Testing Your Installation

After the collector has been deployed in your environment, your logs and anomaly detection will be available in the Skylar Automated RCA user interface.

Troubleshooting

In the event that Skylar Automated RCA requires the collector logs for troubleshooting, the logs are located in the following locations:

1. Collector installation log: /tmp/zlog-collector-install.log.*
2. Collector runtime log: /var/log/td-agent/td-agent.log

In case of an HTTP connection error, check the spelling of the Skylar Automated RCA host URL. Also check that any network proxy servers are configured appropriately.

Contact Skylar Automated RCA at support@sciencelogic.com if you need any assistance.

Environment Variables

If the environment is using a proxy server to access the Internet then standard variables, such as http_proxy, should be configured prior to installation. For more information, see Operating with a Proxy Server.

Operating with a Proxy Server

If the agent environment requires a non-transparent proxy server to be configured, you should do this at two points:

• The standard http_proxy and https_proxy environment variables must be set in the local environment when the installer is run. This allows the installer to access the Internet to download necessary components.
• After installation is run, the system service also needs to have the same environment variables available. This allows the Skylar Automated RCA agent to communicate with the log host to send logs.

Setting the Proxy Server in a systemd Environment

If the agent service is run from systemd and a proxy server is in use, the service needs to have the appropriate proxy configuration added to systemd. This may not be needed if your system is already configured, so that all systemd services globally use a proxy.

To do this, after the installation is performed, edit the file /etc/systemd/service/td-agent.service.d/override.conf to add environment configuration lines for the proxy server. For example:

Environment=http_proxy=myproxy.example.com:8080

After this is done, run the following commands to reload the systemd daemon and start the service:

sudo systemctl daemon-reload

sudo systemctl restart td-agent