Monitoring Device Logs Using theSL1 Agent

This section describes how to use the SL1 Agent to monitor logs with Log File Monitoring policies. SL1 supports multiple methods for ingesting log data, which you can use to generate events.

The following video explains log file monitoring with the SL1 agent:

Use the following menu options to navigate the SL1 user interface:

To view a pop-out list of menu options, click the menu icon ().
To view a page containing all of the menu options, click the Advanced menu icon ().

What is a Log File Monitoring Policy?

A Log File Monitoring policy specifies:

A file or Windows log on the host device that an agent will monitor
The logs from the file or Windows log that an agent will send to SL1

You can create, edit, and delete Log File Monitoring policies from the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies). After creating a Log File Monitoring policy, you must align the policy to one or more devices either from the Log File Monitoring page or by using a Device Template.

You can view logs collected by the SL1 agent on the Logs panel on the Device Investigator page for the device on which the agent is installed. The same logs also appear on the Logs tab in the Device Properties and Device Summary pages for that device. You can define event policies that specify how logs collected by an agent will trigger events.

Log File Monitoring policies can be included in a PowerPack. For information about including a Log File Monitoring Policy in a PowerPack, see the PowerPacks section.

When running the agent as a dedicated user on Windows or Linux systems, the user requires Read access to any directories being monitored. To monitor system events on Windows, the user must be in the Event Log Readers group. See Installing a Windows Agent for more information.

Viewing the List of Log File Monitoring Policies

The Log File Monitoring Policies page displays a list of all Log File Monitoring policies. From this page, you can also create, edit, and delete Log File Monitoring policies.

To sort the list of Log File Monitoring policies, click on a column heading. The list will be sorted by the column value, in ascending order. To sort by descending order, click the column heading again. The Last Edited column sorts by descending order on the first click; to sort by ascending order, click the column heading again.

For each Log File Monitoring Policy, the page displays:

Name. Name of the Log File Monitoring policy.
Policy ID. Unique numeric ID, automatically assigned by SL1 to each Log File Monitoring policy.
Source Type. The source of the logs on the monitored device. Possible values are:

File. The agent will monitor a file on the file system of the device(s).
Event Log. The agent will monitor the Windows log on the device(s).

Source. The full path of the log file or the name of the log that the agent will monitor.

For Linux or Unix operating systems, use "/" in the file paths. For Windows you can use "\" in the file paths, but a double slash "\\" will escape the "\" to ensure the file path ends up with a legitimate slash in it.
Filter. The regular expression that the agent uses to determine whether a log message is sent to SL1.
Subscribers. The number of devices with which the policy is aligned.
Edited By. SL1 user who created or last edited the Log File Monitoring policy.
Last Edited. Date and time the Log File Monitoring policy was created or last edited.

Creating a Log File Monitoring Policy

To create a Log File Monitoring policy:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).

Click Create. The Log Monitoring Policy modal appears:

Supply values in the following fields:

Name. Enter a name for the policy.

Type. Select the source of the logs on the monitored device. Choices are:

File. The agent will monitor a file on the file system of the device(s).
Event Log. The agent will monitor the Windows log on the device(s).

File Path. If you selected File in the Type field, this field is displayed. Enter the full path of the file to monitor. You can use a * to match multiple files, such as /var/log/em7/*.log.
Source. If you selected Event Log in the Type field, this field is displayed. Select the Windows log to monitor. Choices are:

application
system
security
other

Description. If you selected other in the Source field, this field is displayed. Type the name of the event log source type.
Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute.

For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
Filter. Specify a regular expression that will be used to evaluate the log messages in the specified file or Windows log. If a log message matches this regular expression, the agent will send that log message to SL1. If a log message does not match this regular expression, the agent will not send that log message to SL1.

For Windows event logs, the SL1 Agent adds the Event ID to the value in the Message portion of the Windows log before applying the value in the Filter field. The agent does not apply the value in the Filter field to the Instance ID or any other property of a Windows event log entry.

Avoid adding a leading ".*" in a filter, such as ".ERROR", as that character might increase the time it takes the agent to execute the filter, and on busy SL1 systems, that character in the filter can negatively impact the CPU.

Click Save.
Before you can use this Log File Monitoring policy, you will need to align the policy with one or more devices. For more information, see Aligning a Log File Monitoring Policy to Devices.

Aligning a Log File Monitoring Policy to Devices

You can align Log File Monitoring policies to devices either from the Log File Monitoring page or by using a Device Template.

This section describes how to align a Log File Monitoring policy from the Log File Monitoring page. It also describes how to use a one-off Device Template to align a Log File Monitoring policy. For more information on Device Templates, including the other methods you can use to create, save, and apply Device Templates, see the Device Groups and Device Templates section.

To align Log File Monitoring policies to one or more devices from the Log File Monitoring page:

Go to the Log File Monitoring page (Registry > Monitors > Logs).

Click Create. The Log File Monitor modal appears:

In the Log File Monitor modal, supply values in the following fields:

Device. Select a device to align with the Log File Monitoring policy.
Log Policy. Select the Log File Monitoring policy to align with the selected device. Only policies that are appropriate for the selected device will appear. For example, if you chose a Linux device in the Device field, the Log Policy field will not show policies of the Event Log type.

If desired, click the names of the following fields to enable and edit them. These fields allow you to override settings of the policy you selected in the Log Policy field for the device selected in the Device field:

File Path. Enter the full file path or the file name to monitor. This field appears only if the type of the policy is File.

Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute. For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
File. Specify a regular expression that will be used to evaluate the log messages in the specified file or Windows log. If and only if a log message matches this regular expression, the agent will send the log message to SL1.

Click Save.

To align Log File Monitoring policies to one or more devices using a Device Template:

Go to the Device Manager page (Devices > Device Manager).

Select the checkboxes for the devices with which you want to align Log File Monitoring policies.
In the Select Action drop-down list, select MODIFY by Template.

Click Go. The Device Template Editor modal appears:

Click the Logs tab.
Click the Add New Log Policy Sub-Template icon ().

Supply values in the following fields:

Align Log Monitoring Policy With. Select the devices to which the Log File Monitoring policy will be applied.
Log Monitoring Policy. Select the Log File Monitoring policy you want to align with the selected devices.

Optionally, you can override one or more settings from the Log File Monitoring policy specifically for the selected devices. To do this, click the field label for each setting you want to override to enable the fields and supply a value in those fields. For a description of each field, see the Creating a Log File Monitoring Policy section.
Repeat steps 6 and 7 for each Log File Monitoring policy you want to align with the devices you selected in step 2.
If you want to save this Device Template for future use, select the Save When Applied & Confirmed checkbox and enter a name for the Device Template in the Template Name field.
Click Apply. The Setting Confirmation page is displayed.
Click Confirm. The aligned Log File Monitoring policy will appear on the Log File Monitoring page (Registry > Monitors > Logs).

Unaligning Log File Monitoring Policies from Devices

To delete Log File Monitoring Policies, you must first unalign the policy from any devices. You can unalign a Log File Monitoring policy by from the Log File Monitoring page.

To unalign devices from a Log File Monitoring policy:

Go to the Log File Monitoring page (Registry > Monitors > Logs)
Select the checkboxes for the devices from which the policy must be unaligned.
In the Select Action drop-down menu, choose Delete Log File Monitors.

This action does not delete the Log File Monitoring policy.

Click Go to unalign the Log File Monitoring policy from the devices.

Editing a Log File Monitoring Policy

To edit a Log File Monitoring policy:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).
Click the wrench icon () for the Log File Monitoring Policy you want to edit. The Log Monitoring Policy modal appears.

Edit the value in one or more fields. For a description of each field, see the Creating a Log File Monitoring Policy section.
Click Save.

Deleting Log File Monitoring Policies

Before you delete a Log File Monitoring Policy, you must unalign that policy from all devices. See Unaligning Log File Monitoring Policies for more information.

To delete one or more Log File Monitoring policies:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).
Select the checkboxes for the Log File Monitoring Policies you want to delete.
In the Select Action drop-down list, select DELETE Log FIle Monitoring Policies.
Click Go.

Viewing the List of Log File Monitoring Policies and Aligned Devices

The Log File Monitoring page (Registry > Monitors > Logs) displays a list of existing relationships between devices and Log File Monitoring policies. From the Log File Monitoring page, you can also align and unalign devices and Log File Monitoring policies.

For each aligned Log File Monitoring policy and device, the page displays:

Name. The name of the Log File Monitoring policy.
Device Name. The name of the device aligned to the Log File Monitoring policy.
ID. The unique numeric ID of the Log File Monitoring policy. The ID is automatically assigned by SL1.
Source Type. The source of the logs in the monitored device. The possible values are:

File. The agent monitors a file on the file system of the device. Usually, this is used to monitor Linux log files.
Event Log. The agent monitors to Windows log on the device.

Source. The full path of the log file or the name of the Windows log that the agent monitors.
Filter. The regular expression the agent uses to determine if a log should be sent to SL1.
Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute. For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
Edited By. The user who created or last edited the alignment between the device and Log File Monitoring policy.
Last Edited. The date and time the alignment between the device and Log File Monitoring policy was created or last edited.

Filtering the List of Log File Monitoring Policies and Aligned Devices

You can filter the list of Log File Monitoring policies and aligned devices on the Log File Monitoring page using the search fields at the top of each column. When you type in each search field, the list of results on the page is automatically updated to match the text, including partial matches.

You can use special characters in each search field to filter. Fore more information about filtering using special characters, see the Filtering the List of Log File Monitoring Policies section.

Creating an Event Policy for Agent Logs

To trigger events based on log messages collected by the agent, you must create an event policy on the Event Policies page, and then associate that event policy with a Log File Monitoring policy.

If you are using the classic user interface for SL1, see Creating an Event Policy for Agent Logs in the Classic User Interface.

To create an event policy that triggers based on log data collected by the agent:

Go to Event Policies page (Events > Event Policies).

On the Event Policies page, click the Create Event Policy button. The Event Policy Editor page appears:

On the Event Policy Editor page and set of tabs, you can define a new event. The Event Policy Editor page contains the following fields and tabs:

Policy Name. At the top left of the Policy Description tab, type a name for the event policy in this field.
Enable Event Policy. This toggle allows you to enable and disable the event policy.
Policy Description. In this field, type a description of the event policy.

Click the Match Logic tab to define pattern-matching for the event:
Complete the following fields:

Event Source. Select ScienceLogic Agent as the source for the event. The fields below this field will change based on your selection.
Log Policy. Select the Log File Monitoring Policy the agent will use to collect the log message.

After selecting and defining your Event Source, enter values in the fields on the right side of the Match Logic tab specify the text that must appear in the log message for the event policy to trigger:

String/Regular Expression. Use this dropdown to select String or Regular Expression.
Match String. A string used to correlate the event with a log message. Can be up to 512 characters in length. To match this event policy, the text of a log message or alert must match the value you enter in this field. Can be any combination of alpha-numeric and multi-byte characters. SL1's expression matching is case-sensitive. This field is required for events generated with a source of Syslog, API, and Email.
Second Match String (Optional). A secondary string used to match against the originating log message. Can be up to 512 characters in length. Can be any combination of alpha-numeric and multi-byte characters. To match this event policy, the text of a log message or alert must match the value you enter in this field and the value you entered in the Match String field. This field is optional.
Allow event to expire if it doesn't reoccur within a time frame. If toggled on, enter the time in which an active event will be cleared automatically if there is no reoccurrence of the event in the fields that appear. You can enter time in minutes or hours.
Require multiple triggers within a time frame. If toggled on, enter the number of events and the time in which an event requires multiple triggers to occur in the fields that appear. You can enter time in minutes or hours.
Detection Weight. If two event definitions are very similar, the weight field specifies the order in which SL1 should match messages against the similar event definitions. The event definition with the lowest weight will be matched first. This field is most useful for events that use expression matching. Options range from 0 (first) - 20 (last).
Multi Match. By default, SL1 will match a log message or alert to only one event policy. If a log message or alert matches multiple event polices, SL1 will use the Detection Weight setting to determine which event policy the log message or alert will match. If you select the Multi Match checkbox in all event policies that can match the same log message or alert, SL1 will generate an event for every event policy that matches that single log message or alert.
Message Match. If SL1 has generated an event and then a second log message or alert matches the same event policy for the same entity, SL1 will not generate a second event, but will increase the count value for the original event on the Events page and in the Viewing Events page. By default, this behavior occurs regardless of whether the two log messages or alerts contain the same message. If you select the Message Match checkbox, this behavior will occur only if the log messages or alerts contain the same message.

Click the Event Message tab to configure the event:
Complete the following fields:

Event Message. The message that appears in the Event Console page or the Viewing Events page when this event occurs. This field defaults to "%M" for new event policies upon creation. The message can be any combination of alphanumeric and multi-byte characters. Variables include the characters "%" (percent) and "|" (bar). You can also use regular expressions and variables that represent text from the original log message to create the Event Message.

To include regular expressions in the Event Message, surround the regular expression with %R and %/R. For example, %RFilename: .*? %/R would search for the first instance of the string "Filename: " (Filename-colon-space) followed by any number of any characters up to the line break. The %R indicates the beginning of a regular expression. The %/R indicates the end of a regular expression.

SL1 uses the regular expression to search the log message and use the matching text in the event message. For details on the regular expression syntax allowed by SL1, see http://www.python.org/doc/howto/.

You can also use the following variables in the Event Message field:
- %I ("eye"). This variable contains the value that matches the Identifier Pattern field in the Advanced tab.
- %M. The full text of the log message that triggered the event will be displayed in Event Message field.
- %T. Threshold value from the log file will be displayed in Event Message field.

Event Severity. Defines the severity of the event. Choices are:

Healthy. Healthy events indicate that a device or condition has returned to a healthy state. Frequently, a healthy event is generated after a problem has been fixed.

Notice. Notice events indicate a condition that does not affect service but about which users should be aware.
Minor. Minor events indicate a condition that does not currently impair service, but the condition needs to be corrected before it becomes more severe.
Major. Major events indicate a condition that impacts service and requires immediate investigation.
Critical. Critical events indicate a condition that can seriously impair or curtail service and requires immediate attention (i.e., service or system outages).

Use Modifier. If selected, when the event is triggered, SL1 will check to see if the interface associated with this event has a custom severity modifier. If so, the event will appear in the Event Console with that custom severity modifier applied to the severity in the Event Severity field. For example, if an interface with an Event Severity Adjust setting of Sev -1 triggers an event with an Event Severity of Major and that event has the Use Modifier checkbox selected, the event will appear in the Event Console with a severity of Minor.

Optionally, supply values in the other fields on this page. For more information on the remaining fields, as well as the Suppressions tab, see Defining an Event Policy.
Click Save.

Creating an Event Policy for Agent Logs in the Classic User Interface

To trigger events based on log messages collected by the agent in the classic user interface for SL1, you must create an event policy that is associated with a Log File Monitoring policy.

To create an event policy in the classic user interface based on log data collected by the agent:

Go to Event Policy Manager page (Registry > Events > Event Manager in the classic user interface).

In the Event Policy Manager page, click Create. The Event Policy Editor page appears:

In the Event Policy Editor page and set of tabs, you can define a new event. The Event Policy Editor page contains three tabs:

Policy. Define basic parameters for the event.

Advanced. Define pattern-matching for the event and also define event roll-ups and suppressions.
Suppressions. Suppress the event on selected devices. When you suppress an event, you are specifying that, in the future, if this event occurs again on a specific device, the event will not appear in the Event Console page or the Viewing Events page for the device.

Supply values in the following fields:

Event Source. Select ScienceLogic Agent.

Policy Name. The name of the event. Can be any combination of alphanumeric characters, up to 48 characters in length.
Operational State. Specifies whether event is to be operational or not. Choices are Enabled or Disabled.

Event Message. The message that appears in the Event Console page or the Viewing Events page when this event occurs. Can be any combination of alphanumeric characters.

You can use regular expressions that represent text from the original log message to create the Event Message:

%R. Indicates a regular expression. Surround the regular expression with %R and %/R. For example, %RFilename: .*? %/R would search for the first instance of the string "Filename: " followed by any number of any characters up to the line break. For details on the regular expression syntax allowed by SL1, see http://www.python.org/doc/howto/.

%I ("eye"). This variable contains the value that matches the Identifier Pattern field in the Advanced tab.
%M. The full text of the log message that triggered the event will be displayed in Event Message field.
%T. Threshold value from the log file will be displayed in Event Message field.

Event Severity. Defines the severity of the event. Choices are:

Healthy. Healthy Events indicate that a device or condition has returned to a healthy state. Frequently, a healthy event is generated after a problem has been fixed.

Notice. Notice Events indicate a condition that does not affect service but about which users should be aware.
Minor. Minor Events indicate a condition that does not currently impair service, but the condition needs to be corrected before it becomes more severe.
Major. Major Events indicate a condition that is service impacting and requires immediate investigation.
Critical. Critical Events indicate a condition that can seriously impair or curtail service and require immediate attention (i.e. service or system outages).

Use Modifier. If selected, when the event is triggered, SL1 will check to see if the interface associated with this event has a custom severity modifier. If so, the event will appear in the Event Console with that custom severity modifier applied to the severity in the Event Severity field. For example, if an interface with an Event Severity Adjust setting of Sev -1 triggers an event with an Event Severity of Major and that event has the Use Modifier checkbox selected, the event will appear in the Event Console with a severity of Minor.
Policy Description. Text that explains what the event means and what possible causes are.

Select the Advanced tab.
In the Log Policy field, select the Log File Monitoring policy that the agent will use to collect the log message.

Enter values in the following fields to specify specific text that must appear in the log message for the event policy to trigger:

First Match String. A string used to match against the originating log message. To match this event policy, the text of a log message must match the value you enter in this field. Can be any combination of alphanumeric characters. Expression matching in SL1 is case-sensitive.

Second Match String. A secondary string used to match against the originating log message. To match this event policy, the text of a log message must match the value you enter in this field and the value you entered in the First Match String field. This field is optional.

The Match Logic field specifies whether SL1 should process First Match String and Second Match String as simple text matches or as regular expressions.

Optionally, supply values in the other fields on this page. For more information on the remaining fields, as well as the Suppressions tab, see the Events section.
Click Save.