Monitoring Device Logs Using an Agent

This section describes how to use the SL1 Agent to monitor logs with Log File Monitoring policies. SL1 supports multiple methods for ingesting log data, which you can use to generate events.

The following video explains log file monitoring with the SL1 agent:

Use the following menu options to navigate the SL1 user interface:

To view a pop-out list of menu options, click the menu icon ().
To view a page containing all of the menu options, click the Advanced menu icon ().

What is a Log File Monitoring Policy?

A Log File Monitoring policy specifies:

A file or Windows log on the host device that an agent will monitor
The logs from the file or Windows log that an agent will send to SL1

The Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies) displays a list of all Log File Monitoring policies. You can create, edit, and delete Log File Monitoring policies from the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).

After creating a Log File Monitoring policy, you must align the policy to one or more devices either from the Log File Monitoring page or by using a Device Template.

You can view logs collected by the SL1 Agent on the Logs panel on the Device Investigator page for the device on which the agent is installed. The same logs also appear on the Logs tab in the Device Properties and Device Summary pages for that device. You can define event policies that specify how logs collected by an agent will trigger events.

Log File Monitoring policies can be included in a PowerPack. For information about including a Log File Monitoring Policy in a PowerPack, see the PowerPacks section.

Viewing the List of Log File Monitoring Policies

The Log File Monitoring Policies page displays a list of all Log File Monitoring policies. From this page, you can also create, edit, and delete Log File Monitoring policies.

To sort the list of Log File Monitoring policies, click on a column heading. The list will be sorted by the column value, in ascending order. To sort by descending order, click the column heading again. The Last Edited column sorts by descending order on the first click; to sort by ascending order, click the column heading again.

For each Log File Monitoring Policy, the page displays:

Name. Name of the Log File Monitoring policy.
Policy ID. Unique numeric ID, automatically assigned by SL1 to each Log File Monitoring policy.
Source Type. The source of the logs on the monitored device. Possible values are:

File. The agent will monitor a file on the file system of the device(s).
Event Log. The agent will monitor the Windows log on the device(s).

Source. The full path of the log file or the name of the log that the agent will monitor.

For Linux or Unix operating systems, use "/" in the file paths. For Windows you can use "\" in the file paths, but a double slash "\\" will escape the "\" to ensure the file path ends up with a legitimate slash in it.
Filter. The regular expression that the agent uses to determine whether a log message is sent to SL1.
Subscribers. The number of devices with which the policy is aligned.
Edited By. SL1 user who created or last edited the Log File Monitoring policy.
Last Edited. Date and time the Log File Monitoring policy was created or last edited.

Filtering the List of Log File Monitoring Policies

To filter the list of credentials in the Log File Monitoring Policies page, use the search fields at the top of each column. The search fields are find-as-you-type filters; as you type, the page is filtered to match the text in the search field, including partial matches. Text matches are not case-sensitive. Additionally, you can use the following special characters in each filter:

, (comma). Specifies an "or" operation. For example:

"dell, micro" would match all values that contain the string "dell" OR the string "micro".

& (ampersand). Specifies an "and" operation. For example:

"dell & micro" would match all values that contain the string "dell" AND the string "micro".

! (exclamation mark). Specifies a "not" operation. For example:

"!dell" would match all values that do not contain the string "dell".

^ (caret mark). Specifies "starts with." For example:

"^micro" would match all strings that start with "micro", like "microsoft".

"^" will include all rows that have a value in the column.

"!^" will include all rows that have no value in the column.

$ (dollar sign). Specifies "ends with." For example:

"$ware" would match all strings that end with "ware", like "VMware".

"$" will include all rows that have a value in the column.

"!$" will include all rows that have no value in the column.

min-max. Matches numeric values only. Specifies any value between the minimum value and the maximum value, including the minimum and the maximum. For example:

"1-5" would match 1, 2, 3, 4, and 5.

- (dash). Matches numeric values only. A "half open" range. Specifies values including the minimum and greater or including the maximum and lesser. For example:

"1-" matches 1 and greater, so it would match 1, 2, 6, 345, etc.

"-5" matches 5 and less, so it would match 5, 3, 1, 0, etc.

> (greater than). Matches numeric values only. Specifies any value "greater than." For example:

">7" would match all values greater than 7.

< (less than). Matches numeric values only. Specifies any value "less than." For example:

"<12" would match all values less than 12.

>= (greater than or equal to). Matches numeric values only. Specifies any value "greater than or equal to." For example:

"=>7" would match all values 7 and greater.

<= (less than or equal to). Matches numeric values only. Specifies any value "less than or equal to." For example:

"=<12" would match all values 12 and less.

= (equal). Matches numeric values only. For numeric values, allows you to match a negative value. For example:

"=-5 " would match "-5" instead of being evaluated as the "half open range" as described above.

Creating a Log File Monitoring Policy

To create a Log File Monitoring policy:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).

Click Create. The Log Monitoring Policy modal appears:

Supply values in the following fields:

Name. Enter a name for the policy.

Type. Select the source of the logs on the monitored device. Choices are:

File. The agent will monitor a file on the file system of the device(s).
Event Log. The agent will monitor the Windows log on the device(s).

File Path. If you selected File in the Type field, this field is displayed. Enter the full path of the file to monitor. You can use a * to match multiple files, such as /var/log/em7/*.log.
Source. If you selected Event Log in the Type field, this field is displayed. Select the Windows log to monitor. Choices are:

application
system
security
other

Description. If you selected other in the Source field, this field is displayed. Type the name of the event log source type.
Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute.

For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
Filter. Specify a regular expression that will be used to evaluate the log messages in the specified file or Windows log. If a log message matches this regular expression, the agent will send that log message to SL1. If a log message does not match this regular expression, the agent will not send that log message to SL1.

For Windows event logs, the SL1 Agent adds the Event ID to the value in the Message portion of the Windows log before applying the value in the Filter field. The agent does not apply the value in the Filter field to the Instance ID or any other property of a Windows event log entry.

Avoid adding a leading ".*" in a filter, such as ".ERROR", as that character might increase the time it takes the agent to execute the filter, and on busy SL1 systems, that character in the filter can negatively impact the CPU.

Click Save.
Before you can use this Log File Monitoring policy, you will need to align the policy with one or more devices. For more information, see Aligning a Log File Monitoring Policy to Devices.

Editing a Log File Monitoring Policy

To edit a Log File Monitoring policy:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).
Click the wrench icon () for the Log File Monitoring Policy you want to edit. The Log Monitoring Policy modal appears.

Edit the value in one or more fields. For a description of each field, see the Creating a Log File Monitoring Policy section.
Click Save.

Deleting Log File Monitoring Policies

Before you delete a Log File Monitoring Policy, you must unalign that policy from all devices. See Unaligning Log File Monitoring Policies for more information.

To delete one or more Log File Monitoring policies:

Go to the Log File Monitoring Policies page (System > Manage > Log File Monitoring Policies).
Select the checkboxes for the Log File Monitoring Policies you want to delete.
In the Select Action drop-down list, select DELETE Log FIle Monitoring Policies.
Click Go.

Viewing the List of Log File Monitoring Policies and Aligned Devices

The Log File Monitoring page (Registry > Monitors > Logs) displays a list of existing relationships between devices and Log File Monitoring policies. From the Log File Monitoring page, you can also align and unalign devices and Log File Monitoring policies.

For each aligned Log File Monitoring policy and device, the page displays:

Name. The name of the Log File Monitoring policy.
Device Name. The name of the device aligned to the Log File Monitoring policy.
ID. The unique numeric ID of the Log File Monitoring policy. The ID is automatically assigned by SL1.
Source Type. The source of the logs in the monitored device. The possible values are:

File. The agent monitors a file on the file system of the device. Usually, this is used to monitor Linux log files.
Event Log. The agent monitors to Windows log on the device.

Source. The full path of the log file or the name of the Windows log that the agent monitors.
Filter. The regular expression the agent uses to determine if a log should be sent to SL1.
Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute. For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
Edited By. The user who created or last edited the alignment between the device and Log File Monitoring policy.
Last Edited. The date and time the alignment between the device and Log File Monitoring policy was created or last edited.

Filtering the List of Log File Monitoring Policies and Aligned Devices

You can filter the list of Log File Monitoring policies and aligned devices on the Log File Monitoring page using the search fields at the top of each column. When you type in each search field, the list of results on the page is automatically updated to match the text, including partial matches.

You can use special characters in each search field to filter. Fore more information about filtering using special characters, see the Filtering the List of Log File Monitoring Policies section.

Aligning a Log File Monitoring Policy to Devices

You can align Log File Monitoring policies to devices either from the Log File Monitoring page or by using a Device Template.

This section describes how to align a Log File Monitoring policy from the Log File Monitoring page. It also describes how to use a one-off Device Template to align a Log File Monitoring policy. For more information on Device Templates, including the other methods you can use to create, save, and apply Device Templates, see the Device Groups and Device Templates section.

To align Log File Monitoring policies to one or more devices from the Log File Monitoring page:

Go to the Log File Monitoring page (Registry > Monitors > Logs).

Click Create. The Log File Monitor modal appears:

In the Log File Monitor modal, supply values in the following fields:

Device. Select a device to align with the Log File Monitoring policy.
Log Policy. Select the Log File Monitoring policy to align with the selected device. Only policies that are appropriate for the selected device will appear. For example, if you chose a Linux device in the Device field, the Log Policy field will not show policies of the Event Log type.

If desired, click the names of the following fields to enable and edit them. These fields allow you to override settings of the policy you selected in the Log Policy field for the device selected in the Device field:

File Path. Enter the full file path or the file name to monitor. This field appears only if the type of the policy is File.

Limit. The maximum log messages the agent sends to SL1 per minute. If the number of matching logs exceeds this value, the agent will stop sending logs to the platform for the remainder of the minute. The limit resets at the beginning of the next minute. For example, suppose you set this field to 10,000. Suppose the agent monitors a device that has 30,000 log messages. The agent will retrieve 10,000 logs and then wait until the beginning of the next minute. The agent will then retrieve the next 10,000 logs and then wait until the beginning of the next minute. The agent will continue to retrieve 10,000 logs per minute until it has retrieved all the logs from the device.
File. Specify a regular expression that will be used to evaluate the log messages in the specified file or Windows log. If and only if a log message matches this regular expression, the agent will send the log message to SL1.

Click Save.

To align Log File Monitoring policies to one or more devices using a Device Template:

Go to the Device Manager page (Devices > Device Manager).

Select the checkboxes for the devices with which you want to align Log File Monitoring policies.
In the Select Action drop-down list, select MODIFY by Template.

Click Go. The Device Template Editor modal appears:

Click the Logs tab.
Click the Add New Log Policy Sub-Template icon ().

Supply values in the following fields:

Align Log Monitoring Policy With. Select the devices to which the Log File Monitoring policy will be applied.
Log Monitoring Policy. Select the Log File Monitoring policy you want to align with the selected devices.

Optionally, you can override one or more settings from the Log File Monitoring policy specifically for the selected devices. To do this, click the field label for each setting you want to override to enable the fields and supply a value in those fields. For a description of each field, see the Creating a Log File Monitoring Policy section.
Repeat steps 6 and 7 for each Log File Monitoring policy you want to align with the devices you selected in step 2.
If you want to save this Device Template for future use, select the Save When Applied & Confirmed checkbox and enter a name for the Device Template in the Template Name field.
Click Apply. The Setting Confirmation page is displayed.
Click Confirm. The aligned Log File Monitoring policy will appear on the Log File Monitoring page (Registry > Monitors > Logs).

Unaligning Log File Monitoring Policies from Devices

To delete Log File Monitoring Policies, you must first unalign the policy from any devices. You can unalign a Log File Monitoring policy by from the Log File Monitoring page.

To unalign devices from a Log File Monitoring policy:

Go to the Log File Monitoring page (Registry > Monitors > Logs)
Select the checkboxes for the devices from which the policy must be unaligned.
In the Select Action drop-down menu, choose Delete Log File Monitors.

This action does not delete the Log File Monitoring policy.

Click Go to unalign the Log File Monitoring policy from the devices.

Creating an Event Policy for Agent Logs

To trigger events based on log messages collected by the agent in the classic user interface for SL1, you must create an event policy that is associated with a Log File Monitoring policy.

To create an event policy in the classic user interface based on log data collected by the agent:

Go to Event Policy Manager page (Registry > Events > Event Manager in the classic user interface).

In the Event Policy Manager page, click Create. The Event Policy Editor page appears:

In the Event Policy Editor page and set of tabs, you can define a new event. The Event Policy Editor page contains three tabs:

Policy. Define basic parameters for the event.

Advanced. Define pattern-matching for the event and also define event roll-ups and suppressions.
Suppressions. Suppress the event on selected devices. When you suppress an event, you are specifying that, in the future, if this event occurs again on a specific device, the event will not appear in the Event Console page or the Viewing Events page for the device.

Supply values in the following fields:

Event Source. Select ScienceLogic Agent.

Policy Name. The name of the event. Can be any combination of alphanumeric characters, up to 48 characters in length.
Operational State. Specifies whether event is to be operational or not. Choices are Enabled or Disabled.

Event Message. The message that appears in the Event Console page or the Viewing Events page when this event occurs. Can be any combination of alphanumeric characters.

You can use regular expressions that represent text from the original log message to create the Event Message:

%R. Indicates a regular expression. Surround the regular expression with %R and %/R. For example, %RFilename: .*? %/R would search for the first instance of the string "Filename: " followed by any number of any characters up to the line break. For details on the regular expression syntax allowed by SL1, see http://www.python.org/doc/howto/.

%I ("eye"). This variable contains the value that matches the Identifier Pattern field in the Advanced tab.
%M. The full text of the log message that triggered the event will be displayed in Event Message field.
%T. Threshold value from the log file will be displayed in Event Message field.

Event Severity. Defines the severity of the event. Choices are:

Healthy. Healthy Events indicate that a device or condition has returned to a healthy state. Frequently, a healthy event is generated after a problem has been fixed.

Notice. Notice Events indicate a condition that does not affect service but about which users should be aware.
Minor. Minor Events indicate a condition that does not currently impair service, but the condition needs to be corrected before it becomes more severe.
Major. Major Events indicate a condition that is service impacting and requires immediate investigation.
Critical. Critical Events indicate a condition that can seriously impair or curtail service and require immediate attention (i.e. service or system outages).

Use Modifier. If selected, when the event is triggered, SL1 will check to see if the interface associated with this event has a custom severity modifier. If so, the event will appear in the Event Console with that custom severity modifier applied to the severity in the Event Severity field. For example, if an interface with an Event Severity Adjust setting of Sev -1 triggers an event with an Event Severity of Major and that event has the Use Modifier checkbox selected, the event will appear in the Event Console with a severity of Minor.
Policy Description. Text that explains what the event means and what possible causes are.

Select the Advanced tab.
In the Log Policy field, select the Log File Monitoring policy that the agent will use to collect the log message.

Enter values in the following fields to specify specific text that must appear in the log message for the event policy to trigger:

First Match String. A string used to match against the originating log message. To match this event policy, the text of a log message must match the value you enter in this field. Can be any combination of alphanumeric characters. Expression matching in SL1 is case-sensitive.

Second Match String. A secondary string used to match against the originating log message. To match this event policy, the text of a log message must match the value you enter in this field and the value you entered in the First Match String field. This field is optional.

The Match Logic field specifies whether SL1 should process First Match String and Second Match String as simple text matches or as regular expressions.

Optionally, supply values in the other fields on this page. For more information on the remaining fields, as well as the Suppressions tab, see the Events section.
Click Save.