Daily Health Tasks

The tasks in this section help you monitor the health of your SL1 system. You should perform these tasks daily (or more frequently, if you require) to gather information about the overall status of your SL1 system and to maintain operational stability.

The following video explains the daily maintenance process and how to troubleshoot common issues related to that process:

Use the following menu options to navigate the SL1 user interface:

To view a pop-out list of menu options, click the menu icon ().
To view a page containing all of the menu options, click the Advanced menu icon ().

What is a Healthy SL1 System?

The following table presents a broad list of focus areas for SL1 health that are important to track for all SL1 systems. Where a specific automated check is available, it is included in the table.

Focus Area	Check	Background	Specific Operation	Result
Patch level	Version has been updated within last 12 months	Software updates are released at least quarterly and include security and stability improvements.	Quarterly manual review of available and planned software updates from ScienceLogic	Plan to keep all SL1 platforms updated within 12 months of the latest release.
Response time	API response times for standard requests are within five seconds	API response times are highly dependent on the size of the response, however all SL1 systems should respond to a simple request without delay.	/api/organization/0	Returns a response set for the “system” organization (id 0).
Central storage capacity	At least 20% of local database storage is free and available for new data	The InnoDB database file will auto-expand but never shrink. When data is removed from the database, space is made available for future use.	Support PowerPack - Support: InnoDB Size	Built-in alerts for the Support PowerPack default to 80% used for major and 90% for critical.
CPU Consumption	System CPU utilization and load average	Both Collectors and Databases can become CPU constrained, leading to unhealthy performance characteristics.	Operating system measures of CPU utilization and load average	Load average should be at or below the system’s available core count. CPU utilization of a 5 min collection should not be above 80%, 70% if hyperthreading is enabled.
Memory Consumption	Avoid swap usage	The MariaDB database will make use of available memory for caching purposes over time, but no SL1 system should require regular swapping, which can lead to extremely poor performance.	Operating system measure of swap usage	Swap usage < 50%
Performance Data Processing	The central system is keeping up with all collection processing.	It is normal to have some backlog of “MF” data, a busy system may normally have 10,000 rows or more between each processing cycle, but they should be completely processed within each cycle (backlog should not build).	Built-in MF rows-behind compared with MF object processing rate	Backlog time < 1 processing cycle
Event Processing	The central system is keeping up with all event processing.	It is normal to have some backlog of “HF” data. A busy system may normally have 10,000 rows or more between each processing cycle, but these rows should be completely processed within each cycle (backlog should not build)	Built-in HF rows-behind compared with HF object processing rate	Backlog time < 1 processing cycle
Run Book Automation (RBA)	The central system is keeping up with all RBA processing.	The built-in RBA engine supports parallel execution and queuing of operations. This can be critical for time-sensitive notification and integration with external systems.	Built-in alerting in the RBA scheduler will notify if the system is falling behind.	No critical events starting with the following phrase: “The automation engine is still processing…”
Performance Data Collection	Collection of data is completing as scheduled.	Collection that is unbalanced or overloaded, or target devices that are misconfigured or unresponsive can result in collection not completing successfully.	SL1 Operational Insights PowerPack	Check for occurrences of “sigterming” collection. The Operational Insights PowerPack makes this easy to navigate using a dashboard.
Asynchronous Message Processing	Message collection is keeping up with asynchronous syslog and SNMP trap messages.	Data Collectors, Message Collectors, and All-in-One appliances receive	Built-in alert for suppressing of messages from “spamming” devices	By default SL1 will suppress messages from devices generating at a rate of > 25/sec/device with a built-in alert.
System Maintenance	Daily maintenance tasks are completing normally	The primary daily maintenance task (scheduled nightly outside of core business hours) is to prune old data from the SL1 database, which is an essential activity for long term health.	Regular check of the system log	Daily maintenance tasks not being terminated due to an incomplete status.
System Backup	Backups completing per schedule.	SL1 supports both configuration-only and full backups. Both should be used since they support different recovery models	Regular check of the system log	The system log will show reports of backup completion and duration.

SL1 Self-Healing

For the SL1 classic User Interface, SL1 provides three self-healing jobs that are performed on the Database Server or All-In-One Appliance. These jobs:

automatically set "s-em7-core:s-em7-core" as the owner and group for the file "silo.log"
automatically restart the process em7_patch_manager if it is stuck in deactivating mode
automatically kill queries that have run for longer than 1 hour and logs each query

SL1 Self-Monitoring

SL1 includes a self-monitoring PowerPack that is included by default with SL1 releases. This PowerPack is called the ScienceLogic Support Pack PowerPack.

Version 103 of the ScienceLogic Support Pack PowerPack replaced the "Support: MySQL Performance" Dynamic Application with the "Support: MariaDB Performance" Dynamic Application.

This self-monitoring PowerPack:

ensures that all SL1 appliances are monitored and running the same version of SL1
ensures that all SL1 appliances appear in the Device Manager page (Registry > Devices > Device Manager)
ensures that configuration files on SL1 appliances are kept in sync
ensures that configuration files on SL1 clusters are kept in sync
monitors the health and configuration of MariaDB
generates alerts when system resources like disk-space are approaching capacity
monitors DNS entries for Database Servers and All-In-One appliances
monitors long-running processes and queries
monitors out-of-memory conditions
ensures that CRM and corosync configurations are up to date
ensures that configuration files match among HA Database Servers
monitors crucial SL1 processes like datapull and config_push
monitors "rows behind" situations and generates alerts
monitors and fixes backlogs of email messages
generates alerts if any of the following files differ:

/etc/my.cnf.d/silo_mysql.cnf
/etc/silo.conf
/etc/siteconfig/mysql.siteconfig
/etc/siteconfig/siloconf.siteconfig

collects telemetry related to Device Groups

Monitoring System Events

To view the entries in the System Logs:

Go to the System Logs page (System > Monitor > System Logs).

On the System Logs page, pay special attention to any log entry tagged as Critical or Major. These entries might require additional diagnostics.
For each log entry, the System Logs page displays:

Date. Date and time the log entry was generated.
Module. Name of the appliance that generated the log entry.
Severity. Specifies the severity assigned to the log entry. The choices are:

Healthy
Notice
Minor
Major
Critical

Message. Descriptive text included in the log entry.

Searching the System Logs

When viewing the System Logs, you might want to sort the entries by date or by log message. This is helpful when you want to view information about a specific occurrence of a system event.

You can also filter the list of logs by appliance, severity, and date.

To search the system logs:

Go to the System Logs page (System > Monitor > System Logs).

The first two filters at the top of the System Logs page let you filter the list of logs by appliance, severity, and date. Select a sort type from the first filter (Appliance, Severity, or From Date) and then type or select filter criteria in the next filter to the right.
Click the Search button to see the results of the filters.
The next two search fields at the top of the System Logs page allow you to search for log entries by message, date, or module.

Search where. Specifies the parameter you want to search by. You can select from the following:

Search Message. Searches all log entries for those that match the text that you enter in the regular expression field.
Search Module ID. Searches all log entries for those that have the same module ID text as that entered in the regular expression field.
Search Date = (Y-m-d H:i:s). Searches all log entries for those that have a date and time that is equal to the date entered in the regular expression field.
Search Date > (Y-m-d H:i:s). Searches all log entries for those that have a date and time that is later than the date entered in the regular expression field.
Search Date Like (Y-m-d H:i:s). Searches all log entries for those that have a date and time that is similar to the date entered in the regular expression field.
Search Date Like != (Y-m-d H:i:s). Searches all log entries for those that have a date and time that is not similar to the date entered in the regular expression field.

regular expression. In this field you manually enter the text to search for. You can use the following special characters in this field:

* Match zero or more characters preceding the asterisk. For example:

"dell*" would match "dell", "dell2650", "dell7250" and "dell1700N".

"*dell*" would match "mydell", "dell", "dell2650", "dell7250" and "dell1700N".

% Match zero or more characters preceding the asterisk. This special character behaves in the same way as the asterisk.

When you click the Search button, the System Logs page will be refreshed and will display only the log entries that match the search parameters.

Deleting Entries from the System Logs

To save space, you might want to remove some or all log entries from the system log.

There are two ways to delete entries from the System Logs page:

Go to the System Logs page (System > Monitor > System Logs).
In the System Logs page, click the Purge button to delete all entries from the System Logs.

Or:

Go to the System Logs page (System > Monitor > System Logs).
In the System Logs page, highlight each entry you want to delete. To select multiple entries, right-click while holding down the <Ctrl> key.
Click the Delete button to delete all the selected entries from the System Logs.

Monitoring System Processes

The System Processes page (System > Monitor > Admin System Processes or System > Monitor > System Processes in the classic user interface) allows you to view read-only information about the execution of SL1's system processes. System Processes gather, manipulate, and publish the data used in SL1. These system processes can be configured and debugged in the Process Manager page (System > Settings > Admin Processes).

If you work frequently with ScienceLogic Support, ensure that you verify periodically which processes are running in debug mode, and disable debug mode as soon as is reasonable. This will reduce the noise generated to your log files and will reduce load on the /var/log/em7 partition on the central database.

Viewing the List of System Processes

To view the list of system processes for all appliances:

Go to the System Processes page (System > Monitor > Admin System Processes).
The System Processes page displays the following for each process:

Appliance. The appliance where the process ran or is currently running. This field will contain the device name of the appliance.
Process. Name of the process.
ID. Unique numeric ID automatically assigned to the process by SL1.
Start Time. Date and time at which the process started running.
End Time. Date and time at which the process stopped running.
Duration. Amount of time, in hours, minutes, and seconds, for which the process ran.
Frequency. Frequency with which SL1 launches the process. Possible values are:

Asynchronous. The process is launched in response to a system event or user request. Asynchronous events display a value of "-1" (negative one) in this column.
Always. The process always runs while SL1 is running. Always running processes display a value of "0" (zero) in this column.
The process runs at intervals in minutes ranging from 1 Minute to 1440 Minutes (Daily).

Percent. Percent of Run Length (defined in the Process Manager page) currently in use by the process.
Instances. This field is not currently in use.
Max Instances. Maximum number of instances of the process that have run in parallel.
Processed. Number of records processed by this run of the process.
Errors. Number of errors encountered by this run of the process.

Recommended System Maintenance

ScienceLogic recommends that you take the following actions on a regular basis to reduce outages as much as possible.

Daily:

Review "SL1 Operational Insights: Database Performance" classic dashboard
Review "SL1 Operational Insights: Collector Performance" classic dashboard
Review "SL1 Operational Insights: System Log Summary" classic dashboard
Review "SL1 Operational Insights: Backup History" classic dashboard

Weekly:

Run the System Status Script and review
- Address every error item in the report
- Read Knowledge Base articles
- Open tickets for issues when help from SL1 Support is needed

Monthly:

Review capacity items
- You must understand License Usage and how to project future capacity

Quarterly:

Audit User Profile access to verify that it meets expected needs
Audit DNS servers and Timeservers on all collectors

Searching and Filtering the List of System Processes

The System Processes page includes ten filters. You can filter the list of processes by one or multiple of the following parameters: appliance, process name, start time, end time, duration, frequency, percent, max instances, processed, and errors. Only processes that meet all the filter criteria will be displayed in the System Processes page.

You can filter by one or more of the following parameters. The list of system processes is dynamically updated as you select each filter.

For eight of the filters, you must enter text to match against. The user interface will search for processes that match the text, including partial matches. Text matches are not case sensitive. You can use the following special characters in each filter:

, Specifies an "or" operation. For example:

"dell, micro" would match all values that contain the string "dell" OR the string "micro".

! Specifies a "not" operation. For example:

"!dell" would match all values that do not contain the string "dell".

Appliance. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching appliance name.
Process. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching process name.
ID. You can enter text to match, and the System Processes page will display only processes that have a matching ID.

Start Time. Only those processes that match all the previously selected fields and have the specified start date and time will be displayed. The choices are:

All. Display processes with all start dates and times.

Last Minute. Display only processes that started within the last minute.
Last Hour. Display only processes that started within the last hour.
Last Day. Display only processes that started within the last day.
Last Week. Display only processes that started within the last week.
Last Month. Display only processes that started within the last month.
Last Year. Display only processes that started within the last year.

End Time. Only those processes that match all the previously selected fields and have the specified end date and time will be displayed. The choices are:

All. Display processes with all end dates and times.
Last Minute. Display only processes that ended within the last minute.
Last Hour. Display only processes that ended within the last hour.
Last Day. Display only processes that ended within the last day.
Last Week. Display only processes that ended within the last week.
Last Month. Display only processes that ended within the last month.
Last Year. Display only processes that ended within the last year.

Duration. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching duration.
Frequency. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching frequency.
Percent. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching percent.
Instances. This field is not currently in use. It is not recommended to filter the System Processes by this field.
Max Instances. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching number of Max Instances.
Processed. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching number of records processed.
Errors. You can enter text to match, including special characters, and the System Processes page will display only processes that have a matching number of errors.

Monitoring the Status of Each Appliance

The Appliance Manager page (System > Settings > Appliances) provides for global appliance configuration and management for your entire system or stack. This includes collector group and load distribution, version information, license status, and other items that are important when you upgrade.

From the Appliance Manager page, you can also access the Web Configuration Utility for each ScienceLogic appliance by clicking the toolbox icon(), or you can access the database administration tool for each Database Server or All-In-One Appliance by clicking the gear icon ().

For each Database Server, Data Collector, and Message Collector, you can click on the magnifying-glass icon () to view the output of the system status script for that appliance.

During upgrade, table cells will highlight known, pending action items that must be done to successfully complete an upgrade, such as highlighting an SL1 appliance that is running a different version of SL1 than the Database Server.

This page is useful for ensuring that every Data Collector is assigned to a Collector Group before you begin an upgrade. In some cases, the Data Collector might be assigned to an empty Collector Group, if the collector is new.

You can also use this page to ensure that Data Collector load is near or below the system requirements for each collector.

To edit and view information about an SL1 appliance:

Go to the Appliance Manager page (System > Settings > Appliances).
Locate the SL1 appliance you want to edit. Click its wrench icon (). The fields in the top pane are populated with values from the selected SL1 appliance.
You can edit one or more of the following fields:

Host Name. Name of the appliance.
IP Address. Primary IP address for the appliance.
Model Type/Module Type. Type of appliance. If an appliance is added with the wrong appliance type, SL1 generates a critical error to notify the user. The types include:

All In One Server
Database
Administration Portal
Data Collection Unit
Message Collection Unit

The combination appliance with a Database Server and an Administration Portal on a single appliance will appear with Module Type of Database. The combination appliance with a Message Collection Unit and a Data Collection Unit will appear with Module Type of Data Collection Unit.
Integration Server (SL1PowerFlow)

Description. Description of the appliance.

You can edit two optional fields for Data Collector or Message Collector.

DB User. User name that can access the MariaDB database on the Data Collector or Message Collector.
DB Password. Password that allows access the MariaDB database on the Data Collector or Message Collector.

If you are using AWS RDS with your SL1 System, you must define the DB User and DB Password for each Data Collector or Message Collector.

ScienceLogic recommends that you vary the Data Collector and Message Collector database credentials for enhanced per-appliance security. This greatly enhances the security of your central database by disallowing a successful attack to go unnoticed on your Data Collector and then succeed without failure on the central database.

The Appliance Manager page also displays the following columns:

Collector Group. For Data Collectors and All-In-One Appliances, specifies the Collector Group associated with the appliance.
Build. Specifies the latest build installed on the appliance.

If an SL1 appliance is running a different version of SL1 than the Database Server, the corresponding cell in the Build column will be highlighted.
MariaDB. Specifies the version of MariaDB running on the All-In-One Appliance, Database Server, Data Collector, or Message Collector.
Capacity. For Database Servers, specifies the licensed capacity of the appliance.
Allocation. For Data Collectors, specifies the number of devices aligned with the appliance.
ID. Unique numeric ID, automatically assigned by the platform to each appliance in the Appliance Manager page.
Validated. Specifies whether the license is valid.
Endpoint. SL1 Agent endpoint for the Gen 1 Agent.
Needs Reboot?. Specifies whether the appliance requires reboot to add latest kernel or security updates. This column is updated every 30 minutes. Hover your mouse to determine why the reboot is required and information about kernel version, packages, and last reboot.
Task Manager Paused? . Specifies whether the task manager service (em7) is paused. This value is updated every two minutes.
Edit Date. Date the appliance's information was discovered or last edited.
Edit User. User who last edited the appliance's information.
Create Date. Date and time the appliance was registered and licensed.

To view the Web Configuration Utility for an appliance, where you can track license data, interfaces, and other device settings, click the Appliance Manager icon (). Use the same login credentials that you used to log into SL1, and close the pop-up window for the Utility when you are done.
If an SL1 appliance is running a different version of SL1 than the Database Server, that appliance is highlighted in the Appliance Manager page. The version number, if known, is listed in the Build column.
For all SL1 appliances, SL1 runs the system status script every 15 minutes. You can click the logs icon () to view the results of the latest system status script.
If you are logging in to the "sl1admin" account on an appliance, you can click the padlock () icon for that appliance to get a one-time password. For more information, see Using the sl1admin Account.
For Database Servers, you can click the gear icon () to access the phpMyAdmin interface for the Database Server. In this interface, you can view all the database tables on the Database Server.
For Data Collectors and Message Collectors, you can click the lightning bolt icon () to manually force the Database Server to send the latest configuration information.

The bomb icon () does not appear for Database Servers that are not configured for High Availability or Disaster Recovery. The bomb icon does not appear for Database Servers that are configured as the primary database in a High Availability or Disaster Recovery configuration.
Click the Save button to save any changes. Click the Save As button to save your changes to a new appliance name.

Logging in SL1 Version 11.3.0 and Later

In SL1 version 11.3.0 and later, configuration files for Rsyslog were completely updated. This new configuration gives you the option of configuring TLS to send or receive syslog message, forwarding logs to a security information and event management (SIEM) tool, filtering inbound logs, and other features. These options are described in detail in the following sub-topics.

Any existing modifications you made to your rsyslog configurations to support log forwarding, filtering, or TLS reception before SL1 version 11.3.0 will be removed. To re-configure any custom rules using the appropriate syntax, see the sub-topics below.

The following options are available:

Configure TLS to send or receive syslog messages
Forward locally generated syslogs to one or more external systems for auditing or processing
Specify alternate inbound TCP or UDP ports for listening for syslog messages
Adjust the priority filter for inbound messages
Filter or discard inbound messages

Configuring TLS Certificates

You will need to configure the TLS certificates before you can send or receive syslogs using SSL/TLS.

Requirements:

A PEM-encoded CA Certificate
A PEM-encoded Certificate
A PEM-encoded Private Key
All three files uploaded to the /etc/pki/rsyslog/ directory

To configure the TLS certificates:

Go to the console of the SL1 server or use SSH to access the server and log in as user em7admin with the password you configured during setup.
Locate the /etc/rsyslog.d/siteconfig.d/global_tls.conf file and edit the file to reference the CA Certificate, Certificate, and Private Key. Uncomment the global section if needed. For example:

# TLS certificates must be defined here when using inbound TLS or TLS forwarding.

global(

DefaultNetstreamDriver="gtls"

DefaultNetstreamDriverCAFile="/etc/pki/rsyslog/exampleCA.crt"

DefaultNetstreamDriverCertFile="/etc/pki/rsyslog/mycert.crt"

DefaultNetstreamDriverKeyFile="/etc/pki/rsyslog/mycert.key"

)
Save the file and check the configuration syntax by running the following command:

sudo rsyslogd -N1
You can optionally restart the rsyslog service:

sudo service rsyslog restart

Forwarding Local Syslog Messages to Remote Systems

Requirements:

One or more destinations that can accept syslog messages with either UDP, TCP or TCP w ith TLS
The IP address or FQDN of each destination system; the FQDN must be resolvable from the sending appliance
The port number for each destination system
The protocol for each destination system

To forward local syslog messages to remote systems:

Go to the console of the SL1 server or use SSH to access the server and log in as user em7admin with the password you configured during setup.
Locate the /etc/rsyslog.d/siteconfig.d/log_forwarding.conf file and edit the file to add the destinations.

The file contains several examples in addition to the following examples:

Example 1: If you are forwarding to a server using UDP, add the following line to the bottom of the configuration file, substituting in your IP or FQDN for the Target and your port for the Port:

action(name="UDP_Forward" type="omfwd" Target="192.0.2.5" Port="514" Protocol="udp")

Example 2: If you are forwarding a subset of high-priority messages using TCP with TLS, add the following line to the bottom of the configuration file, substituting in your IP/FQDN for the Target and your port for the Port.

if prifilt("*.err;*.emerg;*.alert;") then {

action(name="TCP_Forward" type="omfwd" Target="192.0.2.200" Port="1514" Protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon")

}
Repeat step 2 if you want to add multiple destinations. When adding multiple destinations you will need to alter the name property so it is unique. It can be as simple as TCP_Forward1, TCP_Forward2, etc… ScienceLogic strongly recommends using the UDP protocol and having no more than 2 destinations. Each destination increases the bandwidth usage and using TCP or TCP w/TLS will increase the processing overhead.
Save the file and check the configuration syntax by running the following command:

sudo rsyslogd -N1
You can optionally restart the rsyslog service:

sudo service rsyslog restart

Specifying Alternate Inbound TCP or UDP Ports

Go to the console of the SL1 server or use SSH to access the server and log in as user em7admin with the password you configured during setup.
Locate the /etc/rsyslog.d/siteconfig.d/inbound_alternates.conf file and edit the file to add the destinations. The file contains several examples in addition to the example here:

The file contains several examples in addition to the following examples:

Example 1: To accept TCP with TLS-encrypted inbound messages, add the following section (there should only be one module section in the configuration):

module(

load="imtcp"

StreamDriver.Name="gtls"

StreamDriver.Mode="1"

StreamDriver.Authmode="anon"

)

Add the input section below the module section to specify the network port you would like to use (the default 514 is already in use, and ScienceLogic does not support disabling it in rsyslog):

input(

type="imtcp"

port="1514"

ruleset="ruleset-networksocket"

)

Example 2: To accept TCP inbound messages on an alternate port, add the following section to specify the network port you would like to use (the default 514 is already in use, and ScienceLogic does not support disabling it in rsyslog):

input(

type="imptcp"

port="9514"

ruleset="ruleset-networksocket"

)

The type is correct as imptcp.
When setting up alternate input ports, do not change the ruleset line. It should always read as ruleset="ruleset-networksocket". If it does not read this, inbound messages will not be processed properly.
When opening up additional inbound ports you will also be required to allow the ports through the local firewall as well.
Save the file and check the configuration syntax by running the following command:

sudo rsyslogd -N1
You can optionally restart rsyslog now or proceed to other sections to configure inbound or outbound messages:

sudo service rsyslog restart

Adjusting the Priority Filter for Inbound Messages

The following procedure lets you adjust the priority filter for Inbound messages from Message Collectors and Data Collectors:

Go to the console of the SL1 appliance or use SSH to access the appliance and log in as user em7admin with the password you configured during setup.
Locate the /etc/rsyslog.d/siteconfig.d/inbound_message_filter.conf file and edit the file to include or uncomment the example line and adjust the priority filter to meet your needs. You can refer to the following examples, but there should only be one line to adjust the filter:

Example 1: To accept all messages, use the following line:

set $.priority_filter = prifilt("*.*");

ScienceLogic does not recommend using this setting, as it can result in message floods.

Example 2: Accept "auth" messages regardless of severity, in addition to the default priorities:

set $.priority_filter = prifilt("auth.*,*.err,*.emerg,*.alert,local7.*,local6.*,local5.*,local4.*,local3.*,local2.*,local1.*,local0.*");
Save the file and check the configuration syntax by running the following command:

sudo rsyslogd -N1
You can optionally restart rsyslog now or proceed to other sections to configure inbound or outbound messages:

sudo service rsyslog restart

Filtering or Discarding Inbound Messages

The following procedure lets you filter or discard inbound messages from Message Collectors and Data Collectors:

Go to the console of the SL1 appliance or use SSH to access the appliance and log in as user em7admin with the password you configured during setup.
Locate the /etc/rsyslog.d/siteconfig.d/inbound_message_filter.conf file and edit the file to include a block to match against the desired messages and a stop statement.

For example:

if ($programname startswith "noisy_program") then {

stop

}

This is not the recommended way to limit the inbound messages to the system. The prefered method is to alter the sending device to only send relevant syslog messages to the system. This functionality is provided for limited use, because sending large amounts of messages in to the system only to be discarded increases processing load and can impact performance. Always limit messages at the source if available.

The matching syntax is fairly flexible. Please refer to the official Rsyslog documentation at https://www.rsyslog.com/doc/master/index.html for available variables and match logic.
Save the file and check the configuration syntax by running the following command:

sudo rsyslogd -N1
You can optionally restart rsyslog now or proceed to other sections to configure inbound or outbound messages:

sudo service rsyslog restart

Sending Logs via Syslog to a Remote Server in SL1 Version 11.2.x and Earlier

This method of sending logs to a remote server was deprecated in SL1 version 11.3.0 and later. For more information on the new method of sending logs to a remote server, see Logging in SL1 Version 11.3.0 and Later.

To send logs to a syslog server or a security information and event management (SIEM) tool in SL1 version 11.2.0 or earlier:

Edit the file /etc/rsyslog.conf and include the following text to send the audit.log to rsyslog:

#audit log

$ModLoad imfile

$InputFileName /var/log/audit/audit.log

$InputFileTag tag_audit_log:

$InputFileStateFile audit_log

$InputFileSeverity info

$InputFileFacility local6

$InputRunFileMonitor
If rsyslog is not already configured to send to a remote server, configure the remote server at this time.
Restart rsyslog for the changes to take effect:

[root@server ~]# systemctl restart rsyslog

Monitoring User Actions and Events on the Audit Logs Page

The Audit Logs page (System > Monitor > Audit) provides an audit trail for SL1. The Audit Logs page displays a record of actions in SL1 that are generated by users or by managed elements. These actions are organized by organization.

Some of the actions that are logged in the Audit Logs page include:

User logins to SL1
The sl1admin user requests a one-time password
Organization name changes
The addition, editing, or deletion of elements in SL1

Entries for the addition, editing, and deletion of elements includes the affected device ID, when applicable.

The installation, editing, or uninstallation of PowerPacks, including when a PowerPack is imported or installed from Global Manager to a Stack
Manually triggered discovery sessions
Events and cleared events
Devices being set to maintenance mode or devices no longer being in maintenance mode
The unalignment of Dynamic Applications from devices and the deletion of that data
The creation, editing, or deletion of Dynamic Applications
The creation, editing, or deletion of Run Book Automation policies
The addition or deletion of Reports
The situation where a user gets an "Access Denied" page as a result of attempting to access a page for which he or she does not have permission
Asset Record changes
User-defined changes to settings on the Data Retention Settings page (System > Settings > Data Retention)
Changes to settings on the System Threshold Defaults page (System > Settings > Thresholds > System)
API requests that use a PUT, POST, or DELETE method

NOTE: By default, the Audit Logs page displays a list of actions associated with all organizations.

Viewing the List of Audit Logs

To view the list of log entries in the Audit Logs page:

Go to the Audit Logs page (System > Monitor > Audit Logs).

The Audit Logs page displays all actions that are performed by users or managed elements in SL1. For each action, the Audit Logs page displays:

Date/Time. Date and time the action occurred and the log entry was created.
Source. Source of the log entry. This usually describes where the action took place. For example, if you change the contact information for your account, an entry will be made in the audit log, and the source will be "Contact Information."
Organization. Organization associated with the action.
Message. Text of the log entry.

Searching and Filtering the List of Audit Logs

The Filter-While-You-Type fields appear as a row of blank fields at the top of the list. These fields let you filter the items that appear in the list.

The list is dynamically updated as you select each filter. For each filter, you must make a selection from a drop-down menu or type text to match against. SL1 will search for entries that match the text, including partial matches. Text matches are not case-sensitive, and you can use special characters in each text field.

By default, the cursor is placed in the first Filter-While-You-Type field. You can use the <Tab> key or your mouse to move your cursor through the fields.

You can filter by one or more of the following parameters. Only items that meet all of the filter criteria are displayed on the page.

The following describes each filter on the Audit Logs page:

Date/Time. Only those audit logs that have the specified creation date will be displayed. The choices are:

All. Display all audit logs that match the other filters.

Last Minute. Display only audit logs that have been created within the last minute.
Last Hour. Display only audit logs that have been created within the last hour.
Last Day. Display only audit logs that have been created within the last day.
Last Week. Display only audit logs that have been created within the last week.
Last Month. Display only audit logs that have been created within the last month.
Last Year. Display only audit logs that have been created within the last year.

Source. You can enter text to match, including special characters, and the Audit Logs page will display only audit logs that have a matching source.
Organization. You can enter text to match, including special characters, and the Audit Logs page will display only audit logs that have a matching organization.
Message. You can enter text to match, including special characters, and the Audit Logs page will display only audit logs that have a matching message.

Generating Reports on Audit Logs

You can export the entries on the Audit Logs page as one of the following report types:

Acrobat document (.pdf)

Web page (.html)
Excel spreadsheet (.xlsx)
OpenDocument Spreadsheet (.ods)
Comma-separated values (.csv)

When you create a report in the Audit Logs page, SL1 includes only those logs that appear in the current view of the page. If you filter the entries on the Audit Logs page, only those logs that meet the filter criteria and currently appear on the page will appear in the report.

To generate an audit logs report:

From the Audit Logs page, click the Report button. The Export current view as a report window appears.

In the Output Format field, select the report format type.
Click Generate.

Using auditd to Monitor Sensitive Files

As an additional security measure, the ScienceLogic Security team monitors the SL1 SaaS environment with the Linux Auditing System (auditd). The auditd system provides enhanced logging that identifies when changes are made to specific files and directories in SL1 that contain sensitive data. Starting with SL1 version 11.1.0, you can send audit logs externally to a security information and event management (SIEM) tool when needed.

The auditd logging at /var/log/audit/audit provides logging based on the configured rules that are shipped with SL1 version 11.1.0 or later. You can send syslog messages to a syslog server or a SIEM tool.

SL1 monitors specific files and directories for changes. The topics below describe what is logged by default, how to modify what is logged, and how to send logs externally.

Files Logged by Default

Starting in SL1 version 11.1.0, a file called rules.d/70_silo_watchlist.rules is created and added to the /etc/audit directory.

The following files are monitored:

-w /etc/silo.conf -p war -k silowatch

-w /etc/php.ini -p wa -k silowatch

-w /etc/yum.conf -p wa -k silowatch

-w /etc/postfix/main.cf -p wa -k silowatch

-w /etc/postfix/master.cf -p wa -k silowatch

-w /etc/nginx/nginx.conf -p wa -k silowatch

-w /etc/yum.conf -p wa -k silowatch

-w /etc/ntp.conf -p wa -k silowatch

-w /etc/my.cnf -p wa -k silowatch

-w /etc/my.cnf.d/ -p wa -k silowatch

-w /etc/php.d/ -p wa -k silowatch

-w /etc/skel/ -p wa -k silowatch

-w /etc/yum/ -p wa -k silowatch

-w /etc/yum.repos.d/ -p wa -k silowatch

-w /etc/ntp/ -p wa -k silowatch

-w /etc/chrony.d/ -p wa -k silowatch

-w /etc/chrony.keys -p wa -k silowatch

-w /etc/chrony.conf -p wa -k silowatch

Modifying the List of Files to be Logged

To add or change files that are monitored:

Log in to the SL1 appliance with the admin account and sudo to the root user.
Edit the file called /etc/audit/rules.d/70_silo_watchlist.rules. Add the following line to the file.

-w /etc/.custom_alignment.conf -p wa -k silowatch

This file is not included in SL1 version 11.1.0 by default.
Run the following command to load the files into the audit.rules file:

[root@server ~]# augenrules --load

You can add or remove specific files that you want to monitor from this list. Including the slash at the end of the monitored path (such as /etc/ntp/) will monitor the entire directory and everything stored within it.

Monitoring the Status of Data Collectors

The Collector Status page displays the status of each Data Collector and Message Collector in your system.

NOTE: This page does not appear in All-In-One Appliances.

Data Collectors retrieve data from managed devices and applications. This collection occurs during initial discovery, during nightly updates, and in response to policies defined for each managed device. The collected data is used to trigger events, display data in the user interface, and generate graphs and reports.

Message Collectors receive and process inbound, asynchronous syslog and trap messages from monitored devices. In most distributed systems, dedicated Message Collector appliances perform message collection. A single Message Collector can handle syslog and trap messages from devices that are monitored by multiple Data Collectors.

To perform collection, you must define a Collector Group and align it with at least one Data Collector. If your Collector Group includes multiple Data Collectors, you can configure the Collector Group for high-availability. For details, see the section on Collector Groups.

To ensure the health of your system, you should periodically check on the status of the Data Collectors and Message Collectors. To access the Collector Status page:

Go to the Collector Status page (System > Monitor > Collector Status).

For each Data Collector in your system, the Collector Status page displays the following:

Collector Name. Name of the Data Collector or Message Collector.
Collector ID. Unique numeric ID automatically assigned to the Data Collector or Message Collector by SL1.
Collector Address. IP address of the Data Collector or Message Collector.
Group ID. Unique numeric ID of the Collector Group associated with the Data Collector or Message Collector.
Group Name. Name of the Collector Group associated with the Data Collector or Message Collector.
Last State Change. Date and time the platform last polled the status of the Data Collector or Message Collector.
Collector State. Operating state of the Data Collector or Message Collector.