SL1 Self-Monitoring

SL1 includes various tools that enable your SL1 system to monitor the appliances within your system as well as your other SL1 stacks. This proactive system health monitoring helps prevent outages and other major issues.

Use the following menu options to navigate the SL1 user interface:

• To view a pop-out list of menu options, click the menu icon ().
• To view a page containing all of the menu options, click the Advanced menu icon ().

The Workflow for SL1 Self-Monitoring

The workflow for SL1 self-monitoring is:

1. Ensure you have the latest version of the required PowerPacks.
2. Create the appropriate credentials.
3. Discover your SL1 devices.
4. Enable connectivity on port 7707 (not required in all scenarios)
5. Align additional Dynamic Applications to your SL1 devices using device templates.
6. Ensure the correct credentials are aligned to the Dynamic Applications.
7. Configure run book automations to populate dashboards with data about your SL1 devices.

PowerPacks Required for Self-Monitoring

To self-monitor SL1, you must first have the latest versions of the following PowerPacks:

• ScienceLogic Support Pack
• Data Pull Support
• SL1: Operational Insights

ScienceLogic Support Pack

SL1 includes the "ScienceLogic Support Pack" PowerPack by default with all SL1 releases.

This PowerPack was developed by ScienceLogic Support to help you monitor your SL1 systems. It allows an SL1 stack to monitor itself and other SL1 stacks.

The "ScienceLogic Support Pack" PowerPack:

• ensures that all SL1 appliances are monitored and running the same version of SL1
• ensures that all SL1 appliances appear on the Devices page
• ensures that configuration files on SL1 appliances and clusters are kept in sync
• monitors the health and configuration of MariaDB 
• generates alerts when system resources like disk-space are approaching capacity
• monitors DNS entries for Database Servers and All-In-One Appliances
• monitors long-running processes and queries
• monitors out-of-memory conditions
• ensures that CRM and corosync configurations are up to date
• ensures that configuration files match among high-availability Database Servers
• monitors crucial SL1 processes like data pull and config_push
• monitors "rows behind" situations and generates alerts
• monitors and fixes backlogs of email messages
• generates alerts if any of the following files differ:
• /etc/my.cnf.d/silo_mysql.cnf
• /etc/silo.conf
• /etc/siteconfig/mysql.siteconfig
• /etc/siteconfig/siloconf.siteconfig
• collects telemetry related to Device Groups

The PowerPack is designed to work "out of the box" with very little configuration required. The Dynamic Applications included in the PowerPack should automatically align when you discover your SL1 appliances for self-monitoring.

For more information about the Dynamic Applications that are included in the "ScienceLogic Support Pack" PowerPack, see the following knowledge base article: https://support.sciencelogic.com/s/article/2951.

Because the PowerPack is included in SL1 installations and upgrades by default, you cannot download it from the ScienceLogic Support site. Whenever you upgrade your SL1 stack to the latest release, you should go to the PowerPack Manager page (System > Manage > PowerPacks) to check for a new version of the "ScienceLogic Support Pack" PowerPack. For more information about updating default PowerPacks, see the section on How SL1 Updates Default PowerPacks.

Data Pull Support

The "Data Pull Support" PowerPack is also included by default in all SL1 releases.

It includes several Dynamic Applications that use SNMP to collect configuration and performance data related to data pull, which is the process by which SL1 Database Servers retrieve collected data from Data Collectors and Message Collectors.

Because the PowerPack is included in SL1 installations and upgrades by default, you cannot download it from the ScienceLogic Support site. Whenever you upgrade your SL1 stack to the latest release, you should go to the PowerPack Manager page (System > Manage > PowerPacks) to check for a new version of the "Data Pull Support" PowerPack. For more information about updating default PowerPacks, see the section on How SL1 Updates Default PowerPacks.

SL1: Operational Insights PowerPacks

To self-monitor SL1, you will also need either the "SL1: Operational Insights - On Premise" or "SL1: Operational Insights - SaaS" PowerPack, depending on your system configuration:

• "SL1: Operational Insights - On-Premise" is required for MariaDB databases.
• "SL1: Operational Insights - SaaS" is required for Aurora RDS databases or SaaS stacks.

These PowerPacks include a variety of Dynamic Applications, run book actions and automation policies, dashboards, and other tools that provide additional SL1 platform health visibility using data collected by the "ScienceLogic Support Pack".

Unlike the "ScienceLogic Support Pack" and "Data Pull Support" PowerPacks, the "SL1: Operational Insights" PowerPacks are not included by default in SL1 releases. To use them, you must first download them from the ScienceLogic Support site and then install them on your SL1 system.

You can periodically check the ScienceLogic Support Site at https://support.sciencelogic.com/s/powerpacks for any newer versions of the "SL1: Operational Insights - On-Premise" or "SL1: Operational Insights - SaaS" PowerPacks.

By default, installing a new version of a PowerPack overwrites all content from a previous version of that PowerPack that has already been installed on the target system. You can use the Enable Selective PowerPack Field Protection setting in the Behavior Settings page (System > Settings > Behavior) to prevent new PowerPacks from overwriting local changes for some commonly customized fields. For more information, see the section on Global Settings.

To download and install the PowerPack:

1. Search for and download the PowerPack from the PowerPacks page (Product Downloads > PowerPacks & SyncPacks) at the ScienceLogic Support Site.
2. In SL1, go to the PowerPacks page (System > Manage > PowerPacks).
3. Click the Actions button and choose Import PowerPack. The Import PowerPack dialog box appears.
4. Click [Browse] and navigate to the PowerPack file from step 1.
5. Select the PowerPack file and click Import. The PowerPack Installer modal displays a list of the PowerPack contents.
6. Click Install. The PowerPack is added to the PowerPacks page.

If you exit the PowerPack Installer modal without installing the imported PowerPack, the imported PowerPack will not appear in the PowerPacks page. However, the imported PowerPack will appear in the Imported PowerPacks modal. This page appears when you click the Actions menu and select Install PowerPack.

Credentials Required for Self-Monitoring

To self-monitor SL1, you must ensure that you have all of the required credentials:

• The "EM7 Default V2" SNMP credential
• A copy of the "SL1 CDB" credential
• A copy of the "SL1 Collector Database" credential

All three of these credentials are included by default on your SL1 system.

For self-monitoring purposes, the "EM7 Default V2" SNMP credential does not need to be updated or copied.

However, you must make a copy of the two database credentials. This is done to ensure those credentials are not accidentally overwritten whenever you update your SL1 system. After you have made copies of those credentials, you can then assign the appropriate database IP address, username, and password to each of those copies.

If you have an on-premises SL1 system, then you should enter your MariaDB IP address, username, and password in the copied credentials. If you have a SaaS SL1 system, then you should enter your Aurora RDS IP address, username, and password in the copied credentials. SaaS customers can obtain the correct Aurora RDS credential information from ScienceLogic.

To copy and update the database credentials:

1. Go to the Credentials page (Manage > Credentials).
2. Click the Actions icon () of the credential you want to duplicate ("SL1 CDB" or "SL1 Collector Database") and then select Duplicate. A copy of the credential appears on the Credentials page.
3. To edit the copied credential, click its Actions icon () and then select Edit/Test. The Edit Credential modal appears.
4. In the Edit Credential modal, supply values in the following fields:
   • Name. Type a new name for the copied credential.
   • Database User. Type the database username associated with a valid account on the database.
     • For the "SL1 CDB" credential, enter the database username for the primary Database Server.
     • For the "SL1 Collector Database" credential, enter the database username for the Data Collector or Message Collector.
   • Password. Type the password associated with the database user you entered in the Database User field.
   • Hostname/IP. Type the database hostname or IP address.
5. After editing the fields in the Edit Credential modal, click the Save & Close button.

Discovering Your SL1 Devices

After creating your credentials, you will then use those credentials to discover your SL1 appliances.

To do so, follow the instructions found in the section Adding Devices Using Unguided Discovery.

While configuring your discovery policy, make sure to do the following:

• Select the "EM7 Default V2" SNMP credential.
• Additionally, select your copies of the "SL1 CDB" credential and/or "SL1 Collector Database" credentials, depending on whether you are discovering the Database Server, one or more Data Collectors or Message Collectors, or both the Database Server and one or more collectors.
• When listing the IP addresses that you want to discover, include the IP addresses of all SL1 appliances that you want to discover.
• Under Advanced Options, select the Discover non-SNMP toggle to turn it on.

Upon discovering your SL1 appliances, most SNMP-based Dynamic Applications should automatically be aligned to the appropriate devices.

After discovering your SL1 appliances, you can go to the Appliance Manager page (System > Settings > Appliances) to confirm that they all appear on the page and that their settings are correct. For example, the Device Name on the Devices and Device Manager (Devices > Device Manager) pages should match the Device Name on the Appliance Manager page.

Enabling Connectivity on Port 7707

On-premises Data Collector and Message Collector appliances do not allow connections to port 7707 (MariaDB) from an external device. Therefore, if you want one on-premises Data Collector to monitor another Data Collector or Message Collector, then there are some additional steps you must take to enable connectivity on port 7707.

If this scenario does not apply to your SL1 configuration, then you can skip this section.

For the instructions below, consider the following scenario:

• "Collector A" is the Data Collector or Message Collector you want to monitor.
• "Collector B" is the Data Collector on which you want to monitor Collector A.

To enable connectivity on port 7707 in this scenario:

1. Log in to the Web Configuration Utility for Collector A using any web browser supported by SL1. The address of the Web Configuration Utility is in the following format:

   https://<Collector-A-IP-address>:7700

   For AWS instances, Collector-A-IP-address is the public IP for the AWS instance for Collector A. To locate the public IP address for an AWS instance, go to AWS, go to the Instances page, and highlight an instance. The Description tab in the lower pane will display the public IP.

2. When prompted to enter your username and password, log in as the "em7admin" user with the appropriate password. After logging in, the main Configuration Utility page appears.

3. Click the Device Settings button. The Settings page appears.

4. On the Settings page, add the IP address for Collector B to the Database IP Address field, immediately after the IP address that is already listed for Collector A, separated by a comma and no space. For example, if the IP address for Collector A is 10.10.10.1 and the IP address for Collector B is 10.20.20.2, then you would enter "10.10.10.1,10.20.20.2" (without the quotation marks) in the Database IP Address field.

5. Click Save and log out of the Web Configuration Utility. With this change saved, the firewalld rules on Collector A will be updated to allow connectivity to port 7706 (MariaDB) from the IP address for Collector B.

Aligning SL1 Self-Monitoring Dynamic Applications

After discovering your SL1 appliances, you must ensure that they are properly set up for monitoring. The easiest way to do this is to apply the device templates that are included in the "ScienceLogic Support Pack," "Data Pull Support," and "SL1: Operational Insights" PowerPacks to ensure your SL1 device classes have the necessary Dynamic Applications aligned to them.

To align the appropriate self-monitoring Dynamic Applications using device templates:

1. Go to the Device Manager page (Devices > Device Manager).

2. In the Device Class column, use the filter field to search for the appropriate device class based on the table at the end of this section.

3. Select the checkbox for each row that contains the device class type that you want to modify.

   Because each of these device classes requires a different configuration, you should modify devices of only one device class at a time.

4. From the Select Action field at the bottom of the page, select MODIFY by Template and then click Go. The Bulk Device Configuration modal appears.

5. In the Template drop-down field at the top of the Bulk Device Configuration modal, select the appropriate device template based on the table at the bottom of this section.

   You can select only one device template at a time. For device classes that have multiple device templates, complete steps 5-7 for one template, then repeat the steps for the other templates.

6. Click the Dyn Apps tab and verify that a list of Dynamic Applications appear in the Subtemplate Selection pane.

7. Click Apply, and then click Confirm.

8. Repeat steps 1-7 as needed until all of the device classes needed for self-monitoring have all of the appropriate device templates applied to them, based on the following table:

Appliance Type	Device Class	Device Templates
Primary Database Server or cluster IP device	SL1 Database	Support: SL1 Active DB/VIP SL1: Database - On Premise SL1 System Processes
Administration Portal	EM7 Admin Portal	Support: SL1 Application Portal
Data Collector	SL1 Data Collector	Support: SL1 Collector SL1: Collectors SL1 System Processes
Message Collector	SL1 Message Collector	Support: SL1 Message Collector SL1 System Processes

It can take Dynamic Applications up to 15 minutes to start collecting data after they are aligned.

Aligning the Correct Credentials to Your Devices

When you align the self-monitoring Dynamic Applications to your SL1 appliances using the device templates that are included in the self-monitoring PowerPacks, those Dynamic Applications use the unedited versions of the "SL1 CDB" and "SL1 Collector Database" credentials that are included in the PowerPacks.

Now, you must edit those Dynamic Applications to use the copies you made for those credentials instead of the original versions that are included in the self-monitoring PowerPacks. This will prevent SL1 from overwriting your self-monitoring configuration whenever you install updates for those PowerPacks in the future.

To ensure the correct credentials are aligned to your SL1 devices:

1. Go to the Device Manager page (Devices > Device Manager).
2. Locate one of your SL1 devices and click its wrench icon (). The Device Administration panel appears.
3. On the Device Administration panel, click the Collections tab. The Dynamic Application Collections page appears.
4. On the Dynamic Application Collections page, locate and select the checkboxes for any Dynamic Applications that have "SL1 CDB" listed in the Credential column.
5. From the Select Action field at the bottom of the page, select your copied version of the "SL1 CDB" credential from the list and then click Go. The Dynamic Application Collections page will refresh and your copied version of the "SL1 CDB" credential should now be listed in the Credential column for the Dynamic Applications you selected.
6. Repeat steps 4 and 5 to replace the credential for any Dynamic Applications that currently have the "SL1 Collector Database" credential in the Credential column with your copy of that credential.

Configuring Run Book Automations to Populate Dashboards

The "SL1: Operational Insights" PowerPacks include two run book actions and automation policies: "SL1: Collector Data Collection" and "SL1: System Log Data Collection."

These automations collect data that is then used to populate the "Collector Performance" and "System Logs Summary" dashboards, respectively, with data relating to your monitored SL1 appliances.

SL1: Collector Data Collection

The "SL1: Collector Data Collection" automation is responsible for collecting SL1 Collector-specific data that is used in the "Collector Performance" dashboard. The automation policy is configured to trigger the "SL1: Collector Data Collection" run book action four times once per hour. The automation helps identify SL1 Collectors that have been discovered and match the following criteria required to appear in the "Collector Performance" dashboard:

• Data Collectors and Message Collectors should be discovered as managed devices.

• The discovered SL1 Collector's Device Name on the Devices and Device Manager (Devices > Device Manager) pages should match the name of the same collector from the Appliance Manager page (System > Settings > Appliances).

• The SL1 Collectors should have the following two Dynamic Applications aligned and successfully collecting data:

  • Support: File System

  • Host Resource: Configuration

• The Data Collectors should be in a collector group (CUG). This does not apply to Message Collectors.

When the automation runs, it stores the data in a custom table, which is then read and displayed by the dashboard widget.

SL1: System Log Data Collection

The "SL1: System Log Data Collection" automation is responsible for collecting system log-specific data that is used in the "System Logs Summary" dashboard. The Automation is configured to trigger the "SL1: System Log Data Collection" run book action five times once per hour.

The automation parses the top problem logs, including SIGTERMs, PoolWorker logs, and unhandled exceptions.

When the automation runs, it stores the data in a custom table, which is then read and displayed by the dashboard widget.

If the total system log count is greater than 6 million, the automation will not collect data; instead, the "System Logs Summary" dashboard will display a message stating that the log count is too high. If this occurs, delete older logs to bring the total count below 6 million.

Verifying Your Devices

Before you configure the run book automation policies required to populate the "Collector Performance" and "System Logs Summary" dashboards, you must verify that you have the necessary devices in your SL1 system. The method for doing so will vary based on whether you have an on-premises SL1 system or a SaaS SL1 system. Both methods are described below.

On-Premises SL1 Systems

To verify that you have the necessary devices:

1. Go to the Device Manager page (Devices > Device Manager).
2. In the Device Class column, use the filter field to search for devices with the "SL1 Database" device class.
3. Verify that all Database Servers are discovered.
4. If the Database Servers are not discovered, then you must discover them. Otherwise, if they are, you can skip to the next section.

SaaS SL1 Systems

To verify that you have the necessary device:

1. Go to the Device Manager page (Devices > Device Manager).
2. In the Device Name column, use the filter field to search for a device with the name "SL1 Stats".
3. Verify that the "SL1 Stats" device exists. If it does not yet exist, proceed to the next step. Otherwise, if it does, you can skip to the next section.

4. From the Actions menu, select Create Virtual Device. The Create Virtual Device modal appears.

5. Complete the following fields:

   • Device Name. Type "SL1 Stats".

   • Organization. Select System.
   • Device Class. Select Virtual Device | Dynamic App Emissary.
   • Collector. Select Self Monitoring Collector.
6. Click Add and then exit the modal.

Configuring the Run Book Automations

To configure the run book automation policies required to populate the "Collector Performance" and "System Logs Summary" dashboards:

1. Go to the Automation Policy Manager page (Registry > Run Book > Automation).
2. In the Automation Policy Name column, use the filter field to search for a policy with the name "SL1: Collector Data Collection".
3. Click the wrench icon () for the "SL1: Collector Data Collection" automation policy. The Automation Policy Editor modal appears.

4. In the Available Devices field, use the filter field to search for the appropriate device:

   • For on-premises SL1 deployments, search for the active SL1 Database Server device.
   • For SaaS SL1 deployments, search for the "SL1 Stats" device.
5. Select the device from the Available Devices field and then click the right arrow icon to move it to the Aligned Devices field.
6. Click Save and then exit the Automation Policy Editor modal.
7. Repeat steps 1-6 to search for and configure the "SL1" System Log Data Collection" automation policy.

Additional Self-Monitoring Resources

For information about additional steps you can take in SL1 to monitor your SL1 system, see the following sections:

• Daily Health Tasks
• Monitoring and Maintaining SL1
• Diagnostic Tools