Monitoring Linux with SSH

This section describes how to configure and discover Linux devices for monitoring by SL1 using SSH and the "Linux Base Pack" PowerPack.

Use the following menu options to navigate the SL1 user interface:

• To view a pop-out list of menu options, click the menu icon ().
• To view a page containing all of the menu options, click the Advanced menu icon ().

Prerequisites for Monitoring Linux Devices with SSH

Before you can monitor Linux devices using the "Linux Base Pack" PowerPack, you must have the following information about the devices that have already been properly configured:

• IP addresses of the devices you want to monitor
• Username with an SSH key or a username with a password for the devices you want to monitor

To monitor devices with the "Linux Base Pack" PowerPack, you must do the following:

1. Configure your Linux Devices
2. Create the Credentials
3. Configure the Template
4. Discover the Linux Devices

The "Linux Base Pack" PowerPack currently supports 425 devices per Data Collector.

Configuring Linux Devices

Before creating your credentials, you must add the following permission to the sudo config file (/etc/sudoers) so the "Linux: Hardware Configuration" Dynamic Application will run without asking for the sudo password:

<username> ALL=(ALL) NOPASSWD:/usr/sbin/dmidecode

If you cannot enable DMIDECODE, you must disable the "Linux: Hardware Configuration" Dynamic Application.

To avoid error messages, check that a home directory exists for the Linux user.

To monitor Linux devices with an IPv6 address in SL1 versions prior to 12.2.4, you must create a soft link in any Data Collector that you plan to monitor a device via an IPv6 address.

To monitor Linux devices via an IPv6 address:

1. Connect by SSH to the Data Collector using your credentials.
2. Run the following command: sudo ln -s /bin/ping /bin/ping6

If this command is not applied, the Linux devices with IPv6 start to display the event "Device Failed Availability Check TCP Port (22)" and collection will stop.

Creating an SSH/Key Credential

To configure SL1 to monitor Linux devices using SSH, you must first create an SSH/Key credential. This credential allows the Dynamic Applications in the "Linux Base Pack" PowerPack) to connect with a Linux device.

To define an SSH/Key credential:

1. Go to the Credentials page (Manage > Credentials).
2. Locate the sample credential you want to use, then click its Actions icon () and select Duplicate. A copy of the credential called "Linux Example Credential- copy" appears.
3. Click the Actions icon () for the credential copy and select Edit. The Edit Credential modal page appears.

4. Supply values in the following fields:

• Name. Type a new name for your Linux credential.

• All Organizations. Toggle on (blue) to align the credential to all organizations, or toggle off (gray) and then select one or more specific organizations from the What organization manages this service? drop-down field to align the credential with those specific organizations.
• Timeout (ms). Keep the default value.
• Hostname/IP. Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
• Port. Type the port number associated with the data you want to retrieve.

  The default TCP port for SSH servers is 22.

• Username and Password. Type the username and password for an SSH or user account on the device to be monitored.
• Username and Private Key (PEM Format). Type or paste the username and SSH private key that you want SL1 to use, in PEM format.

  For PEM Keys with a passphrase, you can use the Password field to set the passphrase.

5. Click Save & Close.

Creating an SSH/Key Credential in the Classic SL1 User Interface

To configure SL1 to monitor Linux devices using SSH, you must first create an SSH/Key credential. This credential allows the Dynamic Applications in the "Linux Base Pack" PowerPack to connect with a Linux device.

To create an SSH/Key credential in the classic SL1 user interface:

1. Go to the Credential Management page (System > Manage > Credentials).
2. Locate the Linux Example Credential credential and click its wrench icon (). The Credential Editor modal page appears:

3. Supply values in the following fields:

• Credential Name. Type a new name for the credential.
• Hostname/IP. Keep the default value. SL1 will replace the variable with the IP address of the device that is currently using the credential.

• Port. Type the port number associated with the data you want to retrieve.

  The default TCP port for SSH servers is 22.

• Timeout (ms). Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
• Username. Type the username for an SSH or user account on the device to be monitored.
• Password. Type the password for an SSH user account on the device to be monitored.
• Private Key (PEM Format). Type or paste the SSH private key that you want SL1 to use, in PEM format.

  In the classic user interface, the private key field will accept only RSA formatted / styled keys to be saved. If you want to create SSH credentials with a key in the OpenSSH format, you must do so in the default SL1 user interface.

  For PEM Keys with a passphrase, you can use the Password field to set the passphrase.

4. Click the Save As button, and then click OK.

Creating a PowerShell Credential

If you are monitoring Linux devices using Windows Active Directory and the Generic Security Services API (GSSAPI), you must create a PowerShell credential. This credential allows the Dynamic Applications in the "Linux Base Pack" PowerPack to connect with a Linux device using an Active Directory user.

Before you begin monitoring with this type of credential, you must configure the following:

• Active Directory server with the Linux Machines included.
• DNS server with the Linux machines included.
• The GSSAPI option must be enabled in the /etc/ssh/sshd_config file of the target Linux machine:

GSSAPIAuthentication yes

GSSAPICleanupCredentials yes # optional

To create a PowerShell credential:

1. Go to the Credentials page (Manage > Credentials).
2. Locate the "Linux Kerberos - Example" credential, then click its Actions icon () and select Duplicate. A copy of the credential called "Linux Kerberos - Example - copy" appears.
3. Click the Actions icon () for the credential copy and select Edit. The Edit Credential modal page appears.
4. Supply values in the following fields:

• Name. Type a new name for the credential. Can be any combination of alphanumeric characters, up to 64 characters. This field is required.
• All Organizations. Toggle on (blue) to align the credential to all organizations, or toggle off (gray) and then select one or more specific organizations from the What organization manages this service? drop-down field to align the credential with those specific organizations. This field is required.

  To learn more about credentials and organizations, see the section Aligning Organizations With a Credential.

• Timeout (ms). Time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server. For collection to be successful, SL1 must connect to the authenticating server, execute the PowerShell command, and receive a response within the amount of time specified in this field.
• Hostname/IP. Keep the default value. SL1 will replace the variable with the IP address of the device that is currently using the credential.

• Port. Type the port number associated with the data you want to retrieve; it will be used to authenticate by SSH using the GSSAPI option. The default TCP port for SSH servers is 22.
• Username. Type the Active Directory username for an SSH on the device to be monitored.

  If the option use_fully_qualified_names is enabled in the target Linux machine, you must type the username in the credential, including the domain. For example: user@DOMAIN.COM

• Password. Type the Active Directory password for an SSH on the device to be monitored.
• Account Type. Select Active Directory.
• Use SSL (HTTPS) / Encrypted. Select whether SL1 will communicate with the device using an encrypted HTTP or HTTPS connection:
  • Toggle on (blue) if SL1 will communicate with the device using an encrypted connection over HTTPS. If toggled on, when communicating with the Windows server, SL1 will use a local user account with authentication of type "Basic Auth". You must then use HTTPS and can use a Microsoft Certificate or a self signed certificate.

    In SL1 versions prior to 12.3.7, this field is labeled Encrypted. In versions 12.3.7 and above, it is labeled Use SSL (HTTPS).

    In SL1 versions 11.3.0 and later, a newer Kerberos library is used that allows for message encryption over HTTP. This feature is on by default and may eliminate the need for you to configure an HTTPS certificate depending on your security requirements.

  • Toggle off (gray) . The credential is encrypted over HTTP rather than HTTPS.
• Validate Certificate (when HTTPS is used). This field is visible when the Use SSL (HTTPS) toggle field is enabled for the connection and enables you to select whether a certificate is validated for the credential. Choices are Ignore or Validate.
• Active Directory Host/IP. Type the hostname or IP address of the Active Directory server that will authenticate the credential.
• Active Directory Domain. Type the domain where the monitored Windows device resides.
• Message Encryption Setting. Select whether Kerberos packages sent over PowerShell Remoting Protocol (PSRP) or Windows Remote Management (WinRM) are encrypted. Choices are Auto, Never, or Always.
• PowerShell Proxy Hostname/IP. If you use a proxy server in front of the Windows devices you want to communicate with, type the fully-qualified domain name or the IP address of the proxy server in this field.

4. Click Save & Close.

Creating a PowerShell Credential in the Classic SL1 User Interface

To create a PowerShell credential in the classic SL1 user interface:

1. Go to the Credential Management page (System > Manage > Credentials).
2. Locate the "Linux Kerberos - Example" credential and click its wrench icon (). The Credential Editor modal page appears.

3. Supply values in the following fields:

• Credential Name. Type a new name for the credential.
• Hostname/IP. Keep the default value. SL1 will replace the variable with the IP address of the device that is currently using the credential.
• Port. Type the port number associated with the data you want to retrieve; it will be used to authenticate by SSH using GSSAPI option. The default TCP port for SSH servers is 22.
• Timeout (ms). Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
• Username. Type the Active Directory username for an SSH on the device to be monitored.

If the option use_fully_qualified_names is enabled in the target Linux machine, you must type the username in the credential, including the domain. For example: user@DOMAIN.COM

• Password. Type the Active Directory password for an SSH on the device to be monitored.
• Active Directory Hostname/IP. Type the Active Directory hostname, IP, or fully qualified domain name (FQDN).
• Domain. Type the domain of the network.

4. Click the Save As button, then click OK.

Configuring the Linux Device Template

A device template allows you to save a device configuration and apply it to multiple devices. You must use the "Linux: Dynamic Applications Template" device template in the discovery session to align all of the PowerPack's Dynamic Applications.

NOTE: When using the device template, ensure that only Linux devices will be discovered. Any device found during discovery will cause SL1 to apply the template to the device, resulting in Linux Dynamic Applications aligning to non-Linux devices.

To configure the Linux device template:

1. Go to the Configuration Templates page (Devices > Templates, or Registry > Devices > Templates in the classic SL1 user interface).

2. Locate the "Linux: Dynamic Applications Template" device template and click its wrench icon (). The Device Template Editor page appears.
3. Click the Dyn Apps tab. The Editing Dynamic Application Subtemplates page appears.

4. Click the "Linux: Template Discovery" Dynamic Application listed in the Subtemplate Selection section on the left side of the page and then click the Credentials field label to enable editing.
5. Select the Linux credential you created in the Credentials field. Repeat this step for all Dynamic Applications. All Dynamic Applications should be aligned to the credentials you created.
6. Enter a new name for the template in the Template Name field.
7. Click Save As.
8. Optionally, you can use the template to pre-configure Process policies and TCP/IP Port policies. To do this while configuring the template, click the Port Policies or the Proc Policies tabs and fill out the relevant fields for your policy. For more information on creating port monitoring policies and process monitoring policies with the device template, see the Creating a Device Template section.

You must rename the sample templates and click Save As to save it. If you do not rename the device template, then your device template will be overwritten the next time you upgrade the "Linux Base Pack" PowerPack.

Configuring the Linux: IC Port Performance Dynamic Application

To use the "Linux: IC Port Performance" Dynamic Application, you will need to create a TCP/IP Port monitoring policy after running the discovery session. To create the TCP/IP Policy:

1. After running your discovery session, go to the TCP/IP Port Monitoring page (Registry > Monitors > TCP-IP Ports).
2. Click the Create button to open the Create New TCP/IP Port Policy page.
3. In the Create New TCP/IP Port Policy page, fill out the following fields:

• Select IP Device. Select the Linux device with the ports you want to monitor.
• Port/Service. Select the port you want to monitor from the drop-down menu.
• Click Save.

4. You will see the ports monitored in the Performance tab of the Device Summary page.

Discovering Linux Devices

To discover Linux devices, perform the following steps:

1. On the Devices page () or the Discovery Sessions page (Devices > Discovery Sessions), click the Add Devices button. The Select page appears:

2. Click the Unguided Network Discovery button. Additional information about the requirements for discovery appears in the General Information pane to the right.

3. Click Select. The Add Devices page appears.
4. Complete the following fields:

• Name. Type a unique name for this discovery session. This name is displayed in the list of discovery sessions on the Discovery Sessions tab.
• Description. Optional. Type a short description of the discovery session. You can use the text in this description to search for the discovery session on the Discovery Sessions tab.
• Select the organization to add discovered devices to. Select the name of the organization to which you want to add the discovered devices.

5. Click Next. The Credentials page of the Add Devices wizard appears:

6. On the Credentials page, locate and select the SSH/Key credential you created for the Linux devices.

7. Click Next. The Discovery Session Details page of the Add Devices wizard appears:

8. Complete the following fields:

• List of IPs/Hostnames. Type the IP addresses for the Linux devices you want to monitor.

• Which collector will monitor these devices?. Select an existing collector to monitor the discovered devices. Required.
• Run after save. Select this option to run this discovery session as soon as you save the session.

In the Advanced options section, click the down arrow icon () to complete the following fields:

• Discover Non-SNMP. Enable this setting.
• Model Devices. Enable this setting.

• Select Device Template. Select the device template that you configured.

9. Click Save and Run if you enabled the Run after save setting, or Save and Close to save the discovery session. The Discovery Sessions page (Devices > Discovery Sessions) displays the new discovery session.
10. If you selected the Run after save option on this page, the discovery session runs, and the Discovery Logs page displays any relevant log messages. If the discovery session locates and adds any devices, the Discovery Logs page includes a link to the Device Investigator page for the discovered device.

Discovering Linux Devices in the SL1 Classic User Interface

To discover Linux devices using a classic discovery session:

1. Go to the Discovery Control Panel page (System > Manage > Classic Discovery or System > Manage > Discovery in the classic user interface).
2. In the Discovery Control Panel, click the Create button.

3. The Discovery Session Editor page appears. On this page, define values in the following fields:

• IP Address Discovery List. Type the IP addresses for the Linux devices you want to monitor, separated by a comma.

• Other Credentials. Select the SSH/Key credential you created for the Linux devices.
• Initial Scan Level. Select 0. Model Device Only.
• Discover Non-SNMP. Select this checkbox.
• Model Devices. Select this checkbox.
• Apply Device Template. Select the device template that you configured.

4. Optionally, you can enter values in the other fields on this page. For more information about the other fields on this page, see the Discovery & Credentials section.

5. Click Save to save the discovery session and then close the Discovery Session Editor window.
6. The discovery session you created appears at the top of the Discovery Control Panel page. Click its lightning-bolt icon () to run the discovery session.
7. The Discovery Session window appears. When the Linux devices are discovered, click their device icons () to view the Device Properties pages for the Linux devices.

Configuring Dynamic Applications for Monitoring

Process Monitoring with the Linux Base Pack

You can utilize the "Linux Base Pack" PowerPack for process monitoring in SL1. To learn more about system processes and creating system process monitoring policies, see the Monitoring System Processes section .

Configuring Collection Frequency for Linux IC Dynamic Applications

The Linux IC Dynamic Applications use results from a different command from the rest of the Dynamic Applications in the PowerPack. The results of the command create a list of filesystems mounted on the target Linux machine that is updated every two hours.

To change the collection frequency of the "Linux: IC Filesystem Inventory" Dynamic Application:

1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
2. Search for the "Data Collection: Host Filesystem Inventory" process and click its wrench icon ().
3. In the Process Editor window, use the Frequency drop-down field to select a new frequency.
4. Click the Save button.

To change the collection frequency of the "Linux: IC Filesystem Performance" Dynamic Application:

1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
2. Search for the "Data Collection: Filesystem statistics" process and click its wrench icon ().
3. In the Process Editor window, use the Frequency drop-down field to select a new frequency.
4. Click the Save button.

To change the collection frequency of the "Linux: IC Detail" Dynamic Application:

1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
2. Search for the "Data Collection: SNMP Detail" process and click its wrench icon ().
3. In the Process Editor window, use the Frequency drop-down field to select a new frequency.
4. Click the Save button.

Unhiding Linux File Systems

In theDevice Hardware page (Devices > Hardware), you can see view the size of the file system, the mount point with the name of the mounted file system, the format of the file system, and whether or not the file system is hidden.

To unhide the file system:

1. Go to the Device Hardware page (Devices > Hardware).
2. Find the file system you want to hide and select its checkbox.
3. In the Select Actions menu, select UNHIDE File systems.
4. Click the Go button to apply your changes.
5. Click the graph icon () next to the file system to open the Device Summary page.
6. Click the Performance tab.
7. You will see the unhidden file system listed in the left pane.

Configuring Linux File System Thresholds

To change the file system threshold:

1. Go to the Device Hardware page (Devices > Hardware).
2. Find the file system you want to hide and select its checkbox.
3. In the Select Actions menu, select UNHIDE File systems.
4. Click the Go button to apply your changes.
5. Click the wrench icon () next to the file system to open the Device Properties page.
6. Click the Thresholds tab.
7. In the Device Thresholds page, scroll down to the File System Thresholds section.
8. Find the threshold you want to edit and drag the sliders to adjust the threshold(s).
9. Click Save to save the threshold(s).

Aligning the Linux: SSH Cache Worker Dynamic Application

If you are upgrading the "Linux Base Pack" PowerPack from version 110 or earlier, you must align the "Linux: SSH Cache Worker" Dynamic Application to continue monitoring. This Dynamic Application acts as a cache producer for all of the Dynamic Applications in the PowerPack.

Devices discovered through a discovery session with the "Linux: Configuration Discovery" Dynamic Application aligned will automatically align with the "Linux: SSH Cache Worker" Dynamic Application in the next poll. However, if the Dynamic Applications were aligned using a template, you will need to set up the "Linux: SSH Cache Worker" Dynamic Application manually.

To align the "Linux: SSH Cache Worker" Dynamic Application using a template:

1. Create a new template adding the "Linux: SSH Cache Worker" Dynamic Application and credential. To do so:

• Go to the Device Template page (Registry>Devices>Template) and click Create. The Device Template Editor modal opens.
• In the Template Name field, enter a template name.
• On the Dyn Apps tab, click Add New Dynamic App Sub-Template in the left Subtemplate menu.
• In the Dynamic Application drop-down field, select Linux: SSH Cache Worker.
• In the Credentials drop-down field, select ssh-cred.
• Click Save.

2. Apply the template to align the "Linux: SSH Cache Worker" Dynamic Application to multiple devices.

• Go to the Device Manager page (Registry>Device Manager) and select the checkboxes of the devices you want to align to the device template.
• In the Select Action menu at the bottom of the page, select MODIFY by Template and then click Go. The Bulk Device Configuration modal appears.

3. In the Template field, select the template you created in the previous steps and then click Apply.
4. Click Confirm to align the "Linux: SSH Cache Worker" Dynamic Application to your selected devices.

Aligning the "Linux: Large Open Files Configuration" and the "Linux: Memory Pressure Performance" Dynamic Applications

The "Linux: Large Open Files Configuration" and "Linux: Memory Pressure Performance" Dynamic Applications do not use cache to collect data and therefore, they do not use the "Linux: SSH Cache Worker" Dynamic Application.

After updating to version 113 of the PowerPack from an earlier version, these Dynamic Applications do not align automatically. If you want to monitor memory pressure or open files, ScienceLogic recommends you align them manually or use the template.

To align the Dynamic Applications using a template, you must first create a new template adding the "Linux: Large Open Files Configuration " and "Linux: Memory Pressure Performance" Dynamic Applications and a credential.

To add a new template:

1. Go to the Device Template page (Registry > Devices > Template) and click Create. The Device Template Editor modal appears.
2. In the Template Name field, enter a template name.
3. On the Dyn Apps tab, click Add New Dynamic App Sub-Template in the left Subtemplate menu.
4. In the Dynamic Application drop-down field, select Linux: Large Open Files Configuration.
5. In the Credentials drop-down field, select ssh-cred.
6. Repeat steps 3-5 for the "Linux: Memory Pressure Performance" Dynamic Application.
7. Click Save.

To apply the template to align the Dynamic Applications to multiple devices:

1. Go to the Device Manager page (Registry > Device Manager) and select the checkbox for all the devices you want to align to the device template.
2. In the Select Action drop-down menu, select MODIFY by Template and then click Go. The Bulk Device Configuration modal appears.
3. In the Template field, select the template you created earlier and then click Apply.
4. Click Confirm to align the Dynamic Applications to your selected devices.

Monitoring Large Open Files

To monitor large open files with the "Linux: Large Open Files Configuration" Dynamic Application, you must first:

• Verify the List of Open Files (lsof) is installed on the Linux device so the Dynamic Application can collect data. To verify installation, run one of the following commands in the Linux server:
  • lsof -v
  • which lsof
• The "Linux: Large Open Files Configuration " Dynamic Application collects data using elevated privileges (sudo). To function properly, the user must be added to the /etc/sudoers file as follows:
  • username ALL=(ALL:ALL) NOPASSWD: /usr/bin/lsof

Specifications

The large open files monitoring process has the following specifications:

• Excludes systemd journal processes, journald, rsyslog, and journal files.
• Filters to show only regular files.
• Displays only the top 20 large open files by default. You can change the default number in the snippet code of Dynamic Application by updating the LIMIT constant.
• Filters files according to the configuration set in the snippet code using FILTER_KEY and EXCLUDE_VALUES.
  • FILTER_KEY. The snippet argument of the collection object you want to use to filters its values.
  • EXCLUDE_VALUES. The value must be applicableto the collection object selected for filtering. You can set it for one element or many elements separated by square brackets and commas.

If LIMIT is set and some of the file names specified in EXCLUDE_VALUES are included in the default top 20 display, the Dynamic Application will collect the top 20 large open files, but you will only see the included files.

The Dynamic Application could present gaps when the device is overloaded.

Monitoring Memory Pressure

To monitor large open files with the "Linux: Memory Pressure Performance" Dynamic Application, you must first:

• Verify that the Linux kernel is version 4.20 or later as this Dynamic Application only collects from these versions. To check which kernel is currently running, enter the following command on your Linux device:

  • sudo uname -r

• Check that the Pressure Stall Information (PSI) feature is enabled by running the following command:

  • grep CONFIG_PSI/boot/config-$(uname -r)

    If the PSI is enabled, you should receive a result similar to CONFIG_PSI_DEFAULT_DISABLED=n

  The PSI feature can be disabled in the kernel configuration, even though the kernel version supports it. ScienceLogic recommends that you check if the PSI is enabled to expect collection data in the Dynamic Application.

Relationships Between Component Devices

The Dynamic Applications in the "Linux Base Pack" PowerPack can automatically build relationships between Linux servers and other associated devices:

• If you discover AppDynamics applications using the Dynamic Applications in the "Cisco: AppDynamics" PowerPack, SL1 will automatically create relationships between Linux Servers and AppDynamics Nodes.
• If you discover Dynatrace environments using the Dynamic Applications in the "Dynatrace" PowerPack, SL1 will automatically create relationships between Linux Servers and Dynatrace Hosts.
• If you discover New Relic devices using the Dynamic Applications in the "New Relic: APM" PowerPack, SL1 will automatically create relationships between Linux Servers and New Relic Servers.