Monitoring Linux with SSH

Download this manual as a PDF file

Use the following menu options to navigate the SL1 user interface:

  • To view a pop-out list of menu options, click the menu icon ().
  • To view a page containing all of the menu options, click the Advanced menu icon ().

The following sections describe how to configure and discover Linux devices for monitoring by SL1 using SSH and the Linux Base Pack PowerPack:

Prerequisites for Monitoring Linux Devices with SSH

Before you can monitor Linux devices using the Linux Base Pack PowerPack, you must have the following information about the devices that have already been properly configured:

  • IP addresses of the devices you want to monitor
  • Username with an SSH key or a username with a password for the devices you want to monitor

To monitor devices with the Linux Base Pack PowerPack, you must do the following:

  1. Configure your Linux Devices
  2. Create the Credentials
  3. Configure the Template
  4. Discover the Linux Devices

NOTE: The Linux Base Pack PowerPack currently supports 425 devices per Data Collector.

The PowerPack has been tested with the following ciphers:

(Host-key algorithms): ssh-ed25519,ecdsa-sha2-nistp521, ecdsa-sha2-nistp384, ecdsa-sha2-nistp256, rsasha2-512, rsa-sha2-256. SSH-RSA and SSH-DSS are not supported.

(MACs): hmac-sha2-256-etm@openssh.com, hmac-sha2-512-etm@openssh.com, hmac-sha2-256, hmac-sha2-512

(KexAlgorithms): curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, diffie-hellman-group18-sha512, diffie-hellman-group16-sha512, diffie-hellman-group-exchange-sha256, diffie-hellman-group14-256

By default, the “Linux: Configuration Cache (Discovery)” snippet code is disabled in the “Linux: Configuration Discovery” Dynamic Application. If you want to perform alignment and classification using the run book action policy, it will be necessary to enable the “Linux: Configuration Cache (Discovery)” snippet code before running the discovery process. This ensures that the “Linux: Configuration Discovery” Dynamic Application aligns with the devices automatically. ScienceLogic recommends that you use a template to align the Dynamic Applications.

Linux Distributions Supported by the Linux Base Pack PowerPack

Distribution Supported Versions

Ubuntu

23

22

20

CentOS

8

7

Red Hat Linux Enterprise

9

8

7

Oracle Linux Server

9

8

7

Debian GNU Linux

12

11

10

9

Fedora Server

39

38

37

36

35

Amazon Linux

Amazon Linux 2

Amazon Linux

SUSE Linux Enterprise Server

15

12

Rocky Linux

9

8

Configuring Linux Devices

Before creating your credentials, you must add the following permission to the sudo config file (/etc/sudoers) so the "Linux: Hardware Configuration" Dynamic Application will run without asking for the sudo password:

<username> ALL=(ALL) NOPASSWD:/usr/sbin/dmidecode

If you cannot enable DMIDECODE, you must disable the "Linux: Hardware Configuration" Dynamic Application.

NOTE: If you see the "Sorry, you must have a tty to run sudo" error message in your device logs, or your "Linux: Hardware Configuration" Dynamic Application is not collecting data even when configured with the "sudo dmidecode", you will need to configure the Tty Requirement in /etc/sudoers, in order to collect hardware configuration information. To do so,add the following line to the sudo config file:

 

Defaults:<username> !requiretty

 

NOTE: To collect information about password expiration, run the following command on the terminal of your Linux device (does not need sudo): chage -l $(whoami)

 

If the chage -l $(whoami) command asks for a password, you will need to disable it by editing the /etc/pam.d/chage file with the following:

 

from: auth required pam_shells.so

to: auth sufficient pam_shells.so

NOTE: To avoid error messages, check that a home directory exists for the Linux user.

To monitor Linux devices with an IPv6 address in SL1 versions prior to 12.1.3 and 12.2.4, you must create a soft link in any Data Collector that you plan to monitor a device via an IPv6 address.

To monitor Linux devices via an IPv6 address:

  1. Connect by SSH to the Data Collector using your credentials.
  2. Run the following command: sudo ln -s /bin/ping /bin/ping6

If this command is not applied, the Linux devices with IPv6 start to display the event "Device Failed Availability Check TCP Port (22)" and collection will stop.

Creating an SSH/Key Credential

To configure SL1 to monitor Linux devices using SSH, you must first create an SSH/Key credential. This credential allows the Dynamic Applications in the Linux Base Pack PowerPack) to connect with a Linux device.

NOTE: If you are on an SL1 system prior to version 11.1.0, you will not be able to duplicate the sample credential. It is recommended that you create your new credentials using the SL1 classic user interface so you do not overwrite the sample credential.

To define an SSH/Key credential:

  1. Go to the Credentials page (Manage > Credentials).
  2. Locate the sample credential you want to use, then click its Actions icon () and select Duplicate. A copy of the credential, called Linux Example Credential- copy appears.
  3. Click the Actions icon () for the credential copy and select Edit. The Edit Credential modal page appears.

An image of the Edit Credential page.

  1. Supply values in the following fields:
  • Name. Type a new name for your Linux credential.

  • All Organizations. Toggle on (blue) to align the credential to all organizations, or toggle off (gray) and then select one or more specific organizations from the What organization manages this service? drop-down field to align the credential with those specific organizations.
  • Timeout (ms). Keep the default value.
  • Hostname/IP. Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
  • Port. Type the port number associated with the data you want to retrieve.

The default TCP port for SSH servers is 22.

  • Username and Password. Type the username and password for an SSH or user account on the device to be monitored.
  • Username and Private Key (PEM Format). Type or paste the username and SSH private key that you want SL1 to use, in PEM format.

For PEM Keys with a Passphrase, you can use the "Password" field to set the Passphrase.

  1. Click Save & Close.

Creating an SSH/Key Credential in the Classic SL1 User Interface

To configure SL1 to monitor Linux devices using SSH, you must first create an SSH/Key credential. This credential allows the Dynamic Applications in the Linux Base Pack PowerPack to connect with a Linux device.

To create an SSH/Key credential:

  1. Go to the Credential Management page (System > Manage > Credentials).
  2. Locate the Linux Example Credential credential and click its wrench icon (). The Credential Editor modal page appears:

  1. Supply values in the following fields:
  • Credential Name. Type a new name for the credential.
  • Hostname/IP. Keep the default value. SL1 will replace the variable with the IP address of the device that is currently using the credential.
  • Port. Type the port number associated with the data you want to retrieve.

The default TCP port for SSH servers is 22.

  • Timeout (ms). Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
  • Username. Type the username for an SSH or user account on the device to be monitored.
  • Password. Type the password for an SSH user account on the device to be monitored.
  • Private Key (PEM Format). Type or paste the SSH private key that you want SL1 to use, in PEM format.

In the classic user interface, the private key field will only accept RSA formatted / styled keys to be saved. If you wish to create SSH credentials with a key in the OpenSSH format you must do so in the default SL1 user interface.

For PEM Keys with a Passphrase, you can use the "Password" field to set the Passphrase.

  1. Click the Save As button, and then click OK.

Creating a PowerShell Credential in the Classic SL1 User Interface

To configure SL1 to monitor Linux devices using Windows Active Directory and GSSAPI, you must first create a PowerShell credential. This credential allows the Dynamic Applications in the Linux Base PackPowerPack to connect with a Linux device using an Active Directory user.

To create a PowerShell credential:

  1. Go to the Credential Management page (System > Manage > Credentials).
  2. Locate the "Linux Kerberos - Example" credential and click its wrench icon (). The Credential Editor modal page appears:

  1. Supply values in the following fields:
  • Credential Name. Type a new name for the credential.
  • Hostname/IP. Keep the default value. SL1 will replace the variable with the IP address of the device that is currently using the credential.
  • Port. Type the port number associated with the data you want to retrieve; it will be used to authenticate by SSH using GSSAPI option. The default TCP port for SSH servers is 22.
  • Timeout (ms). Type the time, in milliseconds, after which SL1 will stop trying to communicate with the authenticating server.
  • Username. Type the Active Directory username for an SSH on the device to be monitored.
  • Password. Type the Active Directory password for an SSH on the device to be monitored.
  • Active Directory Hostname/IP. Type the Active Directory hostname, IP, or fully qualified domain name (FQDN).
  • Domain. Type the Domain of the network.
  1. Click the Save As button, then click OK.

Before you begin monitoring with this type of credential, it's necessary to configure the following:

  • Active Directory Server with the Linux Machines included.
  • DNS Server with the Linux Machines included.
  • GSSAPI option enabled in the /etc/ssh/sshd_config file of the target Linux machine.

GSSAPIAuthentication yes

GSSAPICleanupCredentials yes # optional

If the option use_fully_qualified_names is enabled in the target Linux machine, you need to type the username in the credential including the domain, for example: user@DOMAIN.COM

Configuring the Linux Device Template

A device template allows you to save a device configuration and apply it to multiple devices. You must use the Linux: Dynamic Applications Template device template in the discovery session to align all of the PowerPack's Dynamic Applications.

NOTE: When using the device template, ensure that only Linux devices will be discovered. Any device found during discovery will cause SL1 to apply the template to the device, resulting in Linux Dynamic Applications aligning to non-Linux devices.

To configure the Linux device template:

  1. Go to the Configuration Templates page (Devices > Templates or Registry > Devices > Templates in the SL1 classic user interface).
  1. Locate the "Linux: Dynamic Applications Template" device template and click its wrench icon (). The Device Template Editor page appears.
  2. Click the Dyn Apps tab. The Editing Dynamic Application Subtemplates page appears.

  1. Click the "Linux: Template Discovery" Dynamic Application listed in the Subtemplate Selection section on the left side of the page and then click the Credentials field label to enable editing.
  2. Select the Linux credential you created in the Credentials field. Repeat this step for all Dynamic Applications. All Dynamic Applications should be aligned to the credentials you created.
  3. Enter a new name for the template in the Template Name field.
  4. Click Save As.
  5. Optionally, you can use the template to pre-configure Process policies and TCP/IP Port policies. To do this while configuring the template, click the Port Policies or the Proc Policies tabs and fill out the relevant fields for your policy. For more information on creating port monitoring policies and process monitoring policies with the device template, see the Creating a Device Template section.

You must rename the sample templates and click Save As to save it. If you do not rename the device template, then your device template will be overwritten the next time you upgrade the Linux Base PackPowerPack.

Configuring the Linux: IC Port Performance Dynamic Application

To use the "Linux: IC Port Performance" Dynamic Application, you will need to create a TCP/IP Port monitoring policy after running the discovery session. To create the TCP/IP Policy:

  1. After running your discovery session, go to the TCP/IP Port Monitoring page (Registry > Monitors > TCP-IP Ports).
  2. Click the Create button to open the Create New TCP/IP Port Policy page.
  3. In the Create New TCP/IP Port Policy page, fill out the following fields:
  • Select IP Device. Select the Linux device with the ports you want to monitor.
  • Port/Service. Select the port you want to monitor from the dropdown menu.
  • Click the Save button.
  1. You will see the ports monitored in the Performance tab of the Device Summary page.

Discovering Linux Devices

To discover Linux devices, perform the following steps:

  1. On the Devices page () or the Discovery Sessions page (Devices > Discovery Sessions), click the Add Devices button. The Select page appears:

Image of the Add Devices wizard, page 1

  1. Click the Unguided Network Discovery button. Additional information about the requirements for discovery appears in the General Information pane to the right.
  1. Click Select. The Add Devices page appears.
  2. Complete the following fields:
  • Name. Type a unique name for this discovery session. This name is displayed in the list of discovery sessions on the Discovery Sessions tab.
  • Description. Optional. Type a short description of the discovery session. You can use the text in this description to search for the discovery session on the Discovery Sessions tab.
  • Select the organization to add discovered devices to. Select the name of the organization to which you want to add the discovered devices.

  1. Click Next. The Credentials page of the Add Devices wizard appears:

Image of the Add Devices wizard, page 2

  1. On the Credentials page, locate and select the SSH/Key credential you created for the Linux devices.
  1. Click Next. The Discovery Session Details page of the Add Devices wizard appears:

Image of the Add Devices wizard, page 2

  1. Complete the following fields:
  • List of IPs/Hostnames. Type the IP addresses for the Linux devices you want to monitor.

  • Which collector will monitor these devices?. Select an existing collector to monitor the discovered devices. Required.
  • Run after save. Select this option to run this discovery session as soon as you save the session.

In the Advanced options section, click the down arrow icon () to complete the following fields:

  • Discover Non-SNMP. Enable this setting.
  • Model Devices. Enable this setting.

  1. Click Save and Run if you enabled the Run after save setting, or Save and Close to save the discovery session. The Discovery Sessions page (Devices > Discovery Sessions) displays the new discovery session.
  2. If you selected the Run after save option on this page, the discovery session runs, and the Discovery Logs page displays any relevant log messages. If the discovery session locates and adds any devices, the Discovery Logs page includes a link to the Device Investigator page for the discovered device.

Discovering Linux Devices in the SL1 Classic User Interface

To discover Linux devices using a discovery session, perform the following steps:

  1. Go to the Discovery Control Panel page (System > Manage > Classic Discovery).
  2. In the Discovery Control Panel, click the Create button.

  1. The Discovery Session Editor page appears. On this page, define values in the following fields:

  • IP Address Discovery List. Type the IP addresses for the Linux devices you want to monitor, separated by a comma.

  • Other Credentials. Select the SSH/Key credential you created for the Linux devices.
  • Initial Scan Level. Select 0. Model Device Only.
  • Discover Non-SNMP. Select this checkbox.
  • Model Devices. Select this checkbox.
  • Apply Device Template. Select the device template that you configured.
  1. Optionally, you can enter values in the other fields on this page. For more information about the other fields on this page, see the Discovery & Credentials section.
  1. Click the Save button to save the discovery session and then close the Discovery Session Editor window.
  2. The discovery session you created appears at the top of the Discovery Control Panel page. Click its lightning-bolt icon () to run the discovery session.
  3. The Discovery Session window appears. When the Linux devices are discovered, click their device icons () to view the Device Properties pages for the Linux devices.

The "Linux: IC Interface Inventory" Dynamic Application runs during nightly discovery. If you want to force discovery of interfaces at a time outside of nightly discovery, run the following command on the collector:

sudo -u s-em7-core /opt/em7/bin/python /opt/em7/backend/discover_update.py

Configuring Dynamic Applications for Monitoring

Configuring Collection Frequency for Linux IC Dynamic Applications

The Linux IC Dynamic Applications use results from a different command from the rest of the Dynamic Applications in the PowerPack. The results of the command create a list of Filesystems mounted on the target Linux machine that is updated every two hours.

To change the collection frequency of the "Linux: IC Filesystem Inventory" Dynamic Application:

  1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
  2. Search for the "Data Collection: Host Filesystem Inventory" process and click its wrench icon ().
  3. In the Process Editor window, use the Frequency dropdown field to select a new frequency.
  4. Click the Save button.

To change the collection frequency of the "Linux: IC Filesystem Performance" Dynamic Application:

  1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
  2. Search for the "Data Collection: Filesystem statistics" process and click its wrench icon ().
  3. In the Process Editor window, use the Frequency dropdown field to select a new frequency.
  4. Click the Save button.

To change the collection frequency of the "Linux: IC Detail" Dynamic Application:

  1. Go to the Process Manager page (System > Settings > Admin Processes or System > Settings > Processes in the SL1 classic user interface).
  2. Search for the "Data Collection: SNMP Detail" process and click its wrench icon ().
  3. In the Process Editor window, use the Frequency dropdown field to select a new frequency.
  4. Click the Save button.

Unhiding Linux File Systems

In theDevice Hardware page (Registry > Devices > Hardware), you can see view the size of the file system, the mount point with the name of the mounted file system, the format of the file system, and whether or not the file system is hidden.

To unhide the file system:

  1. Go to theDevice Hardware page (Devices > Hardware or Registry > Devices > Hardware in the SL1 classic user interface).
  2. Find the file system you want to hide and select its checkbox.
  3. In the Select Actions menu, select UNHIDE File systems.
  4. Click the Go button to apply your changes.
  5. Click the graph icon () next to the file system to open the Device Summary page.
  6. Click the Performance tab.
  7. You will see the unhidden file system listed in the left pane.

Configuring Linux File System Thresholds

To change the file system threshold:

  1. Go to the Device Hardware page (Devices > Hardware or Registry > Devices > Hardware in the SL1 classic user interface).
  2. Find the file system you want to hide and select its checkbox.
  3. In the Select Actions menu, select UNHIDE File systems.
  4. Click the Go button to apply your changes.
  5. Click the wrench icon () next to the file system to open the Device Properties page.
  6. Click the Thresholds tab.
  7. In the Device Thresholds page, scroll down to the File System Thresholds section.
  8. Find the threshold you want to edit and drag the sliders to adjust the threshold(s).
  9. Click Save to save the threshold(s).

Aligning the Linux: SSH Cache Worker Dynamic Application

After updating to Linux Base Pack PowerPack version 111, you must align the "Linux: SSH Cache Worker" Dynamic Application to continue monitoring. This Dynamic Application acts as a cache producer for all the Dynamic Applications.

Devices discovered through a discovery session with the "Linux: Configuration Discovery" Dynamic Application aligned will automatically align with the "Linux: SSH Cache Worker" Dynamic Application in the next poll. However, if the Dynamic Applications were aligned using a template, you will need to set up the "Linux: SSH Cache Worker" Dynamic Application manually.

To align the "Linux: SSH Cache Worker" Dynamic Application using a template:

  1. Create a new template adding the "Linux: SSH Cache Worker" Dynamic Application and credential.
  • Go to the Device Template (Registry>Devices>Template) and click Create. The Device Template Editor modal opens.
  • Enter a template name in the Template Name field.
  • On the Dyn Apps tab click Add New Dynamic App Sub-Template in the left Subtemplate menu.
  • In the Dynamic Application drop-down field, select "Linux: SSH Cache Worker".
  • In the Credentials drop-down field, select ssh-cred.
  • Click Save.
  1. Apply the template to align the "Linux: SSH Cache Worker" Dynamic Application to multiple devices.
  • Go to the Device Manager (Registry>Device Manager) and select the checkbox of the devices you want to align.
  • In the Select Action menu at the bottom of the page, select MODIFY by Template. Next, click Go. The Bulk Device Configuration modal appears.
  1. In the Template field, select the template you created in the previous steps and then click Apply.
  2. Click Confirm to align the "Linux: SSH Cache Worker" Dynamic Application to your selected devices.

Relationships Between Component Devices

The Dynamic Applications in the Linux Base Pack PowerPack can automatically build relationships between Linux servers and other associated devices:

  • If you discover AppDynamics applications using the Dynamic Applications in the Cisco: AppDynamics PowerPack, SL1 will automatically create relationships between Linux Servers and AppDynamics Nodes.
  • If you discover Dynatrace environments using the Dynamic Applications in the Dynatrace PowerPack, SL1 will automatically create relationships between Linux Servers and Dynatrace Hosts.
  • If you discover New Relic devices using the Dynamic Applications in the New Relic PowerPack, SL1 will automatically create relationships between Linux Servers and New Relic Servers.