Configuring Linux SSH Automations

This section describes how to use the automation policies, run book actions, and the custom action type found in the "Linux SSH Automations" PowerPack.

Linux SSH Automation Policies

The "Linux SSH Automations" PowerPack includes eight run book automation policies. Each automation policy triggers a run book action that collects diagnostic data or runs a remediation command over SSH for events associated with devices in the "Linux Automation" device group, and an action that formats the output. The "Linux Automation" device group is included in this PowerPack.

All run book actions use the "Execute Shell Commands" custom action type, which is also included in the PowerPack.

All automation policies are tied to SL1 events generated by:

• Dynamic Applications from the "Host Resources" PowerPack
• Dynamic Applications from the "Net-SNMP" PowerPack
• Dynamic Applications from the "Linux Base" PowerPack
• Interface collection
• File System Collection
• System Process Monitoring Policies

Several of the run book actions use the substitution character feature of the "Execute Shell Commands" custom action type. If an event variable is included in a command, such as %Y for the sub-entity name, the custom action type automatically replaces that variable with the value from the triggering event.

The following table shows the standard automation policies, their aligned events, and the run book actions that runs in response to the events.

The aligned events are included as part of this PowerPack and are not installed with the SL1 platform. You must install the PowerPack to obtain these events.

Automation Policy Name	Aligned Device Group	Aligned Events	Aligned Run Book Actions
Linux SSH: Illicit Process Remediation	Linux Automation	Poller: Illicit process running	Linux Illicit Process Remediation Datacenter Automation: Format Output as HTML
Linux SSH: Process Restart Remediation	Linux Automation	AKCP: DC Voltage sensor High Critical Poller: required process not running	Linux Process Restart Remediation Datacenter Automation: Format Output as HTML
Linux SSH: Run CPU Diagnostic Commands	Linux Automation	AKCP: DC Voltage sensor Low Critical Linux SSH: CPU utilization has Exceeded Threshold Host Resource: CPU has exceeded threshold Net-SNMP: CPU has exceeded threshold	Linux CPU Diagnostic Commands Datacenter Automation: Format Output as HTML
Linux SSH: Run File System Diagnostic Commands	Linux Automation	Poller: File system usage exceeded (critical) threshold Poller: File system usage exceeded (major) threshold	Linux File System Diagnostic Commands Datacenter Automation: Format Output as HTML
Linux SSH: Run Interface Error/Discard Diagnostic Commands	Linux Automation	Interface inbound discards has exceeded the Falling-High threshold Interface inbound discards has exceeded the Rising-Low threshold Interface inbound discards percentage has exceeded the Falling-High threshold Interface inbound discards percentage has exceeded the Rising-High threshold Interface inbound errors has exceeded the Falling-High threshold Interface inbound errors has exceeded the Rising-High threshold Interface inbound errors has exceeded Rising-Medium threshold Interface inbound errors has exceeded the Falling-Low threshold Interface inbound errors has exceeded the Falling-Medium threshold Interface inbound errors has exceeded the Rising-Low threshold Interface inbound discards has exceeded the Rising-Low threshold Interface inbound discards has exceeded the Rising-Medium threshold Interface inbound discards percentage has exceeded the Falling-Low threshold Interface inbound discards percentage has exceeded the Falling-Medium threshold Interface inbound discards percentage has exceeded the Rising-High threshold Interface inbound discards percentage has exceeded the Rising-Low threshold Interface inbound discards percentage has exceeded the Rising-Medium threshold Interface outbound discards has exceeded the Falling-High threshold Interface outbound discards has exceeded the Falling-Low threshold Interface outbound discards has exceeded the Falling-Medium threshold Interface outbound discards has exceeded the Rising-High threshold Interface outbound discards has exceeded the Rising-Low threshold Interface outbound discards has exceeded the Rising-Medium threshold Interface outbound discards percentage has exceeded the Falling-High threshold Interface outbound discards percentage has exceeded the Falling-Low threshold Interface outbound discards percentage has exceeded the Falling-Medium threshold Interface outbound discards percentage has exceeded the Rising-High threshold Interface outbound discards percentage has exceeded the Rising-Low threshold Interface outbound discards percentage has exceeded the Rising-Medium threshold Interface inbound errors percentage has exceeded the Falling-High threshold Interface inbound errors percentage has exceeded the Falling-Low threshold Interface inbound errors percentage has exceeded the Falling-Medium threshold Interface inbound errors percentage has exceeded the Rising-High threshold Interface inbound errors percentage has exceeded the Rising-Low threshold Interface inbound errors percentage has exceeded the Rising-Medium threshold Interface outbound errors has exceeded the Falling-High threshold Interface outbound errors has exceeded the Falling-Low threshold Interface outbound errors has exceeded the Falling-Medium threshold Interface outbound errors has exceeded the Rising-High threshold Interface outbound errors has exceeded the Rising-Low threshold Interface outbound errors has exceeded the Rising-Medium threshold Interface outbound errors percentage has exceeded the Falling-High threshold Interface outbound errors percentage has exceeded the Falling-Low threshold Interface outbound errors percentage has exceeded the Falling-Medium threshold Interface outbound errors percentage has exceeded the Rising-High threshold Interface outbound errors percentage has exceeded the Rising-Low threshold Interface outbound errors percentage has exceeded the Rising-Medium threshold Poller: Interface reporting discards Poller: Interface reporting packet errors	Linux Interface Error/Discard Diagnostic Commands Datacenter Automation: Format Output as HTML
Linux SSH: Run Interface Utilization Diagnostic Commands	Linux Automation	Interface inbound usage percentage has exceeded the Falling-High threshold Interface inbound usage percentage has exceeded the Rising-High threshold Interface inbound usage percentage has exceeded the Falling-Low threshold Interface inbound usage percentage has exceeded the Falling-Medium threshold Interface inbound usage percentage has exceeded the Rising-Low threshold Interface inbound usage percentage has exceeded the Rising-Medium threshold Interface inbound usage rate has exceeded the Falling-High threshold Interface inbound usage rate has exceeded the Falling-Low threshold Interface inbound usage rate has exceeded the Falling-Medium threshold Interface inbound usage rate has exceeded the Rising-High threshold Interface inbound usage rate has exceeded the Rising-Low threshold Interface inbound usage rate has exceeded the Rising-Medium threshold Interface outbound usage percentage has exceeded the Falling-High threshold Interface outbound usage percentage has exceeded the Falling-Low threshold Interface outbound usage percentage has exceeded the Falling-Medium threshold Interface outbound usage percentage has exceeded the Rising-High threshold Interface outbound usage percentage has exceeded the Rising-Low threshold Interface outbound usage percentage has exceeded the Rising-Medium threshold Interface outbound usage rate has exceeded the Falling-High threshold Interface outbound usage rate has exceeded the Falling-Low threshold Interface outbound usage rate has exceeded the Falling-Medium threshold Interface outbound usage rate has exceeded the Rising-High threshold Interface outbound usage rate has exceeded the Rising-Low threshold Interface outbound usage rate has exceeded the Rising-Medium threshold	Automation Utilities: Calculate Memory Size for Each Action (from the Datacenter Automation Utilities PowerPack) Linux Interface Utilization Diagnostic Commands Linux Tcpdump Command Datacenter Automation: Format Output as HTML
Linux SSH: Run Memory/Swap Diagnostic Commands	Linux Automation	Host Resource: Swap Memory has exceeded threshold Linux SSH: Swap usage has Exceeded Threshold Net-SNMP: Swap has exceeded threshold Host Resource: Physical Memory has exceeded threshold Net-SNMP: Physical Memory exceeded threshold	Automation Utilities: Calculate Memory Size for Each Action (from the "Datacenter Automation Utilities" PowerPack) Linux Memory/Swap Diagnostic Commands Linux Memory Dmidecode Command Datacenter Automation: Format Output as HTML
Linux SSH: Run System-Storage Diagnostic Commands	Linux Automation	Linux SSH: File System over usage threshold	Linux System-Storage Diagnostic Commands Datacenter Automation: Format Output as HTML

The following figure shows a file system usage threshold exceeded event with a Severity of Major on the Events page. Click the [Actions] button () for an event and select View Automation Actions to see the run book actions triggered by the events.

The results shown for this event, in the Event Actions Log, include the automation policy that ran (shown at the top of the following figure), along with the run book actions (commands) that ran. Results for each command are also displayed. The following figure shows an example of this output.

To learn more about which commands are executed by default for a given run book action, see Configuring Linux SSH Run Book Actions.

Although you can edit the automation policies described in this section, it is a best practice to use "Save As" to create a new automation policy, rather than to customize the standard automation policies.

User-initiated Automation Policies

All Linux SSH automation policies have a Policy Type of Active Events/User Initiated, which enables all of the features of the "Active Events" and the "User Initiated" policy types. As a result, these automation policies can be triggered by active events that meet the criteria in the policy, or you can manually trigger the automation.

You can run these automation policies as needed from the Devices page, the Events page, and the Service Investigator page. If there is an event policy specified in the automation policy, that event must be active for the policy to be run manually, and the policy can only be run on that event type. The same applies for the device groups list.

For these automation policies to be visible from the Tools panel in the Device Summary modal , the following three bullets must be true between the event and the automation policy configuration:

• Organization. The organization associated with the event must match the organization configured in the automation policy. Policies in the "System" organization match all organizations.
• Aligned Devices. The device for which the event is triggered must be configured as an Aligned Device in the automation policy.
• Aligned Event. The event must match one of the Aligned Events configured in the automation policy.

In most situations, you would run a user-initiated automation in response to an event that just occurred. If you have Automation PowerPacks installed on your SL1 system, the Event Actions Log window for that event might contain diagnostic information from other automations that have already run, including information that helps you determine which user-initiated automation you should run next to address the cause of the event.

To run a user-initiated automation policy, click the open icon () to open the Device Summary modal for the event and click in the Tools section. Any available user-initiated automation policy will be listed there, available to run on-demand.

Creating and Customizing Automation Policies

You can use the default run book automation policies in this PowerPack, or you can create and customize the policies as needed.

You might need to configure a run book action policy before you can add it to the automation policy. For more information, see Customizing Linux SSH Run Book Actions.

Prerequisites

Before you create an automation policy using the run book actions in the "Linux SSH Automations" PowerPack, you must determine:

• Which set of commands you want to run on a monitored device when an event occurs. The run book actions in this PowerPack can run the "Execute Shell Commands" action type with different commands and output formats. You can also create your own run book actions using the custom action type supplied in the PowerPack.
• What event criteria you want to use to determine when the run book actions will trigger, or the set of rules that an event must match before the automation is executed. This can include matching only specific event policies, event severity, associated devices, and so on. For a description of all the options that are available in the automation policies, see the Run Book Automation section.

Creating or Customizing an Automation Policy

In this PowerPack, the automation policies are disabled by default, so at the minimum you will need to set the Policy State to Enabled before you can run a policy.

To create or customize an automation policy that uses the run book actions in the "Linux SSH Automations" PowerPack:

1. Go to the Automation Policy Manager page (Registry > Run Book > Automation).

2. Click the Create button to create an automation policy, or search for an existing automation policy that you want to edit and click the wrench icon () for that policy. The Automation Policy Editor page appears:

   

3. Complete the following fields:

• Policy Name. Enter a name for the automation policy.

• Policy Type. Select whether the automation policy will match events that are active, match when events are cleared, or run on a scheduled basis. Typically, you would select Active Events in this field.

• Policy State. Specifies whether the policy will be evaluated against the events in the system. If you want this policy to begin matching events immediately, select Enabled. The default is Disabled.

• Policy Priority. Specifies whether the policy is high-priority or default priority. These options determine how the policy is queued.

• Organization. Select one or more organizations to associate with the automation policy. The automation policy will execute only for devices in the selected organizations (that also match the other criteria in the policy). To configure a policy to execute for all organizations, select System without specifying individual devices to align to.

• Align With. Select Device Groups.

• Aligned Device Groups. The "Linux Automation" device group needs to be aligned. To add the device group to the Aligned Device Groups field, select the "Linux Automation" device group in the Available Device Groups field and click the right arrow (>>).

• Aligned Actions. This field includes the actions from the "Linux SSH Automations" PowerPack. To add an action to the Aligned Actions field, select the action in the Available Actions field and click the right arrow (>>). To re-order the actions in the Aligned Actions field, select an action and use the up arrow or down arrow buttons to change that action's position in the sequence.

  You must have at least two Aligned Actions: one that runs the run book action and one that provides the output format. The actions providing the output formats are contained in the "Datacenter Automation Utilities" PowerPack, which is a prerequisite for running automations in this PowerPack.

  If you are selecting multiple collection actions that use the "Execute Shell Commands" action type, you may want to include the "Calculate Memory Size for Each Action" run book action, found in the "Datacenter Automation Utilities" PowerPack, in your automation policy.

4. Optionally, supply values in the other fields on this page to refine when the automation will trigger.

5. Click Save for a new policy, or click Save As if you are customizing an existing policy. If you modify one of the included automation policies and save it with the original name, any customizations you made to that policy will be overwritten when you upgrade the PowerPack.

Removing an Automation Policy from a PowerPack

If you have customized an automation policy from the PowerPack, you might want to remove that policy from that PowerPack to prevent your changes from being overwritten if you update the PowerPack later. If you have the license key with author's privileges for a PowerPack or if you have owner or administrator privileges with your license key, you can remove content from a PowerPack.

To remove content from a PowerPack:

1. Go to the PowerPack Manager page (System > Manage > PowerPacks).
2. Find the "Linux SSH Automations" PowerPack. Click its wrench icon ().
3. In the PowerPack Properties page, in the navigation bar on the left side, click Run Book Policies.
4. In the Embedded Run Book Polices pane, locate the policy you updated, and click the bomb icon () for that policy. The policy will be removed from the PowerPack and will now appear in the bottom pane.