Monitoring Web Content

SL1 allows you to create policies that monitor a website for specific content. This is helpful:

To determine if a website is up and running.
To determine if the connection between a webserver and a database is up and running.
To monitor system tools that can be accessed through a browser.
To monitor content on a website.

If SL1 cannot match the expression in the content policy with the text on the website, SL1 generates an event.

SL1 uses cURL to send and receive data from the website.

NOTE: Web content monitoring policies cannot monitor web sites larger than 1 MB.

Use the following menu options to navigate the SL1 user interface:

To view a pop-out list of menu options, click the menu icon ().
To view a page containing all the menu options, click the Advanced menu icon ().

This section includes the following topics:

Viewing the Web Content Monitoring Policies

You can view a list of web content monitoring policies from the Web Content Monitoring page (Registry > Monitors > Web Content). The Web Content Monitoring page displays the following information for each web content monitoring page:

Web Content Policy Name. Name of the policy.
Policy URL. The URL that SL1 will monitor for specified content.
Policy ID. Unique, numeric ID, assigned to the policy automatically by SL1.
State. Whether SL1 will monitor the external website. This column will either show "Enabled" (SL1 will monitor the external website) or "Disabled" (SL1 will not monitor the external website).
Device Name. Name of the device associated with the policy.
IP Address. IP address of the device associated with the policy. This is the IP address SL1 uses to communicate with the device.
Device Category. Device category of the device associated with the policy.
Organization. Organization for the device associated with the policy.

From the list of policies, you can select the checkbox for one or more policies and choose one of the following bulk actions from the Select Action drop-down at the bottom right of the page:

Delete Monitors. Deletes the selected policies from SL1. The associated reports (from the Device Reports > Performance tab) are also deleted.
Enable Monitors. Enables the selected policies so that SL1 can collect the data for these policies.
Disable Monitors. Disables the selected policies. SL1 will not collect the data specified in these policies.

Filtering the List of Web Content Monitoring Policies

You can filter the list of policies on the Web Content Monitoring page by one or more parameters. Only policies that meet all the filter criteria will be displayed in the Web Content Monitoring page.

To filter by parameter, enter text into the desired filter-while-you-type field. The Web Content Monitoring page searches for policies that match the text, including partial matches. By default, the cursor is placed in the left-most filter-while-you-type field. You can use the <Tab> key or your mouse to move your cursor through the fields. The list is dynamically updated as you type. Text matches are not case-sensitive.

You can also use special characters to filter each parameter.

Filter by one or more of the following parameters:

Policy Name. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies with a matching name.
Policy URL. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies that monitor URLs that match the text.
Policy ID. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies that have a matching policy ID.
State. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies that have a matching state (enabled or disabled).
Device Name. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies aligned with a device with a matching device name.
IP Address. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies aligned with a device with a matching IP address.
Device Category. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies aligned with a device with a matching device category.
Organization. You can enter text to match, including special characters, and the Web Content Monitoring page will display only policies that have a matching organization.

Defining a Web Content Policy

You can define a web content monitoring policy for a device on the Monitors tab of the Device Investigator.

To define a web content monitoring policy:

Go to the Devices page and click the Device Name of the device for which you want to define a web content monitoring policy. The Device Investigator displays.

Click the Monitors tab.

Click Create, and then select Create Web Content Policy. The Web Content Policy modal appears:

In the Web Content Policy modal, supply values in the following fields:

Select Device. From this drop-down list, select a device to align with this policy. By default, the current device is selected in this field.

NOTE: Before you can define a content policy, you must decide which managed device you want to associate with the policy. You might want to associate the policy with the web server you will be monitoring with the policy, but you aren't required to do so. The requests to the web server will be sent from an appliance, but you must still associate the policy with a device.

Policy Name. Name of the new policy. This can be any combination of letters and numbers.

State. Specifies whether SL1 should start collecting data specified in this policy from the device. Choices are:

Enabled. SL1 will collect the data specified in this policy, from the device, at the frequency specified in the Process Manager page (System > Settings > Processes) for the Data Collection: Web Content Verifier process.
Disabled. SL1 will not collect the data specified in this policy, from the device, until the State field is set to Enabled.

Port. Port on web-server to which SL1 will send queries. This is usually port 80 (the HTTP port), or port 443 (the HTTPS port).
Timeout. Specify the number of seconds after which SL1 should stop trying to connect to the server. If the timeout period elapses before SL1 can connect to the server, an event is generated.
HTTP Status Code. Specify the HTTP status code you expect to receive in the response. If any other status code is returned, SL1 will generate an event.

Proxy Server:Port. For companies or organizations that use proxy servers, enter the URL and port for the proxy server in this field. Use the format:

URL:port_number

Proxy Username:Password. For companies or organizations that use proxy servers, enter the username and password for the proxy server in this field. Use the format:

user name:password

Proxy Auth Method. For companies or organizations that use proxy servers, specify the type of authentication:

Default. By default, no authentication parameters are sent. Use this option for proxy servers that do not require authentication. However, if you supply a value in another field that requires authentication, e.g. Proxy Username:Password, the Any authentication parameter will be used.

Basic. Most widely compatible authentication across platforms. Sends a Base64-encoded string that contains a user name and password for the client. Base64 is not a form of encryption and should be considered the same as sending the username and password in clear text.
Digest. Password is transmitted as encrypted text, but the username and content of the message are not encrypted. Digest authentication is a challenge-response scheme that is intended to replace Basic authentication. The server sends a string of random data called a nonce to the client as a challenge. The client responds with a hash that includes the username, password, and nonce, among additional information.
GSS-Negotiate. Authenticates using Kerberos and the GSS-API. Kerberos authentication is faster than NTLM and allows the use of mutual authentication and delegation of credentials to remote machines.
NTLM. NT LAN Manager (NTLM) authentication is a challenge-response scheme that is a more secure variation of Digest authentication. NTLM uses Windows credentials to transform the challenge data instead of the unencoded user name and password. NTLM authentication requires multiple exchanges between the client and server. The server and any intervening proxies must support persistent connections to successfully complete the authentication
Any. Accept any type of authentication.
Any except Basic (Any Safe). Accept any type of authentication except Basic.

Location Redirect. Specifies how you want the policy to behave when it encounters an HTTP redirect in a target website. Choices are:

Default. If you selected 301, 302, or 303 in the HTTP Status Code field, the web content policy will not follow redirection by default. The default behavior for all other web content policies is to follow redirection and search for the regular expression on the website to which SL1 has been redirected.
Always Follow. When you select this option, web content policies follow redirection and search for the regular expression on the website to which SL1 has been redirected.
Never Follow. When you select this option, web content policies never follow redirection. This option allows the web content policy to search for a 301, 302, or 303 HTTP status code.

Uniform Resource Locator (URL). URL or IP address where the website is located. If the website requires login and the login is forms based (user enters username and password in the index page), include the username and password in the URL.

You can include the variable %D in this field. SL1 will replace the variable with the IP address of the device that this policy is aligned to.
You can include the variable %N in this field. SL1 will replace the variable with the name of the device that this policy is aligned to.
You can include the variable %H in this field. SL1 will replace the variable with the hostname of the device that this policy is aligned to. If the device was not discovered by hostname, SL1 will replace this variable with the IP address of the device.

Post String. If the URL is very long or requires data that cannot be transferred with a standard "GET" request (that is, data that cannot be included in the URL), you can enter a POST string in this field. The data will be sent with the cURL equivalent of an HTTP POST command. Data should be formatted as follows:

variable=value

If you are going to include more than one variable/value pair, separate each pair with an ampersand (&). For example, suppose you want to send values for the fields "Birthyear" and "Value". You could enter the following in the Post String field:

Birthyear=1980&Value=OK

If you want to include non-alphanumeric characters in the Post String field, make sure you encode the characters using appropriate URL encoding.

Cookie Value. For pages that require a cookie value to be set, enter the cookie value in this field.
Browser Emulation. Specifies how to format the query. Select the agent that is compatible with the web server.
HTTP Auth Username:Password. For websites that pop-up a dialog box asking for username and password, use this field. Enter the username and password in this field. Use the format "username:password".

HTTP Auth Method. For websites that require authentication, use one of the selected methods:

Default. By default, no authentication parameters are sent. Use this option for websites that do not require authentication. However, if you supply a value in another field that requires authentication, e.g. HTTP Auth Username:Password, the Any authentication parameter will be used.

Basic. Most widely compatible authentication across platforms. Sends a Base64-encoded string that contains a username and password for the client. Base64 is not a form of encryption and should be considered the same as sending the username and password in clear text.
Digest. Password is transmitted as encrypted text, but the username and content of the message are not encrypted. Digest authentication is a challenge-response scheme that is intended to replace Basic authentication. The server sends a string of random data called a nonce to the client as a challenge. The client responds with a hash that includes the username, password, and nonce, among additional information.
GSS-Negotiate. Authenticates using Kerberos and the GSS-API. Kerberos authentication is faster than NTLM and allows the use of mutual authentication and delegation of credentials to remote machines.
NTLM. NT LAN Manager (NTLM) authentication is a challenge-response scheme that is a more secure variation of Digest authentication. NTLM uses Windows credentials to transform the challenge data instead of the unencoded username and password. NTLM authentication requires multiple exchanges between the client and server. The server and any intervening proxies must support persistent connections to successfully complete the authentication
Any. Accept any type of authentication.
Any except Basic (Any Safe). Accept any type of authentication except Basic.

SSL Encryption. Specifies whether SL1 should use SSL when communicating with the website. If login for the website is forms-based, enable this option.

Expression Check #1. Text to search for:

If you select the Invert checkbox, SL1 will trigger an event if the text is found.
If you do not select the Invert checkbox, SL1 will trigger an event if the text is not found.

Expression Check #2. Another text string to search for:

If you select the Invert checkbox, SL1 will trigger an event if the text is found.
If you do not select the Invert checkbox, SL1 will trigger an event if the text is not found.

Referrer String. URL of the website. Some load-balanced configurations will not allow a request for a specific IP address. If you entered a specific IP address in the URL field, you can spoof a URL in this field.

Host Resolution. Hostname of the website. Some load-balanced configurations will not allow a request for a specific IP address. If you entered a specific IP address in the URL field, you can spoof a fully-qualified hostname in this field.

You can include the variable %N in this field. SL1 will replace the variable with hostname of the device that this policy is aligned to. If SL1 cannot determine the hostname, SL1 will replace the variable with the primary, management IP address for the current device.

Min Page size (Kb). Page size means the size of the page, in Kb, specified in the URL of the policy. If the returned page is not at least the size specified in this field, SL1 generates an event. This threshold triggers the event "Page size below minimum threshold."
Max Page size (Kb). Page size means the size of the page, in Kb, specified in the URL of the policy. If the returned page is larger than the size specified in this field, SL1 generates an event. This threshold triggers the event "Page size above maximum threshold."
Min Download speed (kb/s). Download speed is the speed, measured in Kb/s, at which data was downloaded from the server (specified in the policy) to SL1. If the download speed is not at least the speed specified in this field, SL1 generates an event. This threshold triggers the event "Download speed below threshold."
Max nslookup time (msec). NSlookup speed is the speed at which your DNS system was able to resolve the name of the server specified in the policy. If the lookup time exceeds the value in this field, SL1 generates an event. This threshold triggers the event "DNS hostname resolution time above threshold."
Max TCP connect time (msec). TCP connect time is the time it takes for SL1 to establish communication with the external server. In other words, the time it takes from the beginning of the HTTP request to the TCP/IP connection. If the connection time exceeds the value in this field, SL1 generates an event. This threshold triggers the event "TCP connection time above threshold."
Max Overall transaction time (msec). Overall transaction time is the total time it takes to make a connection to the external server, send the HTTP request, wait for the server to parse the request, receive the requested data from the server, and close the connection. If the overall transaction time exceeds the value in this field, SL1 generates an event. This threshold triggers the event "Total transaction time above threshold."

Click Save.

Example Web Content Policy

This policy is aligned with the device "vms".
This policy will search for the expression "Prosecutor", entered in the Expression Check #1 field, in www.msnbc.com ("http://www.msnbc.com/").

Defining a Web Content Policy in the Classic SL1 User Interface

There are two places in SL1 from which you can define a policy for monitoring web content:

From the Device Manager page (Devices > Device Manager):

In the Device Manager page, find the device that you want to associate with the monitoring policy. Select the wrench icon () for the device.
In the Device Administration panel, select the Monitors tab.
From the Create menu in the upper right, select Create Web Content Policy.

Or:

From the Web Content Monitoring page (Registry > Monitors > Web Content):

In the Web Content Monitoring page, select the Create button.

The Web Content Policy modal appears.

Editing a Web Content Policy

To edit a web content monitoring policy:

Go to the Devices page and click the name of the device for which you want to edit a monitoring policy. The Device Investigator displays.
Click the Monitors tab.
Find the policy you want to edit and click its wrench icon (). The Web Content Policy modal appears.
In the Web Content Policy modal, you can change the values in one or more of the fields described in the section on Defining a Web Content Monitoring Policy.
Click Save.

Editing a Web Content Policy in the Classic SL1 User Interface

There are two places in SL1 from which you can edit a policy to monitor web content:

From the Device Manager page (Devices > Device Manager):

In the Device Manager page, find the device that you want to associate with the monitoring policy. Click the wrench icon () for the device.
In the Device Administration panel, click the Monitors tab.
In the Monitoring Policies page, find the policy you want to edit and click its wrench icon ().

Or:

From the Web Content Monitoring page (Registry > Monitors > Web Content):

In the Web Content Monitoring page, find the policy you want to edit and click its wrench icon ().

The Web Content Policy modal appears.
In the Web Content Policy modal, you can change the values in one or more of the fields described in the section on Defining a Web Content Monitoring Policy.
Click Save.

Executing the Web Content Monitoring Policy

After creating or editing a web content monitoring policy, you can manually execute the policy and view detailed logs of each step during the execution.

NOTE: After you define a web content monitoring policy and enable the policy, SL1 will automatically execute the policy every five minutes. However, you can use the steps in this section to execute the policy immediately and see debug information about the execution of the policy.

To execute a web content monitoring policy:

Go to the Devices page and click the name of the device for which you want to execute the monitoring policy. The Device Investigator displays.
Click the Monitors tab.
Find the policy you want to run manually and click its lightning bolt icon ().
The Session Logs modal opens while the policy is executing. The Session Logs page provides detailed descriptions of each step during the execution. This is very helpful for diagnosing possible problems with a policy.

Executing the Web Content Monitoring Policy in the Classic SL1 User Interface

To execute a web content monitoring policy in the classic SL1 user interface:

In the Web Content Monitoring page (Registry > Monitors > Web Content), find the policy you want to run manually.
Click the lightning bolt icon () to manually execute the policy.
While the policy is executing, SL1 spawns a modal page called Session Logs. The Session Logs page provides detailed descriptions of each step during the execution. This is very helpful for diagnosing possible problems with a policy.

Deleting a Web Content Monitoring Policy

You can delete a web content monitoring policy from the Monitors tab of the Device Investigator. When you delete a monitoring policy, SL1 no longer uses the policy to collect data from the aligned device. Deleting a monitoring policy will also remove all data that was previously collected by the policy.

To delete a web content policy:

Go to the Devices page and click the name of the device for which you want to delete the monitoring policy. The Device Investigator displays.
Click the Monitors tab.
Find the policy you want to delete and click its bomb icon (). A confirmation prompt appears.
Click OK.

Deleting a Web Content Monitoring Policy in the Classic SL1 User Interface

You can delete one or more web content monitoring policies from the Web Content Monitoring page. When you delete a monitoring policy, SL1 no longer uses the policy to collect data from the aligned device. Deleting a monitoring policy will also remove all data that was previously collected by the policy.

To delete a web content monitoring policy in the classic SL1 user interface:

Go to the Web Content Monitoring page (Registry > Monitors > Web Content).
In the Web Content Monitoring page, select the checkbox(es) for each web content monitoring policy you want to delete. Click the checkmark icon () to select all of the web content monitoring policies.
In the Select Action menu in the bottom right of the page, select Delete Monitors.

Click the Go button to delete the web content monitoring policy.
The policy is deleted from SL1. The associated reports (from the Device Reports > Performance tab) are also deleted.

Viewing Reports on a Web Content Policy

See the section on Viewing Performance Graphs for information and examples of reports for monitoring port availability.

Viewing ASCII Page Content

From the Web Content Monitoring page, you can view the ASCII content (from the web page) that was retrieved by the web content monitoring policy. The ASCII content is returned only when the policy is manually executed.

The Content Page Dump page displays:

The regular expression(s) used in the web-content monitoring policy. SL1 searches the web content for these text strings.
The text (from the website) that was searched.

There are two ways to access the Content Page Dump page:

From the Device Manager page (Devices > Device Manager):

In the Device Manager page, find the device that you want to associate with the monitoring policy. Click the wrench icon () for the device.
In the Device Administration panel, click the Monitors tab.
In the Monitoring Policies page, find the policy you want to edit and click the page icon ().

Or:

From the Web Content Monitoring page (Registry > Monitors > Web Content):

Click the lightning bolt icon () to manually execute the policy.
In the Web Content Monitoring page, find the policy you want to edit and select its page icon ().

The Content Page Dump page appears.

In the Content Page Dump page, you can view the content that is searched and the regular expressions that SL1 searched for.
If the Web Content policy has not yet completed, this page will display the message:

"Web content verification data may take up to 5 minutes to appear. Try again later."

Viewing the Monitored Website

In some cases, you might want to view the website being monitored, directly from the user interface. To do this:

Go to the Web Content Monitoring page (Registry > Monitors > Web Content).
Find the policy for which you want to view the website. Click its globe icon ().
SL1 will spawn a new browser page and display the monitored website.