Diagnostic Tools

This section describes some diagnostic tools for troubleshooting and diagnosing problems in Skylar One (formerly SL1).

Use the following menu options to navigate the Skylar One user interface:

• To view a pop-out list of menu options, click the menu icon ().
• To view a page containing all of the menu options, click the Advanced menu icon ().

Viewing Information About ScienceLogic Processes

The Process Manager page allows you to view a list of ScienceLogic processes and optionally define parameters for those processes. These processes gather, manipulate, and publish the data used in Skylar One.

ScienceLogic recommends that you do not edit the values in this page without first consulting ScienceLogic. Incorrect values can severely disrupt ScienceLogic platform operations.

ScienceLogic processes fall into three scheduling categories or Frequencies:

• Asynchronous. The process is launched in response to a system event or user request.
• Scheduled. The process is launched on a regular schedule.
• Always. The process always runs while Skylar One is running.

Skylar One performs many tasks in parallel:

• Through a modular design, allowing functions to be distributed to multiple processing platforms.
• Through multi-processing, where multiple instances of a process run simultaneously.

Viewing the List of ScienceLogic Processes

To view the list of process in the Process Manager page:

1. Go to the Process Manager page (System > Settings > Admin Processes).
2. The Process Manager page displays information about each ScienceLogic process. The Process Manager page displays the following for each process:

• Process Name. Name of the process.
• Program File. Name of the executable file associated with the process.
• Frequency. Frequency with which the platform launches the process. Possible values are:

• Asynchronous. The process is launched in response to a system event or user request.
• Always. The process always runs while Skylar One is running.
• Scheduled. The process runs at intervals ranging from 1 Minute to Daily.

• Runtime Offset. This field applies only to scheduled processes and allows the platform to stagger the launch of a process. The field specifies the number of minutes after the default scheduled time to execute a process. The default scheduled time at which processes are initially executed is midnight UTC. So if a process has a Frequency of 5 Minutes and the Runtime Offset is set to "2", the process will execute at two minutes past UTC midnight, seven minutes past UTC midnight, 12 minutes past UTC midnight, 17 minutes past UTC midnight, etc. Choices range from 0–1439.
• Async Throttle. This field applies only to asynchronous processes. This field indicates the number of jobs per process that can run simultaneously.

• Batch Factor. This field applies only to scheduled processes and determines how many multithreaded child processes are spawned on each execution of the process.

number of tasks a process is responsible for completing/Batch Factor = number of child processes that will be spawned

• The number of tasks is typically determined by the number of devices the process is collecting data from.
• The maximum number of child processes is limited by the number of CPUs installed in the Skylar One appliance that runs the process.

NOTE: Batch Factor defines the maximum number of worker processes or child processes. This value has precedence over the value specified in dynamic_collect_num_request_workers.

• Time Factor. Determines how long the process can run before being stopped by the process manager. This setting only applies to asynchronous processes and scheduled processes. For asynchronous processes, this is the length of time an instance of the process can run. For scheduled processes, the value of Time Factor is used to calculate Run Length.

(Frequency * Time Factor) + Frequency = Run Length

For example, suppose a process runs every 15 minutes (as specified in the Frequency field). A Time Factor of 2 means the process is allowed to run for 45 minutes. A Time Factor of 0 means the process is allowed to run for 15 minutes.

• Run Length. Specifies how long the process can run before being stopped by the process manager. This number is based on the Time Factor for the process.
• State. Current operational state of the process. Possible values are:

• Enabled. Process can run.
• Disabled. Process cannot run.

• Debug. Specifies whether debugging information is enabled for the process. For more details on debugging a process, see the section Debugging a Process.
• ID. Unique numeric ID assigned to each process by Skylar One.
• Edited By. Date and time the process settings were last edited.
• Edit Date. Date and time the process settings were last edited.

Debugging a Process and Viewing Debug Logs

When you debug a process, you tell Skylar One to use verbose logging for that process. You can then view Skylar One log file to view the logs.

There might be circumstances where you have narrowed down a problem to a specific ScienceLogic process (for example, based on an error message or event). When this happens, you might find it helpful to turn on debugging for that process and view the debug logs.

ScienceLogic recommends that you enable the debug option only while troubleshooting a problem while working with ScienceLogic Support or while following a troubleshooting guide, and that you then immediately turn off debugging when you have completed troubleshooting. Do not leave the debug option enabled during normal operation of Skylar One. When you turn on debugging, Skylar One will run significantly more slowly.

You cannot enable debug mode for the "Message Collection: SNMP Trap" or "Message Collection: Syslog" processes.

To enable the debug option for a process:

1. In the Process Manager page, find the process you want to edit. Select its wrench icon ().
2. The Process Editor page appears and is populated with values for the selected process.
3. Edit the following field:

• Debug. Enables or disables debugging information for a process. Select Enabled.

4. Click the Save button in the Process Editor page.
5. Log in to the console of the appliance where the process is running. Alternately, you can use SSH to open a shell session on the appliance. Log in as em7admin with the appropriate password.

TIP: To view a list of IP addresses for all appliances in your system, go to the Appliance Manager page (System > Settings > Appliances).

6. If the process you are debugging is a process that has a Frequency of Always, you must restart the process to make it pick up the new debug status (enabled). To restart the process, enter the following at the shell prompt:

sudo service process_name restart

For example, if you were debugging the process for the event engine, you would enter:

sudo service em7_event restart

7. Navigate to the directory /var/log/em7. View the file silo.log. The most recent entries will be posted at the end of the file.
8. After you have finished troubleshooting the process, remember to disable debugging. If the process has a Frequency of Always, you must restart the process to make it pick up the new debug status (disabled).

Viewing Information About Unhandled Exceptions

An exception specifies that something happened "out of the norm" that is preventing the software from executing the next step. Exceptions are a specific type of error, usually the result of invalid input, missing input, or a network error that prevents communication between software modules. For most exceptions, Skylar One will handle the exception by logging a specific error in the System Logs and will continue to run the process. However, if the platform does not handle the exception, the process will stop running, and Skylar One will generate an error message describing the unhandled exception.

Viewing the List of Unhandled Exceptions

To view the list of unhandled exceptions for all appliances:

1. Go to the Unhandled Exceptions page (System > Monitor > Unhandled Exceptions).
2. The Unhandled Exceptions page displays the following for each unhandled exception:

• Exception Filename. Full path of the file where the exception occurred.
• Line. Line number of the line in the file where the exception occurred.
• Exception Information. Error message associated with the exception.
• First Occurrence. Date and time of the first occurrence of the exception.
• Last Occurrence. Date and time of the last occurrence of the exception.
• Count. Number of times the exception has occurred.

Viewing the Output of the System Status Script

For each Database Server, Data Collector, and Message Collector, you can view the output of the system status script for that appliance. To do this:

1. Go to the Appliance Manager page (System > Settings > Appliances).
2. Locate the Database Server, Data Collector, or Message Collector that you want to view diagnostic information about.
3. Click on its magnifying-glass icon () to view the output of the system status script for that appliance.

Viewing the Database Tables on the Database Server

In some circumstances, you might need to view the contents of the database tables (the permanent tables are stored on the Database Server). There are two ways to do this:

• Using the built-in Database Tools in the Database Tool page (System > Tools > DB Tool).

  The Database Tool page is available only in versions of Skylar One prior to 12.2.1 and displays only for users that have sufficient permissions to access the page.

• Using the link to the phpMyAdmin interface in the Appliance Manager page (System > Settings > Appliances).

Accessing the Database Tool

The Database Tool page allows administrators to view information about the internal ScienceLogic databases and run SQL queries against those internal databases.

The Database Tool page is available only in versions of Skylar One prior to 12.2.1 and displays only for users that have sufficient permissions to access the page.

CAUTION: Contact ScienceLogic for details on using the Database Tool page and troubleshooting databases. Do not make changes to the database or run the Optimizer Tool without guidance from ScienceLogic.

To access the database tool:

1. Go to the Database Tool page (System > Tools > DB Tool).
2. To run an SQL query from the Database Tool page, enter values in the following fields:

• Select Database. Select a database to query.
• SQL Query. Enter an SQL query to execute against the selected database. For more information on each database and each table, use the options in the Actions menu.

NOTE: You must be familiar with SQL and know how to build a proper query before using the Database Tool page.

3. Click the Go button to execute the query.
4. The results from the query are displayed in the pane at the bottom of the page.
5. To view the reports about the a database(s), click the Actions menu. The following options are available:

• Engines. Displays status information about the server's storage engines. For each engine, the modal page displays a description of the engine, whether the engine is supported by Skylar One, and whether or not the engine supports transactions, XA, and save points.
• Global Status. Displays a list of global variables used in the database tables and the current value for each global variable.
• InnoDB Variables. Displays a list of InnoDB variables used in Skylar One and the value for each variable.
• Open Tables. Displays a list of currently open tables. For each table, the modal page displays the database name, table name, whether the table is currently in use, and whether the table is currently locked.
• Optimizer Tool. Leads to the Database Optimizer Tool page, where you can choose to optimize, repair, check, flush, or analyze all the tables in a database.

Contact ScienceLogic for details on using the Database Optimizer Tool page. Do not run the Optimizer Tool without guidance from ScienceLogic.

• Processes. Displays a list of running threads on the databases and tables. For each process, the modal page displays the connection ID, the database user who issued the statement, the host name of the client that issued the statement, the affected database, the command, the time in seconds that the thread has been in its current state, the state of the thread, and any available description of the process.
• Table Status. Displays the status of each database table in the platform. For each table, the modal page displays the table name, the database engine, database version, row format, number of rows, average row-length, length of the data file, maximum length of the data file, length of the index file, number of allocated but unused bytes, the next auto-increment value, the create time for the table, the update time for the table, the table's character set and collation, the live checksum value, options used with CREATE TABLE, and any comments.
• Variables. Displays a list of all database system variables used in Skylar One and the value of each variable.

Disabling Normalization, Re-Enabling Normalization, and Backfilling Raw Data

ScienceLogic does not recommend stopping normalization on Data Collectors. However, there are rare occasions where ScienceLogic Customer Support might ask you to disable normalization as part of troubleshooting.

To disable normalization:

1. Either go to the console of the Database Server or use SSH to access the Database Server.

2. Log in as user em7admin with the appropriate password.

3. Type the following at the command line:

   sudo visilo

This is the file where users can customize the silo.conf file. In step #6, you will execute a command that sends these changes to the system silo.conf file.

4. In the LOCAL section, add the following line:

   rollups_disabled=ON

5. Save your changes and exit the file (:wq).
6. Restart the data collection process to ensure they receive the change. Type the following at the command line:

   sudo service em7_hfpulld restart

   sudo service em7_lfpulld restart

   sudo service em7_mfpulld restart

To re-enable normalization and normalize data that was collected while normalization was disabled:

1. Either go to the console of the Database Server or use SSH to access the Database Server.
2. Log in as user em7admin with the appropriate password.

3. Type the following at the command line:

   sudo visilo

4. In the LOCAL section, add the following line:

   rollups_disabled=OFF

5. Save your changes and exit the file (:wq).

6. Restart the data collection process to ensure collectors receive the change. Type the following at the command line:

   sudo service em7_hfpulld restart

   sudo service em7_lfpulld restart

   sudo service em7_mfpulld restart

7. At the command line, type the following to normalize the data that was collected while normalization was disabled:

   [/opt/em7/backend/data_normalizer_backfill.py --database, <database> --dids <[device IDs]> --start <start date> --end <end date> --workers <number of workers>

NOTE: To get help, at the shell prompt, type "/opt/em7/backend/data_normalizer_backfill.py -h".

where:

• --database database. Specifies the database that you want to backfill with normalized data. The choices are:

• data_avail. Table that stores normalized data for availability.
• data_cv. Table that stores normalized data for Web Content policies.
• data_dns. Table that stores normalized data for DNS policies.
• data_email. Table that stores normalized data for Email Round-Trip policies.
• data_ports. Table that stores normalized data for TCP-IP Ports policies.
• data_procs. Table that stores normalized data for System Processes policies.
• data_services. Table that stores normalized data for Windows Services policies.
• data_storage. Table that stores normalized data for file systems.
• data_tv. Table that stores normalized data for SOAP/XML Transaction policies.
• dynamic_app_data_appID. Table that stores normalized data for a Dynamic Application. Specify the application ID for the Dynamic Application.

• --dids device IDs. Specifies the device ID of the device or devices for which you want to normalize data.

• You can specify a single device ID.
• You can specify multiple device IDs, separated by commas and surrounded by square brackets.
• If you do not specify any device IDs, Skylar One will normalize the specified data for all devices in your system.

• --start start date. The timestamp that specifies the data to normalize. Raw data with a time stamp at this time or later will be normalized. Skylar One will normalize data starting with this timestamp and ending with the end-date timestamp.

• Specify the timestamp in the format yyyy-mm-dd hh:mm:ss, using a 24-hour clock. Surround the timestamp with single quotes.

• --end end date. The timestamp that specifies the data to normalize. Raw data with a time stamp at this time or earlier will be normalized. Skylar One will normalize data starting with the start-date timestamp and ending with this timestamp.

• Specify the timestamp in the format yyyy-mm-dd hh:mm:ss, using a 24-hour clock. Surround the timestamp with single quotes.

• --workers workers. Number of worker processes to assign to this task. This field is optional. Please consult ScienceLogic Customer Support for suggestions on worker processes.

For example:

python /opt/em7/backend/data_normalizer_backfill.py --database dynamic_app_data_16 --start '2017-10-01 00:00:00' --end '2017-10-10 00:00:00' --workers 10

This command normalizes raw data collected by the Dynamic Application with an application ID of 16, associated with all subscriber devices (no device IDs specified, so defaults to "all devices"), and that was collected between midnight on October 1, 2017 and midnight on October 10, 2017. The data_normalizer_backfill.py code uses ten worker processes to perform the normalization.

Enable Logging for Data Pull Storage Objects

To investigate missed polls or slow database queries, you can temporarily enable logging for data pull storage objects. After you complete the diagnostics, you must disable logging for data pull storage objects, because the logging can affect the performance of data pull.

Enable

To enable logging for data pull storage objects:

1. Either go to the console of the Database Server or use SSH to access the Database Server.
2. Log in as an administrator.

3. At the shell prompt, enter the following:

   sudo visilo

4. In the silo.conf file, add the following lines:

   [DATAPULL]

   log_storage_object_stats = 1

5. Save your changes to the file (:wq).

6. You must restart the data collection processes to ensure they receive the change. To do this, enter the following at the shell prompt:

   sudo service em7_hfpulld restart

   sudo service em7_lfpulld restart

   sudo service em7_mfpulld restart

Disable

When you have completed your diagnostics, disable logging for data pull storage objects. To do this:

1. Either go to the console of the Database Server or use SSH to access the Database Server.
2. Log in as an administrator.

3. At the shell prompt, enter the following:

   sudo visilo

4. In the silo.conf file, edit the following:

   [DATAPULL]

   log_storage_object_stats = 0

5. Save your changes to the file (:wq).

6. You must restart the data collection processes to ensure they receive the change. To do this, enter the following at the shell prompt:

   sudo service em7_hfpulld restart

   sudo service em7_lfpulld restart

   sudo service em7_mfpulld restart

Controlling Developer Log Settings

In rare cases, you may need to modify log levels or suppression of certain logs in Skylar One, usually at the request of ScienceLogic Customer Support. To do so, you can use the following pages:

• Skylar One Developer Logs (System > Tools > Skylar One Developer Logs), which captures logs related to the default user interface (AP2), GraphQL, and associated services.

• PHP Developer Logs (System > Tools > PHP Developer Logs), which captures logs related to PHP execution, errors, and database queries for the classic Skylar One user interface.

Both of these pages are helpful for debugging and troubleshooting issues in Skylar One.

This section describes the options included on the Skylar One Developer Logs and PHP Developer Logs pages.

These pages are available only for Administrator-level users in Skylar One.

Generating Developer Logs for the Default User Interface (AP2)

You can configure developer log settings for the default user interface (AP2) from the Skylar One Developer Logs page (System > Tools > Skylar One Developer Logs) and then generate logs filtered based on your configuration.

This page is only available for Administrator-level users in Skylar One.

To generate developer logs for the default user interface (AP2):

1. Go to the Skylar One Developer Logs page (System > Tools > Skylar One Developer Logs).

2. In the Log Levels field, select the maximum log level you want to capture. In order of severity, your options are: 

   • Error

   • Warning

   • Info

   • Debug

   • Trace

   • Log

   When you select a maximum log level, the log will capture messages of that severity level and higher. For example, if you select Info, the log will include messages with severities of Info, Warning, and Error.

3. In the Capture log for the field, select the log duration. This field determines the length of time during which the log info will be captured. Your options range from next 1 minute to next 24 hours.

4. Click Capture Log. The log will appear in the Completed Logs pane, along with its status and other information.

5. When the log file has finished compiling, you can click its hyperlink in the Completed Logs pane to download it. The completed log file includes a version of the /var/log/sl1/nextui.log file, with the contents filtered based on the selections you made on the Skylar One Developer Logs page.

Log files are automatically deleted 24 hours after they are created.

Generating Developer Logs for the Classic User Interface

You can configure developer log settings for the default user interface (AP2) from the PHP Developer Logs page (System > Tools > PHP Developer Logs) and then generate logs filtered based on your configuration.

This page is only available for Administrator-level users in Skylar One.

To generate developer logs for the classic user interface:

1. Go to the PHP Developer Logs page (System > Tools > PHP Developer Logs).

2. In the UI Developer Log Levels section, select the types of messages that you want written to the user interface log file (em7php.log). Each type of message has an associated number; the log level is the sum of all enabled messages. The numbers and associated message types are:

   • 1. Critical

   • 2. Error

   • 4. Warning

   • 8. Info

   • 16. Debug

   • 32. Trace

   To determine the log level, sum the numbers associated with each type of message you want to enable. For example, if you want to enable Critical, Error, and Warning messages, then you would sum one, two, and four to get a log level value of seven.

3. In the UI/REST MySQL Query Log Levels section, select the types of messages you want written to the mysqli.log file. This log file collects every PHP-based call to MySQL and includes general information about the query. Determine the granularity of data you want by selecting one or more of the following checkboxes:

   • Error

   • Warning

   • Info (non-error)

   • Request URI (if you want the mysqli.log file to include the request uniform resource identifier)

4. In the Advanced Settings section, configure the suppressions and the date/time format you want to use by editing the following fields:

   • Suppression List. This list acts as a bitmask to log entries. For example, to suppress all entries for css-em7, you would enter "css.em7::127", where 127 is the sum of all possible log levels. You can specify multiple suppressions in the list, separated by commas.

   • Datetime Format. Specifies a user-defined date format that will be used for system logs. You can use any date variables supported by the PHP date function in this field.

     Seconds and milliseconds are always appended to the date/time stamp.

   • Include IP in log filenames. Select this option to add the IP address from which the user is logged in to the name of each log file.

5. In the Set Log Config field, enter a length of time (in minutes) during which the log info will be captured.

6. Click Set Log Config to confirm your log configuration settings.

7. To download a log file, click the name of the log you want to download under Download Logs.