.. include:: ../external_links.txt .. _reference-logging-debugging: ##################################### Reference: Logging/Debugging ##################################### This document is a guide to troubleshooting collection issues with the |FRAMEWORK_NAME|. The log message format is as follows: .. code-block:: none - - - - * asctime: Local time of the log message * unique_id: Unique ID for the execution of the Snippet Framework * log_location: Location of the logging statement * log_level: Level of log message (debug, info, etc) * context: (Optional) Information to help identify which ResultContainer produced the log message. Example context: ``[a:,d:,c:,s:]`` * message: Message containing helpful information related to the execution Some logs may provide contextually aware messages. They use shorthand to denote information in the header. The character definitions are as follows: .. list-table:: Character Definitions :widths: 25 35 :header-rows: 1 * - Character - Named Value * - a - Application ID * - d - Device ID * - c - Collection ID * - s - Step Name When tracing a logfile, log messages may contain the collection name. If a name is not provided by the syntax, the |FRAMEWORK_NAME| will auto-generate one based on the Collection Object ID and Device ID. The following format will be used: ``.`` ****************** Available Logfiles ****************** Errors can be logged to one of four log locations. Familiarity with these locations will enable you to debug most collection and development issues. * Device log messages are generated during the communication process. The |FRAMEWORK_NAME| writes to the log if any issues are encountered. * |STEP_NAME_PLURAL_CAPITAL| use a standard Python module for logging. Logs are stored in the filesystem. See details below for the location of these logs. * System logs are populated when the |FRAMEWORK_NAME| throws an exception or does not operate as expected. Please contact ScienceLogic Support if you see any log messages in system logs. * The |FRAMEWORK_NAME| logs show all pertinent information related to collection issues. For example, any data gaps not related to SIGTERM are logged to the |FRAMEWORK_NAME| logs. .. _logfile-locations: Logfile Locations ================= The following logs are generated during execution of the |FRAMEWORK_NAME|: .. code-block:: none # shows logs by device that execute a Dynamic Application /var/log/em7/silo.log # shows all logs sent by the ScienceLogic Libraries /var/log/em7/sl1.log # shows messages from the Framework and Module loggers /var/log/em7/snippet_framework.log # shows messages that are aligned to any log policy /var/log/em7/snippet_framework.steps.log # shows messages for a specific log policy. will be replaced with the name from the database: /var/log/em7/snippet_framework.policy_.log Logging Detailed or Specific Information ======================================== When a Dynamic Application is executed by SL1, the framework will create logs of the collection process internally in specific locations, as described above. If you would like to see detailed information on the entire collection process, you can execute the process in debug mode, which lists the detailed |STEP_NAME_PLURAL|, input, and output along with any errors encountered during the collection. To log specific information to a collection object, or Dynamic Application on a device, you can set up a Log Policy that writes all of the information (not limited to errors only) to a specific file. For details on how to set up log policies, see the section :ref:`Log Policy`. How to View the Log File ======================== When a collection error occurs, the error message is logged to the log files described above :ref:`Logfile Locations`. If you have configured a Log Policy, you can view the specified log file to see collection details and any exceptions encountered. .. note:: The various log files are written at execution time. To view them, you can use a variety of Linux commands such as cat, nano, tail, and so on. For example: tail -f /var/log/em7/log_file.log ***************************************************************** Logging Detailed or Specific Information for Dynamic Applications ***************************************************************** Logging for Dynamic Applications ================================ When a Dynamic Application executes at its poll frequency, it automatically writes to logs only if any exceptions or errors occur. Alternatively, you can run Dynamic Applications in Dynamic Single/UI Debug mode, which runs the Dynamic Application such that you will see all of the process details. Additionally, if you want to see all collection logged, you can configure a `Log Policy`_. The example below is a template for the exception message when the Dynamic Application is run with the details of device id, collection name, and so on, which could be used to debug. .. code-block:: none A blocking error occurred for the collection [] and can no longer continue. Executor: Action Arguments: Data: Type: Exception: In the next table, you can see the meaning for each entry on the exception template: .. list-table:: * - **collection_name** - Collection Name * - **did** - Device Id * - **executor** - |STEP_NAME| where the error occurred * - **action_arguments** - Configuration parameters for the |STEP_NAME| * - **data** - Result information from the previous |STEP_NAME| * - **type** - Type of the result * - **exception** - Full StackTrace of the issue that stopped the collection The following location contains the log file for the |FRAMEWORK_NAME|: .. code-block:: none /var/log/em7/snippet_framework.log .. warning:: Using the **Dynamic Single** option could cause permission errors at collection time if it is not used with the correct user. For more information, see `How to Run a Single Dynamic Application in DEBUG Mode`_. .. _log-policy: Log Policy ========== The |FRAMEWORK_NAME| is designed to enable deeper analysis and provide information to help you solve possible issues. It gives you the capability to create dedicated logs that contain specific information from the collection process based on the configuration. For this purpose, the framework contains the Log Policy CLI, a command-line tool that allows CRUD (Create, Read, Update and Delete) operations for logging policies. The Log Policy CLI tool is included with the |FRAMEWORK_NAME| content library and can be run in any execution environment to which the library is aligned. The CLI tool can be executed by running the following command: .. code-block:: none sudo -u s-em7-core /opt/em7/envs//bin/python -m silo.low_code.cli log-policy .. note:: It is important to run the command using the ``s-em7-core`` user to avoid collection issues. You can see the execution environment GUID in the Environment Manager in SL1, as follows: .. code-block:: none System > Customize > ScienceLogic Libraries Actions > Execution Environments To better understand the Log Policy CLI tool, use the help option, as shown below: .. code-block:: none sudo -u s-em7-core /opt/em7/envs//bin/python -m silo.low_code.cli log-policy --help Each command provides additional information when you use the help option after the command name, as shown below. .. code-block:: none sudo -u s-em7-core /opt/em7/envs//bin/python -m silo.low_code.cli log-policy --help The Log Policy CLI tool creates a dedicated log file for each log policy. The log file is written to the following location: .. code-block:: none /var/log/em7/snippet_framework.policy_.log .. warning:: Keep in mind that logging can affect collector CPU, memory, and disk performance. Be careful when creating a Log Policy that will match many different criteria. *************** Troubleshooting *************** When a collection is executed in debug mode, using the SL1 UI, or dynamic single, you can see the full collection process. Dynamic Single is a tool that allows for direct execution of a Dynamic Application for a specific device. This script must be ran on the Collector where the Dynamic Application runs for the particular device. To execute Dynamic Single, the following command must be run: ``sudo -u s-em7-core /opt/em7/backend/dynamic_single.py `` There are additional environment variables that you can specify when running a command. These options enable additional functionality of the framework for more in-depth debugging options. * DEBUG: Enables debug output if this value is set to True. Default: False * SF_LOG_FULL_RESULT: Log the entire result after each step, instead of the first 1024 characters. Please take care when using this option as it can greatly increase log sizes. Default: False * SF_LOG_FULL_RC: The log message of a ResultContainer will contain all characters instead of the first 1024 characters. Please take care when using this option as it can greatly increase log sizes. Default: False To execute Dynamic Single with options, the following command must be run: ``sudo -u s-em7-core /opt/em7/backend/dynamic_single.py `` ``sudo -u s-em7-core DEBUG=1 SF_LOG_FULL_RESULT=1 /opt/em7/backend/dynamic_single.py `` Dynamic Single Permission Error =============================== If the execution environments were created with an invalid user, you could see an error related to permissions at execution time. This could happen if debug mode was executed with the dynamic single option incorrectly. To solve this error, set the permissions manually using the following commands. .. code-block:: none chown -R s-em7-core:s-em7-core /opt/em7/envs/ chown s-em7-core:s-em7-core /var/log/em7/snippet_framework*.log ********************************** Framework Exceptions and Solutions ********************************** This section describes the most common exceptions found when debugging collection issues or developing new snippet arguments. The section below describes what each exception means, along with possible troubleshooting methods to resolve them. .. note:: The `Snippet Framework PowerPack`_ includes Dynamic Applications with some of the exceptions mentioned below and can be run as a reference. ************************* Unhandled Framework Error ************************* When an unhandled error occurs in the |FRAMEWORK_NAME|, collect the following logs from the collector, and contact ScienceLogic Support: * /var/log/em7/snippet_framework.log .. image:: Screenshots/UnhandledFrameworkExceptions.png ************** Snippet Errors ************** You may encounter some other errors and exceptions while using the |FRAMEWORK_NAME| like the ones mentioned below. By modifying the *User Editable* area, if there are some mistakes in the source code you may see a `Syntax Error`. To solve the error you should review the syntax of your custom code. .. image:: Screenshots/SyntaxtError.png Another common error can be the `Import Error`. If this is the case, you should review and fix your execution environment. .. image:: Screenshots/ImportError.png