.. include:: ../external_links.txt .. _creating-custom-step: ########################################## Creating a Custom |STEP_NAME_CAPITAL| ########################################## The |FRAMEWORK_NAME| executes reusable snippets known as |STEP_NAME_PLURAL|. A |STEP_NAME| should consist of a single purpose (for example, parse string to JSON, select a value from a dict, etc). |STEP_NAME_PLURAL_CAPITAL| are chained together during execution to collect, process, and select the correct values with little to no coding required. A valid |STEP_NAME| can be a class or function, depending on where the |STEP_NAME| is registered. If a |STEP_NAME| does not exist for your scenario, you can extend the |FRAMEWORK_NAME| to add that functionality. The two available |STEP_NAME| types are as follows: * :ref:`Requestor ` - Retrieves the required data from the datasource. * :ref:`Processor ` - Perform an action on the result. For example, a processor may parse, transform, or format the data. These are only a few examples of what actions a processor may perform. ************** Considerations ************** Before developing a |STEP_NAME|, ensure that there is not already a |STEP_NAME| or a way to chain |STEP_NAME_PLURAL| together to accomplish the task. This helps maintain a healthy set of |STEP_NAME_PLURAL| without overlap and reduces the chances of outdated code being propagated to different Dynamic Applications and/or SL1 stacks. A core concept is the difference between a ``Parser`` and ``Selector``. A ``Parser`` should convert a data structure into a consumable format for a ``Selector``. By separating these two concepts you can produce a |STEP_NAME| that is more reusable than if a single |STEP_NAME| performed both actions. Ensure to use security best practices when developing a |STEP_NAME|. For example, avoid logging any secure information (such as credentials). Determine if the |STEP_NAME| you are developing will be used by one or many Dynamic Applications. If you know that this |STEP_NAME| will not be used by multiple Dynamic Applications, having the source code in the Snippet may prove to be more beneficial for quick adjustments and ease of deployment. If the |STEP_NAME| will be used by multiple Dynamic Applications it will be beneficial to include the code in a ScienceLogic Library to eliminate the need to copy, paste, and potentially update it in multiple locations. ************** Pre-requisites ************** There are several important concepts to understand prior to starting development of a |STEP_NAME|. These concepts help ensure the |STEP_NAME| can be written with a minimal amount of code by de-duplicating repeated code. Type Checking ============= .. note:: |TYPECHECKING| is disable by default. To enable it, go to :ref:`Enable Type Checking ` The framework can validate the parameters data type of a |STEP_NAME| using `pytypes`_, a typing toolbox. For this validation to occur, you must specify the argument name and expected type(s). This type checking is available when you utilize the decorator ``@typechecked``. If the data type is not correct an exception will be raised which will stop the |CONTEXT_SAVER_NAME_CAMEL_CASE| from continuing throughout the collection pipeline. .. autofunction:: silo.low_code.typechecked :noindex: Below we can see an example utilizing **typechecked** to ensure the input, output, and action_arg are all of the correct type. .. jinja:: step_namespace .. literalinclude:: /{{ step_namespace }}/standard/selector.py :start-after: dev_example_typecheck_start :end-before: dev_example_typecheck_end .. note:: ``String23`` is a defined type that makes Python 2 and Python 3 strings compatible. .. warning:: ``@typechecked`` does not work on Python3 .. _enable_typechecking: Enable |TYPECHECKING| --------------------- To enable |TYPECHECKING|, set the ``typechecking`` flag to ``True`` in the entrypoint. .. code-block:: python :emphasize-lines: 6 snippet_framework( collections, custom_substitution, snippet_id, app=self, typechecking=True, ) .. warning:: Note that enabling |TYPECHECKING| affects the collection performance. Below are the times of |TYPECHECKING| enable and then disable. .. code-block:: bash :emphasize-lines: 7, 15 [em7admin@CRUCIBLE-CU-188-201 framework]$ silo_mysql -e "select avg(elapsed_time), avg(cpu_time) from silo_metrics.da_collection_jobs where collection_time = '2022-03-23 19:30:00' and complete = 1" +-------------------+---------------+ | avg(elapsed_time) | avg(cpu_time) | +-------------------+---------------+ | 739.5584 | 732.1279 | +-------------------+---------------+ [em7admin@CRUCIBLE-CU-188-201 framework]$ silo_mysql -e "select avg(elapsed_time), avg(cpu_time) from silo_metrics.da_collection_jobs where collection_time = '2022-03-23 21:30:00' and complete = 1" +-------------------+---------------+ | avg(elapsed_time) | avg(cpu_time) | +-------------------+---------------+ | 95.7932 | 91.2887 | +-------------------+---------------+ [em7admin@CRUCIBLE-CU-188-201 framework]$ Available Parameters ==================== |STEP_NAME_PLURAL_CAPITAL| use inspection to determine the parameters for the functions. This allows you to determine what information is required for your |STEP_NAME| and not receive any arguments that you will not consume for the operation. Some |STEP_NAME_PLURAL| may have a different set of available parameters which will be identified in the list. All |STEP_NAME_PLURAL| have the following parameters available: * object action_arg: Action argument for the current |STEP_NAME| * int action_number: Action number assigned to the execution of the action * object result: Result from the previous action * |CONTEXT_SAVER_NAME_CAMEL_CASE| result_container: |STEP_NAME_PLURAL_CAPITAL| from the previous action * object snippet_arg: The Snippet Argument for the collection (after substitution) * silo.apps.cache.CacheManager step_cache: CacheManager for interacting with step cache Requesters require one of the parameters below to be used (in addition to any of the parameters in the previous section): * request: A single |CONTEXT_SAVER_NAME_CAMEL_CASE| will be given to the Requestor. The Requestor will be called in a loop to handle each request. All error handling occurs in the |FRAMEWORK_NAME|. * requests: List of all |CONTEXT_SAVER_NAME_CAMEL_CASE_PLURAL| for the Requestor will be provided. The Requestor must perform error handling to ensure that all requests are processed. A Requestor's ``rewind`` function has access to additional parameters: * object data_request: Data raised by a step that contains information regarding the next request. Suppose we are writing a |STEP_NAME| call function and need access to the result and action argument to perform a replace. To accomplish that task, we would have the following signature: .. code-block:: python def call(self, result, action_arg): return result.replace(action_arg[0], action_arg[1]) call("Hello, World!", ["Hello", "Hi"]) # returns "Hi, World!" Logging ======= There are two available loggers, contextually-aware loggers and module-level loggers. Both loggers can be requested from silo.low_code and will be outlined in their respective sections. If you need to log messages and are not within context (for example, initializing an object) you can log messages that will not be contextually aware. This logger is acquired by requesting a :py:mod:`Module Logger `. If you require contextually-aware messages you must use the contextually-aware logger. This logger uses information from the current |STEP_NAME| and |CONTEXT_SAVER_NAME_CAMEL_CASE| to generate a header to identify the message. It is also used to log to the specific policy logfile (if configured). This logger can be acquired by requesting a :py:func:`Step Logger ` to add contextually-aware information to the log message. To see additional information about log files you can review :ref:`Logfile Locations`. By default, the |FRAMEWORK_NAME| will attempt to redact any information related to secrets. If additional secrets are required, you can add to secret by calling :py:func:`Add Secret `. This will ensure that any Snippet Framework based logger will redact this information and replace it with `*`'s. ScienceLogic Libraries and Execution Environments ================================================= The |FRAMEWORK_NAME| makes use of reusable |STEP_NAME_PLURAL| through ScienceLogic libraries. When developing a |STEP_NAME| you should strive to use these libraries for the |STEP_NAME| code. To learn more about `ScienceLogic Libraries and Execution Environments`_ you can review the docs. |CONTEXT_SAVER_NAME_CAMEL_CASE| =============================== The |CONTEXT_SAVER_NAME_CAMEL_CASE| is the standard data interface for moving information between the |STEP_NAME_PLURAL| / |STEP_NAME_PLURAL|. It is a pure data-class with no attached functions so its only purpose is to move information around. To learn more about the |CONTEXT_SAVER_NAME_CAMEL_CASE| you can review the :py:mod:`ResultContainer ` documentation. Collection ========== A Collection contains all pertinent information regarding a specific Dynamic Application Collection Object. To learn more about Collection you can review the `Collection`_ docs. .. _low_code_decorators: Decorators ========== The |FRAMEWORK_NAME| uses decorators to help users create and register |STEP_NAME_PLURAL| or add functions to specific |STEP_NAME_PLURAL| of the collection. The purpose of decorators is to provide a mechanism to create definitions without the need to modify the content library and to use the snippet code directly. After calling a decorator, the defined function must follow the signature. .. code-block:: python @decorator_name(parameters) def name_function(result, action_arg, resultcontainer): |STEP_NAME_CAPITAL| Registration Decorators ------------------------------------------------ |STEP_NAME_CAPITAL| registration decorators are used for adding custom |STEP_NAME_PLURAL| into the |FRAMEWORK_NAME|. This enables you to easily extend the base functionality of the |FRAMEWORK_NAME|. The available registration decorators are as follows: .. autofunction:: silo.low_code.register_requestor :noindex: .. autofunction:: silo.low_code.register_processor :noindex: .. autofunction:: silo.low_code.register_cacher :noindex: *********** Development *********** .. _custom_dev_requestor: Requestor ========= A Requester states how to retrieve information from a single source-type. The |STEP_NAME| will handle any device issues (credential, connection, response, etc) and will raise the appropriate exception to be handled by the |FRAMEWORK_NAME| automatically. This |STEP_NAME| should not do any processing on the retrieved data and return an object that can be serialized. Requestors must be a class that inherits from ``silo.low_code.RequestMaker``. Requestors utilize a de-duplication process to ensure the minimum number of calls are made to the device which reduces load on SL1 and the device. This process uses the function `get_request_id` to determine which calls are duplicates and makes the call only once to the device. If writing a Requestor to handle all requests at once, you must implement error handling for each request. If the Requestor handles a single request at a time, error handling is provided by the |FRAMEWORK_NAME|. In this scenario, if a `silo.apps.errors.DeviceError` is raised a device log message will be created on the device. A Requestor can be configured to support the ``rewind`` capability. This allows a Processor to ``rewind`` to the previous Requestor. The ``rewind`` function should generate the new ``action_arg`` for the |STEP_NAME|. After it is executed, the request_ids will be regenerated as new request will be sent. To utilize this feature, the |STEP_NAME| must be registered with a keyword argument ``rewind`` that is a callable function. This feature is used in conjuction with the parameter, ``data_request`` that supplies all information for the request. .. note:: When developing a Requestor that supports rewind, ensure that you communicate everything required to rewind. This allows Processor developers to supply the correct format and information to the rewind function. Example ------- This sample Data Requester will return the first argument as the value from the data request. .. jinja:: step_namespace .. literalinclude:: /{{ step_namespace }}/standard/network_request.py :start-after: reg_req_example_start :end-before: reg_req_example_end .. _custom_dev_processor: Processor ========= A Processor should perform a single operation on a result. Their input should be type-checked to reduce duplicate code and reduce potential issues. A Processor can be a class or function. A Processor can additionally request more data if the previous Requestor supports the rewind functionality. A |STEP_NAME| that utilizes this feature must register with the correct type. .. code-block:: python @register_processor(type=silo.low_code.REQUEST_MORE_DATA_TYPE) Before you create a new step that utilizes rewind, you must first understand how the requestor expects the data. Once you understand how the data must be sent, the step must raise the exception ``silo.low_code.RequestMoreData``. When raising this exception you should specify any values you want set as keyword arguments. For example, if you wanted to reference ``data_request.beans`` you would use the following: .. code-block:: python raise silo.low_code.RequestMoreData(beans="Cool Beans") The processor can optionally set the index (storage key) when performing the loop. If this is not specified, the request_id will be used in its place. To specify the index, use the function, ``result_container.set_index(index_to_use)``. The processor can optionally set the maximum number of iterations when performing the loop. If this is not set, the default value of 100 will be used. To set this value, update the attribute at ``result_container.max_iterations``. Example ------- This sample Data Parser will convert the input, a JSON structure, into a Python Dictionary. .. jinja:: step_namespace .. literalinclude:: /{{ step_namespace }}/standard/selector.py :start-after: dev_example_typecheck_start :end-before: dev_example_typecheck_end .. _custom_dev_cacher: Cacher ======= A Cacher is similar to a Processor except the operation should be writing out the current result. A Cache |STEP_NAME| does not have the ability to modify the result as its not included in the automatic cache key that is generated. A Cacher can optionally specify the ability to ``read`` which allows the |FRAMEWORK_NAME| to fast-forward to the step after the ``Cacher``. This can be specified the in registration decorator utilizing the keyword argument ``read``. Example ------- This sample Cacher will write the current data to the specified key. If a key is not specified, the automatic cache key will be used. .. code:: python def get_key(action_arg, result_container): try: key = action_arg.get("key", result_container.request_id) except AttributeError: key = result_container.request_id return key def cache_read(action_arg, result_container, step_cache): return step_cache.read(get_key(action_arg, result_container)) @register_cacher(read=cache_read) def cache_write(result, action_arg, result_container, step_cache): step_cache.write(get_key(action_arg, result_container), result)