Creating a Custom Step
The Snippet Framework executes reusable snippets known as steps. A step should consist of a single purpose (for example, parse string to JSON, select a value from a dict, etc). Steps are chained together during execution to collect, process, and select the correct values with little to no coding required. A valid step can be a class or function, depending on where the step is registered. If a step does not exist for your scenario, you can extend the Snippet Framework to add that functionality. The two available step types are as follows:
Considerations
Before developing a step, ensure that there is not already a step or a way to chain steps together to accomplish the task. This helps maintain a healthy set of steps without overlap and reduces the chances of outdated code being propagated to different Dynamic Applications and/or SL1 stacks.
A core concept is the difference between a Parser and
Selector. A Parser should convert a data structure
into a consumable format for a Selector. By separating
these two concepts you can produce a step that is more
reusable than if a single step performed both actions.
Ensure to use security best practices when developing a step. For example, avoid logging any secure information (such as credentials).
Determine if the step you are developing will be used by one or many Dynamic Applications. If you know that this step will not be used by multiple Dynamic Applications, having the source code in the Snippet may prove to be more beneficial for quick adjustments and ease of deployment. If the step will be used by multiple Dynamic Applications it will be beneficial to include the code in a ScienceLogic Library to eliminate the need to copy, paste, and potentially update it in multiple locations.
Pre-requisites
There are several important concepts to understand prior to starting development of a step. These concepts help ensure the step can be written with a minimal amount of code by de-duplicating repeated code.
Type Checking
Note
Type Checking is disable by default. To enable it, go to Enable Type Checking
The framework can validate the parameters data type of a step
using pytypes, a typing toolbox. For this validation to occur, you must
specify the argument name and expected type(s). This type checking is
available when you utilize the decorator @typechecked. If the data type
is not correct an exception will be raised which will stop the
ResultContainer from continuing throughout the collection
pipeline.
- silo.low_code.typechecked(*v, **k)
Decorator to check the input and output type of a function or method.
Perform type-checking at run-time to ensure the parameters have the correct types. If an incorrect type is provided, an exception will be raised stating the parameter and type issue.
- Parameters
input (object) – The expected input type. If the typechecking is running against a Requestor it will use request as input. If the typechecking is running against a Processor it will use result as input. If the input is not correct, the exception InputTypeError will be raised.
output (object) – The expected output type. If the output is not correct, the exception OutputTypeError will be raised.
**extra (object) – Used when checking additional parameters. The name must match the parameter. If the input is not correct, the exception InputTypeError will be raised.
Below we can see an example utilizing typechecked to ensure the input, output, and action_arg are all of the correct type.
@register_processor(
metadata={
"author": "ScienceLogic",
"descr": DESCR_PROCESSOR_SELECTOR_SIMPLE_KEY,
"title": "Simple Key",
"group": "Selectors",
},
arg_required=True,
)
@typechecked(input=Iterable[Any], output=Any, action_arg=Union[Iterable[Any], String23, int])
def simple_key(result, action_arg):
"""For dicts, lists, etc, return the keys specified with periods separating.
For example: Path.to.value.
"""
keys = action_arg
if isinstance(action_arg, (str, unicode)):
keys = action_arg.split(".")
elif not isinstance(action_arg, list):
keys = [action_arg]
response = []
if isinstance(result, list):
if len(keys) == 1 and isinstance(keys[0], int):
return result[action_arg]
for value in result:
response.append(get_parts(value, keys))
return response
return get_parts(result, keys)
Note
String23 is a defined type that makes Python 2 and Python 3 strings compatible.
Warning
@typechecked does not work on Python3
Enable Type Checking
To enable Type Checking, set the typechecking flag to True in the
entrypoint.
snippet_framework(
collections,
custom_substitution,
snippet_id,
app=self,
typechecking=True,
)
Warning
Note that enabling Type Checking affects the collection performance.
Below are the times of Type Checking enable and then disable.
[em7admin@CRUCIBLE-CU-188-201 framework]$ silo_mysql -e "select
avg(elapsed_time), avg(cpu_time) from silo_metrics.da_collection_jobs
where collection_time = '2022-03-23 19:30:00' and complete = 1"
+-------------------+---------------+
| avg(elapsed_time) | avg(cpu_time) |
+-------------------+---------------+
| 739.5584 | 732.1279 |
+-------------------+---------------+
[em7admin@CRUCIBLE-CU-188-201 framework]$ silo_mysql -e "select
avg(elapsed_time), avg(cpu_time) from silo_metrics.da_collection_jobs
where collection_time = '2022-03-23 21:30:00' and complete = 1"
+-------------------+---------------+
| avg(elapsed_time) | avg(cpu_time) |
+-------------------+---------------+
| 95.7932 | 91.2887 |
+-------------------+---------------+
[em7admin@CRUCIBLE-CU-188-201 framework]$
Available Parameters
Steps use inspection to determine the parameters for the functions. This allows you to determine what information is required for your step and not receive any arguments that you will not consume for the operation. Some steps may have a different set of available parameters which will be identified in the list.
All steps have the following parameters available:
object action_arg: Action argument for the current step
int action_number: Action number assigned to the execution of the action
object result: Result from the previous action
ResultContainer result_container: Steps from the previous action
object snippet_arg: The Snippet Argument for the collection (after substitution)
silo.apps.cache.CacheManager step_cache: CacheManager for interacting with step cache
Requesters require one of the parameters below to be used (in addition to any of the parameters in the previous section):
request: A single ResultContainer will be given to the Requestor. The Requestor will be called in a loop to handle each request. All error handling occurs in the Snippet Framework.
requests: List of all ResultContainers for the Requestor will be provided. The Requestor must perform error handling to ensure that all requests are processed.
A Requestor’s rewind function has access to additional parameters:
object data_request: Data raised by a step that contains information regarding the next request.
Suppose we are writing a step call function and need access to the result and action argument to perform a replace. To accomplish that task, we would have the following signature:
def call(self, result, action_arg):
return result.replace(action_arg[0], action_arg[1])
call("Hello, World!", ["Hello", "Hi"])
# returns
"Hi, World!"
Logging
There are two available loggers, contextually-aware loggers and module-level loggers. Both loggers can be requested from silo.low_code and will be outlined in their respective sections.
If you need to log messages and are not within context (for example,
initializing an object) you can log messages that will not be contextually
aware. This logger is acquired by requesting a
Module Logger.
If you require contextually-aware messages you must use the contextually-aware
logger. This logger uses information from the current step and
ResultContainer to generate a header to identify the message.
It is also used to log to the specific policy logfile (if configured). This
logger can be acquired by requesting a Step Logger
to add contextually-aware information to the log message.
To see additional information about log files you can review Logfile Locations.
By default, the Snippet Framework will attempt to redact any information related
to secrets. If additional secrets are required, you can add to secret by calling
Add Secret. This will ensure that any
Snippet Framework based logger will redact this information and replace it with
*’s.
ScienceLogic Libraries and Execution Environments
The Snippet Framework makes use of reusable steps through ScienceLogic libraries. When developing a step you should strive to use these libraries for the step code. To learn more about ScienceLogic Libraries and Execution Environments you can review the docs.
ResultContainer
The ResultContainer is the standard data interface for moving
information between the steps / steps. It is a
pure data-class with no attached functions so its only purpose is to move
information around. To learn more about the ResultContainer
you can review the ResultContainer
documentation.
Collection
A Collection contains all pertinent information regarding a specific Dynamic Application Collection Object. To learn more about Collection you can review the Collection docs.
Decorators
The Snippet Framework uses decorators to help users create and register steps or add functions to specific steps of the collection. The purpose of decorators is to provide a mechanism to create definitions without the need to modify the content library and to use the snippet code directly.
After calling a decorator, the defined function must follow the signature.
@decorator_name(parameters)
def name_function(result, action_arg, resultcontainer):
Step Registration Decorators
Step registration decorators are used for adding custom steps into the Snippet Framework. This enables you to easily extend the base functionality of the Snippet Framework. The available registration decorators are as follows:
- silo.low_code.register_requestor(*args, **kwargs)
Decorator for registering Requesters
Register a Requester that will be validated once the Framework is initialized. A Requester must inherit from the class
silo.low_code.RequestMaker.- Parameters
get_req_id (callable) – Optional. Function used for generating the request id
name (str) – Step Name to use for the Custom Step
metadata (dict) – Metadata related to the step
rewind (callable) – Callable function that updates the action_arg to start the rewinding process
required_args (list) – List of required top-level arguments for the step. If these arguments are not present in the action_arg, the Snippet Framework will not attempt to execute the collection and log a warning message instead.
arg_required (bool) – If an argument is required for a step. This denotes a step must have an argument, regardless of the type. This can be used in conjunction with required_args if your step accepts a dictionary or a single value.
- Returns
Original object
- Return type
object
- silo.low_code.register_processor(*args, **kwargs)
Decorator to register a processor
Register a processor that will be validated once the Framework is initialized. There are no sanity checks on a processor so it should always be valid.
- Parameters
get_req_id (callable) – Optional. Function used for generating the request id
name (str) – Step Name to use for the Custom Step
metadata (dict) – Metadata related to the step
type (str) – Type of the step.
required_args (list) – List of required top-level arguments for the step. If these arguments are not present in the action_arg, the Snippet Framework will not attempt to execute the collection and log a warning message instead.
arg_required (bool) – If an argument is required for a step. This denotes a step must have an argument, regardless of the type. This can be used in conjunction with required_args if your step accepts a dictionary or a single value.
- Returns
Original object
- Return type
object
- silo.low_code.register_cacher(*args, **kwargs)
Decorator to register a cacher
Register a cacher that will be validated once the Framework is initialized.
- Parameters
get_req_id (callable) – Optional. Function used for generating the request id
name (str) – Step Name to use for the Custom Step
metadata (dict) – Metadata related to the step
read (callable) – Callable function that performs a cache read for fast-forwarding
required_args (list) – List of required top-level arguments for the step. If these arguments are not present in the action_arg, the Snippet Framework will not attempt to execute the collection and log a warning message instead.
arg_required (bool) – If an argument is required for a step. This denotes a step must have an argument, regardless of the type. This can be used in conjunction with required_args if your step accepts a dictionary or a single value.
- Returns
Original object
- Return type
object
Development
Requestor
A Requester states how to retrieve information from a single source-type. The step will handle any device issues (credential, connection, response, etc) and will raise the appropriate exception to be handled by the Snippet Framework automatically. This step should not do any processing on the retrieved data and return an object that can be serialized.
Requestors must be a class that inherits from silo.low_code.RequestMaker.
Requestors utilize a de-duplication process to ensure the minimum number of calls are made to the device which reduces load on SL1 and the device. This process uses the function get_request_id to determine which calls are duplicates and makes the call only once to the device.
If writing a Requestor to handle all requests at once, you must implement error handling for each request. If the Requestor handles a single request at a time, error handling is provided by the Snippet Framework. In this scenario, if a silo.apps.errors.DeviceError is raised a device log message will be created on the device.
A Requestor can be configured to support the rewind capability. This allows
a Processor to rewind to the previous Requestor. The rewind function
should generate the new action_arg for the step. After
it is executed, the request_ids will be regenerated as new request
will be sent.
To utilize this feature, the step must be registered
with a keyword argument rewind that is a callable
function. This feature is used in conjuction with the parameter,
data_request that supplies all information for the request.
Note
When developing a Requestor that supports rewind, ensure that you communicate everything required to rewind. This allows Processor developers to supply the correct format and information to the rewind function.
Example
This sample Data Requester will return the first argument as the value from the data request.
@register_requestor(
metadata={
"author": "ScienceLogic",
"descr": DESCR_REQUESTOR_STATIC_VALUE,
"title": "Static Value",
},
)
class StaticValueRequest(RequestMaker):
def validate_request(self, request):
"""Ensure the incoming ResultContainer is valid for the Requestor
Validate the ResultContainer is valid. If it is invalid, raise an exception to indicate
the ResultContainer cannot be collected by the Requestor.
:param ~silo.low_code.ResultContainer request: Request to validate
"""
if not isinstance(request.credential.id, int):
msg = "Somehow you ended here in the test code! Try a different credential"
raise CredTypeNotSupported(msg)
@staticmethod
def get_name():
return "static_value"
@staticmethod
def get_desc():
return "Returns whichever value is in the request"
@typechecked(input=ResultContainer, output=ResultContainer)
def call(self, request, action_number):
"""Returns the first element in the requests
:param ~silo.low_code.ResultContainer request: ResultContainer that
need to be executed
:param int action_number: Action number assigned to the execution of the action
:return: ResultContainer where the result is value passed in
:rtype: ~silo.low_code.ResultContainer
"""
return ResultContainer(request.current_config, previous_result=request)
Processor
A Processor should perform a single operation on a result. Their input should be type-checked to reduce duplicate code and reduce potential issues. A Processor can be a class or function.
A Processor can additionally request more data if the previous Requestor supports the rewind functionality. A step that utilizes this feature must register with the correct type.
@register_processor(type=silo.low_code.REQUEST_MORE_DATA_TYPE)
Before you create a new step that utilizes rewind, you must
first understand how the requestor expects the data. Once you
understand how the data must be sent, the step must raise the exception
silo.low_code.RequestMoreData. When raising this exception
you should specify any values you want set as keyword arguments.
For example, if you wanted to reference data_request.beans
you would use the following:
raise silo.low_code.RequestMoreData(beans="Cool Beans")
The processor can optionally set the index (storage key) when
performing the loop. If this is not specified, the request_id
will be used in its place. To specify the index, use the
function, result_container.set_index(index_to_use).
The processor can optionally set the maximum number of
iterations when performing the loop. If this is not set, the
default value of 100 will be used. To set this value, update
the attribute at result_container.max_iterations.
Example
This sample Data Parser will convert the input, a JSON structure, into a Python Dictionary.
@register_processor(
metadata={
"author": "ScienceLogic",
"descr": DESCR_PROCESSOR_SELECTOR_SIMPLE_KEY,
"title": "Simple Key",
"group": "Selectors",
},
arg_required=True,
)
@typechecked(input=Iterable[Any], output=Any, action_arg=Union[Iterable[Any], String23, int])
def simple_key(result, action_arg):
"""For dicts, lists, etc, return the keys specified with periods separating.
For example: Path.to.value.
"""
keys = action_arg
if isinstance(action_arg, (str, unicode)):
keys = action_arg.split(".")
elif not isinstance(action_arg, list):
keys = [action_arg]
response = []
if isinstance(result, list):
if len(keys) == 1 and isinstance(keys[0], int):
return result[action_arg]
for value in result:
response.append(get_parts(value, keys))
return response
return get_parts(result, keys)
Cacher
A Cacher is similar to a Processor except the operation should be writing out the current result. A Cache step does not have the ability to modify the result as its not included in the automatic cache key that is generated.
A Cacher can optionally specify the ability to read
which allows the Snippet Framework to fast-forward to the step after the
Cacher. This can be specified the in registration decorator
utilizing the keyword argument read.
Example
This sample Cacher will write the current data to the specified key. If a key is not specified, the automatic cache key will be used.
def get_key(action_arg, result_container):
try:
key = action_arg.get("key", result_container.request_id)
except AttributeError:
key = result_container.request_id
return key
def cache_read(action_arg, result_container, step_cache):
return step_cache.read(get_key(action_arg, result_container))
@register_cacher(read=cache_read)
def cache_write(result, action_arg, result_container, step_cache):
step_cache.write(get_key(action_arg, result_container), result)