Available Steps¶

Syntax¶

Low-Code¶

The low_code Syntax is a YAML configuration format for specifying your collections. You explicitly provide the steps you want to run, in the order you need them to be executed. There are multiple versions of the low_code Syntax.

Framework Name

low_code

All versions support specifying configuration. The configuration will be all sub-elements under the step. For example, if you had a step cool_step and wanted to specify two key values, you would provide the following:

low_code:
  id: eastwood1
  version: 2
  steps:
    - cool_step:
        key1: value1
        key2: value2

Note

Notice that, in the example above, key1 and key2 are indented one level beyond cool_step. Setting them at the same indentation level as cool_step, as shown below, will result in an error.

low_code:
  id: eastwood1
  version: 2
  steps:
    - cool_step:
      key1: value1 # key1 is not properly indented
      key2: value2 # key2 is not properly indented

To provide a list in YAML you will need to utilize the - symbol. For example, if you had a step cool_step and wanted to specify a list of two elements, you would provide the following:

low_code:
  id: eastwood1
  version: 2
  steps:
    - cool_step:
      - key1
      - key2

Version 2¶

Version 2 of the low_code Syntax provides more flexibility when defining the order of step execution. This version can utilize multiple versions of Requestors (if supported) and allows for steps to run before a Requestor executes.

Format¶

low_code:
  id: eastwood1
  version: 2
  steps:
    - static_value: '{"key": "value"}'
    - json
    - simple_key: "key"

id: Identification for the request.

Note

If this value is not specified, the Snippet Framework will automatically create one. This allows for easier tracking when debugging when an ID is not required for the collection.
version: Specify the version of the low_code Syntax.
steps: Order of the steps for the Snippet Framework to execute.

Version 1¶

Version 1 was the original low_code syntax. It allowed for a single Requestor and any number of processors. It lacks support for multiple Requestors so it is not preferred.

Format¶

low_code:
  id: my_request
  network:
    static_value: '{"key": "value"}'
  processing:
    - json
    - simple_key: "key"

id: Identification for the request.

Note

If this value is not specified, the Snippet Framework will automatically create one. This allows for easier tracking when debugging when an ID is not required for the collection.
version: Specify the version of the low_code Syntax. If not provided, it will default to 1.
network: Section for the data requester step.
processing: Section for listing the steps required to transform your data to the desired output for SL1 to store.

Requestors¶

HTTP¶

The HTTP Data Requestor provides HTTP request functionality to gather information for the framework. The full URI is built from the credential and argument of the requestor. The credential contains the base URL and the port.

Step details:

Framework Name	`http`
Supported Credentials	Basic, SOAP/XML
Supported Fields of Basic Cred.	Hostname/IP Port Username Password
Supported Fields of SOAP Cred.	HTTP Auth User HTTP Auth Password HTTP Headers CURL Options, only SSLVERIFYPEER Proxy Hostname/IP Proxy Port Proxy User Proxy Password URL (Host, Port) Proxy Port
Parameters	method: delete, get (by default), head, options, patch, post, put uri: The second part of the base url provided in the credential response_type: text (default) [str], raw [returned data] check_status_code: True (default), False

Note

This step supports all the parameters mentioned in requests.Session.request Note that the parameters mentioned above will override the credential. For example, if you define verify: False in the credential but verify: True in the step parameters, the verify=True will be used in the request.

Example of use¶

To access the API of an SL1 System, you would use the URI:

https://SL1_IP_ADDRESS/api/account

The resource path for this example is:

/api/account

The SL1_IP_ADDRESS can be provided with the credential.

The output of this step:

{
  "searchspec":{},
  "total_matched":4,
  "total_returned":4,
  "result_set":
  [
    {
      "URI":"/api/account/2",
      "description":"AutoAdmin"
    },
    {
      "URI":"/api/account/3",
      "description":"AutoRegUser"
    },
    {
      "URI":"/api/account/1",
      "description":"em7admin"
    },
    {
      "URI":"/api/account/4",
      "description":"snadmin"
    }
  ],
}

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - http:
        uri: "/api/account"

Response Status Code Checking¶

When the parameter check_status_code is set to True (default), and the response’s status code meets the following condition:

$400 <= status code < 600$

An exception will be raised, thus stopping the current collection.

When the parameter check_status_code is set to False, no exception will be raised for any status code value.

Pagination Support¶

A custom step is required to raise the exception silo.low_code_steps.rest.HTTPPageRequest to rewind execution back to the previous network requestor. This exception is specific to the http and requires a dictionary as its own argument. The dictionary can either replace or update the existing action_arg dictionary passed to the http step.

If our Snippet Argument looked like this:

low_code:
    id: my_request
    version: 2
    setps:
      - http:
          uri: "/account"
      - pagination_trimmer
      - pagination_request:
          index: "request_key"
          replace: True

Our pagination_request step could look like this:

@register_processor(type=REQUEST_MORE_DATA_TYPE)
def pagination_request(result, action_arg):

  if result:
    # Replacement of action_arg
    raise HTTPPageRequest({"uri": "/account", "params": result}, index=action_arg.get("index"), replace=action_arg.get("replace", False))

This assumes that the result will contain the next pagination action argument. The step issues the HTTPPaginateRequest and sets the new action_arg with the first positional parameter. With the kwarg replace set to True, the http step will receive a new action_arg.

Static Value¶

The Static Value Data Requester is used to mock network responses from a device for testing purposes or when a step needs a static name.

Step details:

Framework Name	`static_value`
Supported Credentials	N/A

Example of use¶

If we wanted to mock:

"Apple1,Ball1,Apple2,Ball2"

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - static_value: "Apple1,Ball1,Apple2,Ball2"
    - firstComponentExecution
    - secondComponentExecution

Processors¶

Data Parsers¶

CSV¶

The CSV Data Parser converts a string into the format requested by the Args.

Step details:

Framework Name	`csv`
Parameters	All the arguments inside: `csv.DictReader`, `csv.reader`
Reference	https://docs.python.org/3/library/csv.html

Example Usage¶

If the incoming data to the step is:

"A1,B1,C1

A2,B2,C2 A3,B3,C3 “

If we wanted to provide these input parameters:

"type": "dict"

The output of this step will be:

[
  ["A1", "B1", "C1"],
  ["A2", "B2", "C2"],
  ["A3", "B3", "C3"],
]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - csv:
        type: dict

If the incoming data to the step is:

"First,Last,Email

Jonny,Lask,jl@silo.com Bobby,Sart,bs@silo.com Karen,Sift,ks@silo.com “

If we wanted to provide these input parameters:

"type": "dict"

The output of this step will be:

[
  {"First": "Jonny", "Last": "Lask", "Email": "jl@silo.com"},
  {"First": "Bobby", "Last": "Sart", "Email": "bs@silo.com"},
  {"First": "Karen", "Last": "Sift", "Email": "ks@silo.com"},
]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - csv:
        type: dict

If the incoming data to the step is:

"booker12,9012,Rachel,Booker"

If we wanted to provide these input parameters:

"fieldnames": ["Username", "Identifier", "First name", "Last name"]
"type": "dict"

The output of this step will be:

[
  {'Username': 'booker12', 'Identifier': '9012', 'First name': 'Rachel', 'Last name': 'Booker' }
]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - csv:
        fieldnames:
          - Username
          - Identifier
          - First name
          - Last name
        type: dict

JSON¶

The JSON Data Parser converts a JSON string into a python dictionary.

Framework Name

json

Example Usage¶

If the incoming data to the step is:

'{
   "project": "low_code", "tickets": {"t123": "selector work",
   "t321": "parser work"}, "name": "Josh", "teams": '["rebel", "sprinteastwood"]
 }'

The output of this step will be:

{
  "name": "Josh",
  "project": "low_code",
  "teams": ["rebel", "sprinteastwood"],
  "tickets": {"t123": "selector work", "t321": "parser work",},
}

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - json

Regex¶

The Regex step uses regular expressions to extract the required information and returns a dictionary.

Step details:

Framework Name	`regex_parser`
Parameters	regex - Regular expression. method - Methods supported by the `re` lib. flags - Flags supported by the `re` lib.
Reference	https://docs.python.org/3/library/re.html

Example Usage¶

If the incoming data to the step is:

"some text where the regex will be applied"

If we wanted to provide these input parameters:

flags: "I","M"
method: search
regex: "(.*)"

The output of this step will be:

{
  "match": "some text where the regex will be applied",
  "groups": ("some text where the regex will be applied",),
  "span": (0, 41)
}

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - regex_parser
        flags:
          - I
          - M
        method: search
        regex: "(.*)"

String Float¶

The String Float parses a string to float. If the string contains multiple floats, it returns a list of them.

Framework Name

stringfloat

Example Usage¶

If the incoming data to the step is:

"1.1 , 2.2"

The output of this step will be:

[1.1, 2.2]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - stringfloat

Paginators¶

paginator_offset_limit¶

The Paginator Offset Limit processor is the step to perform pagination over offset-limit-based APIs that handle GET Request. It raises the exception silo.low_code_steps.rest.HTTPPageRequest to rewind the execution to the previous HTTP network requestor.

Step details:

Framework Name	`paginator_offset_limit`
Parameters	limit - Integer, the maximum number of entries to retrieve. path - Path to the results.

Example of use¶

To use this step, you also need to use the HTTP requestor and provide the corresponding parameters as shown below:

low_code:
  id: eastwood1
  version: 2
  steps:
    - http:
        uri: /api/device
        params:
          limit: 10
          offset: 0
    - json
    - paginator_offset_limit:
        limit: 5
        path: result_set
    - jmespath:
        index: true
        value: "values(@)|[].result_set[].{_index: URI, _value: description}"

The parameters for the Paginator Offset Limit are optional. If the limit is not provided, the defined limit in the HTTP requestor will remain unmodified. The value for this parameter must be greater than 0. If the results are not directly in the root node, you need to provide a path to the results, the same way you would for the Simple Key selector. This processor will enable the rewind capability to return to the HTTP requestor. When there are no more entries to retrieve, the loop will finish. Next, you will require a step to combine the results. In the example above, you can see how the results were combined by using the JMESPath step.

Let’s say that the payload returned by the HTTP looks like this:

{
  'searchspec':{},
  'total_matched':7,
  'total_returned':3,
  'result_set':[
    {
      'URI': '/api/device/4',
      'description': 'AutoRegUser'
    },
    {
      'URI': '/api/device/5',
      'description': 'AutoAdmin'
    },
    {
       'URI': '/api/device/8',
       'description': 'toolkit device'
    },
  ],
}

If we provide the following input parameters:

limit: 3
path: result_set

The step will raise the exception, and the rewind process will allow returning to the HTTP requestor to get the following three entries.

The Snippet Argument should look like this:

- paginator_offset_limit:
     limit: 3
     path: result_set

When there are no more entries to retrieve, the step will return a dictionary with all the results, as you can see below:

{'offset_3': {'result_set': [{'URI': '/api/device/1',
                              'description': 'REST toolkit device'},
                             {'URI': '/api/device/2',
                              'description': 'snadmin'},
                             {'URI': '/api/device/3',
                              'description': 'em7admin'}],
              'searchspec': {},
              'total_matched': 7,
              'total_returned': 3},
 'offset_6': {'result_set': [{'URI': '/api/device/4',
                              'description': 'AutoRegUser'},
                             {'URI': '/api/device/5',
                              'description': 'AutoAdmin'},
                             {'URI': '/api/device/8',
                              'description': 'toolkit device'}],
              'searchspec': {},
              'total_matched': 7,
              'total_returned': 3},
 'offset_9': {'result_set': [{'URI': '/api/device/9',
                              'description': 'device'}],
              'searchspec': {},
              'total_matched': 7,
              'total_returned': 1}}

Note

The Paginator Offset Limit has a default maximum number of iterations of 100. More information about the maximum number of iterations can be found at Rewind / Collect more data

Selectors¶

JMESPath¶

JMESPath is a query language for JSON data. The JMESPath step can be used on data after being JSON parsed. It uses a path expression as a parameter to specify the location of the desired element (or a set of elements). Paths use the dot notation.

The JMESPath step accepts one path expression. Additionally, the JMESPath step provides a capability to build custom indexable output. This requires the expression path to be a multiselect hash with defined _index and _value keys. See below for more information.

Step details:

Framework Name	`jmespath`
Parameters	value: path.to[data] (always required) index: perform index,value merge (default: false)
Reference	jmespath Url

Example Usage¶

If the incoming data to the step is:

{
  "data":
  [
    {
      "id": 1,
      "name": "item1",
      "children": {
        "cname": "myname1"
      }
    },
    {
      "id": 2,
      "name": "item2",
      "children": {
        "cname": "myname2"
      }
    }
  ]
}

If we provide the following input parameters:

merge: true
value: "data[].{_index: id, _value: children.cname}"

The output of this step will be:

[(1, 'myname1'), (2, 'myname2')]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - json
    - jmespath:
        index: true
        value: "data[].{_index: id, _value: children.cname}"

If we provide the following input parameters:

merge: false
value: "data[].children.cname"

The output of this step will be:

["myname1","myname2"]

The Snippet Argument should look like this:

low_code:
  id: my_request2
  version: 2
  steps:
    - <network_request>
    - json
    - jmespath:
        value: "data[].children.cname"

JSONPath¶

The JSONPath is a query language for JSON data. The JSONPath selector can be used on any data once it has been parsed. It uses path expressions as parameters to specify a path to the element (or a set of elements). Paths use the dot notation.

The JSONPath parser can accept one or two paths. If one is given, that is the path to the data. If two are given, that provides the path to the index and the path to the data.

Step details:

Framework Name	`jsonpath`
Parameters	value: $.path.to[data] (always required) index: $.path.to[index] (used for explicit indexing)
Reference	jsonpath Url

Example Usage¶

If the incoming data to the step is:

{
  "data":
  [
    {
      "id": 1,
      "name": "item1",
      "children": {
        "cname": "myname1"
      }
    },
    {
      "id": 2,
      "name": "item2",
      "children": {
        "cname": "myname2"
      }
    }
  ]
}

If we wanted to provide these input parameters:

value: "$.data[*].children.cname"
index: "$.data[*].id"

The output of this step will be:

[(1, 'myname1'), (2, 'myname2')]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - jsonpath:
        value: "$.data[*].children.cname"
        index: "$.data[*].id

Simple Key¶

The Simple Key selector is for dictionaries, lists, etc. It returns the specified key by using periods as separators.

Step details:

Framework Name	`simple_key`
Format	path.to.value

Example Usage¶

If the incoming data to the step is:

{
  "key": {
    "subkey": {
      "subsubkey": "subsubvalue",
      "num": 12
    }
  }
}

If we wanted to provide these input parameters:

"key.subkey.num"

The output of this step will be:

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - simple_key: "key.subkey.num"

If the incoming data to the step is:

{
  "key": {
    1: {
      "2": ["value0", "value1", "value2"]
    }
  }
}

If we wanted to provide these input parameters:

"key.1.2.0"

The output of this step will be:

"value0"

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - simple_key: "key.1.2.0"

If the incoming data to the step is:

[
  {"id": "value0"},
  {"id": "value1"}
]

If we wanted to provide these input parameters:

"id"

The output of this step will be:

["value0", "value1"]

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - simple_key: "id"

Ungrouped¶

Store Data¶

The store_data step allows a user to store the current result into a key of their choosing. This enables a pre-processed dataset to be used at a later time where it may be necessary to have the full result. An example of this could be trimming data you do not need but requiring the whole payload to make a decision in the future.

This step does not update request_id so it will not affect the automatic cache_key generated by the Snippet Framework.

Framework Name	`store_data`
key	storage_key

For example, if you wanted to store the current result into the key storage_key, you would use the following step definition:

store_data: storage_key

To access this data in a later step, you would use the following:

result_container.metadata["storage_key"]

Cachers¶

cache_writer¶

The cache_writer step enables user defined caching (read and write) to SL1’s DB instance.

Step details:

Framework Name	`cache_write`
Parameters	key:	Metadata key that the DB will use for cache R/W (default: request_id)
	reuse_for:	Time in minutes that specifies valid cache entry duration times to be Read. (default: 5)
	cleanup_after:	Time in minutes that specifies when caches will expire and should be removed from the DB (default: 15)

Note

When the parameter reuse_for is defined as 0 minutes, the cache_writer will not allow a fast-forward in the pipeline execution.

Note

When the parameter cleanup_after is defined to be smaller than reuse_for, the cache_writer will fail to find valid data and run through the step execution up to the point of cache_writer.

Example Usage¶

Below is an example where we want to make a network request, process the data (json->dict) and then select a subset of that data.

The Snippet Argument should look like this:

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - json
    - simple_key: "id"

Let’s assume that the network request and json processing are steps that we would like to cache and possibly reuse in another collection and/or dynamic application. I am going use a custom key, here with a cache reuse time, reuse_for, of 5 minutes and a clean up, cleanup_after on my cache entries after 15 minutes.

low_code:
  id: my_request
  version: 2
  steps:
    - <network_request>
    - json
    - cache_writer:
        key: here
        reuse_for: 5
        cleanup_after: 15
    - simple_key: "id"

It there is a cache entry that is 5 minutes or newer since the start of the collection cycle the step will read the cached value and fast forward to the simple_key step.