Steps

Steps are the fundamental building blocks used to create Dynamic Applications. These are the most important aspects of the toolkits. When building Dynamic Applications, a set of Steps are defined that are then executed by the Snippet Framework to obtain the defined collection object.

The snippet framework executes these steps in the order they are defined in the snippet arguments for each collection object in a Dynamic Application.

The following is a list of all Steps which are common to ALL the toolkits and also the steps that are only found in the REST Toolkit.

cache_writer

The cache_writer step enables user defined caching (read and write) to SL1’s DB instance.

Step details:

Framework Name	`cache_write`
Parameters	key:	Metadata key that the DB will use for cache R/W (default: request_id)
	reuse_for:	Time in minutes that specifies valid cache entry duration times to be Read. (default: 5)
	cleanup_after:	Time in minutes that specifies when caches will expire and should be removed from the DB (default: 15)

Note

When the parameter reuse_for is defined as 0 minutes, the cache_writer will not allow a fast-forward in the pipeline execution.

Note

When the parameter cleanup_after is defined to be smaller than reuse_for, the cache_writer will fail to find valid data and run through the step execution up to the point of cache_writer.

Example - Writing to Cache

Below is an example where we want to make a network request, process the data (json->dict) and then select a subset of that data.

The Snippet Argument should look like this:

low_code:
  version: 2
  steps:
    - <network_request>
    - json
    - simple_key: "id"

Let’s assume that the network request and json processing are steps that we would like to cache and possibly reuse in another collection and/or Dynamic Application. I am going use a custom key, here with a cache reuse time, reuse_for, of 5 minutes and a clean up, cleanup_after on my cache entries after 15 minutes.

low_code:
  version: 2
  steps:
    - <network_request>
    - json
    - cache_writer:
        key: here
        reuse_for: 5
        cleanup_after: 15
    - simple_key: "id"

It there is a cache entry that is 5 minutes or newer since the start of the collection cycle the step will read the cached value and fast forward to the simple_key step.

csv

The csv step wraps the python standard library’s CSV module to parse CSV files. It returns the parsed data in a list of dictionaries or lists.

Note

The line terminator must be one of the following \r\n, \r, \n, \n\r.

Note

The jc step provides a csv parser.

Step details:

Framework Name	`csv`
Parameters	All the arguments inside: `csv.DictReader`, `csv.reader`
Reference	https://docs.python.org/3/library/csv.html

The following Step Arguments are passed transparently to the library as defined in csv.DictReader and this should be considered the source of truth.

delimiter: , - This defines the delimiter used. The default value for this is a comma.
fieldnames: - This is only needed when the first row does NOT contain the field names. In the first example the first row has the field names. In the second case the field names are explicitly defined. Note that these values will take precedence when the first row contains labels.
restkey: string - This defines the fieldname that will be used when there are additional entries in a row. All additional entries will be placed in this field. The default is None.
restval: string - This defines the values that will be placed into fields when there are not enough items in the row. The default is None.

Example - Parsing a CSV Response

Dictionary Output

Consider the following Snippet Argument.

low_code:
  id: my_request
  version: 2
  steps:
    - static_value: "Username,Identifier,Onetime password,Recovery Code,First Name,Last Name,Dept,location\r\n
    booker12,9012,12se74,rb9012,Rachel,Booker,Sales,Manchester\r\n
    grey07,2070,04ap67,lg2070,Laura,Grey,Depot,London\r\n
    johnson81,4081,30no86,cj4081,Craig,Johnson,Depot,London\r\n
    jenkins46,9346,14ju73,mj9346,Mary,Jenkins,Engineering,Manchester\r\n
    smith79,5079,09ja61,js5079,Jamie,Smith,Engineering,Manchester\r\n"
    - csv:
      type: dict

Output

[
    OrderedDict(
        [
            ("Username", " booker12"),
            ("Identifier", "9012"),
            ("Onetime password", "12se74"),
            ("Recovery Code", "rb9012"),
            ("First Name", "Rachel"),
            ("Last Name", "Booker"),
            ("Dept", "Sales"),
            ("location", "Manchester"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " grey07"),
            ("Identifier", "2070"),
            ("Onetime password", "04ap67"),
            ("Recovery Code", "lg2070"),
            ("First Name", "Laura"),
            ("Last Name", "Grey"),
            ("Dept", "Depot"),
            ("location", "London"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " johnson81"),
            ("Identifier", "4081"),
            ("Onetime password", "30no86"),
            ("Recovery Code", "cj4081"),
            ("First Name", "Craig"),
            ("Last Name", "Johnson"),
            ("Dept", "Depot"),
            ("location", "London"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " jenkins46"),
            ("Identifier", "9346"),
            ("Onetime password", "14ju73"),
            ("Recovery Code", "mj9346"),
            ("First Name", "Mary"),
            ("Last Name", "Jenkins"),
            ("Dept", "Engineering"),
            ("location", "Manchester"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " smith79"),
            ("Identifier", "5079"),
            ("Onetime password", "09ja61"),
            ("Recovery Code", "js5079"),
            ("First Name", "Jamie"),
            ("Last Name", "Smith"),
            ("Dept", "Engineering"),
            ("location", "Manchester"),
        ]
    ),
]

List Output

Below is the same Snippet Argument as above without the type specified.

low_code:
  id: my_request
  version: 2
  steps:
    - static_value: "Username,Identifier,Onetime password,Recovery Code,First Name,Last Name,Dept,location\r\n
    booker12,9012,12se74,rb9012,Rachel,Booker,Sales,Manchester\r\n
    grey07,2070,04ap67,lg2070,Laura,Grey,Depot,London\r\n
    johnson81,4081,30no86,cj4081,Craig,Johnson,Depot,London\r\n
    jenkins46,9346,14ju73,mj9346,Mary,Jenkins,Engineering,Manchester\r\n
    smith79,5079,09ja61,js5079,Jamie,Smith,Engineering,Manchester\r\n"
    - csv:

Output

[
    [
        "Username",
        "Identifier",
        "Onetime password",
        "Recovery Code",
        "First Name",
        "Last Name",
        "Dept",
        "location",
    ],
    [
        " booker12",
        "9012",
        "12se74",
        "rb9012",
        "Rachel",
        "Booker",
        "Sales",
        "Manchester",
    ],
    [" grey07", "2070", "04ap67", "lg2070", "Laura", "Grey", "Depot", "London"],
    [" johnson81", "4081", "30no86", "cj4081", "Craig", "Johnson", "Depot", "London"],
    [
        " jenkins46",
        "9346",
        "14ju73",
        "mj9346",
        "Mary",
        "Jenkins",
        "Engineering",
        "Manchester",
    ],
    [
        " smith79",
        "5079",
        "09ja61",
        "js5079",
        "Jamie",
        "Smith",
        "Engineering",
        "Manchester",
    ],
]

Using the Snippet Arguments above we can use the JMESPath step to select certain rows or fields for a Collection Object.

No Fieldnames

For the case where the first row does not contain the fieldnames, then the following Snippet Argument can be used:

low_code:
  version: 2
  steps:
    - static_value: "
    booker12,9012,12se74,rb9012,Rachel,Booker,Sales,Manchester\r\n
    grey07,2070,04ap67,lg2070,Laura,Grey,Depot,London\r\n
    johnson81,4081,30no86,cj4081,Craig,Johnson,Depot,London\r\n
    jenkins46,9346,14ju73,mj9346,Mary,Jenkins,Engineering,Manchester\r\n
    smith79,5079,09ja61,js5079,Jamie,Smith,Engineering,Manchester\r\n"
    - csv:
      type: dict
      fieldnames:
          - Username
          - Identifier
          - One-time password
          - Recovery code
          - first name
          - last name
          - department
          - location

Output

[
    OrderedDict(
        [
            ("Username", " booker12"),
            ("Identifier", "9012"),
            ("One-time password", "12se74"),
            ("Recovery code", "rb9012"),
            ("first name", "Rachel"),
            ("last name", "Booker"),
            ("department", "Sales"),
            ("location", "Manchester"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " grey07"),
            ("Identifier", "2070"),
            ("One-time password", "04ap67"),
            ("Recovery code", "lg2070"),
            ("first name", "Laura"),
            ("last name", "Grey"),
            ("department", "Depot"),
            ("location", "London"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " johnson81"),
            ("Identifier", "4081"),
            ("One-time password", "30no86"),
            ("Recovery code", "cj4081"),
            ("first name", "Craig"),
            ("last name", "Johnson"),
            ("department", "Depot"),
            ("location", "London"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " jenkins46"),
            ("Identifier", "9346"),
            ("One-time password", "14ju73"),
            ("Recovery code", "mj9346"),
            ("first name", "Mary"),
            ("last name", "Jenkins"),
            ("department", "Engineering"),
            ("location", "Manchester"),
        ]
    ),
    OrderedDict(
        [
            ("Username", " smith79"),
            ("Identifier", "5079"),
            ("One-time password", "09ja61"),
            ("Recovery code", "js5079"),
            ("first name", "Jamie"),
            ("last name", "Smith"),
            ("department", "Engineering"),
            ("location", "Manchester"),
        ]
    ),
]

http

The HTTP Data Requestor creates and executes an HTTP/HTTPS call to a specified endpoint and returns the results. The endpoint can be defined in 2 ways:

uri: The host and port in the credential will be used as the base endpoint and the uri defined in the step_args will be appended to it.

url: The endpoint defined as the url in the step_args will be called directly.

Step details:

Framework Name	`http`
Supported Credentials	Basic, SOAP/XML
Supported Fields of Basic Cred.	Hostname/IP Port Username Password
Supported Fields of SOAP Cred.	HTTP Auth User HTTP Auth Password HTTP Headers CURL Options, only SSLVERIFYPEER Proxy Hostname/IP Proxy Port Proxy User Proxy Password URL (Host, Port) Proxy Port
Parameters	method: delete, get (default), head, options, patch, post, put uri/url: The endpoint to call using the defined method response_type: text (default) [str], raw [returned data] check_status_code: True (default), False

Note

This step supports all the parameters mentioned in requests.Session.request except the hooks parameter. The parameters defined in the step will take precedent over what is defined in the credential. For example, if you define verify: False in the credential but verify: True in the step parameters, the verify=True will be used in the request.

Note

Any parameters that are specified by the step and the configuration will attempt to be combined.

Dictionary: Merge the two together, with user-provided values as the priority.

Other: Value replacement

Calling a Relative Endpoint

To access the API of an SL1 System, the full URL would be:

https://SL1_IP_ADDRESS/api/account

If the SL1_IP_ADDRESS is defined in the credential, the relative URI can be used instead:

/api/account

For this example, we are assuming the base endpoint (SL1_IP_ADDRESS) is defined in the credential, so we can call the uri endpoint like this:

low_code:
  version: 2
  steps:
    - http:
        uri: "/api/account"

The output of this step:

{
  "searchspec":{},
  "total_matched":4,
  "total_returned":4,
  "result_set":
  [
    {
      "URI":"/api/account/2",
      "description":"AutoAdmin"
    },
    {
      "URI":"/api/account/3",
      "description":"AutoRegUser"
    },
    {
      "URI":"/api/account/1",
      "description":"em7admin"
    },
    {
      "URI":"/api/account/4",
      "description":"snadmin"
    }
  ],
}

Calling a Direct Endpoint

To call an HTTP endpoint directly, define the url in the step_args. For example, say we want to call an API to determine the top 20 most popular board games right now. The full URL would look something like this:

https://boardgamegeek.com/xmlapi2/hot?boardgame

We tell the http step to call that endpoint by setting it as the url. After making the call, we can use the jc step to parse the response and finally use the jmespath selector to select our values.

low_code:
version: 2
steps:
  - http:
      url: https://boardgamegeek.com/xmlapi2/hot?boardgame
  - jc: xml
  - jmespath:
      value: items.item[:20].name

Response Status Code Checking

When the parameter check_status_code is set to True (default), and the response’s status code meets the following condition:

\(400 <= status code < 600\)

An exception will be raised, thus stopping the current collection.

When the parameter check_status_code is set to False, no exception will be raised for any status code value.

Pagination Support

A custom step is required to raise the exception silo.low_code_steps.rest.HTTPPageRequest to rewind execution back to the previous network requestor. This exception is specific to the http and requires a dictionary as its own argument. The dictionary can either replace or update the existing step_args dictionary passed to the http step.

If our Snippet Argument looked like this:

low_code:
    version: 2
    setps:
      - http:
          uri: "/account"
      - pagination_trimmer
      - pagination_request:
          index: "request_key"
          replace: True

Our pagination_request step could look like this:

@register_processor(type=REQUEST_MORE_DATA_TYPE)
def pagination_request(result, step_args):

  if result:
    # Replacement of step_args
    raise HTTPPageRequest({"uri": "/account", "params": result}, index=step_args.get("index"), replace=step_args.get("replace", False))

This assumes that the result will contain the next pagination step arguments. The step issues the HTTPPaginateRequest and sets the new step_args with the first positional parameter. With the kwarg replace set to True, the http step will receive new step_args.

jc

JC is a third-party library that enables easy conversion of text output to python primitives. While this is primarily used for *nix parsing, it includes other notable parsers such as xml, ini, and yaml to name a few. The entire list of supported formats and their respective outputs can be found on their github.

Note

The Snippet Framework does not support streaming parsers (parsers that end in _s). These will not appear in the list of available parsers.

Step details:

Framework Name	`jc`
Parameters	parser_name - Name of the parser to utilize
Reference	https://github.com/kellyjonbrazil/jc/tree/v1.25.0

There are currently 149 parsers available in the installed version of jc v1.25.0. The list of available parsers are as follows:

acpi, airport, arp, asciitable, asciitable_m, blkid, bluetoothctl, cbt, cef, certbot, chage, cksum, clf, crontab, crontab_u, csv, curl_head, date, datetime_iso, debconf_show, df, dig, dir, dmidecode, dpkg_l, du, efibootmgr, email_address, env, file, find, findmnt, finger, free, fstab, git_log, git_ls_remote, gpg, group, gshadow, hash, hashsum, hciconfig, history, host, hosts, http_headers, id, ifconfig, ini, ini_dup, iostat, ip_address, iptables, ip_route, iw_scan, iwconfig, jar_manifest, jobs, jwt, kv, kv_dup, last, ls, lsattr, lsb_release, lsblk, lsmod, lsof, lspci, lsusb, m3u, mdadm, mount, mpstat, netstat, nmcli, nsd_control, ntpq, openvpn, os_prober, os_release, passwd, path, path_list, pci_ids, pgpass, pidstat, ping, pip_list, pip_show, pkg_index_apk, pkg_index_deb, plist, postconf, proc, ps, resolve_conf, route, rpm_qi, rsync, semver, sfdisk, shadow, srt, ss, ssh_conf, sshd_conf, stat, swapon, sysctl, syslog, syslog_bsd, systemctl, systemctl_lj, systemctl_ls, systemctl_luf, systeminfo, time, timedatectl, timestamp, toml, top, tracepath, traceroute, tune2fs, udevadm, ufw, ufw_appinfo, uname, update_alt_gs, update_alt_q, upower, uptime, url, ver, veracrypt, vmstat, w, wc, who, x509_cert, x509_csr, xml, xrandr, yaml, zipinfo, zpool_iostat, zpool_status

Step Arguments

Supplying additional key-value pairs in the step_args will pass them through to the parser. For example, if you needed to run a parser example_parser_name that expected an additional argument such as split, you would use the following:

jc:
  parser_name: example_parser_name
  split: ","

If no additional parameters need to be supplied, you can specify the parser_name directly as the step argument.

jc: example_parser_name

Example - Parsing a CLI Response

One of the commands that the jc step supports parsing for is iostat. We can use the following Snippet Argument to define iostat as the parser_name for the jc step and extract the current io statistics from the machine we ssh into:

low_code:
  version: 2
  steps:
    - ssh:
        command: iostat
    - jc: iostat
    - jmespath:
        value: '[?type==`device`].{_index: device, _value: kb_read_s}'
        index: true

Suppose we received the following result from running the iostat command:

  avg-cpu(perc):   user    nice  system  iowait   steal    idle
                   5.25    0.01    2.74    0.08    0.00   91.93

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              11.25        46.33       176.47   84813251  323042096
scd0              0.00         0.00         0.00          1          0
dm-0              1.34        38.36         2.15   70222779    3930730
dm-1              0.13         0.14         0.36     261788     665732

The jc parser will parse the above output into JSON, which we can then use to extract the desired information out of using a selector. In this example we use the jmespath selector to extract the kb_read_s value for each device.

{
    "percent_user": 5.25,
    "percent_nice": 0.01,
    "percent_system": 2.74,
    "percent_iowait": 0.08,
    "percent_steal": 0.0,
    "percent_idle": 91.93,
    "type": "cpu"
},
{
    "device": "sda",
    "tps": 11.25,
    "kb_read_s": 46.29,
    "kb_wrtn_s": 176.45,
    "kb_read": 84813635,
    "kb_wrtn": 323281369,
    "type": "device"
},
{
    "device": "scd0",
    "tps": 0.0,
    "kb_read_s": 0.0,
    "kb_wrtn_s": 0.0,
    "kb_read": 1,
    "kb_wrtn": 0,
    "type": "device"
},
{
    "device": "dm-0",
    "tps": 1.34,
    "kb_read_s": 38.33,
    "kb_wrtn_s": 2.15,
    "kb_read": 70222907,
    "kb_wrtn": 3931900,
    "type": "device"
},
{
    "device": "dm-1",
    "tps": 0.13,
    "kb_read_s": 0.14,
    "kb_wrtn_s": 0.36,
    "kb_read": 261788,
    "kb_wrtn": 665732,
    "type": "device"
}

After using jmespath to select only device information, the final result would look like:

{
    "dm-0": 38.33,
    "dm-1": 0.14,
    "scd0": 0.0,
    "sda": 46.29
}

jmespath

The JMESPath step is our most performant step for selecting data. This step uses a 3rd party library, JMESPath. See JMESPath URL. JMESPath is a query language for JSON that is used to extract and transform elements from a JSON document.

It is important to understand the multi-select-hash as this is used to index the data. When multiple Collection Objects are placed in a group, it is recommended to always index the data explicitly. That is, each group of Collection Objects that are stored within a Dynamic Application should always use the index: True option and ensure that a unique value is used for the index.

Step details:

Framework Name

jmespath

Parameters

index: True or False (default: False) This specifies if the data is to be explicitly indexed. This should always be used whenever multiple collection objects are grouped.

value: string defining the path to the data (required)

Reference

JMESPath URL

Example - Selecting Attributes with jmespath

Selection

Consider the following Snippet Argument which fetches earthquake data detected within the last hour.

low_code:
  version: 2
  steps:
    - http:
        url: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson

Output

{
  "type": "FeatureCollection",
  "metadata": {
    "generated": 1706266177000,
    "url": "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson",
    "title": "USGS All Earthquakes, Past Hour",
    "status": 200,
    "api": "1.10.3",
    "count": 12
  },
  "features": [
    {
      "type": "Feature",
      "properties": {
        "mag": 1.4,
        "place": "21 km SE of Valdez, Alaska",
        "time": 1706265719345,
        "updated": 1706265802101,
        "tz": "",
        "url": "https://earthquake.usgs.gov/earthquakes/eventpage/ak02417669tq",
        "detail": "https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak02417669tq.geojson",
        "felt": "",
        "cdi": "",
        "mmi": "",
        "alert": "",
        "status": "automatic",
        "tsunami": 0,
        "sig": 30,
        "net": "ak",
        "code": "02417669tq",
        "ids": ",ak02417669tq,",
        "sources": ",ak,",
        "types": ",origin,phase-data,",
        "nst": "",
        "dmin": "",
        "rms": 0.52,
        "gap": "",
        "magType": "ml",
        "type": "earthquake",
        "title": "M 1.4 - 21 km SE of Valdez, Alaska"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [-146.0674, 60.9902, 27.5]
      },
      "id": "ak02417669tq"
    }
  ],
  "bbox": [-150.6143, 33.3551667, 0.1, -116.4315, 64.4926, 115.9]
}

Suppose we wanted to show only the magnitudes of the earthquakes in this list. This can be accomplished by using the JMESPath step given the following query string shown below.

low_code:
version: 2
steps:
  - http:
      url: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson
  - json
  - jmespath:
      value: features[].properties.mag

Output

[2.18000007, 0.9, 0.67, 1.4, 1.2, 0.71, 1.5, 0.76, 1]

Filtering

Using the same example Snippet Argument as above, we wish to only display magnitudes greater than 1.0. For this we will use the filtering feature of JMESPath.

low_code:
  version: 2
  steps:
    - http:
        url: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson
    - json
    - jmespath:
        value: features[?properties.mag > `1.0`]

Output

[
  {
      "type": "Feature",
      "properties": {
          "mag": 2.4,
          "place": "10 km W of Point MacKenzie, Alaska",
          "time": 1706268933977,
          "updated": 1706269019706,
          "tz": None,
          "url": "https://earthquake.usgs.gov/earthquakes/eventpage/ak024176qd6d",
          "detail": "https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak024176qd6d.geojson",
          "felt": None,
          "cdi": None,
          "mmi": None,
          "alert": None,
          "status": "automatic",
          "tsunami": 0,
          "sig": 89,
          "net": "ak",
          "code": "024176qd6d",
          "ids": ",ak024176qd6d,",
          "sources": ",ak,",
          "types": ",origin,phase-data,",
          "nst": None,
          "dmin": None,
          "rms": 0.43,
          "gap": None,
          "magType": "ml",
          "type": "earthquake",
          "title": "M 2.4 - 10 km W of Point MacKenzie, Alaska",
      },
      "geometry": {"type": "Point", "coordinates": [-150.1724, 61.3705, 34.9]},
      "id": "ak024176qd6d",
  },
  ...
]

Reducing Payload Sizes

One common performance issue is working a large payload from an API. In most cases, this data must be reduced to a few columns for further processing. We will again reuse the previous Snippet Argument and reduce the data to just the mag, time, and place columns. This query uses a multi-select-hash to save our fields of interest.

low_code:
  version: 2
  steps:
    - http:
        url: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson
    - json
    - jmespath:
        value: "features[].properties.{mag: mag, place: place, time: time}"

The output is a list of dictionaries with the fields that were requested.

[
    {"mag": 1.6, "place": "10 km E of Willits, CA", "time": 1706276615270},
    {"mag": 1.6, "place": "44 km NNW of Beluga, Alaska", "time": 1706276105156},
    {"mag": 2.3, "place": "30 km NW of Karluk, Alaska", "time": 1706275723157},
    {"mag": 1.8, "place": "42 km W of Tyonek, Alaska", "time": 1706275673972},
    {"mag": 1.74000001, "place": "17 km W of Volcano, Hawaii", "time": 1706275545280},
    {"mag": 1.9, "place": "4 km SSW of Salcha, Alaska", "time": 1706274091738},
    {"mag": 1.12, "place": "7 km NE of Pala, CA", "time": 1706273608330},
]

Alternatively, we could also use a multi-select-list.

low_code:
  version: 2
  steps:
    - http:
        url: https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson
    - json
    - jmespath:
        value: "features[].properties.[mag, place, time]"

The output is now a list of the three fields that were specified.

[
    [1.1, "6 km SW of Salcha, Alaska", 1706276986399],
    [1.6, "10 km E of Willits, CA", 1706276615270],
    [1.6, "44 km NNW of Beluga, Alaska", 1706276105156],
    [2.3, "30 km NW of Karluk, Alaska", 1706275723157],
    [1.8, "42 km W of Tyonek, Alaska", 1706275673972],
    [1.74000001, "17 km W of Volcano, Hawaii", 1706275545280],
    [1.9, "4 km SSW of Salcha, Alaska", 1706274091738],
    [1.12, "7 km NE of Pala, CA", 1706273608330],
]

Indexing

For the following examples the static_value step provides the equivalent converted json output.

[
  { "URI": "/api/account/2", "color": "red", "description": "AutoAdmin" },
  { "URI": "/api/account/3", "color": "yellow", "description": "AutoRegUser" },
  { "URI": "/api/account/1", "color": "green", "description": "user" },
  { "URI": "/api/account/4", "color": "blue", "description": "snadmin" }
]

This first Collection Object’s Snippet Argument selects the description for each URI.

low_code:
  version: 2
  steps:
    - static_value:
        -   URI: "/api/account/2"
            color: red
            description: AutoAdmin
        -   URI: "/api/account/3"
            color: yellow
            description: AutoRegUser
        -   URI: "/api/account/1"
            color: green
            description: user
        -   URI: "/api/account/4"
            color: blue
            description: snadmin
    - jmespath:
        index: true
        value: '[].{_index: URI, _value: description}'

Output

{
  "/api/account/1": "user",
  "/api/account/2": "AutoAdmin",
  "/api/account/3": "AutoRegUser",
  "/api/account/4": "snadmin"
}

The next Collection Object’s Snippet Argument selects color for each URI.

low_code:
  version: 2
  steps:
    - static_value:
        -   URI: "/api/account/2"
            color: red
            description: AutoAdmin
        -   URI: "/api/account/3"
            color: yellow
            description: AutoRegUser
        -   URI: "/api/account/1"
            color: green
            description: user
        -   URI: "/api/account/4"
            color: blue
            description: snadmin
    - jmespath:
        index: true
        value: '[].{_index: URI, _value: color}'

Output

{
  "/api/account/1": "green",
  "/api/account/2": "red",
  "/api/account/3": "yellow",
  "/api/account/4": "blue"
}

In both Collection Objects, URI is used as the index. In order to properly ingest the data into SL1, we need create a label object with the following Snippet Argument.

low_code:
  version: 2
  steps:
    - static_value:
        -   URI: "/api/account/2"
            color: red
            description: AutoAdmin
        -   URI: "/api/account/3"
            color: yellow
            description: AutoRegUser
        -   URI: "/api/account/1"
            color: green
            description: user
        -   URI: "/api/account/4"
            color: blue
            description: snadmin
    - jmespath:
        index: true
        value: '[].{_index: URI, _value: URI}'

Output

{
  "/api/account/1": "/api/account/1",
  "/api/account/2": "/api/account/2",
  "/api/account/3": "/api/account/3",
  "/api/account/4": "/api/account/4"
}

Make sure that all Collection Objects shown above are within the same group.

Notice that the index between the Collection Objects is fixed. Doing so will ensure that the collected data is properly associated within SL1.

json

The json step is used to convert a JSON string into a python object. It is commonly used to transform results from http API calls into a format we can then use with a selector step like jmespath.

Framework Name

json

Example - Converting a JSON String

If the incoming data to the step is:

'{
  "project": "low_code",
  "tickets": { "t123": "selector work", "t321": "parser work" },
  "name": "Josh", "teams": ["rebel_scrum", "sprint_eastwood"]
}'

The output of this step will be:

{
    "name": "Josh",
    "project": "low_code",
    "teams": ["rebel_scrum", "sprint_eastwood"],
    "tickets": {
        "t123": "selector work",
        "t321": "parser work"
    }
}

The Snippet Argument should look like this:

low_code:
  version: 2
  steps:
    - static_value: '{
        "project": "low_code",
        "tickets": { "t123": "selector work", "t321": "parser work" },
        "name": "Josh", "teams": ["rebel_scrum", "sprint_eastwood"]
      }'
    - json
    - jmespath:
        value: project

jsonpath

JSONPath has been deprecated. Use JMESPath instead.

low_code

The low_code Syntax is a YAML configuration format for specifying your collections. You explicitly provide the steps you want to run, in the order you need them to be executed. There are multiple versions of the low_code Syntax.

Framework Name

low_code

All versions support specifying configuration. The configuration will be all sub-elements under the step. For example, if you had a step cool_step and wanted to specify two key values, you would provide the following:

low_code:
  version: 2
  steps:
    - cool_step:
        key1: value1
        key2: value2

Note

Notice that, in the example above, key1 and key2 are indented one level beyond cool_step. Setting them at the same indentation level as cool_step, as shown below, will result in an error.

low_code:
  version: 2
  steps:
    - cool_step:
      key1: value1 # key1 is not properly indented
      key2: value2 # key2 is not properly indented

To provide a list in YAML you will need to utilize the - symbol. For example, if you had a step cool_step and wanted to specify a list of two elements, you would provide the following:

low_code:
  version: 2
  steps:
    - cool_step:
      - key1
      - key2

Version 2

Version 2 of the low_code Syntax provides more flexibility when defining the order of step execution. This version can utilize multiple versions of Requestors (if supported) and allows for steps to run before a Requestor executes.

Format

low_code:
  version: 2
  steps:
    - static_value: '{"key": "value"}'
    - json
    - simple_key: "key"

id: Identification for the request.

Note

If this value is not specified, the Snippet Framework will automatically create one. This allows for easier tracking when debugging when an ID is not required for the collection.
version: Specify the version of the low_code Syntax.
steps: Order of the steps for the Snippet Framework to execute.

Version 1

Version 1 was the original low_code syntax. It allowed for a single Requestor and any number of processors. It lacks support for multiple Requestors so it is not preferred.

Format

low_code:
  network:
    static_value: '{"key": "value"}'
  processing:
    - json
    - simple_key: "key"

id: Identification for the request.

Note

If this value is not specified, the Snippet Framework will automatically create one. This allows for easier tracking when debugging when an ID is not required for the collection.
version: Specify the version of the low_code Syntax. If not provided, it will default to 1.
network: Section for the data requester step.
processing: Section for listing the steps required to transform your data to the desired output for SL1 to store.

paginator_offset

The paginator_offset step works in conjunction with the http step to support offset- or paged-based API queries.

Step details:

Framework Name	`paginator_offset`
Parameters	limit - Integer, the maximum number of entries to retrieve. (always required) path - Path to the results. limit_qs - Key within the URL query string that contains the limit offset_qs - Key within the URL query string that contains the offset pagination_increment - Override the offset increase a specific amount. Default: limit from URL

Note

The Paginator Offset Limit has a default maximum number of iterations of 50.

Note

When specifying the limit within the step, it must be less than or equal to then limit provided in the http step params (step_limit <= http_limit). Specifying a limit greater than the value from the http step params will cause the paginator to not collect additional data.

Example - Offset Pagination

The following example of the paginator working with SL1 API. In this example the limit is set to a low value to show the effects of pagination. There are 6 accounts on the target SL1 and thus 3 API calls will be made. The output of each API call appears as follows:

[{"URI": "/api/account/2", "description": "AutoAdmin"}, {"URI": "/api/account/3", "description": "AutoRegUser"}]
[{"URI": "/api/account/1", "description": "em7admin"}, {"URI": "/api/account/4", "description": "snadmin"}]
[{"URI": "/api/account/5", "description": "Test Account"}, {"URI": "/api/account/6", "description": "test person"}]

  low_code:
    version: 2
    steps:
      - http:
          uri: /api/account
          params:
            limit: 2
            hide_filterinfo: true
      - json
      - paginator_offset:
          limit: 2

The paginator step then combines the result into an ordered dictionary as shown below:

OrderedDict(
    [
        ("offset_2", [{"URI": "/api/account/2", "description": "AutoAdmin"}, {"URI": "/api/account/3", "description": "AutoRegUser"}]),
        ("offset_4", [{"URI": "/api/account/1", "description": "em7admin"}, {"URI": "/api/account/4", "description": "snadmin"}]),
        ("offset_6", [{"URI": "/api/account/5", "description": "Test Account"}, {"URI": "/api/account/6", "description": "test person"}
    ]
)

Example - Overriding Offset

In this example the API will use page instead of offset for its pagination technique. Lets assume that there are 3 pages that return results. The following requests would be made:

/api/something?limit=2

/api/something?page=2&limit=2

/api/something?page=3&limit=2

  low_code:
    version: 2
    steps:
      - http:
          uri: /api/something
          params:
            limit: 2
      - json
      - paginator_offset:
          limit: 2
          offset_qs: page
          pagination_increment: 1

regex_parser

The regex_parser enables the use of regular expression processing to select data from a string. It wraps the python standard library re module.

Note

All re methods are supported except for purge, compile, and escape. For methods that require additional parameters, check the re documentation to find usage and parameter names, which can be passed directly into the step as shown in example 2.

Step details:

Framework Name	`regex_parser`
Parameters	`regex` - This defined the regular expression. `method` - All methods as defined by the `re` lib are supported except purge, compile, and escape. `flags` - All flags as defined by the `re` library are supported.
Reference	Re URL

Example - Using RegEx Methods

Search

static_value is returning a string where an IP address is within a block of text. The regex_parser’s search method is used to find the IP address.

low_code:
  version: 2
  steps:
    - static_value: an IP 192.168.0.42 where the regex will be applied
    - regex_parser:
        flags:
          - I
          - M
        method: search
        regex: (192.168.0.\d{1,3})

Output

{
  'match': '192.168.0.42',
  'groups': ('192.168.0.42',),
  'span': (6, 17)
}

Substitution

static_value is returning a string where a block of text contains tab separated data. Using regex_parser’s sub feature we will change tabs into commas.

low_code:
  version: 2
  steps:
    - static_value: "Sepal length\tSepal width\tPetal length\tPetal width\tSpecies\n
    5.1\t3.5\t1.4\t0.2\tI. setosa\n
    4.9\t3.0\t1.4\t0.2\tI. setosa\n
    4.7\t3.2\t1.3\t0.2\tI. setosa\n
    4.6\t3.1\t1.5\t0.2\tI. setosa\n
    5.0\t3.6\t1.4\t0.2\tI. setosa\n"
    - regex_parser:
        flags:
          - I
          - M
        method: sub
        regex: "\t"
        repl: ","
        count: 0

Output

{
  'groups': '',
  'match': 'Sepal length,Sepal width,Petal length,Petal width,Species\n'
          ' 5.1,3.5,1.4,0.2,I. setosa\n'
          ' 4.9,3.0,1.4,0.2,I. setosa\n'
          ' 4.7,3.2,1.3,0.2,I. setosa\n'
          ' 4.6,3.1,1.5,0.2,I. setosa\n'
          ' 5.0,3.6,1.4,0.2,I. setosa\n',
  'span': ''
}

simple_key

The simple_key step can be used to select a value from a dictionary or a list. simple_key has been deprecated. See examples below for migrating simple_key to JMESPath.

Note

The simple_key step cannot be used to select indexed data. Indexed data is required grouping like sets of collection objects. This step should not be used for the aforementioned usecase.

Step details:

Framework Name	`simple_key`
Format	path to the key

Example - Selecting Attributes with simple_key

Nested Key

{
  "key": {
    "subkey": {
      "subsubkey": "subsubvalue",
      "num": 12
    }
  }
}

From the provided data above we would like to select num. The Snippet Argument would be as follows.

low_code:
  version: 2
  steps:
    - static_value:
        key:
            subkey:
                subsubkey: subsubvalue,
                num: 12
    - simple_key: "key.subkey.num"

Output

The equivalent jmespath Snippet Argument is shown below.

low_code:
  version: 2
  steps:
    - static_value:
        key:
            subkey:
                subsubkey: subsubvalue,
                num: 12
    - jmespath:
        value: key.subkey.num

List Selection

{
  "key": {
    1: {
      "2": ["value0", "value1", "value2"]
    }
  }
}

From the provided data we would like to select value0. Then our Snippet Argument that would select the data is as follows.

low_code:
  version: 2
  steps:
    - static_value:
        key:
          '1':
            '2':
              - value0
              - value1
              - value2
    - simple_key: key.1.2.0

Output

"value0"

The equivalent jmespath Snippet Argument is shown below.

low_code:
  version: 2
  steps:
    - static_value:
        key:
          '1':
            '2':
              - value0
              - value1
              - value2
    - jmespath:
        value: key."1"."2"[0]

Object Selection

[
  {"id": "value0", "cheese": "swiss"},
  {"id": "value1", "cheese": "goat"}
]

From the provided data above we would like to select only the id values. Then our Snippet Argument that would select the data is as follows.

low_code:
  version: 2
  steps:
    - static_value:
        - id: value0
          cheese: swiss
        - id: value1
          cheese: goat
    - simple_key: "id"

Output

["value0", "value1"]

The equivalent jmespath Snippet Argument is shown below.

low_code:
  version: 2
  steps:
    - static_value:
        - id: value0
          cheese: swiss
        - id: value1
          cheese: goat
    - jmespath:
        value: "[*].id"

SL1 API

Below is a Snippet Argument whose static_value step is mocking a SL1 REST API payload.

low_code:
    version: 2
    steps:
      - static_value: '[{"URI":"\/api\/account\/2","description":"AutoAdmin"}
        ,{"URI":"\/api\/account\/3","description":"AutoRegUser"},
        {"URI":"\/api\/account\/1","description":"user"},
        {"URI":"\/api\/account\/4","description":"snadmin"}]'
      - json:
      - simple_key: "description"

Output

['AutoAdmin', 'AutoRegUser', 'user', 'snadmin']

The output of the simple_key step is only the description field. This could be assigned to a collection object in SL1 and would list out the descriptions.

Note

If both the URI and description fields were collected in two seperate collection objects then simple key should not be used. See JMESPath <steps-jmespath>.

The equivalent jmespath Snippet Argument is shown below.

low_code:
    version: 2
    steps:
      - static_value: '[{"URI":"\/api\/account\/2","description":"AutoAdmin"}
        ,{"URI":"\/api\/account\/3","description":"AutoRegUser"},
        {"URI":"\/api\/account\/1","description":"user"},
        {"URI":"\/api\/account\/4","description":"snadmin"}]'
      - json:
      - jmespath:
          value: "[*].description"

static_value

The static_value step is used to mock network responses. For example, instead of making the api request with the http step, the response could be mocked by using the static_value step and hardcoding the data.

Step details:

Framework Name	`static_value`
Supported Credentials	N/A

Example - Mocking a Network Response

String Input

If we wanted to mock:

"Apple1,Ball1,Apple2,Ball2"

The Snippet Argument would look like this:

low_code:
  version: 2
  steps:
    - static_value: "Apple1,Ball1,Apple2,Ball2"

Output

'Apple1,Ball1,Apple2,Ball2'

Stringified JSON

By copying a stringified JSON response you can avoid repeated REST API accesses when developing a Snippet Argument. Below is an example of a stringified JSON response that is used in conjuction with a json step.

low_code:
  version: 2
  steps:
    - static_value: "[
      {"URI":"/api/account/2","description":"AutoAdmin"},
      {"URI":"/api/account/3","description":"AutoRegUser"},
      {"URI":"/api/account/1","description":"user"},
      {"URI":"/api/account/4","description":"snadmin"}
    ]"

Output

'[
  {"URI": "/api/account/2", "description": "AutoAdmin"},
  {"URI": "/api/account/3", "description": "AutoRegUser"},
  {"URI": "/api/account/1", "description": "user"},
  {"URI": "/api/account/4", "description": "snadmin"},
]'

Adding the json step will convert the string into a list of dictionaries.

low_code:
  version: 2
  steps:
    - static_value: "[
      {"URI":"/api/account/2","description":"AutoAdmin"},
      {"URI":"/api/account/3","description":"AutoRegUser"},
      {"URI":"/api/account/1","description":"user"},
      {"URI":"/api/account/4","description":"snadmin"}
    ]"
    - json

Output

[
  {"URI": "/api/account/2", "description": "AutoAdmin"},
  {"URI": "/api/account/3", "description": "AutoRegUser"},
  {"URI": "/api/account/1", "description": "user"},
  {"URI": "/api/account/4", "description": "snadmin"},
]

Formatted Data

The static_value allows YAML expressed data structures such as lists or dictionaries to be outputted. Below is the previous JSON stringified example Snippet Argument.

low_code:
  version: 2
  steps:
    - static_value: "[
      {"URI":"/api/account/2","description":"AutoAdmin"},
      {"URI":"/api/account/3","description":"AutoRegUser"},
      {"URI":"/api/account/1","description":"user"},
      {"URI":"/api/account/4","description":"snadmin"}
    ]"
    - json

Below is the equivalent YAML expressed step argument for static_value.

low_code:
  version: 2
  steps:
    - static_value:
      - URI: /api/account/2
        description: AutoAdmin
      - URI: /api/account/3
        description: AutoRegUser
      - URI: /api/account/1
        description: user
      - URI: /api/account/4
        description: snadmin

The outputted data type of this snippet argument is a list of dictionaries. Notice that the json step is no longer needed.

store_data

The store_data step allows a user to store the current result into a key of their choosing. This enables a pre-processed dataset to be used at a later time where it may be necessary to have the full result. An example of this could be trimming data you do not need but requiring the whole payload to make a decision in the future.

This step does not update request_id so it will not affect the automatic cache_key generated by the Snippet Framework.

Framework Name	`store_data`
key	storage_key

For example, if you wanted to store the current result into the key storage_key, you would use the following step definition:

store_data: storage_key

To access this data in a later step, you would use the following:

result_container.metadata["storage_key"]