Writing a Snippet Argument

A Snippet Argument provides the execution plan to the Snippet Framework and enables low-code/no-code collections.

Before writing a Snippet Argument, we must first understand how to collect the required data and ensure that step(s) exist for the required tasks. If a step does not exist, we determine if we can chain steps together to get the required result. After we know what steps to use, then we write the Snippet Argument for the given Syntax.

The most basic form of writing a Snippet Argument is through the low_code Syntax which we use for the following examples.

CLI

The following steps are typically used for CLI-based collections:

The following steps are typically used if the previous steps do not satisfy your use-case:

The following CLI examples require collecting data in this sequence:

  • Perform an SSH request using the ssh step

  • Convert the CLI output into a Python data structure

  • Select information from that data structure

Monitoring Zombie Processes

For this example, we ssh into a device and run a command to count the number of zombie process.

In order, to find the number of processes, we run the ps command as follows.

ps -elF

We run this command on an Oracle Linux 8 device and it produces the following output:

F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN    RSS PSR STIME TTY          TIME CMD
4 S root         1     0  0  80   0 - 53678 ep_pol  4708   5 Jan09 ?        08:50:11 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
1 S root         2     0  0  80   0 -     0 kthrea     0   6 Jan09 ?        00:00:20 [kthreadd]
1 S root         4     2  0  60 -20 -     0 worker     0   0 Jan09 ?        00:00:00 [kworker/0:0H]
1 S root         6     2  0  80   0 -     0 smpboo     0   0 Jan09 ?        00:36:22 [ksoftirqd/0]
1 S root         7     2  0 -40   - -     0 smpboo     0   0 Jan09 ?        00:01:06 [migration/0]

Next, we use the jc step which is able to parse the ps command. The jc step transforms the CLI output into the following python data structure.

{0: {'addr': '-',
   'c': 0,
   'cmd': '/usr/lib/systemd/systemd --switched-root --system --deserialize '
          '22',
   'f': '4',
   'ni': '0',
   'pid': 1,
   'ppid': 0,
   'pri': '80',
   'psr': '2',
   'rss': 4708,
   's': 'S',
   'stime': 'Jan09',
   'sz': '53678',
   'time': '08:50:12',
   'tty': None,
   'uid': 'root',
   'wchan': 'ep_pol'}, ...

We receive a list of dictionaries where the key s contains the process state. The value Z in this field indicates a zombie process. Next, we use the jmespath selector to select all the zombie processes. Our query looks like the following:

"[?s == `Z`]"

From here, we find the number of zombie instances using the JMESpath length function.

"length([?s == `Z`])"

Note

The monitored device might not have any zombie processes, so to verify the functionality of the snippet argument, we can change the jmespath query to monitor Sleeping versus a Zombie state. The query to output the number of sleeping processes is length([?s == `S`]).

For this scenario, we use the following steps:

Steps for collection

Operation

Step

Perform a SSH request

ssh

Convert response from a string to a dictionary

jc

Select data from a dictionary

jmespath

Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.

Determining Step Arguments

ssh

Step reference

This step requires the CLI command, ps -elF, as its parameter.

ssh: "ps -elF"
jc

Step reference

This step requires that the parser name, ps, is provided.

- jc: ps
jmespath

Step reference

This step requires the query to be specified with the value key.

- jmespath:
    value: "length([?s==`Z`])"

Creating the Snippet Argument

Our final Snippet Argument using the low_code version 2 Syntax is as follows.

low_code:
  version: 2
  steps:
    - ssh: ps -elF
    - jc: ps
    - jmespath:
        value: "length([?s==`Z`])"

Getting IcmpMsg data from /proc/net/snmp

The Linux kernel provides SNMP counters in the proc filesytem. We are interested in collecting ICMP InType and OutType counters. Those values map to specific ICMP message types.

We start by checking the contents of /proc/net/snmp.

cat /proc/net/snmp

The output is a text string as shown below.

Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 768107856 0 0 0 0 0 767902838 769154056 0 8 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 60534 0 0 9149 0 0 0 0 10209 41176 0 0 0 0 76816 0 25428 0 0 0 0 39773 10209 0 0 0 0
IcmpMsg: InType0 InType3 InType8 OutType0 OutType3 OutType8 OutType69
IcmpMsg: 41176 9149 10209 10209 25428 39773 1406
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 18190

This data is parsed used the parse_proc_net_snmp step resulting in the following structured output.

{'Icmp': {'InAddrMaskReps': 0,
        'InAddrMasks': 0,
        'InCsumErrors': 0,
        'InDestUnreachs': 9149,
        'InEchoReps': 41178,
        'InEchos': 10209,
        'InErrors': 0,
        'InMsgs': 60536,
        'InParmProbs': 0,
        'InRedirects': 0,
        'InSrcQuenchs': 0,
        'InTimeExcds': 0,
        'InTimestampReps': 0,
        'InTimestamps': 0,
        'OutAddrMaskReps': 0,
        'OutAddrMasks': 0,
        'OutDestUnreachs': 25428,
        'OutEchoReps': 10209,
        'OutEchos': 39775,
        'OutErrors': 0,
        'OutMsgs': 76818,
        'OutParmProbs': 0,
        'OutRedirects': 0,
        'OutSrcQuenchs': 0,
        'OutTimeExcds': 0,
        'OutTimestampReps': 0,
        'OutTimestamps': 0}, ...

Next, we select the IcmpMsg key from the dictionary produced by the parser step using this JMESPath query.

"IcmpMsg"

The result is show below.

{
  'InType0': 41180,
  'InType3': 9149,
  'InType8': 10209,
  'OutType0': 10209,
  'OutType3': 25429,
  'OutType8': 39777
}

Below is a list of translated Control Message IDs from the output above.

  • 0 - Echo Reply (used to ping)

  • 3 - Destination Unreachable

  • 8 - Echo Request (used to ping)

For this scenario, we use the following steps:

Steps for collection

Operation

Step

Perform a SSH request

ssh

Convert response from a string to a dictionary

parse_proc_net_snmp

Select data from a dictionary

jmespath

Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.

Determining Step Arguments

ssh

Step reference

This step requires the CLI command, cat /proc/net/snmp.

ssh: "cat /proc/net/snmp"
parse_proc_net_snmp

Step reference

This command does not require arguments.

jmespath

Step reference

This step requires the query to be specified with the value key.

- jmespath:
    value: IcmpMsg

Creating the Snippet Argument

Our final Snippet Argument using the low_code version 2 Syntax is as follows.

low_code:
  version: 2
  steps:
    - ssh:
        command: cat /proc/net/snmp
    - parse_proc_net_snmp
    - jmespath:
        value: IcmpMsg

Parsing lines from lscpu

We are using the linux command line tool lscpu to find the CPU model type of our device.

/usr/bin/lscpu

The command line output is as follows.

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    2
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz
Stepping:              0
CPU MHz:               2893.202
BogoMIPS:              5786.40
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              36608K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca

Using the parse_line step, we can transform this output into a python data structure for selecting our desired field.

{
  'Architecture': 'x86_64',
  'BogoMIPS': '5786.40',
  'Byte Order': 'Little Endian',
  'CPU MHz': '2893.202',
  'CPU family': '6',
  'CPU op-mode(s)': '32-bit, 64-bit',
  'CPU(s)': '8',
  'Core(s) per socket': '2',
  'Flags': 'fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca ',
  'Hypervisor vendor': 'VMware',
  'L1d cache': '32K',
  'L1i cache': '32K',
  'L2 cache': '1024K',
  'L3 cache': '36608K',
  'Model': '63',
  'Model name': 'Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz',
  'NUMA node(s)': '1',
  'NUMA node0 CPU(s)': '0-7',
  'On-line CPU(s) list': '0-7',
  'Socket(s)': '4',
  'Stepping': '0',
  'Thread(s) per core': '1',
  'Vendor ID': 'GenuineIntel',
  'Virtualization type': 'full'
}

Next, we select the Model name key using the following JMESPath query.

- jmespath:
    value: "\"Model name\""

Note

The space in property or key for our search requires that we escape double quote our key value, Model name, in our query.

For this scenario, we use the following steps:

Steps for collection

Operation

Step

Perform a SSH request

ssh

Convert response from a string to a dictionary

parse_line

Select data from a dictionary

jmespath

Now that we know what steps to use, we find what arguments, if any, need to be supplied to the step.

Determining Step Arguments

ssh

Step reference

This step requires the CLI command, /usr/bin/lscpu, as its parameter.

ssh: "/usr/bin/lscpu"
parse_line

Step reference

This step requires the delimiter or split type to be specified. In our case, the split type is a colon, :. Also, we will use the special key from_output that tells the step to treat the split like a key value pair. The key is first string before the split type and the remaining string becomes the value in a newly formed Python dictionary.

- parse_line:
    split_type: colon
    key: from_output
jmespath

Step reference

This step requires the query to be specified with the value key. Again, we are looking for the processor model name and use the key Model name.

- jmespath:
    value: "\"Model name\""

Creating the Snippet Argument

Our final Snippet Argument using the low_code version 2 Syntax is as follows.

low_code:
  id: regex_parse_line
  version: 2
  steps:
    - ssh:
        command: /usr/bin/lscpu
    - parse_line:
        split_type: colon
        key: from_output
    - jmespath:
        value: "\"Model name\""

Prometheus

The following steps are typically used for Prometheus-based collections:

The following steps are typically used if the previous steps do not satisfy your use-case:

Best Practices

  • Prometheus supports many binary operators (arithmetic, trigonometric, comparison, etc.) and aggregation operations (sum, min, max, mean, etc.). Before defining a custom step to perform these operations with the collected data, consider using the operators as part of your query.

  • Prometheus supports several functions to operate on data. Before you define a custom step to do something similar, consider using the function as part of your query.

  • The Snippet Framework truncates indices if they are longer than 512 characters. As a result of this truncation, a new index will be generated, with the first original 50 characters of the index followed by a unique identifier. If possible, try to reduce the index by using the key labels of the PromQL syntax.

Obtaining HTTP API Endpoint Statistics

For this example, we will check a Prometheus HTTP API server to diagnose which API endpoints are returning which status codes, and the respective count of each. The required operations for collecting the data will be as follows:

  • Request the prometheus_http_requests_total metric

  • Interpret the response as a vector

  • Retrieve the handler and status code from the results

After determining how to collect the data, we then need to determine if our tasks have existing steps around them. For this scenario, we will use the following steps:

Steps for collection

Operation

Step

Query Prometheus for prometheus_http_requests_total

query

Interpret results as a vector

result_type

Retrieve the handler and status code labels

labels

Now that we know what steps to use, we will need to determine what arguments, if any, need to be supplied to the step.

Determining Step Arguments

query

After reviewing the available arguments for query, we can determine we only need to provide the step with the name of the metric we want to query for.

query: prometheus_http_requests_total
result_type

After reviewing the available arguments for result_type, we notice that we must specify the format we want the query to be interpreted as, which is a vector format in this instance.

result_type: vector
labels

After reviewing the available arguments for labels, we determine that it takes a list of values. So we will list out all the labels we want to retrieve from the results, which is the handler and code.

labels:
  - handler
  - code

Creating the Snippet Argument

Now that we have the steps and their arguments to perform the collection, we need to write our final Snippet Argument using the promql Syntax.

promql:
  query: prometheus_http_requests_total
  result_type: vector
  labels:
    - handler
    - code

Obtaining HTTP Request Response Time Statistics

For this example, we will collect the average duration of a request within the last 3 minutes to the Prometheus HTTP API server. The result will be a time series that we must aggregate to take the average of the time in seconds for each respective status code. The required operations for collecting the data will be as follows:

  • Request the http_request_duration_seconds_count time series metric for the last 3 minutes

  • Interpret the response as a matrix

  • Calculate the average value of each status code

  • Retrieve the average response time by status

After determining how to collect the data, we then need to determine if our tasks have existing steps around them. For this scenario, we will use the following steps:

Steps for collection

Operation

Step

Query Prometheus for http_request_duration_seconds_count

query

Interpret results as a matrix

result_type

Select the average value of each

aggregation

Retrieve the status code associated with each value

labels

Now that we know what steps to use, we will need to determine what arguments, if any, need to be supplied to the step.

Determining Step Arguments

query

After reviewing the available arguments for query, we can determine we only need to provide the step with the name of the metric we want to query for and the time frame to select.

query: http_request_duration_seconds_count[3m]
result_type

After reviewing the available arguments for result_type, we notice that we must specify the format we want the query to be interpreted as, which is a matrix format in this instance.

result_type: matrix
aggregation

After reviewing the available arguments for aggregation, we determine that we will need to specify how we want our data aggregated. In this case we want the mean value to get the average.

aggregation: mean
labels

After reviewing the available arguments for labels, we determine that it takes a list of values. So we will list out all the labels we want to retrieve from the results, which is only the status.

labels:
  - status

Creating the Snippet Argument

Now that we have the steps and their arguments to perform the collection, we need to write our final Snippet Argument using the promql Syntax.

promql:
  query: http_request_duration_seconds_count[3m]
  result_type: matrix
  aggregation: mean
  labels:
    - status

The output of the Collection would look something like this:

{
    "{status='200'}": 0.24,
    "{status='400'}": 4.57,
    "{status='999'}": 2.12
}

REST

The following steps are typically used for REST-based collections:

Collecting device names from an AIO SL1 System

For this example, we collect device names from a SL1 system. The required operation for collecting the data is as follows:

  • Perform an HTTP request to /api/device on localhost (due to this being an AIO we can use localhost instead of an ip address or hostname)

  • Interpret the text response as JSON

  • Convert the results to a python dictionary with the following structure:

    {
        "device_uri": "device_name"
    }
    

After determining how to collect the data, we determine if our tasks have existing steps around them. For this scenario, we use the following steps:

Steps for collection

Operation

Step

Perform an HTTP request

http

Convert response from JSON

json

Convert the result to a dictionary

jmespath

Now that we know what steps to use, we determine what arguments, if any, need to be supplied to the step.

Determining Step Arguments

http

Step reference

We only provide the step with the url argument. The username / password is automatically applied from the credential.

http:
  url: http://localhost:80/api/device
json

Step reference

We notice that no arguments are needed for this step.

json
jmespath

Step reference

We find that we need to use index and value to construct our Python dictionary result.

Note

This section is not a deep dive into JMESpath. For more information around jmespath, refer to the official documentation.

jmespath:
  index: true
  value: "values(@)|[].{_index: URI, _value: description}"

Creating the Snippet Argument

We write our final Snippet Argument using the low_code version 2 Syntax. We add all our steps under the steps section, which takes a list of step names and their arguments.

low_code:
  version: 2
  steps:
    - http:
        url: http://localhost:80/api/device
    - json
    - jmespath:
        index: true
        value: "values(@)|[].{_index: URI, _value: description}"

Collecting aligned organization for a device

We collect the aligned organization for a given device. Since part of the URI is dynamic, in this case the device id, we utilize substitution that automatically substitutes context-aware information into the Snippet Argument. The required operation for collecting the data is as follows:

  • Perform an HTTP request to /api/device/<device_id> on localhost (due to this being an AIO we can use localhost instead of an ip address or hostname)

  • Interpret the text response as JSON

  • Select a value from the dictionary

We determine if our tasks have existing steps around them. For this scenario, we use the following steps:

Steps for collection

Operation

Step

Perform an HTTP request

http

Convert response from a JSON string to a dictionary

json

Select data from a dictionary

jmespath

Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.

Determining Step Arguments

http

After reviewing the available arguments for http and determining that we will be specifying the entire URL, we only need to provide the step with the url parameter. The username / password is automatically applied from the credential. Since we need context-aware information, device id, we use substitution to get this information. After referring to the available substitution arguments, we use ${silo_did} to construct the url http://localhost:80/api/device/1.

http:
  url: http://localhost:80/api/device/${silo_did}
json

After reviewing the available arguments for json, we notice that no arguments are needed for this step.

json
jmespath

After reviewing the available arguments for jmespath, we find that we need to use value to extract our data.

Note

This section is not a deep dive into jmespath. For more information around jmespath, refer to the official documentation.

jmespath:
  value: organization

Creating the Snippet Argument

Now that we have the steps and their arguments to perform the collection, we write our final Snippet Argument using the low_code version 2 Syntax. We add all our steps under the steps section, which takes a list of step names and their arguments.

low_code:
  version: 2
  steps:
    - http:
        url: http://localhost:80/api/device/${silo_did}
    - json
    - jmespath:
        value: organization

SNMP

Step reference

The following steps are typically used for SNMP-based collections:

Obtaining sysDescr

For this example, we will perform a SNMP get to gather the devices sysDescr, .1.3.6.1.2.1.1.1.0.

Determining Step Arguments

snmp

Step reference

This step requires the the oid, .1.3.6.1.2.1.1.1.0, to be executed as a get operation.

snmp:
    method: get
    oids:
     - .1.3.6.1.2.1.1.1.0

Creating the Snippet Argument

Our final Snippet Argument using the low_code version 2 Syntax is as follows.

low_code:
  version: 2
  steps:
    - snmp:
        method: get
        oids:
         - .1.3.6.1.2.1.1.1.0

Obtaining interface names

For this example, we will perform a SNMP walk to gather the devices sysDescr, .1.3.6.1.2.1.2.2.1.2.

Determining Step Arguments

snmp

Step reference

This step requires the the oid, .1.3.6.1.2.1.2.2.1.2, to be executed as a walk operation.

snmp:
    method: walk
    oids:
     - .1.3.6.1.2.1.2.2.1.2

Creating the Snippet Argument

Our final Snippet Argument using the low_code version 2 Syntax is as follows.

low_code:
  version: 2
  steps:
    - snmp:
        method: walk
        oids:
         - .1.3.6.1.2.1.2.2.1.2