Writing a Snippet Argument
A Snippet Argument provides the execution plan to the Snippet Framework and enables low-code/no-code collections.
Before writing a Snippet Argument, we must first understand how to collect the required data and ensure that step(s) exist for the required tasks. If a step does not exist, we determine if we can chain steps together to get the required result. After we know what steps to use, then we write the Snippet Argument for the given Syntax.
The most basic form of writing a Snippet Argument is through
the low_code
Syntax which we
use for the following examples.
CLI
The following steps are typically used for CLI-based collections:
The following steps are typically used if the previous steps do not satisfy your use-case:
The following CLI examples require collecting data in this sequence:
Perform an SSH request using the ssh step
Convert the CLI output into a Python data structure
Select information from that data structure
Monitoring Zombie Processes
For this example, we ssh into a device and run a command to count the number of zombie process.
In order, to find the number of processes, we run the ps command as follows.
ps -elF
We run this command on an Oracle Linux 8 device and it produces the following output:
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
4 S root 1 0 0 80 0 - 53678 ep_pol 4708 5 Jan09 ? 08:50:11 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
1 S root 2 0 0 80 0 - 0 kthrea 0 6 Jan09 ? 00:00:20 [kthreadd]
1 S root 4 2 0 60 -20 - 0 worker 0 0 Jan09 ? 00:00:00 [kworker/0:0H]
1 S root 6 2 0 80 0 - 0 smpboo 0 0 Jan09 ? 00:36:22 [ksoftirqd/0]
1 S root 7 2 0 -40 - - 0 smpboo 0 0 Jan09 ? 00:01:06 [migration/0]
Next, we use the jc step which is able to parse the ps command. The jc step transforms the CLI output into the following python data structure.
{0: {'addr': '-',
'c': 0,
'cmd': '/usr/lib/systemd/systemd --switched-root --system --deserialize '
'22',
'f': '4',
'ni': '0',
'pid': 1,
'ppid': 0,
'pri': '80',
'psr': '2',
'rss': 4708,
's': 'S',
'stime': 'Jan09',
'sz': '53678',
'time': '08:50:12',
'tty': None,
'uid': 'root',
'wchan': 'ep_pol'}, ...
We receive a list of dictionaries where the key s contains the process state. The value Z in this field indicates a zombie process. Next, we use the jmespath selector to select all the zombie processes. Our query looks like the following:
"[?s == `Z`]"
From here, we find the number of zombie instances using the JMESpath length function.
"length([?s == `Z`])"
Note
The monitored device might not have any zombie processes, so to verify
the functionality of the snippet argument, we can change the jmespath query
to monitor Sleeping versus a Zombie state. The query to output the number
of sleeping processes is length([?s == `S`])
.
For this scenario, we use the following steps:
Operation |
Step |
---|---|
Perform a SSH request |
ssh |
Convert response from a string to a dictionary |
jc |
Select data from a dictionary |
jmespath |
Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.
Determining Step Arguments
ssh
This step requires the CLI command, ps -elF
, as its parameter.
ssh: "ps -elF"
jc
This step requires that the parser name, ps
, is provided.
- jc: ps
jmespath
This step requires the query to be specified with the value
key.
- jmespath:
value: "length([?s==`Z`])"
Creating the Snippet Argument
Our final Snippet Argument using the low_code
version 2 Syntax is as follows.
low_code:
version: 2
steps:
- ssh: ps -elF
- jc: ps
- jmespath:
value: "length([?s==`Z`])"
Getting IcmpMsg data from /proc/net/snmp
The Linux kernel provides SNMP counters in the proc filesytem. We are interested in collecting ICMP InType and OutType counters. Those values map to specific ICMP message types.
We start by checking the contents of /proc/net/snmp
.
cat /proc/net/snmp
The output is a text string as shown below.
Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 1 64 768107856 0 0 0 0 0 767902838 769154056 0 8 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 60534 0 0 9149 0 0 0 0 10209 41176 0 0 0 0 76816 0 25428 0 0 0 0 39773 10209 0 0 0 0
IcmpMsg: InType0 InType3 InType8 OutType0 OutType3 OutType8 OutType69
IcmpMsg: 41176 9149 10209 10209 25428 39773 1406
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 18190
This data is parsed used the parse_proc_net_snmp step resulting in the following structured output.
{'Icmp': {'InAddrMaskReps': 0,
'InAddrMasks': 0,
'InCsumErrors': 0,
'InDestUnreachs': 9149,
'InEchoReps': 41178,
'InEchos': 10209,
'InErrors': 0,
'InMsgs': 60536,
'InParmProbs': 0,
'InRedirects': 0,
'InSrcQuenchs': 0,
'InTimeExcds': 0,
'InTimestampReps': 0,
'InTimestamps': 0,
'OutAddrMaskReps': 0,
'OutAddrMasks': 0,
'OutDestUnreachs': 25428,
'OutEchoReps': 10209,
'OutEchos': 39775,
'OutErrors': 0,
'OutMsgs': 76818,
'OutParmProbs': 0,
'OutRedirects': 0,
'OutSrcQuenchs': 0,
'OutTimeExcds': 0,
'OutTimestampReps': 0,
'OutTimestamps': 0}, ...
Next, we select the IcmpMsg
key from the dictionary produced by the parser
step using this JMESPath query.
"IcmpMsg"
The result is show below.
{
'InType0': 41180,
'InType3': 9149,
'InType8': 10209,
'OutType0': 10209,
'OutType3': 25429,
'OutType8': 39777
}
Below is a list of translated Control Message IDs from the output above.
0 - Echo Reply (used to ping)
3 - Destination Unreachable
8 - Echo Request (used to ping)
For this scenario, we use the following steps:
Operation |
Step |
---|---|
Perform a SSH request |
ssh |
Convert response from a string to a dictionary |
parse_proc_net_snmp |
Select data from a dictionary |
jmespath |
Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.
Determining Step Arguments
ssh
This step requires the CLI command, cat /proc/net/snmp
.
ssh: "cat /proc/net/snmp"
parse_proc_net_snmp
This command does not require arguments.
jmespath
This step requires the query to be specified with the value
key.
- jmespath:
value: IcmpMsg
Creating the Snippet Argument
Our final Snippet Argument using the low_code
version 2 Syntax is as follows.
low_code:
version: 2
steps:
- ssh:
command: cat /proc/net/snmp
- parse_proc_net_snmp
- jmespath:
value: IcmpMsg
Parsing lines from lscpu
We are using the linux command line tool lscpu
to
find the CPU model type of our device.
/usr/bin/lscpu
The command line output is as follows.
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz
Stepping: 0
CPU MHz: 2893.202
BogoMIPS: 5786.40
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 36608K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
Using the parse_line step, we can transform this output into a python data structure for selecting our desired field.
{
'Architecture': 'x86_64',
'BogoMIPS': '5786.40',
'Byte Order': 'Little Endian',
'CPU MHz': '2893.202',
'CPU family': '6',
'CPU op-mode(s)': '32-bit, 64-bit',
'CPU(s)': '8',
'Core(s) per socket': '2',
'Flags': 'fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca ',
'Hypervisor vendor': 'VMware',
'L1d cache': '32K',
'L1i cache': '32K',
'L2 cache': '1024K',
'L3 cache': '36608K',
'Model': '63',
'Model name': 'Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz',
'NUMA node(s)': '1',
'NUMA node0 CPU(s)': '0-7',
'On-line CPU(s) list': '0-7',
'Socket(s)': '4',
'Stepping': '0',
'Thread(s) per core': '1',
'Vendor ID': 'GenuineIntel',
'Virtualization type': 'full'
}
Next, we select the Model name
key using the following JMESPath query.
- jmespath:
value: "\"Model name\""
Note
The space in property or key for our search requires that we escape double quote our key value, Model name, in our query.
For this scenario, we use the following steps:
Operation |
Step |
---|---|
Perform a SSH request |
ssh |
Convert response from a string to a dictionary |
parse_line |
Select data from a dictionary |
jmespath |
Now that we know what steps to use, we find what arguments, if any, need to be supplied to the step.
Determining Step Arguments
ssh
This step requires the CLI command, /usr/bin/lscpu
, as its parameter.
ssh: "/usr/bin/lscpu"
parse_line
This step requires the delimiter or split type to be specified. In our case,
the split type is a colon, :
. Also, we will use the special key
from_output
that tells the step to treat the split like a key value pair.
The key is first string before the split type and the remaining string
becomes the value in a newly formed Python dictionary.
- parse_line:
split_type: colon
key: from_output
jmespath
This step requires the query to be specified with the value
key. Again,
we are looking for the processor model name and use the key Model name
.
- jmespath:
value: "\"Model name\""
Creating the Snippet Argument
Our final Snippet Argument using the low_code
version 2 Syntax is as follows.
low_code:
id: regex_parse_line
version: 2
steps:
- ssh:
command: /usr/bin/lscpu
- parse_line:
split_type: colon
key: from_output
- jmespath:
value: "\"Model name\""
Prometheus
The following steps are typically used for Prometheus-based collections:
The following steps are typically used if the previous steps do not satisfy your use-case:
Best Practices
Prometheus supports many binary operators (arithmetic, trigonometric, comparison, etc.) and aggregation operations (sum, min, max, mean, etc.). Before defining a custom step to perform these operations with the collected data, consider using the operators as part of your query.
Prometheus supports several functions to operate on data. Before you define a custom step to do something similar, consider using the function as part of your query.
The Snippet Framework truncates indices if they are longer than 512 characters. As a result of this truncation, a new index will be generated, with the first original 50 characters of the index followed by a unique identifier. If possible, try to reduce the index by using the key
labels
of the PromQL syntax.
Obtaining HTTP API Endpoint Statistics
For this example, we will check a Prometheus HTTP API server to diagnose which API endpoints are returning which status codes, and the respective count of each. The required operations for collecting the data will be as follows:
Request the
prometheus_http_requests_total
metricInterpret the response as a vector
Retrieve the handler and status code from the results
After determining how to collect the data, we then need to determine if our tasks have existing steps around them. For this scenario, we will use the following steps:
Operation |
Step |
---|---|
Query Prometheus for |
query |
Interpret results as a vector |
result_type |
Retrieve the handler and status code labels |
labels |
Now that we know what steps to use, we will need to determine what arguments, if any, need to be supplied to the step.
Determining Step Arguments
query
After reviewing the available arguments for query
, we can
determine we only need to provide the step with the name of the
metric we want to query for.
query: prometheus_http_requests_total
result_type
After reviewing the available arguments for result_type
,
we notice that we must specify the format we want the query
to be interpreted as, which is a vector format in this instance.
result_type: vector
labels
After reviewing the available arguments for labels
, we determine
that it takes a list of values. So we will list out all the labels
we want to retrieve from the results, which is the handler and code.
labels:
- handler
- code
Creating the Snippet Argument
Now that we have the steps and their arguments to perform the
collection, we need to write our final Snippet Argument using
the promql
Syntax.
promql:
query: prometheus_http_requests_total
result_type: vector
labels:
- handler
- code
Obtaining HTTP Request Response Time Statistics
For this example, we will collect the average duration of a request within the last 3 minutes to the Prometheus HTTP API server. The result will be a time series that we must aggregate to take the average of the time in seconds for each respective status code. The required operations for collecting the data will be as follows:
Request the
http_request_duration_seconds_count
time series metric for the last 3 minutesInterpret the response as a matrix
Calculate the average value of each status code
Retrieve the average response time by status
After determining how to collect the data, we then need to determine if our tasks have existing steps around them. For this scenario, we will use the following steps:
Operation |
Step |
---|---|
Query Prometheus for |
query |
Interpret results as a matrix |
result_type |
Select the average value of each |
aggregation |
Retrieve the status code associated with each value |
labels |
Now that we know what steps to use, we will need to determine what arguments, if any, need to be supplied to the step.
Determining Step Arguments
query
After reviewing the available arguments for query
, we can
determine we only need to provide the step with the name of the
metric we want to query for and the time frame to select.
query: http_request_duration_seconds_count[3m]
result_type
After reviewing the available arguments for result_type
,
we notice that we must specify the format we want the query
to be interpreted as, which is a matrix format in this instance.
result_type: matrix
aggregation
After reviewing the available arguments for aggregation
, we determine
that we will need to specify how we want our data aggregated. In this case
we want the mean value to get the average.
aggregation: mean
labels
After reviewing the available arguments for labels
, we determine
that it takes a list of values. So we will list out all the labels
we want to retrieve from the results, which is only the status.
labels:
- status
Creating the Snippet Argument
Now that we have the steps and their arguments to perform the
collection, we need to write our final Snippet Argument using
the promql
Syntax.
promql:
query: http_request_duration_seconds_count[3m]
result_type: matrix
aggregation: mean
labels:
- status
The output of the Collection would look something like this:
{ "{status='200'}": 0.24, "{status='400'}": 4.57, "{status='999'}": 2.12 }
REST
The following steps are typically used for REST-based collections:
Collecting device names from an AIO SL1 System
For this example, we collect device names from a SL1 system. The required operation for collecting the data is as follows:
Perform an HTTP request to
/api/device
on localhost (due to this being an AIO we can use localhost instead of an ip address or hostname)Interpret the text response as JSON
Convert the results to a python dictionary with the following structure:
{ "device_uri": "device_name" }
After determining how to collect the data, we determine if our tasks have existing steps around them. For this scenario, we use the following steps:
Operation |
Step |
---|---|
Perform an HTTP request |
http |
Convert response from JSON |
json |
Convert the result to a dictionary |
jmespath |
Now that we know what steps to use, we determine what arguments, if any, need to be supplied to the step.
Determining Step Arguments
http
We only provide the step with the url
argument. The
username / password is automatically applied from the credential.
http:
url: http://localhost:80/api/device
json
We notice that no arguments are needed for this step.
json
jmespath
We find that we need to use index
and value
to
construct our Python dictionary result.
Note
This section is not a deep dive into JMESpath. For more information around jmespath, refer to the official documentation.
jmespath:
index: true
value: "values(@)|[].{_index: URI, _value: description}"
Creating the Snippet Argument
We write our final Snippet Argument using the low_code
version 2 Syntax.
We add all our steps under the steps
section, which takes a list of step
names and their arguments.
low_code:
version: 2
steps:
- http:
url: http://localhost:80/api/device
- json
- jmespath:
index: true
value: "values(@)|[].{_index: URI, _value: description}"
Collecting aligned organization for a device
We collect the aligned organization for a given device. Since part of the URI is dynamic, in this case the device id, we utilize substitution that automatically substitutes context-aware information into the Snippet Argument. The required operation for collecting the data is as follows:
Perform an HTTP request to
/api/device/<device_id>
on localhost (due to this being an AIO we can use localhost instead of an ip address or hostname)Interpret the text response as JSON
Select a value from the dictionary
We determine if our tasks have existing steps around them. For this scenario, we use the following steps:
Operation |
Step |
---|---|
Perform an HTTP request |
http |
Convert response from a JSON string to a dictionary |
json |
Select data from a dictionary |
jmespath |
Now that we know what steps to use, we need to find what arguments, if any, need to be supplied to the step.
Determining Step Arguments
http
After reviewing the available arguments for http
and determining
that we will be specifying the entire URL, we only need to provide
the step with the url
parameter. The username / password
is automatically applied from the credential. Since we
need context-aware information, device id, we use substitution
to get this information. After referring to the available
substitution arguments, we use ${silo_did}
to construct the url http://localhost:80/api/device/1.
http:
url: http://localhost:80/api/device/${silo_did}
json
After reviewing the available arguments for json
, we notice
that no arguments are needed for this step.
json
jmespath
After reviewing the available arguments for jmespath
, we find
that we need to use value
to extract our data.
Note
This section is not a deep dive into jmespath. For more information around jmespath, refer to the official documentation.
jmespath:
value: organization
Creating the Snippet Argument
Now that we have the steps and their arguments to perform the collection, we
write our final Snippet Argument using the low_code
version 2 Syntax. We
add all our steps under the steps
section, which takes a list of step names
and their arguments.
low_code:
version: 2
steps:
- http:
url: http://localhost:80/api/device/${silo_did}
- json
- jmespath:
value: organization
SNMP
The following steps are typically used for SNMP-based collections:
Obtaining sysDescr
For this example, we will perform a SNMP get to gather the devices
sysDescr, .1.3.6.1.2.1.1.1.0
.
Determining Step Arguments
snmp
This step requires the the oid, .1.3.6.1.2.1.1.1.0
, to be
executed as a get operation.
snmp:
method: get
oids:
- .1.3.6.1.2.1.1.1.0
Creating the Snippet Argument
Our final Snippet Argument using the low_code
version 2 Syntax is as follows.
low_code:
version: 2
steps:
- snmp:
method: get
oids:
- .1.3.6.1.2.1.1.1.0
Obtaining interface names
For this example, we will perform a SNMP walk to gather the devices
sysDescr, .1.3.6.1.2.1.2.2.1.2
.
Determining Step Arguments
snmp
This step requires the the oid, .1.3.6.1.2.1.2.2.1.2
, to be
executed as a walk operation.
snmp:
method: walk
oids:
- .1.3.6.1.2.1.2.2.1.2
Creating the Snippet Argument
Our final Snippet Argument using the low_code
version 2 Syntax is as follows.
low_code:
version: 2
steps:
- snmp:
method: walk
oids:
- .1.3.6.1.2.1.2.2.1.2