Snippet Framework Collection Pipeline

This section explains how the Snippet Framework runs a collection from start to finish. It introduces the pipeline model, describes the three step types (Syntax, Requestor, and Processor) and what each one does, and walks through a complete example pipeline so you can see how data moves through the framework during execution.

Snippet Framework Collection

The Snippet Framework executes monitoring logic as a structured pipeline of typed steps. Each run of a collection object passes through this pipeline in a defined sequence, with data flowing from one step to the next.

Understanding the pipeline model is important for building well-structured monitoring workflows and for troubleshooting collection behavior.

How the Collection Pipeline Works

When a Snippet Framework Dynamic Application collects data for a collection object, the framework follows the following steps:

1. Reads the snippet argument for that collection object (a YAML block defining the steps to execute and their configuration).
2. Parses the argument using the Syntax step to produce a structured execution plan.
3. Executes the Requestor to fetch data from the target system.
4. Passes the result through one or more Processors that transform or select values from the response.
5. Returns the final value to Skylar Onefor storage, analysis, and visualization.

The framework batches Requestors across all collection objects in the application and executes them together for efficiency before running per-object Processors downstream.

Step Types

The Snippet Framework defines three and only three step types. All steps are provided by toolkits. the Snippet Framework itself does not ship with any built-in steps. Without a compatible toolkit installed in an Execution Environment, the framework has no steps to execute.

Syntax

The Syntax step parses the raw YAML snippet argument into a structured internal format the framework can consume. Every collection pipeline begins with a Syntax step.

In most cases, a default Syntax step is used automatically, and you do not need to specify it. Custom Syntax steps may be registered by advanced toolkit developers.

Requestor

The Requestor step makes the actual request to the target system, which could be an HTTP call, an SSH command,a Prometheus scrape, or similar.

For snippet-based Dynamic Applications that use the http: requestor step, SSL certificate verification during collection is controlled by the Verify SSL for Collections by Default setting on the Behavior Settings page (System > Settings > Behavior). This setting is disabled by default. For more information, see the System Administration manual.

The Requestor is where data is fetched from the external source. The following are some key behaviors of the Requestor:

• Only one Requestor is active per collection pipeline pass.
• The framework batches all pending Requestors across collection objects and executes them together before proceeding to Processor steps.
• The Requestor is the step most closely tied to the collection protocol.
  • REST Toolkit: `rest_get`, `rest_post`, etc.
  • CLI Toolkit: `ssh`, `local_cmd`, etc.
  • Prometheus Toolkit: `prometheus`

Processor

Processor steps transform or select data from the Requestor response. Multiple Processors can be chained in sequence, with each Processor receiving the output of the previous one.

Common Processor examples from the REST Toolkit:

Processor	What It Does
json	Parses a JSON string response into a structured object
jsonpath	Selects a value from a structured object using a JMESPath expression
simple_key	Selects a single key from a dictionary

Processors are the primary way to extract the specific values your Dynamic Application needs from a larger API response or command output.

Worker processes are allocated dynamically by the Data Collector based on available capacity and the jobs being executed. When a collection run terminates without completing (SIGTERM), a termination entry is written to the system logs, including a list of pending device work items. Execution logs, including logs from failed runs, can be downloaded from the user interface.

Collection Pipeline Execution Flow

A complete pipeline for a REST-based collection object might look like this:

1. Syntax will parse YAML argument.
2. The Requestor will parse the response body as JSON.
3. The Processor selects the target metric value using an expression.
4. The result is when the selected value is returned to Skylar One

This pipeline structure is defined entirely within the snippet argument YAML, not in the snippet code itself. Changing the collection behavior requires updating the YAML configuration, not modifying Python code.

Error Handling and Data Flow

The Snippet Framework manages sequencing, batching, and error handling across the pipeline so that developers can focus on defining what to collect rather than implementing execution mechanics. If a step fails, the framework handles the failure condition according to its internal error-handling rules without requiring custom error-handling code in the snippet.