Collection Pipeline

The Collection Pipeline behind the Snippet Framework operates in an event-loop like fashion. This ensures all available device metrics will be completed (or errored out) and enables batching of Requestors. By batching the Requestors, a developer is able to implement functionality such as rate limiting, thread pools, etc. Typically the Pipeline will iterate (maximum number of Requestors in a Device Metric) + 1 times.

There is some terminology that needs to be clarified to fully understand this flow-chart.

  • Device Metric: A Device Metric is a Collection / Device combination (ResultContainer).

  • Pending Requestors: These are Device Metrics where the next step is a Requestor. All Requestors will be executed as a batch job to ensure certain functionality can exist. For example, if there are two Device Metrics and one has a Requestor on step one while the other has a Requestor on step four, they will both be batched at the same time.

Below is a high-level flow-chart on the Collection Pipeline.

../_images/UniversalPipelineProcessing.png