Paginator

Creating a custom paginator is an advanced feature. If you only require basic functionality, the Snippet Framework includes one out-of-the-box paginator by default. As pagination schemes are not standardized, the Snippet Framework supports the ability to create a custom step that provides pagination support. The custom step must leverage the HTTPPageRequest exception in order to rewind execution back to the previous network requestor. That is, when the HTTPPageRequest exception is raised, the Snippet Framework will go back to say, the http step, and re-execute that step, along with any other steps between the original network requestor and the step raising the HTTPPageRequest exception.

Creating a Custom Paginator

In this example, we will make use of the SL1 built-in API. We will set the limit of our network request (e.g. http step) to a low value in order to show the effects of pagination. That is, with the limit set to 2, the http step will request a page size of only 2 at a time. In this example, there are only 3 accounts on the target SL1 and thus only 2 API calls will be made in total. To begin, we must write our own custom step that defines how we want our paginator to behave. Our custom paginator will use the metadata, specifically the url for the API request, that was populated from the http step to extract the offset and limit query parameters. The offset will need to be increased by the page limit on each API call to retrieve the next page. In order to request the next page, we will calculate the next request’s offset by adding the current call’s offset to the limit and then raising the HTTPageRequest exception, causing the rewind to occur with the specified params. Below is an example snippet using the methodology described above.

from silo.low_code_steps.rest import HTTPPageRequest
from urllib.parse import parse_qs, urlsplit

@register_rmd
def custom_paginator(result, action_arg, metadata):
    query_params = parse_qs(urlsplit(metadata["url"]).query)
    limit = int(query_params["limit"][0])
    offset = int(query_params["offset"][0]) + limit
    if len(result) >= limit:
        raise HTTPPageRequest(
            {"params": {"offset": offset}}
        )

The Snippet Framework will continue to re-execute the http step using the new URI created from the HTTPageRequest until the limit is greater than > the number of total results returned.

Note

When the amount of total results returned is divisible by the limit, an extra query will inherently occur with no results retrieved.

The following snippet argument leverages the custom_paginator step created above:

low_code:
  version: 2
  steps:
    - http:
        uri: /api/account
        params:
          limit: 2
          offset: 0
          hide_filterinfo: 1
    - json
    - custom_paginator
    - jmespath:
        value: '*[].{_index: URI, _value: description}'
        index: true

The way this will work is that the http step will be called initially, requesting the first 2 results from the /api/account endpoint. It will then convert the response using the json step. When the execution plan reaches our custom_paginator step, it will verify that the network request needs to run again. Seeing as how in our example we have 3 total accounts to retrieve, the execution will then rewind and begin again at the http step, only this time beginning with an offset of our limit and retaining the previous results. So, the request will retrieve the last result from the endpoint, convert the result using the json step, and finally reach our custom_paginator again for the last time. As there are no more results to retrieve, the execution plan will continue on to the final step and retrieve the expected data using the jmespath selector.