#########
Paginator
#########

Creating a custom paginator is an advanced feature. If you only require basic
functionality, the Snippet Framework includes one
:ref:`out-of-the-box paginator <steps-paginator-offset>`
by default. As pagination schemes are not standardized, the Snippet Framework
supports the ability to create a :ref:`custom step <custom_steps>` that provides
pagination support. The custom step must leverage the ``HTTPPageRequest``
exception in order to rewind execution back to the previous network requestor.
That is, when the ``HTTPPageRequest`` exception is raised, the Snippet
Framework will go back to say, the ``http`` step, and re-execute that step, along
with any other steps between the original network requestor and the step
raising the ``HTTPPageRequest`` exception.


Creating a Custom Paginator
===========================

In this example, we will make use of the SL1 built-in API. We will set the
`limit` of our network request (e.g. :ref:`http step <steps-http>`)
to a low value in order to show the effects of pagination. That is, with the
`limit` set to 2, the ``http`` step will request a page size of only 2 at a
time. In this example, there are only 3 accounts on the target SL1 and thus
only 2 API calls will be made in total. To begin, we must write our own
custom step that defines how we want our paginator to behave. Our custom
paginator will use the metadata, specifically the url for the API request,
that was populated from the ``http`` step to extract the `offset` and `limit`
query parameters. The `offset` will need to be increased by the page `limit`
on each API call to retrieve the next page. In order to request the next page,
we will calculate the next request's `offset` by adding the current call's
`offset` to the `limit` and then raising the ``HTTPageRequest`` exception,
causing the rewind to occur with the specified ``params``. Below is an
example snippet using the methodology described above.

.. code-block:: python

    from silo.low_code_steps.rest import HTTPPageRequest
    from urllib.parse import parse_qs, urlsplit

    @register_rmd
    def custom_paginator(result, action_arg, metadata):
        query_params = parse_qs(urlsplit(metadata["url"]).query)
        limit = int(query_params["limit"][0])
        offset = int(query_params["offset"][0]) + limit
        if len(result) >= limit:
            raise HTTPPageRequest(
                {"params": {"offset": offset}}
            )

The Snippet Framework will continue to re-execute the ``http`` step
using the new URI created from the ``HTTPageRequest`` until the
`limit` is greater than ``>`` the number of total results returned.

.. note::

    When the amount of total results returned is divisible by the `limit`,
    an extra query will inherently occur with no results retrieved.

The following snippet argument leverages the ``custom_paginator`` step created above:

.. code-block:: yaml

    low_code:
      version: 2
      steps:
        - http:
            uri: /api/account
            params:
              limit: 2
              offset: 0
              hide_filterinfo: 1
        - json
        - custom_paginator
        - jmespath:
            value: '*[].{_index: URI, _value: description}'
            index: true

The way this will work is that the ``http`` step will be called initially,
requesting the first 2 results from the ``/api/account`` endpoint. It will then
convert the response using the
:ref:`json step <steps-json>`. When the execution plan
reaches our ``custom_paginator`` step, it will verify that the network request
needs to run again. Seeing as how in our example we have 3 total accounts to
retrieve, the execution will then rewind and begin again at the ``http`` step,
only this time beginning with an offset of our `limit` and retaining the previous
results. So, the request will retrieve the last result from the endpoint,
convert the result using the ``json`` step, and finally reach our ``custom_paginator``
again for the last time. As there are no more results to retrieve, the execution plan
will continue on to the final step and retrieve the expected data using the
:ref:`jmespath selector <steps-jmespath>`.