.. include:: ../common.txt
.. include:: ../links.txt


###########
Pagination
###########

Not all APIs can produce and deliver all the data at once or on a single
page. Pagination is a sequence of pages that are connected and have similar
content. Data can be contained on multiple pages, and to collect all the
content, we need several requests.

The |TOOLKIT_REST| provides pagination support. This document describes the
process in the |SNIPPET_FRAMEWORK|.


***********************
Pagination |STEP_NAME|
***********************

Using the |TOOLKIT_REST|, you have the ability to define a |STEP_NAME| that
requests more information with the HTTP requestor. This functionality is
known as **rewinding**. It allows you to get paginated responses from a
REST API.

The HTTP requestor is registered with a ``rewind`` function that updates the
Action Argument for a new request. To understand how a requestor is registered
with the rewind capability, please look at `Requestor section`_ of the
|STEP_NAME| user guide.

To make pagination happen, you need to create a processor |STEP_NAME| of type
`REQUEST_MORE_DATA_TYPE` as follows:

   * ``@register_processor(type=silo.low_code.REQUEST_MORE_DATA_TYPE)``

And this processor |STEP_NAME| must raise the following exception:

   * ``silo.low_code_steps.rest.HTTPPageRequest``

This exception allows the |SNIPPET_FRAMEWORK| to perform a loop process that
returns to the HTTP requestor and execute the next request. The Processor
|STEP_NAME| determines when there is no more data to collect. When all of the
data is collected, the result of the processor |STEP_NAME| will be an ordered
dictionary of the collected results.

Find more information on and how to request more data and processor |STEP_NAME_PLURAL|
the following `Processor section`_ of the Steps user guide.


******************************
Trimming and Metadata Storage
******************************

Some payloads can get pretty big. You may need to trim your data to remove
values you do not need to reduce the whole payload to keep the minimum
information you need.

There is a way to store the full payload that may be required for the
pagination step. This enables you to perform trimming on data before combining
it all at once when you collect all the results.

The |TOOLKIT_REST| provides a step out-of-the-box for storing the response
payload to the Result Container in a property called Metadata.

You can store the data by using the ``store_data`` |STEP_NAME|.
This |STEP_NAME| allows storing a result in the Result Container with a key
that you select. This enables a pre-processed dataset to be used later.

More detail about the storage data |STEP_NAME| can be found in
`Storage Metadata Step`_.


********************************************************
Paginator object definition within the HTTP |STEP_NAME|
********************************************************

The |SNIPPET_FRAMEWORK| can return to the previous network requestor and
re-execute with a different Action Argument.

The HTTP requestor is registered using the ``http_paginate`` function to
handle the rewind capability and create or update the Action Argument.

The |TOOLKIT_REST| allows the HTTP requestor to use the results to modify the
upcoming request to support pagination.

You will still need to create your custom processor |STEP_NAME|. To create the
processor |STEP_NAME|, follow the example provided in the HTTP |STEP_NAME|,
section`Pagination Support`_.