Pagination

Not all APIs can produce and deliver all the data at once or on a single page. Pagination is a sequence of pages that are connected and have similar content. Data can be contained on multiple pages, and to collect all the content, we need several requests.

The REST: Toolkit provides pagination support. This document describes the process in the Snippet Framework.

Pagination step

Using the REST: Toolkit, you have the ability to define a step that requests more information with the HTTP requestor. This functionality is known as rewinding. It allows you to get paginated responses from a REST API.

The HTTP requestor is registered with a rewind function that updates the Action Argument for a new request. To understand how a requestor is registered with the rewind capability, please look at Requestor section of the step user guide.

To make pagination happen, you need to create a processor step of type REQUEST_MORE_DATA_TYPE as follows:

• @register_processor(type=silo.low_code.REQUEST_MORE_DATA_TYPE)

And this processor step must raise the following exception:

• silo.low_code_steps.rest.HTTPPageRequest

This exception allows the Snippet Framework to perform a loop process that returns to the HTTP requestor and execute the next request. The Processor step determines when there is no more data to collect. When all of the data is collected, the result of the processor step will be an ordered dictionary of the collected results.

Find more information on and how to request more data and processor steps the following Processor section of the Steps user guide.

Trimming and Metadata Storage

Some payloads can get pretty big. You may need to trim your data to remove values you do not need to reduce the whole payload to keep the minimum information you need.

There is a way to store the full payload that may be required for the pagination step. This enables you to perform trimming on data before combining it all at once when you collect all the results.

The REST: Toolkit provides a step out-of-the-box for storing the response payload to the Result Container in a property called Metadata.

You can store the data by using the store_data step. This step allows storing a result in the Result Container with a key that you select. This enables a pre-processed dataset to be used later.

More detail about the storage data step can be found in Storage Metadata Step.

Paginator object definition within the HTTP step

The Snippet Framework can return to the previous network requestor and re-execute with a different Action Argument.

The HTTP requestor is registered using the http_paginate function to handle the rewind capability and create or update the Action Argument.

The REST: Toolkit allows the HTTP requestor to use the results to modify the upcoming request to support pagination.

You will still need to create your custom processor step. To create the processor step, follow the example provided in the HTTP step, section`Pagination Support`_.