5.5 REST Driver Pagination

REST API pagination is the process that helps to divide the datasets into distinct pages. It is better to break up the dataset into smaller chunks which will make the response quicker and it is easy to handle.

5.5.1 Supported Pagination techniques:

There are three pagination techniques that can be used:

Offset Pagination
Cursor Pagination
Custom Pagination

5.5.2 Offset Pagination

Pagination technique that uses an offset parameter to determine the starting point of the next set of results.

Parameters	Description
Offset	Starting point
Limit	Size of the page
Total results	Response attribute in the dataset that gives the total count or manually enter the total count.

To update the pagination details, perform the following steps:

In Identity Console, go to Drivers.
Select the driver that you want to perform pagination.
Navigate to Configuration.
Click Driver parameters drop-down and go to Publisher Settings.
Under Publisher Option, select Publisher setting to Poll Mode.
Under Optional Header Fields, select the Pagination Type to Offset Pagination.
Enter the required details in the following field:
1. Offset
2. Page Size
3. Total Count
Click Save.

Example

For the following SAP Cloud payload, enter the input parameters as given below:

Input Parameters:

Offset = 1
Page Size = 2
Total Count = totalResults (or) 1209

SAP Cloud payload:

{
"schemas": [
"urn:ietf:params:scim:api:messages:2.0:ListResponse"
],
"totalResults": 1209,
"itemsPerPage": 100,
"Resources": [
{
"id": "P000000",
"userUuid": "93876fe3-8f0c-4d2c-9db2-caac1adf83e7",
"displayName": "Adrian Stephens",
"emails": [
{
"value": "adrian.stephens@microfocus.com",
"primary": true
}
],
"name": {
"givenName": "Adrian",
"familyName": "Stephens"
},
"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {}
},
{
"id": "P000016",
"userUuid": "6bb51346-e077-43ef-810c-82bbb64313ec",
"displayName": "Abhishek Prasad",
"emails": [
{
"value": "abhishek.prasad@microfocus.com",
"primary": true
}
],
"name": {
"givenName": "Abhishek",
"familyName": "Prasad"
},
"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {}
},
{
"id": "P000238",
"userUuid": "6055737e-7e18-4732-a9aa-29b4966a569c",
"userName": "tgarrison",
"displayName": "Tana Garrison",
"emails": [
{
"value": "tgarrison@extfocus.com",
"primary": true
}
],
"name": {
"givenName": "Tana",
"middleName": "s",
"familyName": "Garrison"
},
"urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
"employeeNumber": "1850"
}
}
]
}

5.5.3 Cursor Pagination

This uses a cursor parameter to determine the starting point of the next set of results. The cursor can be a unique identifier or a bookmark that points to a specific location in the result set.

The Cursor pagination is further divided into two according to the cursor placement on the dataset:

Link-Header based Pagination
Key set/Token Pagination

Link-Header based Pagination

Link-Header based pagination relies on HTTP headers to provide pagination meta data. The HTTP Link entity-header field provides a means for serializing one or more links in HTTP headers.

To update the pagination details, perform the following steps:

In Identity Console, go to Drivers.
Select the driver that you want to perform pagination.
Navigate to Configuration.
Click Driver parameters drop-down and go to Publisher Settings.
Under Publisher Option, select Publisher setting to Poll Mode.
Under Optional Header Fields, select the Pagination Type to Cursor Pagination.
Select the Cursor location to Response Header.
Enter the required details in the following field:
1. Link Header Relation
2. Link Header Value
3. Absolute URL (Yes/No)
4. (optional) If it is not an Absolute URL, Specify first Page URL.
5. Page Size.
Click Save.

It is further explained by a syntax with an example below:

Syntax: Link: <uri-reference1>; param1=value1; param2="value2", <uri-reference2>; param1=value1; param2="value2"

Parameters	Description
Param	link relation
Value	link relation value

Example:

Enter the following Input Parameter:

Link Header Relation = rel
Link Header Value = “next”
Absolute URL = Yes
Page Size = 2

Link:

<https://api.github.com/repositories/1300192/issues?page=2>; rel="prev", 
<https://api.github.com/repositories/1300192/issues?page=4>; rel="next", 
<https://api.github.com/repositories/1300192/issues?page=515>; rel="last", 
<https://api.github.com/repositories/1300192/issues?page=1>; rel="first".

After parsing the link header, it is stored as key value pair as shown below:

{ rel:
{ prev: 'https://api.github.com/repositories/1300192/issues?page=2',
next: 'https://api.github.com/repositories/1300192/issues?page=4',
last: 'https://api.github.com/repositories/1300192/issues?page=515',
first: 'https://api.github.com/repositories/1300192/issues?page=1'
}
}

Key set/Token pagination

In this technique the cursor is present in the response body.

Parameters	Description
Cursor Key	Attribute in response that contains the cursor
Terminating Key	Attribute in response that indicates the last page
Terminating value	Pagination is done when the Terminating key value matches this value
Page size	Number of records on given page

To update the pagination details, perform the following steps:

In Identity Console, go to Drivers.
Select the driver that you want to perform pagination.
Navigate to Configuration.
Click Driver parameters drop-down and go to Publisher Settings.
Under Publisher Option, select Publisher setting to Poll Mode.
Under Optional Header Fields, select the Pagination Type to Cursor Pagination.
Select the Cursor location to Response Payload.
Enter the required details in the following field:
1. Cursor key
2. Terminating Key
3. Terminating Value
4. Absolute URL (Yes/No)
5. (optional) If it is not an Absolute URL, Specify first Page URL.
6. Page Size.
Click Save.

Example:

Enter the following Input Parameters:

Cursor key = nextPageURL
Terminating Key = nextPageURL
Terminating Value = null
Absolute URL = Yes
Page Size = 2

Salesforce Payload:

{
"currentPageUrl": "/services/data/v23.0/chatter/users",
"nextPageUrl": null,
"previousPageUrl": null,
"users": [
{
"companyName": "Open Text",
"firstName": Tana,
"id": "0055i000004C5CfAAK",
"lastName": "Garrison",
"name": "Tana Garrison",
"type": "User",
"url": "/services/data/v23.0/chatter/users/0055i000004C5CfAAK"
},
{
"companyName": "Open Text",
"firstName": "Adrian",
"id": "0055i000004C5CaAAK",
"lastName": "Stephens",
"name": "Adrian Stephens",
"type": "User",
"url": "/services/data/v23.0/chatter/users/0055i000004C5CaAAK"
},
{
"companyName": "Microfocus",
"firstName": "Tejas",
"id": "0055i000004HQiFAAW",
"lastName": "MP",
"name": "Tejas MP",
"type": "User",
"url": "/services/data/v23.0/chatter/users/0055i000004HQiFAAW"
},
{
"companyName": Microfocus,
"firstName": "Abhishek",
"id": "0055i000004HubrAAC",
"lastName": "Prasad",
"name": "Abhishek Prasad",
"type": "User",
"url": "/services/data/v23.0/chatter/users/0055i000004HubrAAC"
}
]
}

Custom Pagination

In some cases, the implementation details can vary depending on the specific inputs used. There is custom pagination in which you must provide the interface that has to be implemented. The implementation can be done by the following methods:

init(String resourceUrl, HashMap<String, String> paginationParm) throws PaginationException;
getFirstPageUrl() throws PaginationException;
getNextPageUrl(String response, HashMap<String, String> responseHeaders) throws PaginationException;

To update the pagination details, perform the following steps:

In Identity Console, go to Drivers.
Select the driver that you want to perform pagination.
Navigate to Configuration.
Click Driver parameters drop-down and go to Publisher Settings.
Under Publisher Option, select Publisher setting to Poll Mode.
Under Optional Header Fields, select the Pagination Type to Custom Pagination.
Enter the required details in the following field:
1. Java Class
2. Init Parameters
Click Save.

For more details check the following Section Section B.0, Using Java Extensions and check the following JavaDocs as well.