Pagination issue in RESTful API design

Question

I am designing a RESTful API for a mobile application I am working on. My problem is with large collections containing many items. I understand that a good practice is to paginate large number of results in a collection.

I have read the Facebook Graph API doc (https://developers.facebook.com/docs/graph-api/using-graph-api/v2.2), Twitter cursors doc (https://dev.twitter.com/overview/api/cursoring), GitHub API doc (https://developer.github.com/v3/) and this post (API pagination best practices).

Consider an example collection /resources in my API that contains 100 items named resource1 to resource100 and sorted descending. This is the response you will get upon a GET request (GET http://api.path.com/resources?limit=5):

{
    "_links": {
        "self": { "href": "/resources?limit=5&page=1" },
        "last": { "href": "/resources?limit=5&page=7" },
        "next": { "href": "/resources?limit=5&page=2" }
    },

    "_embedded": {
        "records": [ 
            { resource 100 },
            { resource 99 },
            { resource 98 },
            { resource 97 },
            { resource 96 }
        ]
    }
}

Now my problem is a scenario like this:

1- I GET /resources with above contents.

2- After that, something is added to the resources collection (say another device adds a new resource for this account). So now I have 101 resources.

3- I GET /resources?limit=5&page=2 as the initial response suggests will contain the next page of my results. The response would be like this:

{
    "_links": {
        "self": { "href": "/history?page=2&limit=5" },
        "last": { "href": "/history?page=7&limit=5" },
        "next": { "href": "/history?page=3&limit=5" }
    },

    "_embedded": {
        "records": [ 
            { resource 96 },
            { resource 95 },
            { resource 94 },
            { resource 93 },
            { resource 92 }
        ]
    }
}

As you can see resource 96 is repeated in both pages (Or similar problem may happen if a resource gets deleted in step 2, in that case one resource will be lost).

Since I want to use this in a mobile app and in one list, I have to append the resources of each API call to the one before it so I can have a complete list. But this is troubling. Please let me know if you have a suggestion. Thank you in advance.

P.S: I have considered timestamp like query strings instead of cursor based pagination, but that will make problems somewhere else for me. (let me know if you need more info about that.)

Why not use both cursor-based pagination and a timestamp?

Jonathan W
– Jonathan W

2015-01-26 16:58:09 +00:00
Commented Jan 26, 2015 at 16:58 — Jonathan W
– Jonathan W, Commented Jan 26, 2015 at 16:58

Jonathan W · Accepted Answer · 2015-01-25 05:14:18Z

3

We just implemented something similar to this for a mobile app via a REST API. The mobile app passed an additional query parameter which represents a timestamp at which elements in the page should be "frozen".

So your first request would look something like GET /resources?limit=5&page=1&from=2015-01-25T05:10:31.000Z and then the second page request (some time later) would increment the page count but keep the same timestamp: GET /resources?limit=5&page=2&from=2015-01-25T05:10:31.000Z

This also gives the mobile app control if it wants to differentiate a "soft" page (preserving the timestamp of the request of page 1) from a "hard refresh" page (resetting the timestamp to the current time).

answered Jan 25, 2015 at 5:14

Jonathan W

3,79921 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Mepla Over a year ago

Thank you very much for the reply. Yes I am considering that approach but as I said in my P.S that may cause further problems for me. Anyways if I end up using this solution I will mark your answer as accepted. Thanks again.

Mepla Over a year ago

Well I ended up doing both curser based and a before/after timestamp. Add this to your answer and I'll accept it. Thank you very much :)

Jonathan W Over a year ago

Except that I don't understand why you would use both a before and an after timestamp per request, nor did you explain why only using one timestamp would cause you issues.

Petar Over a year ago

@JonathanW so when do you update and save the 'from' value then ?

Jonathan W Over a year ago

@pe60t0 In the case of the mobile app I was involved with, the app itself would do that when a pull-down refresh was issued. When doing infinite scroll towards the bottom of the list, it would send the same timestamp. Make sense?

|

Stephen Souness · Accepted Answer · 2015-01-24 13:01:53Z

0

Why not just maintain a set of seen resources?

Then when you process each response you can check whether the resource is already being presented.

answered Jan 24, 2015 at 13:01

Stephen Souness

2821 silver badge3 bronze badges

1 Comment

Mepla Over a year ago

That will only prevent something to repeat in view, but in a lower level, I am wasting time performing API calls that do not get me what I want. (Imaging my step 2 if twenty resources were added)

Collectives™ on Stack Overflow

Pagination issue in RESTful API design

2 Answers 2

8 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related