API Pagination

Why not send everything

When a collection holds thousands or millions of records, returning them all in one response would be slow to generate, memory-hungry to hold, and fragile to transfer over an imperfect network. Pagination splits the result into manageable pages, so each request transfers a bounded slice and your code processes the collection a chunk at a time rather than all at once.

Most APIs apply a sensible default page size and let you request a different one up to a documented maximum. The response then includes both the current page of items and some metadata that tells you how to fetch the next one — a cursor, a link, or a page number — so you are never left guessing whether more data remains beyond what you can see.

Offset pagination

Offset (or page-number) pagination uses parameters such as page and per_page, or limit and offset. It is wonderfully intuitive — page three of twenty — and it lets you jump straight to an arbitrary page without walking through the ones before it.

GET /v1/items?limit=50&offset=100

Its weakness shows up on data that changes while you are paging through it. If a record is inserted or removed between two requests, the window shifts beneath you, and you can quietly skip a row or see one twice. For static or slowly changing data offset paging is perfectly fine; for busy, live data it is genuinely risky, and the bugs it produces are intermittent and maddening to reproduce after the fact.

Cursor pagination

Cursor (or keyset) pagination hands you an opaque token that marks exactly where you stopped. You pass that token back to fetch the next page, and the server resumes from that precise point regardless of inserts or deletions happening elsewhere in the collection at the same time.

This makes cursors far more reliable on busy collections, and usually faster at depth too, since the database need not count its way past all the rows you have already skipped. The trade-off is that you cannot jump to an arbitrary page — you walk forward from where you are, one page at a time. Treat the cursor as genuinely opaque: never try to construct, decode, or reason about its internal contents, because its format can change without notice.

Walking every page

The loop is the same whichever style you use: fetch a page, process it, ask for the next, and stop when the API signals there are no more results to return. Some APIs expose that signal through a next field in the response body; others use the standard Link header carrying rel='next', which your client follows like any other link.

Keep your page size reasonable rather than maximal, handle the empty final page gracefully without treating it as an error, and respect rate limits while you loop so a large scan does not trip a throttle. Reading the paging metadata the API gives you is always more robust than trying to guess when to stop from a record count.

Choosing an approach

Prefer cursor pagination for anything that changes frequently or that you intend to scan in full, since it gives you correctness for free on moving data. Reach for offset only when stable data and genuine random page access both matter to your use case. Whichever style the API offers, follow its documented conventions rather than inventing your own ad-hoc scheme on top.

Pagination pairs naturally with filtering and sorting, which narrows and orders the collection before you ever start walking it — fewer pages to fetch, in a predictable order, which makes the whole scan cheaper and more reliable.