API Pagination in n8n: Retrieve Complete Records and Resume Safely

API pagination in n8n prevents a limited response from silently becoming an incomplete collection. The workflow must request subsequent pages, recognize the documented stopping condition, and distinguish completion from an interrupted run. A successful HTTP response alone does not establish that every required record was retrieved.

Configure the HTTP Request node for the destination’s actual contract. Page numbers, offsets, cursors, and next-page URLs are different mechanisms; a field used by another provider is not a universal pagination rule.

Inspect one real response before configuring the loop

Use a small authorized read request and inspect the body, headers, and accepted parameters. Determine the documented page size, starting index, ordering, and navigation fields.

A service may expose a cursor, a next link, a page number, or another completion indicator. It may also return an array without enough metadata to infer completion safely. Use the service’s documentation and a controlled example to resolve that ambiguity.

The HTTP Request reference explains the node’s configurable pagination modes. Selecting a mode is only one part of implementing the destination’s contract.

Advance according to the documented mechanism

For a cursor, pass the returned value as specified rather than constructing your own. For numbered pages, verify the expected starting index. For offsets, use the provider’s advancement rule instead of assuming every API uses the requested maximum or the actual count identically.

If a response supplies the next URL, validate the expected destination before allowing subsequent authenticated requests. A link in a response should not silently send a credential to an unrelated host.

Variables such as the node’s pagination count can help compose a request, but check their meaning against the API’s indexing. A one-page error can repeat the beginning or omit an entire section of the collection.

Define completion and safeguards separately

Use an explicit stopping rule

Stop when the contract’s end condition is satisfied, such as an absent next cursor or a documented final-page flag. An empty page is a valid stopping signal only when the service defines it that way.

Set a safety limit to prevent uncontrolled work, and record when that limit is reached. Do not label a collection complete merely because the workflow stopped after a configured number of requests.

Detect repeated navigation values. A cursor that does not progress deserves an alert and investigation rather than an infinite loop that consumes quota while retrieving the same records.

Protect against repeated work after interruption

A workflow can fail after reading initial pages or after the destination accepted a write whose response was lost. Restarting can therefore repeat records even if the source pagination itself is correct.

Use a stable entity identifier with an appropriate upsert, destination lookup, or documented idempotency mechanism. Do not create a fresh identity for the same logical record on every attempt.

Save checkpoints only after the relevant results are persisted successfully. Advancing first can turn a partial failure into an omitted page on the next run. If resuming with overlap, deduplicate by the appropriate stable key.

Account for a collection that changes during retrieval

Records inserted or updated during an extraction can affect ordering, particularly with offset-based navigation. Use stable ordering or a supported time window where available.

A cursor is not automatically a guarantee of an immutable snapshot. Check the provider’s documented consistency behavior and reconcile when the task requires a more complete result.

For recurring updates, retain the confirmed collection position or time and relevant schema context. Periodic broader reconciliation may be necessary to identify changes outside a narrow window.

Respect rate limits and payload constraints

Use the provider’s rate-limit response and retry guidance, including Retry-After where applicable. Bound attempts and add appropriate spacing instead of sending uncontrolled parallel requests.

Reduce unnecessary returned fields and use accepted batch sizes. A fixed record count can still produce a large byte payload when some entries include long descriptions or attachments.

Consider timeout, memory, and incremental storage for larger volumes. Estimate the number of pages per run and the frequency when calls are billed. A daily multi-page extraction has a different operating footprint from a manual lookup.

Check completeness in the persisted result

Compare unique source identifiers, collected count, persisted count, filters, and any authoritative total available. A total reported by your own loop is not an independent proof that the loop covered the source.

Create an appropriate test set spanning more than one page, with a known record at the end. Test missing navigation data, repeated cursors, a rate-limit response, and an interruption during persistence.

Mark a failed or partial run clearly. Keep enough sanitized evidence to identify which page or stage stopped without logging customer payloads and secrets unnecessarily.

Distinguish n8n’s public API from a generic HTTP call

The public n8n API pagination guide documents nextCursor. Follow that contract when listing instance resources. The generic HTTP Request node must instead follow whatever contract its target API defines.

For a workflow inventory, record collection time, page count, filters, unique identifiers, and errors. Do not infer the instance’s size from the first response.

The inventory guide covers API-key handling and administrative scope. A key that lists instance resources should remain protected rather than becoming a convenient public example value.

Plan a recovery path for larger collections

Divide work by a supported time range, account, or resource where useful. Persist confirmed progress and retain a run identifier so the next execution can distinguish recovery from a new extraction.

Normalize the schema deliberately before writing. If an API changes its response, detect and report the mismatch rather than silently dropping fields or producing an apparently empty success.

When volume exceeds the practical scope of one run, assess a queue or suitable database-backed approach. The objective is a verifiable, recoverable collection, not forcing every response into memory because the first example was small.

Document the production contract

Record endpoint, paging mechanism, size, navigation field, completion rule, safety limit, rate handling, deduplication key, checkpoint timing, and completeness evidence. Assign the person who investigates a partial run.

After a version or credential change, repeat the affected tests. Monitor page count, duration, failed runs, duplicates, and reconciled records. A sudden change can reveal a broken assumption even when the transport still returns success.

A useful controlled example could contain 12 known records and use a page size of five. Verify the expected three-page collection and the final known identifier rather than checking only the total. Then interrupt persistence partway through the run and inspect the resumed result for missing or repeated entities. These numbers describe a test design, not a measurement of a real customer database or a guaranteed page size for every provider.

Frequently asked questions

Does HTTP 200 prove I retrieved everything?

No. Check navigation, unique records, persistence, and an appropriate reference.

Are cursor and offset interchangeable?

No. Follow the API’s actual navigation contract.

How do I avoid duplicates after a failure?

Use stable identity and appropriate destination protection, and advance checkpoints only after confirmed persistence.

Is a page limit the completion rule?

It is a safeguard. Reaching it without evidence of the end should remain an incomplete outcome.