Pagination styles (REST source)

The REST source walks multi-page responses automatically. Set pagination.type to one of the styles below. Every style has a loop/termination guard so a misbehaving API can’t loop forever. max_pages is an optional cap across all of them (unset by default). A pass the cap cuts short keeps only a persist_cursor bookmark, which names the next unread page. An incremental or windowed pass the cap cuts short fails the run: a record-derived or window bookmark would skip the pages it never read, and keeping none would re-read the same pages every run while each run looked green. Set assume_ordered: true when the feed returns records in ascending replication_key order and the bookmark is pushed to the server with a replication_bind — a truncated incremental pass then persists the largest key it read and the next run’s first request starts at it. The bind must compare inclusively (>=, e.g. template: "gte|${bookmark}"): rows that share the last key read can sit on the first unread page, and a strict > server filter would skip them for good (the re-read rows at that key arrive again, so pair it with an upsert sink). Without a replication_bind the combination is refused at load time: the next run would start at page 1 again and re-read the same pages. assume_ordered does not apply to windowed passes or across requests: entries.

Style	Stops when
None	after the first page
Cursor	the next-token JSONPath is null/absent (or repeats)
CursorInBody	the next-token JSONPath is null/absent (or repeats) — for POST-search APIs that take the cursor in the request body
PageNumber	a page returns zero records (or repeats the previous page; repeating an earlier, non-adjacent page fails the run, since rows may be unread — set allow_repeated_pages: true with max_pages for a feed whose pages can legitimately repeat)
Offset	the offset reaches total (via total_path), or an empty page arrives (a short page alone does not stop it: servers clamp limit; stop_when_short: true opts in for servers that reject an offset past the end)
OffsetInBody	like Offset (total, an empty page, or stop_when_short: true) — offset/limit are written into the JSON request body (POST-query APIs)
RecordFieldCursor	a short page arrives — keyset paging by the running max/min of a record field
LinkHeader	there’s no rel="next" in the Link response header
NextLinkInBody	the next-page URL in the body is absent, null, or empty

Note

An HTTP 204 No Content (or any 2xx with an empty body) is treated as an empty page under every style, so a feed that ends with a 204 after its last data page (e.g. $top/$skip paging) stops cleanly instead of erroring.

Cursor

pagination:
  type: Cursor
  next_token_path: $.meta.next_cursor  # JSONPath to the next-page token
  param_name: starting_after           # query param to send it back as

Cursor in body (POST search)

For endpoints that page a POST search body — the next cursor comes back in the response and must be written back into the request JSON body (e.g. a CRM POST …/objects/{obj}/search):

source:
  type: rest
  config:
    method: POST
    body: { limit: 100, sorts: ["hs_lastmodifieddate"] }   # base search body
    records_path: $.results[*]
    pagination:
      type: CursorInBody
      next_token_path: $.paging.next.after   # JSONPath to the next cursor
      body_cursor_field: after               # body field to inject it into

The first request sends body unchanged; each later request adds body[body_cursor_field] = <cursor>. Pagination stops when the cursor is null/absent or repeats.

body_cursor_field may be a JSON Pointer for a nested location, so a GraphQL API reached through the REST source can page by variables.after:

    method: POST
    body:
      query: "query($first: Int, $after: String) { orders(first: $first, after: $after) { nodes { id } pageInfo { endCursor } } }"
      variables: { first: 250 }
    records_path: $.data.orders.nodes[*]
    pagination:
      type: CursorInBody
      next_token_path: $.data.orders.pageInfo.endCursor
      body_cursor_field: /variables/after

A pointer must resolve to an existing value or to a new key of an existing object (here variables); intermediate objects and array elements are never created, and a pointer that does not resolve fails the request. The same applies to OffsetInBody’s offset_field / limit_field, and to replication_bind / window binds with into: body + path (see the state cookbook).

Page number

pagination:
  type: PageNumber
  param_name: page
  start_page: 1
  page_size: 500
  page_size_param: per_page

Offset / limit

pagination:
  type: Offset
  limit: 1000
  limit_param: limit
  offset_param: offset
  total_path: $.meta.total             # optional; enables an exact stop

Offset / limit in the body (POST-query APIs)

For POST endpoints that take offset/limit in the JSON request body (not the query string). The offset advances by each page’s record count and paging stops on an empty page or once the offset reaches total_path. A short page alone does not stop it (servers clamp limit) unless stop_when_short: true. When records_path fans out to nested children ($.orders[*].lines[*]) or record_ancestors is set, set rows_path to the server’s row array, or the offset would advance by child records.

pagination:
  type: OffsetInBody
  offset_field: offset
  limit_field: limit
  limit: 500
  stop_when_short: true                # only for servers that never cap a page (default false)

Report APIs return the whole response as one record (headers plus a rows array, e.g. a runReport endpoint with records_path: "$"). Counting records would read every page as short and stop after page 1, so count the rows instead, and stop at the reported total:

pagination:
  type: OffsetInBody
  offset_field: offset
  limit_field: limit
  limit: 10000
  rows_path: "$.rows"                  # advance by the rows on each page
  total_path: "$.rowCount"             # stop once offset reaches the total

rows_path counts an array’s elements, 1 for an object, 0 for null or no match (which ends paging). A missing or non-numeric total_path falls back to the short-page rule.

Keyset (record-field cursor)

Page by the running max (or min) of a record field — the pattern accounting journal APIs use (offset = max(JournalNumber) of the last page). Stops on a short page.

The cursor is taken from every record a page returns, including ones the incremental filter drops, so a page of only-old rows still moves it. Values compare like replication bookmarks: numbers and numeric strings by value ("59" follows "9"), timestamps chronologically, other strings as text. A full page that leaves the cursor where it was fails the run — more than a page of rows share that value, or the feed isn’t ordered by the field — rather than stopping and silently skipping the rest.

pagination:
  type: RecordFieldCursor
  field: JournalNumber
  into: query                          # or `body`
  param: offset
  agg: max                             # or `min`
  page_size: 100
  stop_when_short: true
  inclusive: false                     # true: the server returns rows >= the cursor

By default the cursor is an exclusive bound, so field must be unique: rows sharing the boundary value at a page cut are never read (a warning fires when a full page ends on such a tie). For a non-unique field like updated_at, filter the server on >= the cursor and set inclusive: true; the rows already emitted at the boundary value are dropped by whole-record fingerprint, so each row is read once.

Resumable cursor & multi-array responses

• persist_cursor: true on a Cursor / CursorInBody stream saves the terminal cursor as the run’s bookmark (via a state: store) and seeds it into the next run’s first request — so an envelope-cursor feed (e.g. a /transactions/sync endpoint) resumes incrementally instead of re-pulling from the start.
• records_multi emits several response arrays in one pass (one pagination advance), each stamped with a configurable op_field — pair with a sink write_mode: upsert + delete_marker to route added/modified→upsert and removed→delete from a single sync response.
• record_ancestors lifts fields from an enclosing array-element ancestor onto records unwrapped from a nested records_path (e.g. keep an event’s envelope id on each unwrapped object).

See the faucet-source-rest README for the full field reference.

Link header

pagination:
  type: LinkHeader      # follows the RFC 5988 `Link: <…>; rel="next"` header

Next link in body

pagination:
  type: NextLinkInBody
  next_link_path: $.links.next         # JSONPath to the absolute next-page URL

A server-given link (either style, or an async-job URL) that points at another scheme, host or port than base_url is fetched without the source’s credentials, and an https base never follows a plain-http link. A redirect to another origin likewise keeps only standard request headers (Accept, Content-Type, User-Agent, Range, …): auth headers, a shared provider’s credential and every configured headers: entry stay behind. When the API really serves pages or results from a second host, list it:

trusted_hosts: [files.example.com, "files.example.com:8443", "*.cdn.example.com"]

A link or a redirect to a listed host keeps every header when it keeps the scheme; a bare host covers any port, host:port only that port.

Note

Use faucet schema source rest to see the exact fields and defaults for each style in your installed version.

See also

• Authentication — pair pagination with the right auth for your API.
• REST API → BigQuery (incremental) — a full paginated REST pipeline, end to end.