Summary Port should add first-class support for navigation-style cursor pagination, where APIs return the next page cursor via the HTTP Link header (rel="next") instead of embedding cursor metadata in the JSON body. This is required for full compatibility with APIs like GitGuardian, which implement cursor pagination through Link headers in the same style as GitHub. Background / Motivation Many modern REST APIs paginate large datasets using the Web Linking standard: Responses return a Link header containing one or more navigation URLs Clients retrieve the next page by following the rel="next" URL directly GitGuardian documents this explicitly in their pagination guide: Cursor-based pagination is used across their API Pagination is driven through links exposed in the response headers GitGuardian Pagination Docs: https://docs.gitguardian.com/api-docs/pagination Real-World Example: GitGuardian API Pagination GitGuardian provides a reference Python implementation: while True: response = requests.get(endpoint_url) if "next" not in response.links: break endpoint_url = response.links["next"]["url"] This shows that pagination works by repeatedly following: response.links["next"]["url"] rather than manually constructing cursor= params. Related Industry Standard: GitHub REST API GitHub uses the exact same navigation approach: Paginated responses include: Link: <...page=2>; rel="next", <...page=last>; rel="last" GitHub docs state: "In all cases, you can use the URLs in the link header to fetch additional pages of results." Problem Statement Port’s current pagination support is primarily focused on: explicit cursor parameters (cursor=...) body-based cursor tokens (next_cursor, starting_after, etc.) However, APIs like GitGuardian require clients to: read the Link header follow rel="next" URLs directly Without native support, integrations must implement custom pagination logic, leading to: duplication across connectors incomplete ingestion support difficulty integrating GitGuardian and GitHub-class APIs Proposed Feature Add “Nav Style Link Header Pagination” Strategy Port should support a pagination mode where: Initial request is made normally Port checks the Link response header If rel="next" exists: Port automatically requests that URL Continue until rel="next" disappears This works even when the cursor is opaque or parameter names vary. Configuration Proposal (Example) pagination: strategy: link_header header: Link rel: next Or more simply: pagination: type: nav Functional Requirements Parse RFC-compliant Link headers (rel="next") Support GitGuardian-style response.links["next"] Treat next-page URLs as opaque (no assumptions about param naming) Work across APIs using: page=2 cursor=... after=... mixed/hidden pagination tokens Coexist with existing cursor/body pagination support Benefits Immediate compatibility with GitGuardian APIs Unlocks GitHub-style REST ingestion in Ocean Framework Reduces connector-specific pagination code Aligns Port with widely adopted HTTP navigation conventions Improves robustness when cursor parameters differ across endpoints Success Criteria Port can paginate GitGuardian endpoints without custom integration code Connectors can declaratively enable nav/link_header pagination End-to-end ingestion works across all rel="next"-based APIs
