Repository navigation
feat(pagination): add generic Paginator and public CLOB pagers - #37
Conversation
SebastianBoehler
left a comment
There was a problem hiding this comment.
Thanks Bill. Two issues to address before this is ready:
-
GCC build failure (
include/polymarket/pagination.hpp:188-191): the free friendoperator==defined insideItemIteratoraccessesPaginator's privatebuffer_index_andbuffer_. It does not inherit the nested class's access rights. All three Ubuntu build jobs fail here. Move the private-state check into anItemIteratormember and call that from the comparison, or otherwise provide valid access. -
Offset resume token repeats the final page (
include/polymarket/pagination.hpp:263-267): the short-page and offset-cap branches leaveoffset_pointing at the page just fetched. With limit=2 and max_offset=2, collecting pages at offsets 0 and 2 returns resume_token().offset=2; reconstructing a paginator from that token fetches offset 2 again. I reproduced the duplicate locally. The token needs to represent the next position or exhaustion, including both terminal paths, with regression coverage.
The focused paginator tests pass locally with Apple Clang; the GCC failure is confirmed by the CI logs. The resume issue was reproduced with a small deterministic fetcher.
For future PRs, please make sure all required CI checks pass before requesting review. For this PR, please fix the failures and wait for green CI before requesting another review.
Summary
Each CLOB and Data API list endpoint had its own copy of the paging loop. Errors were handled differently from one to the next, and callers had no way to read one page at a time or resume.
include/polymarket/pagination.hppwithPaginator<T>for cursor and offset paging. It offersnext_page(),collect(max_items), range-for over items witherror()checked afterwards, andresume_token(). Items fetched before a failure are kept. A repeated cursor or more than 1000 pages is an error. Reaching the offset cap setstruncated().ClobClient:markets_pager,sampling_markets_pager,simplified_markets_pager,sampling_simplified_markets_pager,open_orders_pager,trades_pager,positions_pager,redeemable_positions_pagerandmergeable_positions_pager.MarketFetcherloops ontoPaginator. Legacy methods return the same results as before.Verification
Commands run and their actual results:
cmake --build build --parallel 2 && ctest --test-dir build --output-on-failure -LE live: 65/65 passed (Linux, Release, GCC 13.3), includingtest_package_consumerand the two fuzz corpus replay tests.python3 scripts/quality.py main --build-dir build: passed.test_pagination(engine, fake fetcher) andtest_clob_pagers(pagers against the local HTTP fixture).Checks not run and the reason (or none):
-LE live): only read-only paging changed, and the request URLs are checked by fixture tests.Compatibility and evidence
Public API, binary compatibility, or release implications (or none):
ClobClientgains methods but no data members, andclob_client.hppnow includespagination.hpp.src/opaque_cursor_pagination.hppmoved into the public header underpolymarket::detail, and its constantMAX_PAGESis nowmax_pages.get_open_orders_resulthits a repeated cursor, theParseerror no longer carries a response body excerpt.Not a protocol change. All coverage is local fixtures; no live calls were made.