Skip to content

Fix University of Twente scraper (0 events ingested) - #1379

Merged
fbacall merged 1 commit into
ElixirTeSS:masterfrom
ioan-alexandra:fix/utwente-scraper-neopage-api
Oct 1, 2026
Merged

fbacall merged 1 commit into
ElixirTeSS:masterfrom
ioan-alexandra:fix/utwente-scraper-neopage-api

Conversation

@ioan-alexandra

@ioan-alexandra ioan-alexandra commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • By inspecting the site's JS bundle, found the actual WebHare JSON-RPC endpoint the page itself calls to load events: POST /wh_services/utwente_base/rpc/GetNEOItems, paginated via a skip parameter.
  • Rewrote UtwenteIngestor to call this endpoint directly, with proper pagination (capped at MAX_PAGES as a safety net) instead of a single hardcoded request assuming one page.

Verification

  • Called the discovered endpoint directly against the live site - confirmed it returns real, current UT events with pagination (moreitems/skip).
  • Rewrote the VCR cassette and test to cover a 2-page pagination scenario.

Motivation and context

UT rebuilt their events page as a fully client-rendered app at a new URL (/en/about-us/news-events-ceremonies/events/, redirected from the old /en/events/).

Checklist

  • I have read and followed the CONTRIBUTING guide.
  • I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree to license it to the TeSS codebase under the BSD license.

@ioan-alexandra
ioan-alexandra force-pushed the fix/utwente-scraper-neopage-api branch from 2816809 to 24cba8e Compare September 16, 2026 12:05
@ioan-alexandra
ioan-alexandra force-pushed the fix/utwente-scraper-neopage-api branch from 24cba8e to 23ae89c Compare September 16, 2026 12:10
@fbacall
fbacall merged commit 1058c28 into ElixirTeSS:master Oct 1, 2026
7 checks passed
@ioan-alexandra
ioan-alexandra deleted the fix/utwente-scraper-neopage-api branch October 1, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants