Skip to content

Fix review findings in recipe scrapers - #1

Merged
Endogen merged 1 commit into
mainfrom
review/fix-bugs-1-10
Sep 21, 2026
Merged

Endogen merged 1 commit into
mainfrom
review/fix-bugs-1-10

Conversation

@Endogen

@Endogen Endogen commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Summary

Companion to Endogen/web2api#1.

Changes

  • web-reader: block SSRF (private/loopback/link-local/multicast/reserved hosts) with WEB2API_WEBREADER_ALLOW_PRIVATE opt-out.
  • allenai: correct tools_url docs/example to the /mcp base URL (was pointing at non-existent /tools).
  • nvidia: send Authorization header on GET /models too.
  • brave-search / wikipedia / x / openstreetmap: coerce numeric params via coerce_int/coerce_float (surfaces INVALID_PARAMS instead of SCRAPE_FAILED).
  • nvidia: raise InvalidParamsError for malformed numeric params.
  • deepl: keep stability loop under the default SCRAPE_TIMEOUT.
  • x: kill the bird subprocess on timeout.

Tests

  • Recipe scrapers compile clean under Python 3.12.
  • web2api full suite (144 passed) exercises the hackernews recipe live against these recipes.

- Block SSRF in web-reader with a private-host guard and opt-out env var
- Correct allenai tools_url docs/example to the /mcp base URL
- Send NVIDIA Authorization header on GET /models too
- Coerce numeric params via coerce_int/coerce_float in brave-search,
  wikipedia, x, and openstreetmap
- Raise InvalidParamsError for malformed numeric params in nvidia
- Keep DeepL stability loop under the default scrape timeout
- Kill the bird subprocess on timeout in the x scraper
@Endogen
Endogen force-pushed the review/fix-bugs-1-10 branch from e4ab8d0 to f79ac5b Compare September 21, 2026 22:52
@Endogen
Endogen merged commit 6889b25 into main Sep 21, 2026
1 of 3 checks passed
@Endogen
Endogen deleted the review/fix-bugs-1-10 branch September 21, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant