FetchNode passes the page HTML as convert_to_md's baseurl, inflating model input ~74x #1142

Activity

1. dosubot commented on Sep 1, 2026

   

   dosubotbot

   on Sep 1, 2026 – with DosuBot

   More actions

   Hi @julianriegler! I'm Dosu and I'm helping the ScrapeGraphAI, Inc team.

   Your diagnosis is spot-on. On master, handle_web_source calls convert_to_md(document[0].page_content, parsed_content) where parsed_content is document[0].page_content [1], so the full HTML gets assigned to h.baseurl [2]. The use_soup branch has the swapped-arguments problem you flagged too — convert_to_md(source, parsed_content) passes the URL as HTML and vice versa [3] — and parsed_content is indeed unbound when cut=True since it's only assigned inside if not self.cut: [4].

   All of these bugs are addressed in PR #1112, which makes exactly the changes you suggested — convert_to_md(doc.page_content, source) and convert_to_md(parsed_content, source) — plus fixes the cleanup_html call (was receiving a Response object instead of response.text, and its tuple return wasn't being unwrapped) and the unbound parsed_content in the cut=True path [5]. That PR is still open and not yet merged [6].

   Separately, PR #1131 (merged) improved convert_to_md so that when a real URL is passed, relative links resolve against the full document path instead of just the site root [7]. That improves link resolution once #1112 lands, but doesn't fix the caller-side bug you're reporting since convert_to_md still blindly assigns whatever it receives to h.baseurl [2].

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
2. MonsterDylan commented on Sep 3, 2026

   

   MonsterDylan

   on Sep 3, 2026

   More actions

   Confirming this on the released 2.2.2 package (not just master), and flagging that the impact is worse than input inflation — on <table>-based pages it causes complete data loss / empty output, so it's a correctness bug, not just a cost/token one.

   Repro (Hacker News)

   from scrapegraphai.graphs import SmartScraperGraph
   
   SmartScraperGraph(
       prompt="Extract the top stories with rank, title, url, points, comments.",
       source="https://news.ycombinator.com/",
       config={"llm": {"api_key": "...", "model": "openai/gpt-4o-mini"}},
   ).run()
   # -> {'stories': []}

   What happens at each stage

   Stage	Chars	Note
   Playwright fetch	34,133	full HN HTML, no block
   convert_to_md(html, url=<the same HTML>)	44,827	output starts | [![](<html lang="en"... — the whole doc becomes a broken markdown-image "URL" because it's assigned to h.baseurl
   ParseNode Html2TextTransformer	9	collapses to | [![](\n\n
   GenerateAnswer	—	model sees ~nothing → NA / {'stories': []}

   So with a ChatOpenAI model (which triggers the convert_to_md branch in FetchNode), any page whose markdown begins with a linked/image element ahead of a table layout can silently return empty. Modern div-based pages (e.g. books.toscrape.com) happen to survive.

   PR #1112 fixes exactly this (convert_to_md(doc.page_content, source)). Given it turns some sites into silent empty results rather than merely bloated input, it'd be great to get that merged. Happy to help test.

   Workaround until then, for anyone hitting empty results: patch the caller so the bogus base-URL is dropped when it's HTML —

   import scrapegraphai.nodes.fetch_node as fn
   _orig = fn.convert_to_md
   fn.convert_to_md = lambda html, url=None: _orig(
       html, None if (url and (len(url) > 2000 or url.lstrip()[:1] == "<")) else url
   )
3. chenhz01 commented on Sep 22, 2026

   

   chenhz01

   on Sep 22, 2026

   More actions

   To recap the call site: it passes parsed_content, which is the raw HTML, where convert_to_md expects a URL string, so html2text's baseurl becomes the entire document and every relative link gets the full HTML prepended. An open PR already fixes the call site to pass the real source URL; my addition below is the mechanical guard that would have caught this. Beyond that, I'd add a cheap input-hygiene guard that would have caught this mechanically: assert baseurl parses as a URL and is under a sane length, plus a regression test asserting converted output stays within a small factor of the raw HTML length - a 74x blow-up should trip a test, not a user's token bill. Want me to take a crack at it? Happy to open a PR with working code.