feat(agent): opt-in append-note concept_update_mode to avoid full-page rewrites on ingest

[sebastianbraun25]

Note

This PR was created in collaboration between a human and AI: implementation, tests, and
PR text were created by an AI assistant under the guidance and review of the human author.

Problem

openkb add currently regenerates the FULL body of every existing concept/entity page it touches. _gen_update/_gen_entity_update in openkb/agent/compiler.py read the entire current page from disk and send it to the LLM as existing_content, with the instruction "Rewrite the full page incorporating the new information naturally — do not just append." For a short document (e.g. a support ticket) that touches 8-10 already-existing concepts/entities, this means 8-10 full-page rewrites per ingest, even when the actual new information is just 1-2 sentences. Cost scales with page size, and page size only grows as more documents are ingested over time.

Solution / Changes

• New concept_update_mode config key (.openkb/config.yaml, default "rewrite" — current behavior, fully unchanged). KB-only, like debug/insert_mode — not in GLOBAL_SCALAR_KEYS.
• New "append" mode:
  • On update, generates a short 1-2 sentence note about the new document instead of a full-page rewrite. The LLM never sees the existing page (no existing_content, no wikilink whitelist turn — notes stay plain text, deferring cross-linking to a future reconciliation step).
  • Appends the note deterministically (no LLM call for the write itself) under a ## Notes heading, keyed by the [[summaries/{doc_name}]] source marker so re-ingesting an updated version of the same document replaces its own line instead of duplicating it.
  • New concept/entity pages are seeded the same way in append mode (short note + one-line description, no full-page LLM generation).
  • sources: frontmatter tracking, backlinks, related-item cross-refs, and index.md updates are unchanged and shared with the existing "rewrite" path.
  • openkb remove cleanup (_remove_doc_from_pages) extended to also strip a removed document's own note line — a no-op for "rewrite"-mode pages.
• New module openkb/agent/compiler_notes.py (prompts + append/upsert logic) rather than growing compiler.py further (already over the 800-line limit, grandfathered tech debt).
• Backward compatible: default mode is unchanged, existing tests pass unmodified.
• New openkb consolidate [PAGE_NAME] [--all] [--min-notes N] [--dry-run] [--yes] command (mirrors recompile's CLI shape) + new module openkb/agent/consolidator.py, to fold a page's accumulated ## Notes into curated prose — manual/opt-in only, never triggered automatically by openkb add:
  • One LLM call per page: no new source document, no concept/entity classification (the page is already fixed).
  • Contradictions between notes (or notes vs. existing prose) are described directly in the rewritten text rather than silently resolved.
  • The full wikilink whitelist is sent here (once per page per run, not per ticket), so consolidated prose can properly cross-link.
  • The ## Notes section is replaced entirely; sources:/type: are preserved, only description: may be refreshed. A later note-append call re-creates a fresh ## Notes section, so the next run only ever sees what changed since the last one — no extra tracking state needed.

Issues

• Resolves #245