Skip to content

schemas: single-source generation path (dependency add / cache refresh) copies without clearing, diverging from merged generation #418

Description

@stevendborrelli

Problem

The schema manager has two generation paths:

  1. Merged path (added in Project typescript support #170): GenerateFromMultipleSources — freshness check, clearLanguageDirs, generate per source type, merge, copy, record lock.
  2. Single-source path (pre-existing, internal/schemas/manager/manager.go Generate/Add): generates for one source and copies the output without clearing the language directories first (CopyFilesBetweenFs only — main has no RemoveAll anywhere in this file).

Path 2 is what cmd/crossplane/dependency/add.go (AddDependency → addPackage → schemas.Add) and cmd/crossplane/dependency/cache.go (refresh → RefreshAll) use.

Consequences (all pre-date TypeScript; TS widens exposure)

  • Copies never remove. Files for a group that no longer exists in any current source persist across any number of dependency add / dependency cache refresh invocations. Only a subsequent project-wide generation (e.g. project build) clears them — and a workflow that only ever runs dependency commands never clears them.
  • Two traversal implementations. addPackage/addAll duplicate the semantics of the newer collectPackageSource/CollectSources (resolution, claim-based dedupe, version IDs, transitive metadata walks). Fixes and auth changes must be made twice; the visited reset landed in Project typescript support #170's collection code would have to be mirrored here by hand.
  • Same root cause as the k8s-mirror collisions: when a non-k8s source generates over an k8s: OpenAPI dependency in the same directory (copy-over, version may even match), API-server-serialized mirror cases are replaced. Those per-case collisions were patched individually with custom validators on the Go-generated side; they share this root: copy-over nothing.
  • TypeScript amplification (context for Project typescript support #170): on a project with typescript in spec.schemas.languages, dependency add runs the single-source path over a tree the merged path owns. updateVersion sets FromMergedPass=false, so the next build correctly regenerates everything (flag behavior is sound) — but until then the on-disk TypeScript tree holds whichever single source last copied in, and function build reads that tree directly. Stale until next full build, wrong indefinitely if no build ever runs.

Suggested fix

Route dependency add and dependency cache refresh through the merged machinery: CollectSources + GenerateFromMultipleSources (or give the single-source path the same clear + cycle semantics and delete the duplication). Either way, remove the copy-without-clear path and the AddAll/RefreshAll remnants once callers are converted.

Depends on

#170 — CollectSources lands there. Filed now so the follow-up is unblocked immediately after that merge; #170 should reference this issue as its known limitation / scope split.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions