RFE: Seed ArgoCD ssh_known_hosts for private SSH remotes (self-hosted Git)

[cjeanner]

(generated using Cursor, against a working environment with access to patterns-operator code, docs, and related repositories)

Summary

When deploying a pattern from a self-hosted / non-public SSH remote (git@host:… that is not github.com / gitlab.com / bitbucket.org / ssh.dev.azure.com), patterns-operator can clone successfully, but Argo CD fails until argocd-ssh-known-hosts-cm is manually patched.

We need first-class support to supply SSH known hosts so Argo CD can verify the host without requiring insecureIgnoreHostKey or post-install oc surgery.

Problem

Two different consumers, two different behaviors

1. patterns-operator clones the pattern with go-git and always sets HostKeyCallback = ssh.InsecureIgnoreHostKey():

   // internal/controller/checkout.go (getSshPublicKey)
   // FIXME(bandini): in the future we might want to support passing some known hosts
   publicKey.HostKeyCallback = stdssh.InsecureIgnoreHostKey()

2. Argo CD repo-server uses strict host-key checking against the global ConfigMap argocd-ssh-known-hosts-cm (ssh_known_hosts). Unknown hosts fail with:

   ssh: handshake failed: knownhosts: key is unknown

So Pattern reconcile can reach reconcile complete while the parent Application stays Unknown / ComparisonError until an operator patches the ConfigMap and restarts the repo-server.

Why existing knobs are insufficient

Approach	Limitation
tokenSecret / bootstrap_secrets with sshPrivateKey	Auth only; operator copies Secret wholesale to Argo as secret-type: repository
insecureIgnoreHostKey: "true" on the repo Secret	Works, but disables verification (documented escape hatch, not ideal for CI/lab/self-hosted Git)
Pre-seed via bootstrap_secrets	Early phase creates Secrets only, before vp-gitops exists; known hosts live in a ConfigMap owned by Argo CD — chicken-and-egg
Manual post-install merge into argocd-ssh-known-hosts-cm	Works but is outside the Pattern lifecycle and easy to miss

Argo CD’s repository Secret API has no per-repo known-hosts payload (only insecure / insecureIgnoreHostKey). Host keys are global via argocd-ssh-known-hosts-cm.

Upstream/private-repos docs and pattern-specific guidance (e.g. layered-zero-trust) already call out that the framework cannot inject known hosts into that ConfigMap today.

Proposed solution

Add Pattern-level (or gitSpec-associated) support to declare SSH known hosts and have the operator:

1. Use them for the operator’s own go-git clone (replace hard-coded InsecureIgnoreHostKey, addressing the existing FIXME).
2. After the managed Argo CD instance / vp-gitops (or global.vpArgoNamespace) exists, merge those entries into argocd-ssh-known-hosts-cm (data.ssh_known_hosts), without wiping Argo’s default public-host fingerprints.
3. Trigger a repo-server rollout (or equivalent) so the new keys are loaded, then rely on normal Application reconcile/refresh.

Suggested input shape (illustrative)

Any of these would work; pick what fits API/UX best:

• Field on gitSpec / Pattern CR (e.g. sshKnownHosts multiline OpenSSH known_hosts lines), and/or
• Convention on the existing tokenSecret (extra key such as sshKnownHosts that the operator consumes for itself + CM merge, in addition to passthrough of Argo-recognised keys), and/or
• Reference to a ConfigMap/Secret the operator reads once and applies

Requirements:

• Merge, don’t replace the ConfigMap body (preserve github.com / gitlab.com / etc.).
• Idempotent reconcile (re-apply if drifted / missing host lines).
• Works with singleArgoCD: true and the existing tokenSecret / vp-private-repo-credentials copy path.
• Document clearly vs insecureIgnoreHostKey (prefer known hosts when available).

Alternatives considered

• Document insecureIgnoreHostKey only — already possible via bootstrap/repo Secret passthrough; weak for anything beyond throwaway labs.
• Extend rhvp.cluster_utils bootstrap_secrets to create ConfigMaps — still races Argo creating/owning the CM; better owned by the operator after GitOps is up.
• ClusterGroup / values-only ConfigMap — pattern chart cannot reliably patch Argo’s CM before the first sync that needs the host key.

Acceptance criteria

• Operator can clone an SSH remote with strict host-key checking when known hosts are provided (no mandatory InsecureIgnoreHostKey).
• When known hosts are provided and tokenSecret is used, argocd-ssh-known-hosts-cm in the VP Argo namespace contains those host lines (merged) after Argo is ready.
• Parent Application can sync an SSH git@… remote without insecureIgnoreHostKey and without manual oc / ssh-keyscan steps.
• Docs updated (private repos / Pattern CR): how to supply known hosts, merge behavior, and when insecureIgnoreHostKey is still appropriate.
• Unit/e2e coverage for merge idempotency and “unknown host → known host after reconcile”.

Context / motivation

Validated Patterns CI and lab setups often serve the pattern over SSH from a local utility host (e.g. git@controller-0.utility:/var/git/…) or internal Git. Public fingerprints shipped with Argo CD do not cover those hosts. Today every such deploy needs an out-of-band seed of known hosts after vp-gitops appears.

Related

• Existing FIXME in internal/controller/checkout.go (getSshPublicKey)
• Argo CD: argocd-ssh-known-hosts-cm / ssh_known_hosts (global); repository Secret: sshPrivateKey, insecureIgnoreHostKey / insecure only
• Docs: https://validatedpatterns.io/learn/private-repos/ ; layered-zero-trust notes that bootstrap cannot inject known hosts into the CM