feat(links): publish account link events to kafka via outbox

[TheMeinerLP]

Summary

Otis publishes account link changes as events to Kafka through a transactional outbox. Stacked on #184 (feat/account-links).

• Outbox: every link change writes an outbox_event row in the same transaction as the change, through the existing LinkTransactions helper. The covered changes are redeem, setting an unverified link, and unlink.
• Format: CloudEvents 1.0 structured JSON.
  • Types: net.onelitefeather.otis.account.linked / .unlinked.
  • source: /otis. id: the outbox row id. subject: the player uuid.
  • data carries playerUuid, provider, externalId (null for unverified links), verified and occurredAt. It never contains codes or tokens.
• Topic otis.account-links: Kafka key = player uuid, so events per player stay in order.
  • Otis creates the topic at startup through micronaut-kafka NewTopic beans: 3 partitions, replication factor 3.
  • The cluster has auto.create.topics.enable=false and no Strimzi topic operator.
• Relay: @Scheduled every 2 s.
  • Each row is claimed with a 30 s lease, which is safe with 2 replicas. SKIP LOCKED is not available on H2, and MariaDB rejects the self-referencing UPDATE … IN (SELECT …) (error 1093).
  • Rows are sent strictly by created_at, and a run stops at the first failure. Delivery is at-least-once, so consumers deduplicate by event id.
  • Published rows are deleted after 7 days.
• Toggle: KAFKA_ENABLED (default false). When it is off, no Kafka beans start, no outbox rows are written, and the Kafka health indicator stays off.
• Observability:
  • Producer spans and traceparent headers come from micronaut-tracing-opentelemetry-kafka.
  • The relay has its own span.
  • The Micrometer gauge otis.outbox.pending is on /prometheus and answers "are link events stuck?".
  • The relay logs one WARN per failure burst.

Deployment

1. Merge feat(links): link external accounts to players with one-time codes #184 and this PR, then release and deploy Otis (KAFKA_ENABLED stays false, so nothing changes).
2. Then merge feat(otis): enable account link events on kafka Kubernetes-FLUX#360 (KAFKA_ENABLED=true, bootstrap feather-kafka-kafka-bootstrap.kafka.svc:9092; the service exists).

Compatibility and migration (no data loss)

• Flyway V4 outbox_event is additive only.
• H2 test: player and link rows are unchanged across V4.
• V1–V4 applied on a throwaway MariaDB 11 (main session) and on PostgreSQL 17, the latter end to end with a 3-broker Kafka 3.9.1 with auto-create off: the topic was created, 3 events were published with traceparent, and all rows were marked published.
• java-client and velocity-plugin are unchanged.

Known limitations

• Deleting a player through the legacy POST /otis/delete removes links by DB cascade without unlinked events. This is documented in the spec and design.
• If Kafka is unreachable at startup, topic creation fails silently and is retried only on the next start. The relay keeps the events in the outbox until the broker is reachable.

Test plan

• ./gradlew clean build: 355 tests, 0 failures, 1 skipped (pre-existing disabled OtisTest)
• Relay driven by run() with a manual clock and a fake publisher (no sleeps); two-replica claim test on H2; rollback test for the outbox; gauge/span/WARN tests
• Real broker path verified against Kafka 3.9.1 and PostgreSQL 17 containers

https://claude.ai/code/session_012BmpdMdQ5wwagagQvfcq8S