fix: hash-tag rsmq queue keys so retries work on redis cluster

[alexluong]

Fixes scheduled retries on Redis Cluster by hash-tagging the rsmq queue keys, with a Redis migration that moves pending retries to the new keys.

The bug

Each rsmq queue is two keys: a sorted set rsmq:deliverymq-retry and a hash rsmq:deliverymq-retry:Q. Schedule (SendMessage), receive (the receive Lua script) and cancel (DeleteMessage) each touch both in one transaction or script. On Redis Cluster the two keys hash to different slots, so all three fail with CROSSSLOT. Tenant keys were hash-tagged for cluster support in #616; the rsmq keys were missed.

What a Redis Cluster user sees (reproduced with make test/e2e/rediscluster, see #1138):

• Scheduled retries never run. Receive has failed on cluster since cluster support was added.
• Since go-redis v9.11 (bumped in chore(deps): refresh Go dependencies #940, first released in v1.0.7), schedule and cancel fail client-side too. The delivery handler then nacks, and the message is redelivered right away with the same attempt number: a failed delivery is retried ~6 times within milliseconds, and a successful manual retry is delivered up to 7 times.
• The alert.attempt.exhausted_retries operator event never fires.
• After 10 consecutive receive errors (about a minute) the retry worker stops and health reports unhealthy.

Standalone Redis and Dragonfly have no slots, so they're unaffected.

The fix

1. Hash-tagged keys (internal/rsmq/rsmq.go, queueKey). Every rsmq key is now <ns>{<qname>} / <ns>{<qname>}:Q, with the deployment ID inside the tag when set:

   	before	after
   no deployment ID	rsmq:deliverymq-retry, …:Q	rsmq:{deliverymq-retry}, …:Q
   DEPLOYMENT_ID=dp_1	dp_1:rsmq:deliverymq-retry, …:Q	dp_1:rsmq:{dp_1:deliverymq-retry}, …:Q

   The deployment ID goes inside the tag so deployments sharing one cluster spread across slots; it stays at the front too, so SCAN MATCH dp_1:* still finds the queue. This applies everywhere, not only in cluster mode, so there's one key layout. The QUEUES set is untouched.

2. Redis migration 004_rsmq_hash_tags (internal/migrator/migratorredis/004_rsmq_hash_tags, README there). For deliverymq-retry and deliverymq-retry-dlq, under the configured deployment ID:

   • Apply moves every message from the old keys to the tagged keys with a Lua script (1000 messages per call, each call atomic, all keys declared so it runs on Dragonfly): sorted set member and score, and the hash fields <id>, <id>:rc, <id>:fr. Messages being processed keep their score. The tagged hash gets the old queue's settings (vt, delay, maxsize, …) where it doesn't have them. The old hash keeps its settings, so instances still on the previous version see a valid, empty queue and keep working without errors.
   • Cleanup does the same move for retries those instances scheduled during the rollout, then deletes the old keys. Newest wins: if the tagged queue already holds the same retry ID, the old copy is dropped.
   • Verify reports the retries still under the old keys and passes: they are expected until cleanup. The message says to run cleanup once no instance of the previous version is running.
   • On Redis Cluster it is marked not applicable: the old keys can't be read atomically there and never held a retry that could run.

3. outpost migrate cleanup <migration> runs an applied migration's cleanup under the migration lock, with a confirmation prompt (--yes skips it). Without a name it lists the applied migrations that have something to clean up, with the number of keys each would remove. It takes a name because 001_hash_tags' cleanup deletes every untagged tenant:* key, which an operator cleaning up the retry queues shouldn't get as a side effect. Documented in migration.mdoc.

4. NewRedisSMQ takes options (WithLogger, WithHashTagPrefix, WithUntaggedKeys) in place of the variadic logger. WithUntaggedKeys gives the old layout; the tests use it to play the previous version.

5. cmd/redis-debug and its sample output in troubleshooting-redis.mdoc list both layouts.

Rollout

For standalone Redis and Dragonfly. Outpost refuses to start with pending migrations, so step 1 is required:

1. outpost migrate apply, before starting the new version. Moves scheduled retries to the new keys.
2. Roll out the new version.
3. Once no instance of the previous version is left, outpost migrate cleanup 004_rsmq_hash_tags. Moves retries that old instances scheduled during the rollout (newest wins) and deletes the old keys.

In between:

• Old instances see an empty retry queue, with no errors. Retries they schedule stay under the old keys until step 3.
• A retry that an old instance is processing during step 1, or schedules during the rollout, can be delivered twice (at-least-once). Stopping the previous version before step 1 avoids it.
• Skipping step 3 leaves those rollout-window retries unfired.
• Rolling back to the previous version after step 1 leaves the moved retries unfired (it reads only the old keys) until the new version runs again.

On Redis Cluster the migration is a no-op (marked not applicable) and there's nothing to clean up.

Behavior changes

• Redis Cluster: retries schedule, fire on their backoff and exhaust as configured.
• Standalone Redis / Dragonfly: retries, including dead-lettered ones in deliverymq-retry-dlq, live under the tagged keys after the migration.
• New CLI command outpost migrate cleanup.

Tests

• TestE2E_Compat_RedisCluster passes in full, including the 4 subtests that fail on fix: make the redis cluster e2e test target run against the test stack #1138; no CROSSSLOT in the logs. Default e2e (Dragonfly) and the short suite pass.
• TestRSMQHashTags_* (miniredis, Dragonfly, and Redis with TESTCOMPAT=1), with and without a deployment ID:
  • apply moves due, delayed and in-flight messages with their scores, receive counts and the DLQ; the tagged queue serves them;
  • after apply, an old-layout client receives nothing without error and can still schedule;
  • existing tagged queue settings are kept;
  • cleanup moves leftovers, keeps the tagged copy on the same ID, deletes the old keys; verify reports leftovers without failing;
  • more than one batch; an empty store.
• TestRSMQHashTags_NotApplicableOnCluster.
• TestRetryScheduler_MigratedLegacyQueue: a retry scheduled the way the previous version did fires through the real retry scheduler after apply, with attempt 2.
• TestCoordinator_Cleanup: runs only the named migration, refuses pending and unknown ones, nothing for not-applicable ones, fails while the lock is held.
• rsmq: tagged layout, untagged layout, and the prefix inside the tag.

🤖 Generated with Claude Code