cli/command/stack: wait for the tasks to stop before removing the networks

[ssubbotin]

- What I did

docker stack rm --detach=false now waits for the tasks of the stack to stop before it removes the networks of the stack. Until now it waited only after everything, networks included, had been removed, and the wait itself returned early.

Background: the daemon accepts removing a network as soon as every task attached to it is marked for removal, that is, while the containers are still stopping. A task that terminates after one of its networks is gone is never deallocated by the manager, so the addresses it holds on its remaining networks, the ingress network in particular, leak until the manager is restarted (moby/moby#37338; a daemon-side fix is proposed in moby/moby#53563). Every docker stack rm of a stack that publishes ports leaks one ingress address per task; after a few dozen redeploys the pool is exhausted and any new service publishing a port stays in New with could not find an available IP while allocating VIP.

Waiting before the networks are removed gives a supported way to remove a stack without leaking on daemons that do not have the fix, and it matches what --detach=false reads as. The default (detached) behaviour is unchanged.

- How I did it

Two commits:

1. waitOnTasks counted the polls in which any task was terminal: the counter was never reset between polls and the inner loop stopped at the first terminal task, so with N tasks it returned after N polls in which at least one task had stopped, whatever the state of the others, polling the API in a tight loop. It now checks on every poll that no task is left in a non-terminal state, pauses 200 ms between polls (as service progress does) and stops when the context is done.
2. The wait moved between the removal of the services and the removal of the secrets, configs and networks; it is skipped when a service could not be removed, since its tasks would never stop. The reference page of stack rm describes what --detach=false does.

Unit tests cover the wait (every task must be terminal; cancellation), the order (networks removed only after the tasks stopped) and the default (no task polling when detached).

- How to verify it

Unit tests: go test ./cli/command/stack/....

End to end, on a throwaway single-node swarm (docker:28.5.2-dind) with the ingress network shrunk to 10.250.0.0/28 so that exhaustion shows up quickly: deploy a stack with one service publishing a port, remove it, twelve times, then create a probe service publishing a port.

docker stack rm variant	time per stack rm	probe after 12 iterations	dockerd log
stock CLI 28.5.2, default (detached)	0 s	stuck in New; the stack's own task no longer started at the 12th iteration	failed to allocate network IP for task …: could not find an available IP
stock CLI 28.5.2, --detach=false (waits after the networks are removed)	~14 s	stuck in New; same	same
this PR, --detach=false	~14 s	Running	no allocation failure

Daemon 28.5.2 in all three runs; the three runs were started in parallel, each in its own dind container. The script (run with docker exec <dind> sh /cli_leak.sh <mode> 12, with the binary built from this branch copied to /docker-fixed):

cli_leak.sh

#!/bin/sh
# Leak of ingress addresses through `docker stack rm`; runs INSIDE a docker:dind container.
# $1 = mode: stock (stock CLI, detached) | stock-wait (stock CLI, --detach=false)
#            | fixed (/docker-fixed built from this PR, --detach=false); $2 = iterations (12)
set -u
MODE=${1:-stock}; N=${2:-12}
IMG=alpine:3.19
log(){ echo "$(date +%T) [$MODE] $*"; }

for i in $(seq 1 60); do docker info >/dev/null 2>&1 && break; sleep 1; done
log "dockerd $(docker version --format '{{.Server.Version}}'), stock cli $(docker version --format '{{.Client.Version}}')"
docker pull -q $IMG >/dev/null 2>&1 || { log "image $IMG missing"; exit 1; }
docker swarm init --advertise-addr 127.0.0.1 >/dev/null 2>&1 || { log "swarm init failed"; exit 1; }
sleep 3
echo y | docker network rm ingress >/dev/null 2>&1; sleep 5
docker network create --driver overlay --ingress --subnet 10.250.0.0/28 --gateway 10.250.0.1 ingress >/dev/null 2>&1 || { log "ingress not created"; exit 1; }
sleep 3
log "swarm up, ingress = 10.250.0.0/28"
cat > /s.yml <<YML
services:
  web:
    image: $IMG
    command: sleep 300
    ports:
      - "18090:80"
YML

wait_running(){
  for i in $(seq 1 60); do
    st=$(docker service ps "$1" --format '{{.CurrentState}}' 2>/dev/null | head -1)
    case "$st" in Running*) return 0;; esac
    sleep 1
  done
  return 1
}
probe(){
  docker service create -d --name "probe_$1" --publish 18080:80 --restart-condition none $IMG sleep 300 >/dev/null 2>&1
  if wait_running "probe_$1"; then log "PROBE $1: Running (ingress address allocated)"; r=0
  else log "PROBE $1: STUCK in state '$(docker service ps probe_$1 --format '{{.CurrentState}}' | head -1)'"; r=1; fi
  docker service rm "probe_$1" >/dev/null 2>&1; sleep 12
  return $r
}
stack_rm(){
  case "$MODE" in
    stock)      docker stack rm s >/dev/null 2>&1 ;;
    stock-wait) docker stack rm --detach=false s >/dev/null 2>&1 ;;
    fixed)      /docker-fixed stack rm --detach=false s >/dev/null 2>&1 ;;
  esac
}
iter(){
  docker stack deploy -c /s.yml s >/dev/null 2>&1
  wait_running s_web || log "  iteration $1: task did not start"
  t0=$(date +%s)
  stack_rm
  log "  iteration $1: stack rm took $(( $(date +%s) - t0 )) s"
  for i in $(seq 1 60); do [ -z "$(docker ps -aq --filter label=com.docker.stack.namespace=s)" ] && break; sleep 1; done
  sleep 3
}

probe before
log "--- $N iterations of stack deploy + stack rm ($MODE) ---"
for n in $(seq 1 $N); do iter $n; done
probe after
sleep 30
probe after_delayed
log "done"

- Human readable description for the release notes

`docker stack rm --detach=false` now waits for the tasks of the stack to stop before removing its networks, so that the swarm manager releases the addresses of the tasks, and it no longer returns before every task has stopped.

- A picture of a cute animal (not mandatory but encouraged)

🦭