[Bug] Docker Labels Race Condition #83

Activity

1. amanda-wee commented on Jan 22, 2026

   

   amanda-wee

   on Jan 22, 2026

   Contributor

   More actions

   I encountered a similiar issue with Beszel's /version check last year and contributed retry logic, but it looks like that is still insufficient.

   The request by the Beszel agent is blocked based on IP (hasn't even gotten to the allowed paths)

   True, and because we don't see the warnings "cannot get valid IP address from request" or "cannot get valid IP address for client allowlist check", this means that it was blocked because it did not qualify based on the -allowfrom parameter or SP_ALLOWFROM environment variable.

   maybe before blocking a request based on IP, the socket-proxy should do a manual query of that container to see if it has the expected labels instead of relying on the Docker socket event stream to have updated the allowlist in time

   Unfortunately we only know the IP address at that point, and there isn't a Docker Engine API call to directly filter for the container corresponding to an IP address. Nonetheless, this could be possible with one Docker Engine API call to list all the containers in all the networks that the socket-proxy is in, and then iterating over the list until a match is found for the IP address, and then the labels can be found (or not). Is it a good idea to have such a potentially expensive operation when handling the requests? It wouldn't be in the happy path, and we could eliminate this performance penalty for those not using per-container allowlists by only doing this fallback if the proxy container name setting is non-empty.

   That said, given that you're allowing /version for the default allowlist, there is a straightforward fix in your case: add beszel-agent to SP_ALLOWFROM. But that wouldn't work if you didn't want a default allowlist, or if some service was trying to use a far more sensitive endpoint that failed due to this race condition, hence it would be a bad idea to include it in a default allowlist.

   ❤️React with ❤️1Reacted by Wolfgang Ellsässer
2. ElioDiNino commented on Jan 23, 2026

   

   ElioDiNino

   on Jan 23, 2026

   Author

   More actions

   I encountered a similiar issue with Beszel's /version check last year and contributed retry logic, but it looks like that is still insufficient.

   It looks like the problem here is that the retry doesn't kick in if any response is received (regardless of status code), and in this case, the socket-proxy returns a valid response with status code 403. So, the Beszel agent breaks out of the retry loop on line 594 and then proceeds to attempt to decode the Docker version despite the 403 status and fails on line 608. This is supported by the error I see from Beszel running with debug logs:

   2026-01-22T23:58:49.049097073Z 2026/01/22 23:58:49 DEBUG Containers err="invalid character 'F' looking for beginning of value"
   

   Unfortunately we only know the IP address at that point, and there isn't a Docker Engine API call to directly filter for the container corresponding to an IP address. Nonetheless, this could be possible with one Docker Engine API call to list all the containers in all the networks that the socket-proxy is in, and then iterating over the list until a match is found for the IP address, and then the labels can be found (or not).

   Ah, great point! I think querying all the containers that the socket proxy shares a network with is the next best solution. Some additional optimization could be done by first eliminating network possibilities based on their subnet and/or IP range found in the IPAM object of the Docker Engine network inspect endpoint.

   Is it a good idea to have such a potentially expensive operation when handling the requests? It wouldn't be in the happy path, and we could eliminate this performance penalty for those not using per-container allowlists by only doing this fallback if the proxy container name setting is non-empty.

   This is something I was also debating. I think/hope that with my optimization suggestion above, it would not be too much of a penalty, but I agree that it only makes sense to enable this when the proxy container name setting is non-empty.

   That said, given that you're allowing /version for the default allowlist, there is a straightforward fix in your case: add beszel-agent to SP_ALLOWFROM. But that wouldn't work if you didn't want a default allowlist, or if some service was trying to use a far more sensitive endpoint that failed due to this race condition, hence it would be a bad idea to include it in a default allowlist.

   I will do this for now to get my Docker statistics back, but yeah, I think this is not a generally advisable solution for someone if they are trying to fully use Docker labels for configuration and I hope a solution can be implemented :)

   ❤️React with ❤️1Reacted by Wolfgang Ellsässer
3. ElioDiNino

   added 2 commits that reference this issue on Jan 23, 2026

   fix(socket-proxy): Temporary workaround for race condition

   ...

   c5ee729

   chore(tinyauth): Allow socket proxy ping as GET request

   ...

   9845c67
4. ElioDiNino commented on Feb 21, 2026

   

   ElioDiNino

   on Feb 21, 2026

   Author

   More actions

   As seen in the linked PR references (henrygd/beszel#1754), there is now an upstream fix/improvement in Beszel to retry for this race condition, but I will leave this issue open since the core issue in the socket proxy remains.

   👍React with 👍2Reacted by Amanda Wee and Wolfgang Ellsässer
5. johnnyggalt commented on Jul 25, 2026

   

   johnnyggalt

   on Jul 25, 2026

   More actions

   I hit the same issue with Dozzle (ghcr.io/amir20/dozzle:latest), fwiw. Depending on the outcome of the race, Dozzle will either work perfectly or exit with:

   Attaching to dozzle
   dozzle  | {"level":"info","version":"v10.6.11","time":"2026-07-25T05:52:24Z","message":"Dozzle version v10.6.11"}
   dozzle  | {"level":"fatal","version":"v10.6.11","time":"2026-07-25T05:52:24Z","message":"Could not connect to any Docker Engine"}
   dozzle exited with code 0
   
6. wollomatic

   self-assigned this

   on Jul 25, 2026
7. wollomatic

   linked a pull request that will close this issuerefactor and harden Docker label handling #163on Jul 25, 2026
8. wollomatic commented on Jul 25, 2026

   

   wollomatic

   on Jul 25, 2026

   Owner

   More actions

   This is a bigger refactor than I initially thought. Here's the current test image:

   ghcr.io/wollomatic/socket-proxy:testing-4c6077ea7910a90ab2892054cd09e8189ca692ad

   👀React with 👀1Reacted by Amanda Wee
9. johnnyggalt commented on Jul 25, 2026

   

   johnnyggalt

   on Jul 25, 2026

   More actions

   Thanks @wollomatic. I've been playing with that test image. My chances of success seem to be massively improved (though still not perfect) by doing a docker compose down before I docker compose up my Dozzle stack (as opposed to simply docker compose up and ctrl+c to stop the stack, then bring it up again). So it seems that having to completely recreate the network and container of my Dozzle stack reduces the chances of triggering the race condition.

   That said, I honestly didn't try that before switching to the test image.
10. wollomatic commented on Jul 26, 2026

    

    wollomatic

    on Jul 26, 2026

    Owner

    More actions

    HI @johnnyggalt, thanks for the feedback.

    I decreased the cache time, so maybe the chances of success increase further. Here´s the new test image:

    ghcr.io/wollomatic/socket-proxy:testing-719d070a22c2a47bd9d84a1a4c98e33a2b1c441f
11. johnnyggalt commented on Jul 30, 2026

    

    johnnyggalt

    on Jul 30, 2026

    More actions

    I decreased the cache time, so maybe the chances of success increase further. Here´s the new test image:

    ghcr.io/wollomatic/socket-proxy:testing-719d070a22c2a47bd9d84a1a4c98e33a2b1c441f

    Just tried this one. Also get failures every now and then with docker compose down then docker compose up. Maybe 1 in 10 fails. Another observation is that when it does fail, it seems to spend longer at the Attaching to dozzle stage of output before Dozzle spits out any logs.
12. wollomatic

    closed this as completedin #163on Aug 6, 2026
13. wollomatic commented on Aug 6, 2026

    

    wollomatic

    on Aug 6, 2026

    Owner

    More actions

    Thanks everyone for investigating and testing this.

    The changes implemented in #163 significantly reduce the likelihood of this race condition. When a request arrives from an IP address that is not yet known, the proxy now performs a controlled refresh of the label-derived allowlists before denying the request. Concurrent refresh requests are coalesced, and recently refreshed data can be reused within a cooldown period.

    However, this can only minimize the race condition, not eliminate it completely. The underlying problem is the Docker container lifecycle: labels and network information can only be discovered through the Docker API once the container exists and has been attached to its networks. At that point, the application inside the container may already have started and attempted to contact the proxy.

    It would be possible to retry Docker lookups repeatedly whenever a request cannot be associated with an existing allowlist. I do not consider that desirable, particularly from a security perspective. A client that is not currently authorized should not be able to cause an effectively unbounded number of Docker socket queries simply by sending repeated requests. Apart from the additional load and complexity, this would expose a privileged backend operation to externally influenced request frequency and could provide an unnecessary denial-of-service vector.

    The implementation therefore deliberately performs only a controlled refresh and continues to fail closed if the requesting container still cannot be identified. This improves the common startup case without weakening the proxy’s security model or turning denied requests into unrestricted Docker API polling.

    Applications that tolerate an initial 403 and retry may handle the remaining edge case gracefully, but this depends on the individual client and cannot be treated as a reliable access-control guarantee.

    For fully deterministic access control, my recommendation remains to run a separate socket-proxy instance for each service, using an appropriate static configuration. Each proxy can share a private network only with its intended client. The resource overhead of an additional socket-proxy container is small, while the resulting configuration is predictable from startup and provides stronger isolation between services.

    Per-container label configuration remains available as a convenient dynamic option, and #163 makes it considerably more robust. Users should nevertheless be aware that Docker’s lifecycle prevents it from guaranteeing that the very first request during container startup will always be accepted.

    👍React with 👍1Reacted by Amanda Wee

    ❤️React with ❤️1Reacted by Elio Di Nino
14. amanda-wee commented on Aug 6, 2026

    

    amanda-wee

    on Aug 6, 2026

    Contributor

    More actions

    Thanks for working on this. I feel like if we had gone with a static config file as was originally suggested instead of the Traefik-style dynamic container labels then this wouldn't have been an issue to begin with since it'll be either the socket proxy isn't up and hence cannot accept the connection, or is it up and already has the allowlist configuration, but I admit I was intrigued by the idea of implementing Traefik-style labels and put off by the lack of a YAML parser in the Go standard library.
15. ElioDiNino commented on Aug 9, 2026

    

    ElioDiNino

    on Aug 9, 2026

    Author

    More actions

    Thanks for the hard work on this @wollomatic! The solution you landed on makes total sense and greatly improves the race conditions I previously ran into.