Skip to content

High CPU usage when backgrounding CLI with -i #6005

Open
#7249
Open
High CPU usage when backgrounding CLI with -i#6005
#7249
Labels
kind/bugstatus/0-triageversion/28.0

Description

@JennToo
JennToo
opened on Apr 11, 2025

Description

When running a docker container with the -i option combined with backgrounding the process (e.g. using & at the end of the bash command), CPU usage jumps to 100% for the docker CLI process

Reproduce

  1. Open bash, or any other shell that supports & backgrounding
  2. docker run -i --rm alpine:latest sleep 300000 &

Expected behavior

The process should continue to run in the background, but just not accept input unless foregrounded.

docker version

Client: Docker Engine - Community
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:07:16 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:07:16 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.167.4-microsoft-standard-WSL2
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 15.47GiB
 Name: jwilco7670l
 ID: 74516f92-1428-475d-b1e6-b8d039ba0a8b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  docker.adtran.com:5000
  cn-docker.adtran.com:5000
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.120.128.0/17, Size: 24

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional Info

I did some light attempts at diagnosing what is going on.

With an strace capture, I see tons of these events:

read(0, 0xc00027c000, 32768)            = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
read(0, 0xc00027c000, 32768)            = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
read(0, 0xc00027c000, 32768)            = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
read(0, 0xc00027c000, 32768)            = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
read(0, 0xc00027c000, 32768)            = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

If I attach to the process with GDB, the process is seems to live in this call stack pretty consistently:

(gdb) thread apply all bt

Thread 1 (Thread 0x7f899e7fc6c0 (LWP 27495) "docker"):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:558
#1  0x0000558e960fc470 in runtime.futexwakeup (addr=0x1, cnt=1) at /usr/local/go/src/runtime/os_linux.go:82
#2  0x0000558e960d5625 in runtime.notewakeup (n=0x1) at /usr/local/go/src/runtime/lock_futex.go:155
#3  0x0000558e9611963c in runtime.sigsend (s=<optimized out>, ~r0=<optimized out>) at /usr/local/go/src/runtime/sigqueue.go:116
#4  0x0000558e96118048 in runtime.sighandler (sig=21, info=<optimized out>, ctxt=<optimized out>, gp=0xc0004c6380) at /usr/local/go/src/runtime/signal_unix.go:722
#5  0x0000558e9611796a in runtime.sigtrampgo (sig=21, info=0xc00050dbf0, ctx=0xc00050dac0) at /usr/local/go/src/runtime/signal_unix.go:490
#6  0x0000558e96141a29 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:352
#7  <signal handler called>
#8  internal/runtime/syscall.Syscall6 () at /usr/local/go/src/internal/runtime/syscall/asm_linux_amd64.s:35
#9  0x0000558e9619a50d in syscall.RawSyscall6 (trap=0, a1=0, a2=94070891631086, a3=0, a4=824636325888, a5=0, a6=0, r1=<optimized out>, r2=<optimized out>, err=<optimized out>) at /usr/local/go/src/syscall/syscall_linux.go:64
#10 0x0000558e9619a566 in syscall.Syscall (trap=0, a1=0, a2=824636325888, a3=32768, r1=<optimized out>, r2=<optimized out>, err=<optimized out>) at /usr/local/go/src/syscall/syscall_linux.go:86
#11 0x0000558e96198558 in syscall.read (fd=<optimized out>, p=..., n=<optimized out>, err=...) at /usr/local/go/src/syscall/zsyscall_linux_amd64.go:736
#12 0x0000558e961baf6e in syscall.Read (fd=0, p=..., n=<optimized out>, err=...) at /usr/local/go/src/syscall/syscall_unix.go:183
#13 internal/poll.ignoringEINTRIO (fd=0, p=..., fn=<optimized out>) at /usr/local/go/src/internal/poll/fd_unix.go:745
#14 internal/poll.(*FD).Read (fd=0xc00014a000, p=..., ~r0=<optimized out>, ~r0=<optimized out>, ~r1=..., ~r1=...) at /usr/local/go/src/internal/poll/fd_unix.go:161
#15 0x0000558e961c67b2 in os.(*File).read (f=0xc000134098, b=..., n=<optimized out>, err=...) at /usr/local/go/src/os/file_posix.go:29
#16 os.(*File).Read (f=0xc000134098, b=..., n=<optimized out>, err=...) at /usr/local/go/src/os/file.go:124
#17 0x0000558e965d3a22 in github.com/docker/cli/cli/streams.(*In).Read (i=<optimized out>, p=..., ~r0=<optimized out>, ~r1=...) at /go/src/github.com/docker/cli/cli/streams/in.go:21
#18 0x0000558e96189531 in io.copyBuffer (dst=..., src=..., buf=..., written=<optimized out>, err=...) at /usr/local/go/src/io/io.go:429
#19 0x0000558e969b7572 in io.Copy (dst=..., src=..., err=..., written=<optimized out>) at /usr/local/go/src/io/io.go:388
#20 github.com/docker/cli/cli/command/container.(*hijackedIOStreamer).beginInputStream.func1 () at /go/src/github.com/docker/cli/cli/command/container/hijack.go:150
#21 0x0000558e9613ff01 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1700
#22 0x0000000000000000 in ?? ()

Additionally in GDB when I resume the process I can see it is receiving a lot of these signals:

(gdb) c
Continuing.

Program received signal SIGTTIN, Stopped (tty input).
internal/runtime/syscall.Syscall6 () at /usr/local/go/src/internal/runtime/syscall/asm_linux_amd64.s:36
warning: 36     /usr/local/go/src/internal/runtime/syscall/asm_linux_amd64.s: No such file or directory
(gdb)

Activity

  1. JennToo
    added
    kind/bug
    status/0-triage
    on Apr 11, 2025
  2. thaJeztah
    added
    version/28.0
    on Apr 11, 2025

  3. thaJeztah commented on Apr 11, 2025

    @thaJeztah
    thaJeztah
    on Apr 11, 2025
    Member

    Was this something that you used successfully on older versions of the CLI (i.e., is this a regression)? Also slightly curious about your use-case 😅 (e.g., if the --detach option would be an option)

  4. JennToo commented on Apr 11, 2025

    @JennToo
    JennToo
    on Apr 11, 2025
    Author

    Its such an odd corner case that I don't know if we've ever even tried it on previous versions.

    And yeah -d (or really just dropping -i) definitely could be an option in some cases.

    With the specific way we're using it, we have a CLI wrapper around docker to run some developer tools where sometimes we do need -i and sometimes we don't. We don't have a great way to detect whether our user will need the -i or not, so we just add it unconditionally. I might see if there's some way we can detect that we're backgrounded from within the wrapper before we invoke docker. Then we could silently drop the -i

    There's definitely workarounds for it, certainly not something I'd consider a show stopper.

  5. thaJeztah commented on Apr 11, 2025

    @thaJeztah
    thaJeztah
    on Apr 11, 2025
    Member

    Thanks for the extra context!

    Yes, with your reproduction steps, it's probably not too complicated to figure out if it was a regression, or just something that was always there. I was curious if you knew it was a regression, as, even if some obscure use, with <insert large number> users, it's always possible someone else is doing something similar (and may not yet have upgraded).

  6. locker95 commented on Aug 27, 2026

    @locker95
    locker95
    on Aug 27, 2026
    Contributor

    took a swing at the background-job spin — #7249

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugstatus/0-triageversion/28.0

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions