Skip to content

Missing cgroup awareness #6059

Description

@itamarst

There are a variety of ways cgroups can restrict CPU availability, and OpenBLAS does not know about all of them.

Without knowing about these restrictions, the result can be code running in a container (e.g. Kubernetes pod) thinking it has far more cores than it actually does, since it will choose the physical CPU count instead of the restricted access the container has been given. This can cause oversaturation and slowness.

To try this out, you can run a program that checks openblas_get_num_threads() under a container created with systemd-run:

systemd-run --user --scope --property CPUQuota="200%" -- yourprogram --yourarg=yourvalue

The program should only have 2 cores, but will in fact report the number of physical cores on the CPU instead.

You can see logic for cgroups parsing (it's on the filesystem) in https://github.com/seanmonstar/num_cpus/blob/master/src/linux.rs for example.

(Not sure if cgroups v1 is worth supporting at this point, since it's old.)

(I validated this with the threadpoolctl Python library.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions