[SOLVED] not able to restrict cpu/ram of a docker container through systemd slices

vincix · 05-11-2020, 01:28 PM

Hi,

I'm trying to restrict the resources of a docker container through a systemd slice, as follows:
I've created the systemd slice (/etc/systemd/system/my_limits.slice)

Code:

[Unit]
Description=my slice for docker resources
Before=slices.target

[Slice]
CPUAccounting=true
CPUQuota=20%
MemoryAccounting=true
MemoryLimit=200M

Then ran: systemctl daemon-reload

And afterwards ran the container that is supposed to be governed by the cgroup slice:

Code:

docker run --rm -it --cgroup-parent=my_limits.slice docker.io/python

Then I'm running some python code to stress the CPU. Unfortunately the CPU rises to 100%, instead of being limited to 20%.

I'm not sure what I'm doing wrong.

Using --cpus="0.2" does work as expected.

P.S. Is there any way I can confirm that the slice I've created is somehow acknowledged by systemd?
systemctl -t slice --all does not show my own slice, but I'm guessing it might be showing only the "essential"/system ones?

I'm running docker 19.03.8 in an Ubuntu 18.04 VM.

wpeckham · 05-12-2020, 07:06 AM

Because that is not the way to get that done. Check pages like this one https://www.serverlab.ca/tutorials/c...er-containers/ for clues and techniques.

dc.901 · 05-12-2020, 07:10 AM

This is good document that I have used in past: https://docs.docker.com/config/conta...e_constraints/
However, difference is that I am on RHEL7 (running docker-ce-19.03.8) not Ubuntu...

vincix · 05-12-2020, 07:50 AM

I don't think you're actually addressing the problem that I want to solve. I'm talking about limiting resources per container groups, not indivdually. That is to say, two or more containers should not use more than 20% of the cpu and 200MB of RAM.
Individual cpu and ram limitations work without any issues, I've already tried that. The point is to create a sort of profile based on systemd slices.
For instance, this answer is exactly what I'm looking for (but it doesn't work):
https://stackoverflow.com/questions/...host-resources
I'm not saying that it's correct, but that's the idea where I'm starting from.

Moreover I've got these exact instructions from this course on linux academy: https://linuxacademy.com/course/docker-security/ (Resource Restrictions Part 3 - systemd and cgroups).

vincix · 05-13-2020, 07:09 AM

I've recreated the exact environmental conditions of as that of the instructor: Centos 7 and docker installed through the official centos repositories (instead of the docker repositories).

Code:

[root@centos7 etc]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-161.git64e9980.el7_8.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Tue Apr 28 14:43:01 2020
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-161.git64e9980.el7_8.x86_64
 Go version:      go1.10.3
 Git commit:      64e9980/1.13.1
 Built:           Tue Apr 28 14:43:01 2020
 OS/Arch:         linux/amd64
 Experimental:    false

Quote:

[root@centos7 etc]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)

And this works as expected. Of course, that's hardly satisfying. What happens if I want to use a newer version of docker? It obviously works differently.

dc.901 · 05-13-2020, 07:36 AM

Your question is valid... Have you file a bug report?

vincix · 05-13-2020, 09:22 AM

No, I haven't. And I'm not sure if this is a bug. After all it works when using the docker version of the offical centos repository, so the one officially supported by Red Hat (but not when using the official docker repositories).
In the first case yum installs great many other dependencies, which might be relevant in some way:

Quote:

PyYAML.x86_64 0:3.10-11.el7
atomic-registries.x86_64 1:1.22.1-33.gitb507039.el7_8
container-storage-setup.noarch 0:0.11.0-2.git5eaf76c.el7
containers-common.x86_64 1:0.1.40-7.el7_8
device-mapper-event.x86_64 7:1.02.164-7.el7_8.1
device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.1
device-mapper-persistent-data.x86_64 0:0.8.5-2.el7
docker-client.x86_64 2:1.13.1-161.git64e9980.el7_8
docker-common.x86_64 2:1.13.1-161.git64e9980.el7_8
fuse-overlayfs.x86_64 0:0.7.2-6.el7_8
fuse3-libs.x86_64 0:3.6.1-4.el7
libaio.x86_64 0:0.3.109-13.el7
libnl.x86_64 0:1.1.4-3.el7
libyaml.x86_64 0:0.1.4-11.el7_0
lvm2.x86_64 7:2.02.186-7.el7_8.1
lvm2-libs.x86_64 7:2.02.186-7.el7_8.1
oci-register-machine.x86_64 1:0-6.git2b44233.el7
oci-systemd-hook.x86_64 1:0.2.0-1.git05e6923.el7_6
oci-umount.x86_64 2:2.5-3.el7
python-backports.x86_64 0:1.0-8.el7
python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7
python-dateutil.noarch 0:1.5-7.el7
python-dmidecode.x86_64 0:3.12.2-4.el7
python-ethtool.x86_64 0:0.8-8.el7
python-inotify.noarch 0:0.9.4-4.el7
python-ipaddress.noarch 0:1.0.16-2.el7
python-pytoml.noarch 0:0.1.14-1.git7dea353.el7
python-setuptools.noarch 0:0.9.8-7.el7
python-six.noarch 0:1.9.0-2.el7
python-syspurpose.x86_64 0:1.24.26-1.el7.centos
slirp4netns.x86_64 0:0.4.3-4.el7_8
subscription-manager.x86_64 0:1.24.26-1.el7.centos
subscription-manager-rhsm.x86_64 0:1.24.26-1.el7.centos
subscription-manager-rhsm-certificates.x86_64 0:1.24.26-1.el7.centos
usermode.x86_64 0:1.111-6.el7
yajl.x86_64 0:2.0.4-4.el7

When using the docker repository, it install much fewer things, but I think there are different technologies being used (it might not be using containerd in the first case, if that's meaningful in any way).
These are the dependencies of docker from the docker repository:

Quote:

containerd.io x86_64 1.2.13-3.1.el7 docker-ce-stable 23 M
docker-ce x86_64 3:19.03.8-3.el7 docker-ce-stable 25 M
docker-ce-cli x86_64 1:19.03.8-3.el7 docker-ce-stable 40 M
Installing for dependencies:
audit-libs-python x86_64 2.8.5-4.el7 base 76 k
checkpolicy x86_64 2.5-8.el7 base 295 k
container-selinux noarch 2:2.119.1-1.c57a6f9.el7 extras 40 k
libcgroup x86_64 0.41-21.el7 base 66 k
libsemanage-python x86_64 2.5-14.el7 base 113 k
policycoreutils-python x86_64 2.5-34.el7 base 457 k
python-IPy noarch 0.75-6.el7 base 32 k
setools-libs x86_64 3.3.8-4.el7 base 620 k

docker version (centos):
Version: 1.13.1
API version: 1.26

pan64 · 05-13-2020, 10:05 AM

actually docker was never compatible with itself and is growing rapidly. 1.13 is at about 4 years old (jan 2017), so 19.03 definitely may contain something new, which was not taken into account in that CentOS release (officially). Probably something was renamed (a feature, config, option, whatever), something was restructured [definitely] (even the versioning concept was changed since then). Probably you need to set cgroup driver https://stackoverflow.com/questions/...ver-to-systemd.

vincix · 05-13-2020, 12:55 PM

Excellent!

This is exactly what I needed. It works like a charm, although I configured it slightly different than the guy on stackoverflow.

I first copied docker.service from /lib/systemd/system/ to /etc/systemd/system, knowing that this is how you can keep the systemd script persistent after upgrading docker and I simply added the "--exec-opt native.cgroupdriver=systemd" to the ExecStart like so:

Code:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

Many thanks!

Later edit:
This is the systemd script installed from official centos repository. As you can see, the option is added automatically:

Quote:

ExecStart=/usr/bin/dockerd-current \
--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
--default-runtime=docker-runc \
--exec-opt native.cgroupdriver=systemd \
--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
--init-path=/usr/libexec/docker/docker-init-current \
--seccomp-profile=/etc/docker/seccomp.json \
$OPTIONS \
$DOCKER_STORAGE_OPTIONS \
$DOCKER_NETWORK_OPTIONS \
$ADD_REGISTRY \
$BLOCK_REGISTRY \
$INSECURE_REGISTRY \
$REGISTRIES

Very interesting