baseline-operator
baseline-operator
The baseline-operator provides a way to run stress-ng workloads on Kubernetes in various deployment configurations.
It is intended to create an artificial baseline load in a Kubernetes cluster in order to be able to run other Kubernetes tests in more realistic conditions (performance, cluster upgrade, etc.).
Use
Create a baseline CRD, i.e.:
apiVersion: perf.baseline.io/v1
kind: Baseline
metadata:
name: baseline-sample
spec:
cpu: 1 # Cores
mem: 1G # Size of the virtual memory. Can be defined as a % of the available memory
io: 1 # Workers continuously calling sync to commit buffer cache to disk
sock: 1 # Workers exercising socket I/O networking
custom: "--timer 1" # Other custom params
# image: quay.io/cloud-bulldozer/stressng # Custom image
# hostNetwork: true # Directly use host network
# nodeSelector: # Filter nodes with labels
# stress: "true"
# tolerations: # Use the control plane nodes
# - key: node-role.kubernetes.io/control-plane
# operator: Exists
# effect: NoSchedule
# - key: node-role.kubernetes.io/master
# operator: Exists
# effect: NoSchedule
$ kubectl apply -f config/samples/perf_v1_baseline.yaml
baseline.perf.baseline.io/baseline-sample configured
$ kubectl get baseline
NAME COMMAND AGE
baseline-sample stress-ng -t 0 --cpu 1 --vm 1 --vm-bytes 1G --io 1 --sock 1 --sock-if eth0 --timer 1 3s
Check for the DaemonSet:
$ kubectl get daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
baseline-sample 1 1 1 1 1 <none> 1m
Check for the Pods:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
baseline-sample-nnq5b 1/1 Running 0 1m
$ kubectl logs baseline-sample-nnq5b
stress-ng: info: [1] setting to a 0 second run per stressor
stress-ng: info: [1] dispatching hogs: 1 cpu, 1 vm, 1 io, 1 sock, 1 timer
Updating the CRD
Update or remove parameters from the CRD:
$ kubectl patch baseline baseline-sample --type merge -p '{"spec":{"cpu":2}}'
baseline.perf.baseline.io/baseline-sample patched
$ kubectl get po
NAME READY STATUS RESTARTS AGE
baseline-sample-nnq5b 1/1 Terminating 0 5m
baseline-sample-xvxc9 0/1 ContainerCreating 0 1s
$ kubectl logs baseline-sample-xvxc9
stress-ng: info: [1] setting to a 0 second run per stressor
stress-ng: info: [1] dispatching hogs: 2 cpu, 1 vm, 1 io, 1 sock, 1 timer
Some updates (like the above) that get translated into a new command cause the DaemonSet to be recreated.
Check for the CRD events:
$ kubectl describe baseline baseline-sample
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 5m20s Baseline Created daemonset default/baseline-sample
Normal Recreated 4s Baseline Rereated daemonset default/baseline-sample
Other fields of the CRD can be updated without the need of a DaemonSet recreation, like i.e.: image, hostNetwork, nodeSelector and tolerations:
$ kubectl patch baseline baseline-sample --type merge -p '{"spec":{"nodeSelector":{"stress":"true"}}}'
baseline.perf.baseline.io/baseline-sample patched
$ kubects get daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
baseline-sample 0 0 0 0 0 stress=true 3m9s
$ kubectl describe baseline baseline-sample
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 6m20s Baseline Created daemonset default/baseline-sample
Normal Recreated 1m10s Baseline Rereated daemonset default/baseline-sample
Normal Updated 7s Baseline Updated daemonset default/baseline-sample
Node placement
If you specify node selector(s), then the DaemonSet controller will create Pods on nodes which match that node selector(s):
apiVersion: perf.baseline.io/v1
kind: Baseline
metadata:
name: baseline-sample
spec:
cpu: 1
nodeSelector:
stress: "true"
$ kubectl get daemonset
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
baseline-sample 1 1 1 1 1 stress=true 1m
By default, DaemonSet Pods only run in worker nodes. If you want to run stress-ng loads in control plane nodes you can use tolerations:
apiVersion: perf.baseline.io/v1
kind: Baseline
metadata:
name: baseline-sample
spec:
cpu: 1
tolerations:
# these tolerations are to have the daemonset runnable on control plane nodes
# remove them if your control plane nodes should not run pods
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
Custom image
It is possible to select a custom image for stress-ng using the image property:
apiVersion: perf.baseline.io/v1
kind: Baseline
metadata:
name: baseline-sample
spec:
cpu: 1
image: quay.io/cloud-bulldozer/stressng
If not defined, defaults to quay.io/jcastillolema/stressng:0.14.01. For network workloads is important for stress-ng version to be >= 0.14.01, which allows to choose the network interface via --sock-if, sockmany-if, --udp-if and udp-flood-if. The default image was compiled through this Dockerfile:
FROM quay.io/centos/centos:stream8
WORKDIR /root
RUN yum install -y libaio-devel libattr-devel libcap-devel libgcrypt-devel libjpeg-devel keyutils-libs-devel lksctp-tools-devel libatomic zlib-devel cmake gcc
RUN curl -L https://github.com/ColinIanKing/stress-ng/archive/refs/tags/V0.14.01.tar.gz -o V0.14.01.tar.gz && tar -xzvf V0.14.01.tar.gz -C /root --strip-components=1
RUN make clean && make && mv stress-ng /usr/local/bin
Installation
$ git clone https://github.com/josecastillolema/baseline-operator
$ cd baseline-operator
$ make deploy
Comments