OWN7A@reply.github.com> To: kube-burner/kube-burner Cc: Subscribed Message-ID: Subject: [kube-burner/kube-burner] [BUG] Very large kube-burner memory usage for node-density-cni 200 ppn with 750 nodes (Issue #1080) Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="--==_mimepart_6970a1931048f_be10c83391ed"; charset=UTF-8 Content-Transfer-Encoding: 7bit Precedence: list X-GitHub-Sender: dceara X-GitHub-Recipient: pfeifferj X-GitHub-Reason: subscribed List-ID: kube-burner/kube-burner List-Archive: https://github.com/kube-burner/kube-burner List-Post: List-Unsubscribe: , List-Unsubscribe-Post: List-Unsubscribe=One-Click X-GitHub-Notify-Platform: newsies X-GitHub-Labels: X-GitHub-Assignees: X-GitHub-IssueState: open X-Auto-Response-Suppress: All destinations: github@josie.lol X-GitHub-Recipient-Address: github@josie.lol X-DKIM: signer='github.com' status='pass' reason='' DKIMCheck: Server passes DKIM test, 0 Spam score X-Spam-Score: 0.9 (/) X-Spam-Report: Spam detection software, running on the system "witcher.mxrouting.net", has performed the tests listed below against this email. Information: https://mxroutedocs.com/directadmin/spamfilters/ --- Content analysis details: (0.9 points) --- pts rule name description ---- ---------------------- ----------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: githubusercontent.com] 3.0 BODY_CONTAINS_AMAZONAWS BODY: Body contains ".amazonaws.com" 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [192.30.252.210 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [192.30.252.210 listed in list.dnswl.org] 1.5 HTML_MESSAGE BODY: HTML included in message -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.0 T_KAM_HTML_FONT_INVALID Test for Invalidly Named or Formatted Colors in HTML 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager -0.1 DKIMWL_WL_HIGH DKIMwl.org - High trust sender SpamTally: Final spam score: 9 ----==_mimepart_6970a1931048f_be10c83391ed Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit dceara created an issue (kube-burner/kube-burner#1080) # Bug Description ## **Output of `kube-burner` version** ``` KUBE_BURNER_VERSION=1.9.2 ``` ## **Describe the bug** When running a large scale node-density-cni test (750 node OCP cluster) in prow, the CI container running kube-burner was killed because it went over its memory allocation limit: ``` ERRO[2026-01-21T06:25:09Z] Some steps failed: ERRO[2026-01-21T06:25:09Z] * could not run steps: step ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni failed: ["ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" test steps failed: "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" pod "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni" failed: could not watch pod: the pod ci-op-wn7mhtrb/ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni failed after 2h42m1s (failed containers: test): Evicted The node was low on resource: memory. Threshold quantity: 100Mi, available: 95044Ki. Container test was using 25712144Ki, request is 18281250Ki, has larger consumption of memory. Container test exited with code 137, reason Error ``` https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/73587/rehearse-73587-pull-ci-openshift-ovn-kubernetes-master-ovncore-perfscale-aws-ovn-xlarge-node-density-cni/2013742347891773440#1:build-log.txt%3A411 ## **To Reproduce** Steps to reproduce the behavior: Set up an OCP 750 node cluster, run node-density-cni, e.g. ``` /tmp/kube-burner-ocp node-density-cni --log-level=info --qps=20 --burst=20 --gc=false --uuid e51ad3fe-0c39-45ca-911c-53c4500d428c --churn-duration=20m --service-latency --gc=false --gc-metrics=false --pods-per-node=200 --namespaced-iterations=true --iterations-per-namespace=1000 --profile-type=both --pprof=false --es-server=https://XXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXX@search-xxxxxx-perf-scale-test-elk-hcm7wtsqpxy7xogbu72bor4uve.us-east-1.es.amazonaws.com/ --es-index=ripsaw-kube-burner ``` ## **Expected behavior** kube-burner shouldn't use an excessive amount of memory (it was killed at ~25Gi). ## **Screenshots or output** The CI infra killed the pod running kube-burner at this point: ``` time="2026-01-21 01:01:34" level=info msg="Triggering job: node-density-cni" file="job.go:115" time="2026-01-21 01:01:34" level=info msg="Cleaning up previous runs" file="job.go:118" time="2026-01-21 01:01:34" level=info msg="Churning enabled" file="job.go:122" time="2026-01-21 01:01:34" level=info msg="Churn cycles: 0" file="job.go:123" time="2026-01-21 01:01:34" level=info msg="Churn duration: 20m0s" file="job.go:124" time="2026-01-21 01:01:34" level=info msg="Churn percent: 10" file="job.go:125" time="2026-01-21 01:01:34" level=info msg="Churn delay: 2m0s" file="job.go:126" time="2026-01-21 01:01:34" level=info msg="Churn type: objects" file="job.go:127" time="2026-01-21 01:18:43" level=info msg="6862/68622 iterations completed" file="create.go:134" time="2026-01-21 01:35:52" level=info msg="13724/68622 iterations completed" file="create.go:134" time="2026-01-21 01:53:01" level=info msg="20586/68622 iterations completed" file="create.go:134" time="2026-01-21 02:10:11" level=info msg="27448/68622 iterations completed" file="create.go:134" time="2026-01-21 02:27:20" level=info msg="34310/68622 iterations completed" file="create.go:134" time="2026-01-21 02:44:29" level=info msg="41172/68622 iterations completed" file="create.go:134" time="2026-01-21 03:01:39" level=info msg="48034/68622 iterations completed" file="create.go:134" time="2026-01-21 03:18:48" level=info msg="54896/68622 iterations completed" file="create.go:134" time="2026-01-21 03:35:57" level=info msg="61758/68622 iterations completed" file="create.go:134" received signal 15, forwarding ``` with ``` * could not run steps: step ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni failed: ["ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" test steps failed: "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" pod "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni" failed: could not watch pod: the pod ci-op-wn7mhtrb/ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni failed after 2h42m1s (failed containers: test): Evicted The node was low on resource: memory. Threshold quantity: 100Mi, available: 95044Ki. Container test was using 25712144Ki, request is 18281250Ki, has larger consumption of memory. Container test exited with code 137, reason Error ``` -- Reply to this email directly or view it on GitHub: https://github.com/kube-burner/kube-burner/issues/1080 You are receiving this because you are subscribed to this thread. Message ID: ----==_mimepart_6970a1931048f_be10c83391ed Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
dceara created an issue (kube-burner/kube-burner#1080)

Bug Description

Output of kube-burner version

KUBE_BURNER_VERSION=1.9.2

Describe the bug

When running a large scale node-density-cni test (750 node OCP cluster) in prow, the CI container running kube-burner was killed because it went over its memory allocation limit:

 ERRO[2026-01-21T06:25:09Z] Some steps failed:                           
ERRO[2026-01-21T06:25:09Z] 
  * could not run steps: step ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni failed: ["ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" test steps failed: "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" pod "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni" failed: could not watch pod: the pod ci-op-wn7mhtrb/ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni failed after 2h42m1s (failed containers: test): Evicted The node was low on resource: memory. Threshold quantity: 100Mi, available: 95044Ki. Container test was using 25712144Ki, request is 18281250Ki, has larger consumption of memory. 
Container test exited with code 137, reason Error

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/73587/rehearse-73587-pull-ci-openshift-ovn-kubernetes-master-ovncore-perfscale-aws-ovn-xlarge-node-density-cni/2013742347891773440#1:build-log.txt%3A411

To Reproduce

Steps to reproduce the behavior:

Set up an OCP 750 node cluster, run node-density-cni, e.g.

/tmp/kube-burner-ocp node-density-cni --log-level=info --qps=20 --burst=20 --gc=false --uuid e51ad3fe-0c39-45ca-911c-53c4500d428c --churn-duration=20m --service-latency --gc=false --gc-metrics=false --pods-per-node=200 --namespaced-iterations=true --iterations-per-namespace=1000 --profile-type=both --pprof=false --es-server=https://XXXXXX:XXXXXXXXXXXXXXXXXXXXXXXXX@search-xxxxxx-perf-scale-test-elk-hcm7wtsqpxy7xogbu72bor4uve.us-east-1.es.amazonaws.com/ --es-index=ripsaw-kube-burner

Expected behavior

kube-burner shouldn't use an excessive amount of memory (it was killed at ~25Gi).

Screenshots or output

The CI infra killed the pod running kube-burner at this point:

 time="2026-01-21 01:01:34" level=info msg="Triggering job: node-density-cni" file="job.go:115"
time="2026-01-21 01:01:34" level=info msg="Cleaning up previous runs" file="job.go:118"
time="2026-01-21 01:01:34" level=info msg="Churning enabled" file="job.go:122"
time="2026-01-21 01:01:34" level=info msg="Churn cycles: 0" file="job.go:123"
time="2026-01-21 01:01:34" level=info msg="Churn duration: 20m0s" file="job.go:124"
time="2026-01-21 01:01:34" level=info msg="Churn percent: 10" file="job.go:125"
time="2026-01-21 01:01:34" level=info msg="Churn delay: 2m0s" file="job.go:126"
time="2026-01-21 01:01:34" level=info msg="Churn type: objects" file="job.go:127"
time="2026-01-21 01:18:43" level=info msg="6862/68622 iterations completed" file="create.go:134"
time="2026-01-21 01:35:52" level=info msg="13724/68622 iterations completed" file="create.go:134"
time="2026-01-21 01:53:01" level=info msg="20586/68622 iterations completed" file="create.go:134"
time="2026-01-21 02:10:11" level=info msg="27448/68622 iterations completed" file="create.go:134"
time="2026-01-21 02:27:20" level=info msg="34310/68622 iterations completed" file="create.go:134"
time="2026-01-21 02:44:29" level=info msg="41172/68622 iterations completed" file="create.go:134"
time="2026-01-21 03:01:39" level=info msg="48034/68622 iterations completed" file="create.go:134"
time="2026-01-21 03:18:48" level=info msg="54896/68622 iterations completed" file="create.go:134"
time="2026-01-21 03:35:57" level=info msg="61758/68622 iterations completed" file="create.go:134"
received signal 15, forwarding

with

  * could not run steps: step ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni failed: ["ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" test steps failed: "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni" pod "ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni" failed: could not watch pod: the pod ci-op-wn7mhtrb/ovncore-XXXXXXXXX-aws-ovn-xlarge-node-density-cni-openshift-qe-node-density-cni failed after 2h42m1s (failed containers: test): Evicted The node was low on resource: memory. Threshold quantity: 100Mi, available: 95044Ki. Container test was using 25712144Ki, request is 18281250Ki, has larger consumption of memory. 
Container test exited with code 137, reason Error

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <kube-burner/kube-burner/issues/1080@github.com>

----==_mimepart_6970a1931048f_be10c83391ed-- From - Wed Jan 21 10:10:01 2026 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 Return-Path: Delivered-To: hi@josie.lol Received: from witcher.mxrouting.net by witcher.mxrouting.net with LMTP id yNEpFPWlcGkQBDwAYBR5ng (envelope-from ) for ; Wed, 21 Jan 2026 10:09:57 +0000 Return-path: Envelope-to: github@josie.lol Delivery-date: Wed, 21 Jan 2026 10:09:57 +0000 Received: from out-28.smtp.github.com ([192.30.252.211]) by witcher.mxrouting.net with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (E