6

I've been looking all over StackOverflow for this, but can't find a satisfactory answer.

When running kubectl top nodes <node name> I get a memory utilisation of approx. 69% (Kubernetes showing roughly 21Gi of 32Gi being used). But if I go into the system itself and run the free command, as well as the top command, I see a total of 6GB of used memory (i.e. 20% - this is the information under the used column in the output of free) - way less than 69% of the total system memory of 32GB.

Even accounting for the differences in Gi and GB, there's still more than 40% difference unaccounted for. I know that Kubernetes uses the stats reported by /sys/fs/cgroup/memory/memory.usage_in_bytes to report on memory utilisation, but why would this be different than the utilisation reported by other processes on the system (especially sometimes higher)? Which one should I take as the source of truth?

Improve this question
2
  • This link answer your question? Commented May 18, 2020 at 13:57
  • Hi @KoopaKiller, I came across that doc before, but it still left the question unanswered as to why the measure in /sys/fs/cgroup/memory/memory.usage_in_bytes is different to the stats reported by free. I think I found the answer to my question here: serverfault.com/questions/902009/… Commented May 19, 2020 at 14:49

1 Answer 1

Reset to default
7

Found answer to my question here: https://serverfault.com/questions/902009/the-memory-usage-reported-in-cgroup-differs-from-the-free-command. In summary, it seems that Kubernetes uses the cgroup memory utilisation, which is reported in /sys/fs/cgroup/memory/memory.usage_in_bytes. The cgroup memory utilisation calculates not only the currently used memory in RAM, but also the "cached" memory (i.e. any memory no longer required by apps that are subsequently free to be reclaimed by the OS, but hasn't been reclaimed yet). The Linux system commands see "cached" memory as "free" but Kubernetes does not (not sure why).

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank for the answer. The discrepancy still exists in 2024. Hoping that Kubernetes will one day fix their memory usage metric

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.