Although there is all kinds of information out there regarding this very topic, I had a hard time getting all of this straight in my head a couple months back. There just didn’t really seem to be a definitive guide or article about this topic to help anyone. Granted I am not saying that this will be The Definitive Guide by any means, but it certainly doesn’t hurt to give it a shot.  Also it seems like a pretty alright first article to get myself out on those interwebs 🙂

My confusion came when I was recommending to my colleague and supervisor that we lower the allocated memory to several VMs. We were approaching being very overcommited on our hosts in terms of memory and there were several guests that had not even used more 19% of their allocation in the last several months.

Their first instinct is to look at what the guest is reporting in terms of memory usage. This I knew wasn’t really the best idea since the guest is generally unaware of the host on which it is running. Their first stop was the guest Task Manager.

Task Manager Performance Tab

Also in the Guest we took a look at Resource Monitor

ResourceMonitor

Taking a look at these, it can be very, Very misleading as to what the guest OS is using in terms of memory. From this picture it seems as though the guest is using 73% of its memory, which is much higher than the less than 19% I had been reporting from vCenter.

Memory Usage in vCenter

Zooming Up

What exactly is going on here?!?

To understand what is happening, we have to first understand what these terms mean

Inside the Guest OS

  • Commit (KB) =”Amount of virtual memory that is reserved for use by a process.”
    In other words, this is the amount of memory that a guest process has reserved, which doesn’t necessarily mean it is actively using it.
  • Working Set (KB)=”Amount of memory in the private working set plus the amount of memory the process is using that can be shared by other processes.”
    This is obviously a pretty vague description, also defined/described as “set of pages in the virtual address space of the process that are currently resident in physical memory”. This second description is MUCH better. What this is saying is that the working set is memory from the process that has been assigned a location on the memory. Again this isn’t memory necessarily being actively used/accessed, but just memory that has a residence/location. An analogy that just came to me(so be prepared)…Think of it as a hotel room that has been assigned to you. You have that whole room if you want it, but its possible you may only Really use the room for sleeping. Which means even though that location is assigned to you, you aren’t actively using it for a majority of the time you have it. You can be so greedy sometimes!!! 🙂
    Similarly some processes like to acquire and hold on to memory even though they are not actively using it.
  • Shared (KB)= Don’t need a quote for this one. Essentially the memory that is being shared with another process
  • Private (KB)= This is the amount of memory that is assigned only to you.

References
What do the Task Manager memory columns mean?
Working Set

Yes, you figured it out. Private+Shared=WorkingSet

Now for memory as vCenter views it.

  • Granted= The amount of memory vCenter presents to the VM/Guest. This is the amount of  “physical” memory the guest believes it has.
  • Consumed=”..the amount of machine memory allocated to the virtual machine, accounting for the savings in shared memory.”

vSphere Resource Management Guide page 34

  • Active= This is the tricky one…
    “ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated
    as the fraction of the invalidated pages that are re-accessed by the guest during the epoch. ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated as the fraction of the invalidated pages that are re-accessed by the guest during the epoch.”

    In layman’s terms, vCenter and Esx/Esxi have a really complicated algorithm for calculating how much of the memory is actively, yes actively being used by the guest.

Understanding Resource Memory Management….

For me personally, I like the Active metric  a lot. In general though it is important to take a look at several of the metrics in order to really get a sense of what is happening with a guest.

There are many different ways to approach and look at memory usage in a guest/vm. As long as you can take a holistic view to memory everything starts to come together and make a bit more sense.

For a great article on using esxtop for measuring performance, check out Yellow-Bricks

Anyway, I hope this has been helpful and/or informative. Please let me know what you think or if I made some sort of seriously huge mistake somewhere.

Conrad


9 Comments

Caz · June 13, 2011 at 4:58 pm

Well, that’s a great article, and I also really like and use the “Active” metric a lot.

the main issue that my organization has is that how do you explain this to your Cloud Costumers..? I mean, you can arge with them all the day and tell them that our “Active” Metric is the correct, and they we’ll say that thay are seein 90% use of RAM.

What do you suggest for this sittuation?

    C-Rad · June 14, 2011 at 10:54 am

    That is a great question. Generally with those types of customers “seeing is believing” and it may be hard for them to just take your word for it.

    I would suggest a couple of things.

    First if it shows them with a 90% memory usage, I would use the memory performance tester. That is linked on my Resource Loading Page. Using that you can show them that if you active use another 10% of the memory Task Manager still shows there being some left over, meaning task manager was not accurate in the first place.

    Secondly I would show them perfmon. If you run perfmon inside of the VM it has vm specific counters which will give you an accurate reading of the resources being used as it uses VMware Tools.

    Hope this helps!

      Mike · June 1, 2015 at 8:53 am

      Very valuable information. I have a few questions. Given all the information you provided what did you do? What was your final recommendation. Did you lower your VM memory allocation? Did you have the discussion about task Mgr vs VCenter vs Perfmon? What data will you use going forward?

        C-Rad · June 4, 2015 at 1:12 pm

        As with most of the answers in the IT world… “It Depends”

        Generally the first thing I look at is the Active metric in vCenter, then I check out Windows to see what it thinks it is using. Some applications like to hold on to memory. Of those applications, some need to hold on to that memory and some don’t. Just holding on to memory isn’t going to make the Active metric in vCenter increase, because those applications aren’t changing/affecting.

        I am not sure if that helps at all, but hopefully it atleast gives more to consider :\

gb · November 23, 2011 at 11:32 am

For your cloud customers the metric to use for chargeback is “consumed.” Reference http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/memory_counters.html

Ravi · March 8, 2012 at 1:19 pm

The counters for VM memory in the perfmon show the correct picture. Atleast that is what vmware suggested on a call.

    C-Rad · March 8, 2012 at 1:31 pm

    Correct, you can use perfmon with the VM metrics to see a more accurate picture. This utilizes vmware tools and the actual host for its metrics.
    I had not really went into perfmon, but it is a another viable alternative. However it pulls its metrics from the vcenter, so you can look in vcenter for the same stuff.

Tweets that mention VM Memory Usage Stats: vCenter vs. Windows | vNoob -- Topsy.com · February 16, 2011 at 11:54 pm

[…] This post was mentioned on Twitter by jasonboche, Rob Quast. Rob Quast said: RT @jasonboche: VM Memory Usage Stats: vCenter vs. Windows by vnoob http://bit.ly/hwXGkn […]

vNoob » Obligatory Looking Back on 2011 Post · January 13, 2012 at 1:04 pm

[…] been one of my first ones regarding understanding memory usage and statistics inside of the VM. Found Here. That either tells me that I am going under appreciated, or I am not achieving the same quality […]

Leave a Reply