Archive for February, 2011

Although there is all kinds of information out there regarding this very topic, I had a hard time getting all of this straight in my head a couple months back. There just didn’t really seem to be a definitive guide or article about this topic to help anyone. Granted I am not saying that this will be The Definitive Guide by any means, but it certainly doesn’t hurt to give it a shot.  Also it seems like a pretty alright first article to get myself out on those interwebs

My confusion came when I was recommending to my colleague and supervisor that we lower the allocated memory to several VMs. We were approaching being very overcommited on our hosts in terms of memory and there were several guests that had not even used more 19% of their allocation in the last several months.

Their first instinct is to look at what the guest is reporting in terms of memory usage. This I knew wasn’t really the best idea since the guest is generally unaware of the host on which it is running. Their first stop was the guest Task Manager.

Task Manager Performance Tab

Also in the Guest we took a look at Resource Monitor

ResourceMonitor

Taking a look at these, it can be very, Very misleading as to what the guest OS is using in terms of memory. From this picture it seems as though the guest is using 73% of its memory, which is much higher than the less than 19% I had been reporting from vCenter.

Memory Usage in vCenter

Zooming Up

What exactly is going on here?!?

To understand what is happening, we have to first understand what these terms mean

Inside the Guest OS

• Commit (KB) =”Amount of virtual memory that is reserved for use by a process.”
  In other words, this is the amount of memory that a guest process has reserved, which doesn’t necessarily mean it is actively using it.
• Working Set (KB)=”Amount of memory in the private working set plus the amount of memory the process is using that can be shared by other processes.”
  This is obviously a pretty vague description, also defined/described as “set of pages in the virtual address space of the process that are currently resident in physical memory”. This second description is MUCH better. What this is saying is that the working set is memory from the process that has been assigned a location on the memory. Again this isn’t memory necessarily being actively used/accessed, but just memory that has a residence/location. An analogy that just came to me(so be prepared)…Think of it as a hotel room that has been assigned to you. You have that whole room if you want it, but its possible you may only Really use the room for sleeping. Which means even though that location is assigned to you, you aren’t actively using it for a majority of the time you have it. You can be so greedy sometimes!!!
  Similarly some processes like to acquire and hold on to memory even though they are not actively using it.
• Shared (KB)= Don’t need a quote for this one. Essentially the memory that is being shared with another process
• Private (KB)= This is the amount of memory that is assigned only to you.

References
What do the Task Manager memory columns mean?
Working Set

Yes, you figured it out. Private+Shared=WorkingSet

Now for memory as vCenter views it.

• Granted= The amount of memory vCenter presents to the VM/Guest. This is the amount of  ”physical” memory the guest believes it has.
• Consumed=”..the amount of machine memory allocated to the virtual machine, accounting for the savings in shared memory.”

vSphere Resource Management Guide page 34

• Active= This is the tricky one…
  “ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated
  as the fraction of the invalidated pages that are re-accessed by the guest during the epoch. ESX uses a statistical sampling approach to estimate the aggregate virtual machine working set size without any guest involvement. At the beginning of each sampling period, the hypervisor intentionally invalidates several randomly selected guest physical pages and starts to monitor the guest accesses to them. At the end of the sampling period, the fraction of actively used memory can be estimated as the fraction of the invalidated pages that are re-accessed by the guest during the epoch.”

  In layman’s terms, vCenter and Esx/Esxi have a really complicated algorithm for calculating how much of the memory is actively, yes actively being used by the guest.

Understanding Resource Memory Management….

For me personally, I like the Active metric  a lot. In general though it is important to take a look at several of the metrics in order to really get a sense of what is happening with a guest.

There are many different ways to approach and look at memory usage in a guest/vm. As long as you can take a holistic view to memory everything starts to come together and make a bit more sense.

For a great article on using esxtop for measuring performance, check out Yellow-Bricks

Anyway, I hope this has been helpful and/or informative. Please let me know what you think or if I made some sort of seriously huge mistake somewhere.

Conrad