PDA

View Full Version : ESXi - Help with Memory Configuration



WarNox
22-11-2010, 01:33 PM
Hey!

Not sure if anyone here is experienced with ESX but thought I'd give it a try anyway :)

Working configuration:

Host sever: 5gb RAM

VM1: 1gb RAM
VM2: 2gb RAM

Datastore free space: 2.45gb

If I try to incrase the RAM of either so that the total for the two VMs adds to over 3gb I get a 'resource allocation error' when I try to boot up the 2nd VM. I would really like to know why this is happening since the server has over 4gb RAM?

I've set up a resource allocation for both VMs so that the swap file created is 0 bytes, as there isn't much space on the datastore, so that is not the issue. I do understand that ESX itself will use some of that RAM but surely not 1.5gb.

Hope someone can help :)

Thanks!

SolMiester
22-11-2010, 01:41 PM
Hi ESX can over commit, so its not resourse per sa. How many CPU has the host?, i ask this as if dual socket, the RAM will be allocated half to each pCPU, therefore the most you can have depending on vCPU is 2GB....
ESX will only use 256-512Mb RAM itself....

WarNox
22-11-2010, 02:20 PM
The server has 2 CPUs:

Intel Xeon 3.2GHz
Processor Sockets: 2
Processors Cores per Socket: 1
Logical Processors: 4
Hyperthreading: Enabled

---

So does this mean the most I can give to one VM is 2gb? Both the VMs are set to use 4 CPUs.

inphinity
22-11-2010, 02:30 PM
What version of ESXi?

SolMiester
22-11-2010, 02:39 PM
The server has 2 CPUs:

Intel Xeon 3.2GHz
Processor Sockets: 2
Processors Cores per Socket: 1
Logical Processors: 4
Hyperthreading: Enabled

---

So does this mean the most I can give to one VM is 2gb? Both the VMs are set to use 4 CPUs.

Ahh, there's an issue right there, its not recommended assigning 4 cores to VM, especially when you only have 4 cores. The reason being that when a VM calls on its 4 cores, the other VM vCPU calls get queued!....
Secondly, the RAM is associated across the banks 2.5 to 1 pCPU and 2.5 to the other.

After the Host COS has its RAM, you are allocating RAM from BOTH banks back to 1 VM.....
Drop back to 2 cores per VM, ESX will distribute the memory via pCPU bank, and you should be alright!


If thats not the issue, check you havent got reservations adding more than the memory you have....
http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

WarNox
22-11-2010, 03:00 PM
Cool, I will give this a try in a few hours. Will post back then.

ESXi version is 3.5 update 5, could not use 4 or 4.1 because of the whole 64 bit thing.

WarNox
22-11-2010, 07:33 PM
Ok so I shut down the 2 VM's, set both to use 2 CPUs but still can't boot with the following:

VM1: 1536mb ram
VM2: 2560mb ram

Screenshots of everything included below. I had a look at that PDF but still can't figure out what the problem here is.

Let me know if you need any more info and thanks again!

SolMiester
23-11-2010, 09:23 AM
You'll have to post screenshot for me...of error message

SolMiester
23-11-2010, 09:26 AM
Ok so I shut down the 2 VM's, set both to use 2 CPUs but still can't boot with the following:

VM1: 1536mb ram
VM2: 2560mb ram

Screenshots of everything included below. I had a look at that PDF but still can't figure out what the problem here is.

Let me know if you need any more info and thanks again!

Your reservation is also the limit, see mine, limit is more than reservation, increase the limit of memory!

WarNox
23-11-2010, 10:30 AM
Error from the Event Log on the ESX server included below. Also included is a screenshot of the configuration I got to work. 1gb on one VM and 2.25gb on the other but anymore than that and I get the same error.

I set the limit the same as the reservation or a swap file is created on the datastore and there isn't enough space left on the datastore for this. Therefore I want to assign each VM a certain amount of physical ram.

What I got now works fine but I want to know why its not letting me assign more than 3.2gb since there is 5gb in the server.

inphinity
23-11-2010, 11:02 AM
ESXi Console > Configuration tab > System Resource Allocation > Advanced View > Host > VIM > Edit settings > Memory Resources

What is the reserved memory amount?

WarNox
23-11-2010, 11:39 AM
I believe its 512mb, see attached.

Thanks again.

SolMiester
23-11-2010, 12:22 PM
Error from the Event Log on the ESX server included below. Also included is a screenshot of the configuration I got to work. 1gb on one VM and 2.25gb on the other but anymore than that and I get the same error.

I set the limit the same as the reservation or a swap file is created on the datastore and there isn't enough space left on the datastore for this. Therefore I want to assign each VM a certain amount of physical ram.

What I got now works fine but I want to know why its not letting me assign more than 3.2gb since there is 5gb in the server.

VM o?S swap file has nothing to do with Memory limit and reservation. Reservation is amount that VM much have and will not share, limit is total amount of RAM for VM and is usually larger than configured memory in the VM...

You set swap file from within the VM O/S

WarNox
23-11-2010, 01:58 PM
If I set the Memory of the VM on the hardware tab to 1.5GB and leave the Memory Limit as 'Unlimited' it creates a *.vswp file on the datastore.

If I set the memory reservation and the limit to the same value as the Memory for the VM the vswp is still created but its 0 bytes, see attached, which works fine. Although I cannot assign all the physical memory.

---

Are you wanting me to try the following? eg:

VM1: 1024mb, Reserved: 1024, Limit: 1536
VM2: 2048mb, Reserved: 2048, Limit: 2560

SolMiester
23-11-2010, 02:09 PM
If I set the Memory of the VM on the hardware tab to 1.5GB and leave the Memory Limit as 'Unlimited' it creates a *.vswp file on the datastore.

If I set the memory reservation and the limit to the same value as the Memory for the VM the vswp is still created but its 0 bytes, see attached, which works fine. Although I cannot assign all the physical memory.

---

Are you wanting me to try the following? eg:

VM1: 1024mb, Reserved: 1024, Limit: 1536
VM2: 2048mb, Reserved: 2048, Limit: 2560

Yeah, something along those lines....if I bring my limit down to the reserve i get same error!

WarNox
23-11-2010, 02:24 PM
Yeah, something along those lines....if I bring my limit down to the reserve i get same error!

Alright, I'll give it a go tonight and let you know how it goes.

That is weird though, can't see a reason for it.

chiefnz
23-11-2010, 10:24 PM
Couple of things you may also want to try.

1) increase the physical RAM.
2) assign less cores to your VM's, you can't assign all 4 CPU's to them as ESX needs this as well you may need to run 1 core for ESX and then split the other 3 between your VM's

cheers,

WarNox
23-11-2010, 11:44 PM
Couple of things you may also want to try.

1) increase the physical RAM.
2) assign less cores to your VM's, you can't assign all 4 CPU's to them as ESX needs this as well you may need to run 1 core for ESX and then split the other 3 between your VM's

cheers,

1. There is 5gb of physical ram in the server which is enough.
2. Already done that, from 4 down to 2.

---

SolMiester: That last suggestion seemed to have worked, sort of :) Thanks for all your help. Just have one more question :D

New config:

VM1: 1536mb, Reserved: 1024, Limit: 1536
VM2: 3072mb, Reserved: 2048, Limit: 3072

I actually did the opposite of the what was suggested above, it logically makes more sense to me. I set the LIMIT to be the same as the VM Memory and set the reservation to be slightly lower than that, as you can see above.

What exactly does this mean for VM1...
1. Maximum memory that VM1 can ever use is 1536mb?
2. Its guaranteed to always have 1024mb available?
3. I don't really understand what the limit shows? Would there be any point to set the Limit above the VM Memory?

Thanks again!

inphinity
24-11-2010, 08:00 AM
New config:

VM1: 1536mb, Reserved: 1024, Limit: 1536
VM2: 3072mb, Reserved: 2048, Limit: 3072

What exactly does this mean for VM1...
1. Maximum memory that VM1 can ever use is 1536mb?
2. Its guaranteed to always have 1024mb available?
3. I don't really understand what the limit shows? Would there be any point to set the Limit above the VM Memory?

Thanks again!

VM1 has 1GB of Physical RAM permanently reserved exclusively for its use, and is allowed to use other non-reserved Physical RAM to a maximum total of 1.5GB.

VM2 has 2GB of Physical RAM permanently reserved exclusively for its use, and is allowed to use other non-reserved Physical RAM to a maximum total of 3GB.

Guest OS can still use a paging file within it's VHD to provide total virtual memory to the guest OS of greater than the VM Memory Limit - e.g. VM1 can have it's 1.5GB of RAM, as well as creating a pagefile as the OS normally would.

Reserved is the amount of RAM dedicated solely for the use of that VM, Limit is the total maximum RAM that VM is allowed access to. There are ratios and rules around how much of the machines actual physical RAM can be reserved, but the Limit can be much higher as the hypervisor can use its own swapfile to simulate additional physical RAM as far as the Guest OS is concerned, as well as sharing non-reserved RAM among multiple VMs depending on load.

WarNox
24-11-2010, 08:04 AM
Ok, that makes sense but what is the difference between the VM Memory setting (VM1 1536) and the Limit (VM1 1536)?

inphinity
24-11-2010, 08:38 AM
Ok, that makes sense but what is the difference between the VM Memory setting (VM1 1536) and the Limit (VM1 1536)?

The former is the amount of memory the Guest OS is told it has. The latter is the actual limit on memory that the Host OS will ever allocate to the guest. If the limit is higher than the Memory setting, the limit will never be reached. If the limit is lower than the memory setting, the Guest will never have it's full expected amount of physical RAM made available to it, and you are likely to see memory sharing come in to play, or a Host swapfile, if the Guest tries to use the memory it beleives it has above the specified Limit.

WarNox
24-11-2010, 09:46 AM
Cool, that explains it.

Well thanks for your help everyone :) Problem solved!