How to limit the RAM usage of Exchange 2010

Actually, in Exchange 2010, it uses 32-KB page while it is 8-KB page in Exchange 2007.
At this point, If you want to limit it to 8GB (i.e.), you should set the msExchESEParamCacheSizeMax value to 262144. (1048576 pages equal to 32GB in Exchange 2010)

Remeber: you have to set also the msExchESEParamCacheSizeMin attribute.


Reference:

Understanding the Mailbox Database Cache
http://technet.microsoft.com/en-us/library/ee832793.aspx