• Vendor Support
• Performance

To be frank, when possible, I don’t tell users if the machine is virtual or physical. I’m not hiding this, but I don’t volunteer non-relevant information. Part of planning an application has to do with the sizing of the production environment. You read the system requirements and prepare the infrastructure accordingly. The application requirements usually state memory, cpu and OS requirements. They don’t state that you must run the system on a Dell PowerEdge 2950 with dual Intel(R) XEON(R) E5420 @ 2.5 GHz quadcore cpus using “blue” network cables. Part of planning the application is taking the vendor’s information and adjusting it to account for your own infrastructure. Many companies run standard infrastructure such as: SNMP, VIRUS/MalWare protection and Systems Management software on all of their systems. They must account for this in terms of cpu and memory when sizing for a new application. Virtualization is “just” one more variable in the planning and sizing equation. Therefore informing the user that the machine is virtual is simply non-relevant.

I’ve been told that at Cisco all new systems are virtual and you must convince John Chambers, the current CEO, to allow you to use a physical machine.

Vendor Support
Many vendors have confusing support policies regarding virtual machines. I know of one vendor that does all of their development, testing and qa on virtual machines and then does not support running production on virtual machines.

In my mind, vendor support is an academic issue. I don’t volunteer the physical machine information to a vendor, so why would I volunteer hardware info about a virtual machine. If a vendor presses the issue, I can always reproduce the issue on a physical machine. To Date: I have never had to reproduce the issue on physical hardware.

Keep in mind that all major software vendors plan to support “Cloud Computing”. You can’t seriously support “Cloud Computing” without supporting virtualization. So if your vendor does not support virtual machines, they will begin to shortly or they will face technical obsolescence.

Performance
To be honest when it comes to performance, virtualization is your friend. That may come as a shock to you, so I’d better explain.

As I mentioned earlier, part of planning an application is sizing it for production. The sizing takes into account factors such as: memory, # of cpus, minimum recommended cpu speed, the number of concurrent users, the overhead of your common infrastructure and a “judgment factor”. The “judgment factor” attempts to account for business growth, typical usage patterns and historical vendor understatement of the hardware requirements . Over the years, I’ve seen many projects purchase way too much hardware. I’ve also seen projects that were struggling to run on the initial hardware within a year or two of the initial roll-out. This can be caused by a variety of reasons, including unexpected business growth or transaction volume due to a temporary business opportunity. Assuming that there are no application locking issues and that cpu or memory is holding the application back, then you are considering how to relocate the application to newer and more powerful hardware. When this happens physical machines become a “boat anchor” where virtual machines are a “lifeboat”.

• Why Physical machines become a “boat anchor”.
  It is often difficult and sometimes impossible to move an existing application to new hardware. In the best case scenario, you have to procure new hardware, install and configure the operating system,OS, “clone” the application to the new hardware, test the application to ensure the configuration is working correctly and then plan an outage to relocate the application permanently to the new hardware. This is at best a several week project. There can be some show stoppers. You may not be able to relocate the current version of the application, if you must use a later version of an operating system. This can often be the case with new hardware in order to obtain the necessary drivers. Now, you are potentially having to update your entire software stack as well as the application to run on a later version of the OS. You will probably have to retrain users on the new version of the application.
  What a nightmare!
  By the time you are done, you may have missed any temporary business opportunity or the business growth may have been impeded.
• Why virtual machines are a “lifeboat”.
  When you outgrow the virtual machine, you can allocate or dedicate more hardware to the virtual machine. If new hardware is required, then you simply migrate the virtual machine to the new hardware, intact. Often this migration can be performed live. Yes, it can be that simple. If you can’t perform a live migration, then an outage to copy the virtual machine from the old system to the new system will be required.
  With virtual machines, you really can throw hardware a some performance problems.

There are cases where you still need to use physical machines, but these are uncommon. In my opinion, you should always use virtual machines first and then determine that you really need dedicated physical machine.

If after all of this, your user still insists on having a physical machine, then are they also insisting that all programs be written in assembly language?

Unfortunately, obtaining software licenses for the applications that we run on these servers can and does still take many weeks.  The procurement process remains basically unchanged.

Although we’ve been taking this into account and It may be obvious to many others, you’ll want to be sure that you take the software and license procurement time into account when establishing project plans.

As an aside, we have one “well-known” vendor that keeps sending us 90-day temporary licenses for a product that is paid for and the check has cleared the bank.

• CPU utilization
• Memory Free
• Total Memory
• IO activity
• Network traffic

I found that the IO activity and Network traffic could be obtained via standard SNMP. The network traffic can be obtained by summing the results from an snmpwalk of VMWARE-RESOURCES-MIB::netHCKbRx and VMWARE-RESOURCES-MIB::netHCKbTx.  Similarly, the IO can be obtained by summing the results from an snmpwalk of VMWARE-RESOURCES-MIB::kbRead and VMWARE-RESOURCES-MIB::kbWritten.

I did not identify items in the vmware esx mib that I could use via SNMP for the CPU utilization or Free Memory. I did find that these items are available by running esxtop or resxtop in batch mode. Running the utility esxtop on the ESX service console in batch outputs a comma separated file, csv. For example: running the command

esxtop -b -n 2 -d 10 > esxtop-output.csv

will run esxtop in batch, take 2 samples with a delay of 10 seconds and write a 3 row csv file named “esxtop-output.csv”. The 1^st line of the .csv file contains the column headers. The ESX Host cpu utilization time will be in a column named:

“\\[your-esx-hostname]\Physical Cpu(_Total)\% Processor Time”

The ESX Host free memory will be in a column named:

“\\[your-esx-hostname]\Memory\Free MBytes”.

The ESX Host total memory will be in a column named:

“\\[your-esx-hostname]\Memory\Machine MBytes”.

Today, I run 2 cronjobs; 1 on my ESX service console to run esxtop in batch and save the data to a .csv file; and the 2^nd on my Cacti monitoring server. On the Cacti monitoring server, I run a “driver” script to collect data items into an RRDTool database every 5 minutes. My Cacti monitoring server uses identity based ssh/scp to get the .csv file that is produced by the cronjob that runs on the ESX service console.The RRD database is mapped as a datasource and graphed via Cacti.  I’m currently looking at using the VMWare remote cli version of esxtop, rexstop. The use of resxtop, would eliminate the need for a cronjob on the ESX service console. This will allow my utilities to run against both ESX and ESXi.

Scripts

ESX RRDTool Data update Script. Call with Arguments DBName, Hostname and community, this driver ccript calls this script to obtain the individual items

#!/bin/bash
. $HOME/setrrdtool_vars
PERF_BASE=$HOME/perfmon-esx
PATH=${PERF_BASE}/bin:${PATH}
#
# arg 1 - DBNAME
# arg 2 - monitor host
# arg 3 - community
#
if [ "${1}" = "" ]; then
   echo "must supply rrd dbname"
   call_errror="yes"
fi
if [ "${2}" = "" ]; then
   echo "must supply host"
   call_errror="yes"
fi
if [ "${3}" = "" ]; then
   echo "must supply community"
   call_errror="yes"
fi
 
if [ "${call_error}" = "yes" ]; then
   exit
fi
 
DBNAME=${1}
MHOST=${2}
COMMUNITY=${3}
 
RRD_DB=${PERF_BASE}/db/${DBNAME}
RRD_LOG=${PERF_BASE}/log/${DBNAME}.log
 
TOD=`date`
 
for tline in ${T}; do
    itemname=`echo ${tline}|awk -F":" '{print $1}'`
    itemval=`echo ${tline}|awk -F":" '{print $2}'`
 
    case "${itemname}" in
	"total_cpu")
	   CPU=${itemval}
	   ;;
      "available_memory")
         MEMAVAIL=${itemval}
         ;;
      "total_memory")
         TOTMEM=${itemval}
         ;;
      "iokb_total")
         IOKB=${itemval}
         ;;
       "netkb_total")
         NETKB=${itemval}
         ;;
	*)
 
	  ;;
 
    esac
done
 
rrdtool update ${RRD_DB} --template \
CPU_utilization:Memory_available:IO_KBytes:Net_KBytes \
N:${CPU}:${MEMAVAIL}:${IOKB}:${NETKB}
 
exit

ESX Data capture Script. Call with Arguments Hostname and community. This uses SSH/SCP, SNMP and some additional helper scripts to parse the csv data returned by esxtop

#!/bin/bash
. $HOME/setrrdtool_vars
export MIBS=ALL
# arg 1 - monitor host
# arg 2 - community
 
if [ "${1}" = "" ]; then
   echo "must supply host to monitor"
   call_errror="yes"
else
   MHOST=${1}
fi
if [ "${2}" = "" ]; then
   echo "must supply community"
   call_errror="yes"
else
   COMMUNITY=${2}
fi
 
if [ "${call_error}" = "yes" ]; then
   exit
fi
 
PERFDIR=$HOME/perfmon-esx
PATH=${PATH}:${PERFDIR}/bin:${PERFDIR}/bin/esxstats
WORKDIR=${PERFDIR}/tmp
ESXTOPCSV=mrtg-esxtop.csv
CSVfile=${WORKDIR}/${MHOST}-${ESXTOPCSV}
export PATH
 
if [ ! -d ${WORKDIR} ]; then
   mkdir ${WORKDIR}
fi
 
# use identity based ssh to copy latest csv file from ${MHOST}
# could replace with resxtop and output output to ${CSVfile}
scp -Bq ${MHOST}:${ESXTOPCSV} ${CSVfile}
 
# Begin CPU
get_cpustats ${CSVfile} ${PERFDIR}/bin/esxstats >${WORKDIR}/${MHOST}-cpustat.tmp
CPUTOT=`cat ${WORKDIR}/${MHOST}-cpustat.tmp | grep -v ${MHOST}`
 
linenum=1
for line in ${CPUTOT}; do
  if [ "${linenum}" = "1" -o "${line}" = "" -o "${line}" = " " ]; then
     skipping=1
     let "linenum = linenum + 1"
  else
     cpupct_total=`echo ${line} | awk -F'"' '{print $2}' | awk -F'"' '{print $1}'`
     let "linenum = linenum + 1"
  fi
done
#End CPU
 
# Host Available Memory
get_machine_mem ${CSVfile} ${PERFDIR}/bin/esxstats >${WORKDIR}/${MHOST}-machinemem.tmp
MACHINEMEM=`cat ${WORKDIR}/${MHOST}-machinemem.tmp | grep -v ${MHOST}`
 
linenum=1
for line in ${MACHINEMEM}; do
  if [ "${linenum}" = "1" -o "${line}" = "" -o "${line}" = " " ]; then
     skipping=1
     let "linenum = linenum + 1"
  else
     tMachineMem=`echo ${line} | awk -F'"' '{print $2}' | awk -F'"' '{print $1}'`
     # Adjust answer to bytes
     let "MachineMem = tMachineMem * 1024"
     let "linenum = linenum + 1"
  fi
done
let "TotalMem = MachineMem"
 
# Begin Used Memory
 
# Kernel Used Memory
linecnt=0
get_kern_used_mem ${CSVfile} ${PERFDIR}/bin/esxstats >${WORKDIR}/${MHOST}-kernused.tmp
KERNMEM=`cat ${WORKDIR}/${MHOST}-kernused.tmp | grep -v ${MHOST}`
 
linenum=1
for line in ${KERNMEM}; do
  if [ "${linenum}" = "1" -o "${line}" = "" -o "${line}" = " " ]; then
     skipping=1
     let "linenum = linenum + 1"
  else
     tKernMem=`echo ${line} | awk -F'"' '{print $2}' | awk -F'"' '{print $1}'`
     # Adjust answer to bytes
     let "KernMem = tKernMem * 1024"
     let "linenum = linenum + 1"
  fi
done
 
# Non-Kernel Used Memory
get_nonkern_used_mem ${CSVfile} ${PERFDIR}/bin/esxstats >${WORKDIR}/${MHOST}-nonkernused.tmp
NONKERNMEM=`cat ${WORKDIR}/${MHOST}-nonkernused.tmp | grep -v ${MHOST}`
 
linenum=1
for line in ${NONKERNMEM}; do
  if [ "${linenum}" = "1" -o "${line}" = "" -o "${line}" = " " ]; then
     skipping=1
     let "linenum = linenum + 1"
  else
     tNonKernMem=`echo ${line} | awk -F'"' '{print $2}' | awk -F'"' '{print $1}'`
     # Adjust answer to bytes
     let "NonKernMem = tNonKernMem * 1024"
     let "linenum = linenum + 1"
  fi
done
 
# End Used Memory
 
# Begin Free Memory
get_free_mem ${CSVfile} ${PERFDIR}/bin/esxstats >${WORKDIR}/${MHOST}-freemem.tmp
FREET=`cat ${WORKDIR}/${MHOST}-freemem.tmp | grep -v ${MHOST}`
 
linenum=1
for line in ${FREET}; do
  if [ "${linenum}" = "1" -o "${line}" = "" -o "${line}" = " " ]; then
     skipping=1
     let "linenum = linenum + 1"
  else
     tAvailMem=`echo ${line} | awk -F'"' '{print $2}' | awk -F'"' '{print $1}'`
     # Adjust answer to bytes
     let "AvailMem = tAvailMem * 1024"
     let "linenum = linenum + 1"
  fi
 
done
# End Free Memory
 
#IO 
 
#Get IO READ Write each VM
ALLREAD=`snmpwalk -m ALL -c ${COMMUNITY} -v 2c ${MHOST} VMWARE-RESOURCES-MIB::kbRead |awk -F":" '{print $4}'|awk -F" " '{print $1}'`
ALLWRITE=`snmpwalk -m ALL -c ${COMMUNITY} -v 2c ${MHOST} VMWARE-RESOURCES-MIB::kbWritten|awk -F":" '{print $4}'|awk -F" " '{print $1}'`
#
IORead=0
for iobr in ${ALLREAD}; do
    if [ "${iobr}" = "" -o "${iobr}" = " " ]; then
        skipping=1
    else
       let "IORead = IORead + iobr"
    fi
done
IOWritten=0
for iobw in ${ALLWRITE}; do
    if [ "${iobw}" = "" -o "${iobw}" = " " ]; then
        skipping=1
    else
       let "IOWritten = IOWitten + iobw"
    fi
done
 
let "IOTotal=IORead + IOWritten"
 
#Get NetworkBytesTotalPerSec from each interface and calculate the total
ALLNET_RX=`snmpwalk -m ALL -c ${COMMUNITY} -v 2c ${MHOST} VMWARE-RESOURCES-MIB::netHCKbRx |awk -F":" '{print $4}'|awk -F" " '{print $1}'`
ALLNET_TX=`snmpwalk -m ALL -c ${COMMUNITY} -v 2c ${MHOST} VMWARE-RESOURCES-MIB::netHCKbTx |awk -F":" '{print $4}'|awk -F" " '{print $1}'`
#
NetRX=0
for netr in ${ALLNET_RX}; do
    if [ "${netr}" = "" -o "${netr}" = " " ]; then
        skipping=1
    else
       let "NetRX = NetRX + netr"
    fi
done
#
NetTX=0
for nett in ${ALLNET_TX}; do
    if [ "${nett}" = "" -o "${nett}" = " " ]; then
        skipping=1
    else
       let "NetTX = NetTX + nett"
    fi
done
let "NetTotal=NetRX + NetTX"
 
echo total_cpu:${cpupct_total} total_memory:${TotalMem} available_memory:${AvailMem} iokb_total:${IOTotal} netkb_total:${NetTotal}
 
exit

ESX Data parse script(s). Call each one with the esxtop csv file to grab the data. Uses this awk script

each script is a seperate file
# get_cpustats
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="Physical Cpu(_Total)" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'
 
# get_free_mem
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="Free MBytes" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'
 
# get_machine_mem
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="Machine MBytes" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'
 
#get_kern_used_mem
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="Kernel MBytes" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'
 
#get_managed_mem
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="Kernel Managed MBytes" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'
 
#get_nonkern_used_mem
CSVfile=$1
NDX=`cat ${CSVfile} | awk -v str="NonKernel MBytes" -f ${2}/get_index_awk`
cat ${CSVfile} | awk -v i=$NDX -F"," '{print $i}'

awk script used by get_cpustats, etc

BEGIN {FS=","}
      {colndx=0;
       for (i=1; i<=NF; i++) {
           if (index($i,str) >0 ) {
               colndx=i;
               print colndx;
               i=NF;
               break;
               }
           }
      }

Trick #1 — Host OS power management may be causing the host to update the clock incorrectly, tell the VMware server software the real clock speed of the cpus by modifying the config.ini.

Add the following entries and restart the vmware authorization server

• host.cpukHz = “30000000″ (this example is for a 3 GHZ system)
• host.noTSC = “TRUE”
• ptsc.noTSC = “TRUE”

On windows the default location of the VMware config.ini is C:\Documents and Settings\All Users\Application Data\VMware\VMware Server

On Linux this file is in /etc/vmware/config

Trick #2 — set the VMware Tools Clock Synchronization to update more frequently by editing the .vmx file and adding

• tools.syncTime = “TRUE”
• tools.syncTime.period = “1″ (this is for a 1 second update)
• timeTracker.periodicStats=”True”
• timeTracker.statsIntercal=”10″

A word of caution, If you use Trick #2 and later you attempt to convert the VMWare Server VM to ESX , then you may have to comment out or remove these items

• tools.syncTime.period
• timeTracker.periodicStats
• timeTracker.statsIntercal

from the .vmx file. At least one version of VMWare converter did not create a VM that I could use in ESX, until I had removed those items prior to running the converter.