Hey all -
I have a Linux VM that runs a database backup task at a set time every evening. The backup task is IO intensive and the obvious problem seems to be storage latency, but I wanted to get a second opinion to see if I'm missing anything. Here are some observations:
From the guest OS level, the top command shows CPU utilization spikes with the start of the backup task. The "wait" counter in top (that measures time waiting for IO completion) goes from around 0 prior to the backup job to around 70. Load averages also go through the roof in top. This all settles down following backup job completion.
In esxtop, I see the following numbers for the VM's CPU:
- %USED = 41.27
- %WAIT = 1200.00
- %VMWAIT = 258.49 (this is head and shoulders above any other VM's value running on the same ESX host)
- %RDY = 0.14
- %IDLE = 120.39
- %CSTP = 1.01
- %SWPWT = 0.00
And the storage latency numbers for this VM go from less than 10ms normally to as high as 90ms during the backup job. After the backup job, storage latency goes back to less than 10ms.
Is there anything I've missed here or should be looking at additional? The %VMWAIT jumped out at me because that 258.49 value is just so much higher than any other VM during the backup job. Normal %VMWAIT for that VM is around 0.00.
The host does not appear to be saturated in CPU or memory at the time of the performance issue for the VM.
Thanks!