How cool is vscsiStats? Part-I

It has been few days since I published a white paper on the performance characterization of SQL server-based vCenter database. Few people have asked me questions about the vscsiStats graphs in the appendix of the paper. Instead of answering the questions individually I decided to blog here for the benefit of all the readers.

As mentioned in the paper, I observed this rather unusual (yes, I say unusual because I didn’t expect performance of virtual I/O stack to be better than that of native) during some of the experiments. Using the vCenter application to reproduce this behavior was rather complex and involved too many variables. Hence I decided to use a simple I/O benchmark, which most of you are familiar – IOmeter ( But to reproduce the issue, I needed to use the exact same I/O load as that produced by the stored procedures of the vCenter database. To create a custom workload profile in IOmeter I was required to configure outstanding IOs, I/O request size, read percentage and percentage of randomness (at least). Question was – how to get these parameters from the workload?

vscsiStats provided the answer. Scott Drummonds (during his VMware days) wrote a great blog on vscsiStats. I highly encourage the readers to go through the article and understand the basics of vscsiStats (if you are not already familiar with the tool).  This will help you appreciate the content of this multi-part blog series. Instead of dwelling on the details of vscsiStats, I will illustrate the usefulness of vscsiStats here.

First, a quick description of a sample histogram output from vscsiStats. All the histograms have similar format and should be straightforward to understand.

If you are curious about this tool and want to learn more, check out these technical literatures:

  1. Storage Workload Characterization and Consolidation in Virtualized Enviornments” – Ajay Gulati, Chethan Kumar, and Irfan Ahmad presented at VPACT 09 (Yes, I was one of the authors)
  2. vscsiStats: Fast and Easy Disk Workload Characterization on VMware ESX Server” – Presentation by Irfan Ahmad at VMworld 2007 (excellent presentation by one of the creators of this tool)

Up next: The histograms I collected …

Running Virtual Center Database in a Virtual Machine

I just completed an interesting project. For years, we at VMware believed that SQL server databases run well when virtualized. We have illustrated this through several benchmark studies published as white papers. It was time for us to look at real applications. One such application that can be found in most of the vSphere based virtual environments is the database component of the vCenter server (the brain behind a vSphere environment). Using the vCenter database as the application and the resource intensive tasks of the vCenter databases (implemented as stored procedures in SQL server-based databases) as the load generator, I compared the performance of these resource intensive tasks in a virtual machine (in a vSphere 4.1 host) to that in a native server.

From a sheer size perspective, the database doesn’t pop out many eyes. But from the criticality standpoint it can give sleepless nights. The database though ~93GB in size, included an inventory that in reality can represent a very large vSphere based virtual datacenter. With 500 hosts and 8000 virtual machines, performance of this vCenter database becomes extremely important. The tasks whose performance I studied are among the most common operations that can affect the performance of a vCenter database. These operations included:

  1. Rolling up performance statistics
  2. Calculating top resource consumers
  3. Purging old performance statistics

I won’t go into the details of these operations. Refer to the white paper for an explanation. Though I expected the performance of the virtualized database to be very close that of the native database, I didn’t expect it to be this close. Both in terms of CPU utilization and I/O performance, the virtual machine was very close to native (in some cases better than native!). I also observed an unusual behavior of native I/O stack during my experiments (I will blog about it next time, but for now check the appendix of the paper).

Here, this is for you VI admins – if you are thinking of virtualizing vCenter database (why not? you can virtualize anything and everything these days ;-)) here is a study that should give you the confidence to take the next step. If you have any comments to share with me or rest of the community, interesting facts, or tough performance issues, feel free to drop a comment or two here.

Oh!, BTW here is the link to the white paper.

Previous studies:

%d bloggers like this: