Collecting VMware ESX or VCenter Performance Counters during a LoadGen Test

In this guide, we'll explain how to collect ESX performance counters during your LoadGen Load and Performance tests using the LoadGen-ESXCollector tool. This tool gathers performance data from an ESXi or vCenter host and records it directly into your SQL Server database.

Understanding the Data Tables

In the architecture of LoadGen, there are three main types of messages that facilitate communication and data management:

• DUAF (Data User Action Framework): This category includes messages related to user actions within applications or transactions, such as start, stop, and fail events.
• PM (Performance Metrics): These messages contain performance data from the systems under test, including various performance counters.
• SMD (Session Messages Director): These messages have the highest priority and ensure the LoadGen Director receives critical session-related communications promptly. We will not use this data to analyze a test.

This structure allows LoadGen to efficiently manage and analyze the vast amounts of data generated during testing, ensuring high levels of performance and insight.

Each test run generates tables with a DateTime string appended to their names. For instance, DUAF202406141746438149 and PM202406141746438149 might be tables from one test run, and DUAF202406231746422999 and PM202406231746422999 might be from another.

Steps to set up the LoadGen-ESXCollector tool

The LoadGen-ESXCollector utilizes VMware vSphere Management SDK to collect performance metrics, writing them efficiently to your SQL Server using SQLBulkCopy. The data is automatically stored in the latest Performance Metrics SQL Table associated with your current LoadGen test.

Step 1: Download the LoadGen-ESXCollector Tool

 You can download the LoadGen-ESXCollector tool here.

Step 2: Run the LoadGen-ESXCollector Tool

The LoadGen-ESXCollector offers different modes of operation. We advise first running in Encryption Mode to securely generate encrypted credentials. Be sure to store the encrypted string generated during this step, as you will use it in Step 3 when configuring the tool in Secure Mode for your actual data collection.

Plain Text Mode

Use this mode if you're comfortable passing credentials directly via the command line:

When you need to collect many counters (CPU, Memory, Datastore, Networking, etc.), the command line may become too long. In that case, use -esxcountersfile instead of -esxCounters to load counters from a text file:

The counters file should contain one counter per line, for example:

• -silent
  Executes the tool without any interactive prompts or output on screen, suitable for automation scripts and background tasks.
• -esxVCenter
  Specifies the hostname or IP address of the vCenter server from which performance metrics will be collected.
   
• -esxHost
  Specifies the hostname or IP address of the ESXi host server from which performance metrics will be collected.
• -esxUser
  The username for authentication on the vCenter or ESXi host.
• -esxPassword
  Password for the provided ESX user account.
• -sqlServer
  Defines the SQL Server hostname or IP address where collected performance metrics will be stored. A period . can be used to specify localhost.
• -sqlDatabase
  Specifies the target SQL Server database for storing performance data.
• -sqlTrusted
  Indicates whether a trusted (Windows-integrated) connection should be used (yes or no). When running from a LoadGen Agent, use yes if the Agent service account has SQL Server access.
• -sqlUser
  The SQL Server username if using SQL authentication (-sqltrusted no).
• -sqlPassword
  Password for the provided SQL Server user account.
• -esxCounters
  A comma-separated list of specific ESX performance counters you want to retrieve (e.g., cpu.usage,mem.active). Use the -getesxcounters mode to identify available counters. For large counter lists, consider using -esxcountersfile instead.
• -esxcountersfile
  Path to a text file containing performance counters to collect, one counter per line. Use this instead of -esxCounters when you need to collect many counters (CPU, Memory, Datastore, Networking, etc.) and the command line becomes too long. This parameter can also be used in Secure Mode to override the counters stored in the encrypted credentials string, allowing you to manage counter selection separately from your credentials.
• -interval
  Defines the interval (in seconds) between each collection of performance metrics. A shorter interval increases granularity and data volume.
• -timeout
  The data collection session's total duration (in seconds) before the tool stops automatically.
• -stopFile
  Path to a file whose creation acts as a signal for the tool to stop data collection. Useful for manual or automated termination of data collection.
• -encrypt
  Runs the tool in interactive mode to securely input credentials, generating an encrypted credential string to be used later in a secure mode.
• -secure
  Used in secure mode to provide a previously generated encrypted credentials string, enhancing security by avoiding plain-text credentials on the command line. You can override specific settings from the encrypted string using -timeout, -stopFile, -esxCounters, or -esxcountersfile on the command line.
• -getesxcounters
  Lists available ESX counters from the specified ESX host. Use this mode to identify performance counters that are available for collection.
• -M
  Enable MultiSubnetFailover for SQL connections.

Step 3: Preparing Your Load Scenario in LoadGen Director

To automate performance data collection during test execution:

1. Open the LoadGen Director.
2. Open your canvas / Load Profile.
3. Double-click the active Load Scenario you want to modify.
4. Select Load scenario events, then click Scenario Start.
   
5. Click ... to create a new custom action.
   

6. Choose Execute a file, and specify the path to LoadGen-ESXCollector:

   C:\Program Files\LoadGen\LoadGen-ESXCollector\LoadGen-ESXCollector.exe

7. Set parameters using secure mode (recommended):

   -silent -secure <secure_string> -timeout 3600
   Replace <secure_string> with your encrypted credentials string from Encryption Mode.

   If you need to collect many counters, add -esxcountersfile to load them from a file instead of embedding them in the encrypted string:

   -silent -secure <secure_string> -esxcountersfile C:\LoadGen\counters.txt -timeout 3600
   This way you can manage your counter selection independently from your encrypted credentials.

Step 4: Visualizing VMware ESX Performance Metrics in LoadGen Analyzer

The LoadGen Analyzer offers a powerful way to analyze performance data collected from VMware ESX hosts during your load and performance tests. Once you've collected ESX counters using the LoadGen-ESXCollector, you can build detailed and interactive charts to gain insight into host performance over the duration of your test.

1. Launch LoadGen Analyzer from your system.
2. Select the correct Data Source.
   1. Make sure you are connected to the database that contains the Performance Metrics (PM) tables. These tables should have been automatically populated by the LoadGen-ESXCollector tool during your test.
3. Create a Custom Chart
4. In the left-side Chart Elements panel:
   1. Expand Custom charts > Data > Loadtest duration
   2. Drag Loadtest duration onto the X-axis of your chart (this represents the timeline of your test).
      
5. Add Performance Metric Values to the Y-axis
   1. Navigate to Custom charts > Data > Performance metrics > Counter value
   2. Drag and drop Counter value onto the Left Y-axis
6. Filter for Your ESX Host, to focus on the performance data from your specific ESX host:
   1. Click the filter icon next to the Left Y axis
   2. In the popup filter editor, choose MachineName
   3. Set the condition to match your ESX host
   4. Press OK to apply the filter
      
7. Group Metrics for clarity, to display multiple series (such as average, maximum, and minimum values) clearly on the same chart:
   1. Navigate to Grouping > Performance metrics > Instance name
   2. Drag the Instance name into the Grouping box
   3. This ensures different counter types (e.g., cpu.usage.average, disk.usage.maximum) are visualized as separate lines in your graph
      
8. (Optional) Customize Functions
   1. In the Function dropdown, you can choose how to aggregate the values (e.g., Average, Minimum, Maximum). This allows for high-level trend analysis or precise point monitoring

Recommended Practices

Using the LoadGen-ESXCollector allows seamless integration of comprehensive ESX performance insights into your LoadGen testing process, aiding precise and reliable performance analysis.

• You can always use Secure or Encryption modes to protect sensitive credentials.
• Define a suitable timeout to manage data volume.
• Select only the necessary counters to optimize data relevance.
• Use -esxcountersfile when collecting many counters across multiple categories (CPU, Memory, Datastore, Networking). This avoids command line length limitations and makes counter management easier - you can update the file without regenerating encrypted credentials.

MultiSubnetFailover Feature

What is MultiSubnetFailover?

MultiSubnetFailover is a SQL Server connection feature that enables faster failover in multi-subnet environments. When enabled, the SQL client attempts to connect to multiple IP addresses simultaneously, reducing connection time in clustered or Always On availability group scenarios.

When to Use MultiSubnetFailover

• SQL Server Always On Availability Groups across multiple subnets
• Failover Cluster Instances spanning different subnets
• Geographically distributed SQL Server deployments
• Network environments with multiple network paths

How to Enable

Add the `-M` flag to any command line:

LoadGen-ESXCollector.exe -silent -esxVCenter <vcenter host OPTIONAL> -esxHost <host> -esxUser <user> -esxPassword <password> -sqlServer <server> -sqlDatabase <database> -sqlTrusted <yes/no> -sqlUser <sqlUser> -sqlPassword <sqlPassword> -esxCounters <esxCounters comma-separated> -interval <intervalInSeconds> -timeout <timeoutInSeconds> -stopFile <stopFilePath> -M 

Secure Mode

This mode decrypts previously encrypted credentials at runtime: