Load Tester collects an extensive measurements during a load test. This data is collected in samples. Each sample has multiple measurements associated with it. Depending on the measurement type, some measurements may not ba applicable during each sample and may therefore appear empty in some displays, such as the Metrics View. Unless otherwise noted, the value of each measurement in a sample pertains only to events that completed within that sample period.
Please note that metrics were formerly referred to as statistics.
While the term sample is used for our data storage units, it is important to understand that Load Tester does not rely on sampled data as some other load-testing products do. Unlike those products, Load Tester measures the response time of every single page and transaction executed. For instance, every web page duration is measured and the page duration metrics reported reflect every page completed during the test.
Test metrics are collected by the Virtual Users as the testcases are executed. The data is collected at four levels: Test Summary, Testcase, Web Page and Transaction. These levels represent narrowing scopes of data - for example: web pages may contain many transactions and a testcase may contain many web pages. In many cases, the higher levels are not simply aggregations of lower levels but measurements tailored specifically for each level.
Attempts - The total number of times the item was attempted during the sample period. This is a summation of the Success and Failures metrics -- every attempt will result in either a success or failure.
Average CAN Duration - The average of the CAN Durations during the sample period.
Average Duration - The average duration for the item. For transactions, the measured duration starts when the first byte of the request is sent and ends when the final byte of the response is received. If a new connection was initiated for the transaction, the connect time is included as well. For pages, the measured duration starts when the first transaction on the page begins and ends when the final transaction completes. For testcases, the measured duration starts when the first page starts and ends when the final page completes. Note that for transactions that utilize HTTP Authentication, CAN transactions are not included in the measured duration of a transaction. They are however inherently included in page duration, due to the way it is measured.
Average Page Duration - The average page duration for all pages.
Average Page Wait Time - The average amount of time that users have waiting for a web page to complete. This counts all users that are actively working on a web page at the end of the sample period. Because page durations that are used in the min/max/avg page duration measurements are not reported until the page completes, this measurement can be useful in determining backlog on the server as the performance degrades.
Average Speed - The average data transfer rate of the item. For testcases, this measurement includes the total bytes transferred for all web pages and transactions and the total duration of the testcase, including think time but not repeat delay. For web pages, this includes the bytes transferred for all transactions and the total duration of the page, not including think time. For transactions, this includes bytes sent and received (the total of the size of the HTTP request and response messages) and the duration of the transaction.
Average Wait Time - The average amount of time that waiting users have been waiting for the page or URL to complete. See definition of Waiting Users for more details.
Average TTFB - The average TTFB for the transaction. The Time To First Byte measures the time from the beginning of the request to the beginning of the response. On large or slow transactions in some systems, this measurement can be useful to separate the time required for the server to service a request from the time required to transfer the response over the network.
Bytes/sec - The average rate at which data was sent and received. It represents all bytes sent/received during the sample period.
Completions - The number of times the item was completed during the sample period. For transactions, a completion indicates that a valid HTTP response was received from the server. This does not imply that the response was correct - only that a parse-able HTTP message was received. Not every attempt will result in a completion, but every completion will result in either a success or failure. For testcases and web pages, a completion indicates that all child elements (web pages and transactions) were attempted. This applies to Testcases, Web Pages and Transactions.
CAN Attempts - The number of times that one or more CAN transactions were required to in order to complete a transaction.
Detailed Page Durations - This metric holds a list of the time and duration of each page load completed during the sample period. This metric applies only to web pages and is also available at the summary level (for all pages). Note that the option to collect these metrics is off by default, to conserve memory. It can be enabled in the preferences (Window menu).
Duration Standard Deviation - The standard deviation of sampled durations for the item.
Errors - A list of all errors recorded during the test. This is not a numeric value -- it is a list separate from the computed and observed metrics. Note that multiple errors could occur during the execution of a single transaction.
Failed Pages - The number of pages that failed during the sample period. This is a summation of the Failures metric for all web pages in the test.
Failures - The number of times the item failed during the sample period. A transaction may fail in many ways - any of which will result in the recording of a failure. Some common examples of failure conditions are: unable to connect to server, connection closed without receiving response from server, unable to write request to server (e.g. user variable missing or dataset exhausted), validator failed (unable to find content, unexpected status code, etc) and extractor unable to locate content. This applies to Testcases, Web Pages and Transactions. If any transactions failed on a web page, the web page will also be recorded as a failure. Likewise, any failure on a transaction or web page will result in the recording of a failure for the testcase. This applies to Testcases, Web Pages and Transactions.
Hits/sec - The number of transactions completed per second. "Transactions/sec" would be more technically accurate but is not used for two reasons: (1) hits/sec is a standard industry term and (2) the term transaction is already applied to many other levels of interaction in business applications. Note that a completed transaction implies that there is a complete response received from the server. Therefore, transactions which are prematurely terminated due (e.g. due to network errors) are not counted as a hit. Transactions which complete successfully but result in an error due to transaction validators are counted as a hit. Thus you can have errors when hits/sec is zero. This measurement is reported only at the test summary level. Note that required CAN transactions are not counted as individual hits.
Maximum Duration - The highest duration for the item. See Average Duration for more details.
Maximum Page Duration - The highest page duration. See Average Page Duration for more details.
Maximum TTFB - The highest TTFB for the transaction. See Average TTFB for more details.
Minimum Duration - The lowest duration for the item. See Average Duration for more details.
Minimum Page Duration - The lowest page duration. See Average Page Duration for more details.
Minimum TTFB - The lowest TTFB for the transaction. See Average TTFB for more details.
Pages - The number of pages completed during the sample period. This is a summation of each of the Completions metric for each page in the test. Note that a completion does not imply success - only that the Virtual User attempted every transaction in the web page.
Pages/sec - The average rate at which pages were completed.
Page Duration Standard Deviation - The standard deviation of sampled durations for all pages.
Succeeded Pages - The number of pages that succeeded during the sample period. See the Failures metric for more details.
Successes - The number of times that the item was completed during the sample period without recording any failures. See the Failures metric for more details. This applies to Testcases, Web Pages and Transactions.
Time - The time at which the sample period ended. This is expressed as an elapsed time since the start of the test.
Total Pages Failed - The total number of page failures recorded up to the current sample period.
Total Size - The total number of bytes transferred for this item. Includes both request and response. Note that the bytes transferred includes only the bytes that make up the HTTP transactions. Other bytes transferred at lower protocol levels, such as those required for TCP/IP overhead, are not included. For individual transactions, the bytes transferred for required CAN transactions are not included in the target transaction.
TTFB Standard Deviation - The standard deviation of sampled TTFBs for the transaction.
Users - The number of users currently being simulated. At the test summary level, this is the total number of users in the test. At the Testcase level, this is the number of users simulating the testcase.
Waiting Users - The number of users working on a page (or URL) at the end of the sample period. Note that working on refers to the process of requesting and receiving the contents of a page or URL. Users that are not working on pages or URLs will either be executing think time or the repeat delay. For individual transactions, the user is considered to be waiting on the transaction from the time it starts sending the request until the last byte is received, including any required CAN transactions.
Load Engine metrics are collected by the load engines in the process of monitoring its own performance.
Time - (see above)
CPU % - The total CPU utilization as reported by the OS.
Memory % - The amount of VM memory that is available. Note that the load testing and/or the load engine, by default, is configured, by default, to use 200M of RAM. There may be more memory available. If so, see the Configuring Memory Usage page for configuration instructions.
Users - The number of VUs running in this load engine.
Total Capacity - The estimated number of VUs that the load engine can handle. Note that this estimate is very conservative and can be very innaccurate at low numbers. It is not unusual for a load engine to report a total capacity of 15 VUs when 1 VU is running, 80 when 10 are running and 500 when 400 are running.
Engine Status - The state of the load engine (Idle, Initializing, Running, Stopping, Error or Off-line).
Bandwidth Out - Total data transfer rate (out from the load engine) during the sample period.
Bandwidth In - Total data transfer rate (in to the load engine) during the sample period.
Steal % - The time that the guest CPU had something runnable, but the hypervisor chose to run something else instead (applies only to systems running on virtual machines).
Server metrics are collected by the Server Agents to help assess the performance of the server OS and application servers during a load test.
CPU % - The total CPU utilization as reported by the OS. For AIX servers, this is the Logical CPU utilization.
Virtual CPU Usage (AIX only) - the average percentage of time among virtual processors spent non-Idle.
CPU Entitlement (AIX only) - the percentage of entitled CPU time being utilized. This value may exceed 100% when the active partition has time donated or stolen from another partition.
Steal % - The time that the guest CPU had something runnable, but the hypervisor chose to run something else instead (applies only to systems running on virtual machines).
Memory % - The total memory utilization as reported by the OS.
Context Switches / sec - The rate at which the OS kernel switches from one thread to another.
Process Queue Length - The number of system threads in the process queue.
Page reads/sec - Rate of major memory faults resulting in page reads.
Page writes/sec - Rate at which pages are moved to disk to free space in physical memory.
Disk I/O Time % - Elapsed time during which a disk was servicing read or write requests.
Avg Disk Service Time - Average amount of time for each disk I/O transfer (ms)
Disk Queue Length - Number of queued disk operations.
Disk Reads/sec - Rate of disk read operations.
Disk Writes/sec - Rate of disk write operations.
Network bytes sent - Network outbound throughput.
Network bytes received - Network inbound throughput.
Network packets received / sec - Number of network packets received per second.
Network packets sent / sec - Number of network packets sent per second.
Network Packets Received Errors - Number of network packets received with errors.
Network Packets Sent Errors - Number of network packets sent with errors.
Network Collisions - Number of network packet transmission attempts that resulted in a collision.
TCP Connections Established - Number of open TCP connections.
TCP Connections Established / sec- Average number of new TCP connections established per second.
TCP Connection Failures - Number of failed TCP connections.
TCP Segments Retransmitted / sec - Rate of previously transmitted TCP segments being retransmitted.
ASP.NET Active Sessions - The number of sessions active.
ASP.NET Application Restarts - Number of times the application has been restarted during the sample period
ASP.NET Cache Entries - Total number of entries within the ASP.NET application cache (both internal and user added)
ASP.NET Cache Hit Ratio - Ratio of hits from all ASP.NET application cache calls. Note that the Windows counter calculates this value as a historical average since the last restart, but Load Tester does not.
ASP.NET Cache Hits - Number of hits from all ASP.NET application cache calls.
ASP.NET Cache Misses - Number of misses from all ASP.NET application cache calls.
ASP.NET Cache Turnover Rate - Number of additions and removals to the total ASP.NET application cache per second.
ASP.NET Compilations - Number of .asax, .ascx, .ashx, .asmx, or .aspx source files dynamically compiled.
ASP.NET Current Requests - The current number of ASP.NET requests, including those that are queued, currently executing, or waiting to be written to the client. Under the ASP.NET process model, when this counter exceeds the requestQueueLimit defined in the processModel configuration section, ASP.NET will begin rejecting requests.
ASP.NET Disconnected Requests - The number of requests that were disconnected due to communication failure.
ASP.NET Errors/sec - Rate of errors occurred.
ASP.NET Last Request Execution Time - The amount of time that it took to execute the most recent ASP.NET request.
ASP.NET Pipeline Instance count - Number of active ASP.NET application pipeline instances.
ASP.NET Queued Requests - The number of ASP.NET requests waiting to be processed.
ASP.NET Rejected Requests - The number of ASP.NET requests rejected because the request queue was full.
ASP.NET Request Wait Time - The amount of time the most recent request was waiting in the queue.
ASP.NET Requests Executing - The number of ASP.NET requests currently executing
ASP.NET Requests Not Found - The number of ASP.NET requests for resources that were not found.
ASP.NET Requests Not Authorized - Number of requests failed due to unauthorized access.
ASP.NET Requests Queue Length - The number of ASP.NET requests in the application request queue.
ASP.NET Requests Succeeded - The number of requests that executed successfully during the sample period.
ASP.NET Requests Timed Out - The number of requests that timed out.
ASP.NET Requests/sec - The number of ASP.NET requests executed per second.
ASP.NET Sessions Timed Out - The number of sessions that have been closed due to their idle time exceeding the AutoDisconnect parameter for the server. Shows whether the AutoDisconnect setting is helping to conserve resources.
ASP.NET Sessions Abandoned - The number of sessions that have been explicitly abandoned.
ASP.NET Unhandled Execution Errors/sec - Rate of unhandled errors.
.NET CLR Class loader Heap Size - The current size (in bytes) of the memory committed by the class loader across all AppDomains. (Committed memory is the physical memory for which space has been reserved on the disk paging file.)
.NET CLR Current AppDomains - The current number of AppDomains loaded in this application. AppDomains (application domains) provide a secure and versatile unit of processing that the CLR can use to provide isolation between applications running in the same process.
.NET CLR Current Assemblies - The current number of Assemblies loaded across all AppDomains in this application. If the Assembly is loaded as domain-neutral from multiple AppDomains then this counter is incremented once only. Assemblies can be loaded as domain-neutral when their code can be shared by all AppDomains or they can be loaded as domain-specific when their code is private to the AppDomain.
.NET CLR Exceptions / sec - The number of exceptions thrown per second. These include both .NET exceptions and unmanaged exceptions that get converted into .NET exceptions e.g. null pointer reference exception in unmanaged code would get re-thrown in managed code as a .NET System.NullReferenceException; this counter includes both handled and unhandled exceptions. Exceptions should only occur in rare situations and not in the normal control flow of the program; this counter was designed as an indicator of potential performance problems due to large (>100/s) rate of exceptions thrown.
.NET CLR Heap Size - The sum of four other counters; Gen 0 Heap Size; Gen 1 Heap Size; Gen 2 Heap Size and the Large Object Heap Size. This counter indicates the current memory allocated in bytes on the GC Heaps.
.NET CLR Generation 0 Collections - The number of times the generation 0 objects (youngest; most recently allocated) are garbage collected during the sample period. Gen 0 GC occurs when the available memory in generation 0 is not sufficient to satisfy an allocation request. Higher generation GCs include all lower generation GCs. This counter is explicitly incremented when a higher generation (Gen 1 or Gen 2) GC occurs.
.NET CLR Generation 1 Collections - The number of times the generation 1 objects are garbage collected. Higher generation GCs include all lower generation GCs.
.NET CLR Generation 2 Collections - The number of times the generation 2 objects (older) are garbage collected. The counter is incremented at the end of a Gen 2 GC (also called full GC).
.NET CLR Lock Contention Queue Length - The number of threads in the runtime currently waiting to acquire a lock.
.NET CLR Lock Contention / sec - Rate at which threads in the runtime attempt to acquire a managed lock unsuccessfully. Managed locks can be acquired in many ways; by the "lock" statement in C# or by calling System.Monitor.Enter or by using MethodImplOptions.Synchronized custom attribute.
.NET CLR Lock Contentions - The total number of times threads in the CLR have attempted to acquire a managed lock unsuccessfully during the sample period. Managed locks can be acquired in many ways; by the "lock" statement in C# or by calling System.Monitor.Enter or by using MethodImplOptions.Synchronized custom attribute.
.NET CLR Logical Thread Count - The number of current .NET thread objects in the application. A .NET thread object is created either by new System.Threading.Thread or when an unmanaged thread enters the managed environment. This counters maintains the count of both running and stopped threads.
.NET CLR Percent Time in GC - The percentage of elapsed time that was spent in performing a garbage collection (GC) since the last GC cycle. This counter is usually an indicator of the work done by the Garbage Collector on behalf of the application to collect and compact memory.
.NET CLR Physical Thread Count - The number of native OS threads created and owned by the CLR to act as underlying threads for .NET thread objects. This does not include the threads used by the CLR in its internal operations; it is a subset of the threads in the OS process.
.NET CLR Recognized Thread Count - Displays the number of threads that are currently recognized by the CLR; they have a corresponding .NET thread object associated with them. These threads are not created by the CLR; they are created outside the CLR but have since run inside the CLR at least once. Only unique threads are tracked; threads with same thread ID re-entering the CLR or recreated after thread exit are not counted twice.
IIS CGI Requests/sec - The rate CGI requests are received by the Web service.
IIS Connection Attempts/sec - The rate that connections to the Web service are being attempted
IIS Current Connections - Current Connections is the current number of connections established with the Web service.
IIS File Cache Entries - The number of entries in the File Cache.
IIS File Cache Hits - The number of successful lookups in the file cache.
IIS File Cache Hit % - The ratio of file cache hits to total cache requests. Note that the Windows counter calculates this value as a historical average since the last restart, but Load Tester does not.
IIS File Cache Misses - The number of failed lookups in the file cache.
IIS ISAPI Extension Requests/sec - The rate that ISAPI Extension requests are received by the Web service.
IIS Kernel URI Cache Entries - The number of entries in the Kernel URI Cache.
IIS Kernel URI Cache Hits - The number of successful lookups in the Kernel URI cache.
IIS Kernel URI Cache Hit % - The ratio of Kernel URI cache hits to total cache requests. Note that the Windows counter calculates this value as a historical average since the last restart, but Load Tester does not.
IIS Kernel URI Cache Misses - The number of failed lookups in the Kernel URI cache.
IIS Requests/sec - The rate HTTP requests are received by the web server.
IIS URI Cache Entries - The number of entries in the URI Cache.
IIS URI Cache Hits - The number of successful lookups in the URI cache.
IIS URI Cache Hit % - The ratio of URI cache hits to total cache requests. Note that the Windows counter calculates this value as a historical average since the last restart, but Load Tester does not.
IIS URI Cache Misses - The number of failed lookups in the URI cache.
These statistics represent the combined processes for IIS (inetinfo), ASP.NET (aspnet_wp) and the Application Pool Process (w3wp).
IIS+App Handle Count - The number of handles held by known web server processes.
IIS+App Private Bytes - The current size of committed memory owned by known web server processes.
IIS+App Thread Count - The current number of threads active for known web server processes.
IIS+App Virtual Bytes - The current size of virtual address space for known web server processes.
IIS+App % Processor Time - The percentage of elapsed time that threads from known webserver processes used the processor to execution instructions..