 |
Windows Terminal Server, Virtualization, Application Delivery, Windows Software Development, Market Analysis, Training Classes and more... |
 |
Navigation |
 |
|||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Awards |
|
||
|
||||
Microsoft Windows Server 2008 Terminal Services
21. Testing and Quality Assurance
Posted by Benny Tritsch on June 22, 2008
[Testing and QA] [Criteria] [Methodology] [Tools and Products]
21.2. Test Methodology
Read in this lesson...
Testing is a discipline that is reminiscent of classic natural sciences. A test often cannot incorporate an entire target environment, so strong abstraction is necessary. Some tests, by their very nature, change the observed object so fundamentally that general statements are no longer possible. Sometimes, marginal parameters (for example, exact user behavior) are not known or can be mapped only roughly in statistical terms. Because no test environment is exactly like the target environment, tests are often performed with individual components. This, however, requires the components to be prioritized. The key rule here is: do it properly or don't do it at all. To obtain meaningful test results, all of the following tasks must be performed: measuring, creating statistics, documenting, evaluating, interpreting, and communicating.
Quite often, there are not enough real users for tests to take place under controlled conditions. Moreover, scenarios with test users are usually not really reproducible because the individual users behave differently from test to test. Therefore, typical user actions need to be simulated, which requires precise knowledge of user behavior and the type of application. However, it quickly becomes evident that only simple user activity can be simulated because complex behavior modeling takes too much time and effort.
NOTE: In many cases, tests with real users do not create adequate results if the users know that they are part of a test scenario. They tend to work and act differently as if they were in an uncontrolled or unmonitored situation. This has direct influence on test results derived from such rather artificial real users’ work profiles. A solution could be not telling users that they are monitored when collecting information about their work profiles. However, in many countries the act of secretly collecting this sort of personal work profile data this is regarded as being socially unacceptable or it is even illegal.
A solution to the challenges with real users is to accept automated tests and synthetic user work profiles. This implies a certain abstraction of the user behavior and a concentration on the most important aspects of a work environment. In terms of synthetic and thus reproducible terminal server tests, it is necessary to distinguish between different concepts that relate to the location of the test logic represented by the synthetic user simulation engine. The test logic can either run on the server or on the client. Modeling the temporal processes of graphical user interaction is problematic due to the terminal server communication protocol being stateful – a fact that also applies to alternative protocols such as ICA. This is why a terminal server test differs significantly from a Web server test with the stateless HTTP protocol.
Client-Side Testing
If the logic for a synthetic terminal server test is set up on clients, the following constellation results: a script or a binary component is able to simulate all user actions on an RDP client. A suitable runtime environment that can perform mouse clicks and keyboard strokes like a human user needs to be installed on each client platform. The graphical display of applications is, however, only an image of the graphical application elements and does not represent the application elements themselves. The latter are created and managed on the terminal server only.
It is, of course, desirable that the test clients be controlled centrally and that multiple simultaneous instances of the RDP client software can be executed per client platform. In the best case scenario, the terminal server does not even notice that it is not a real user but a script logic requesting a session and interacting with the applications.
In such a scenario based on client-side test logic, the handling of time and space requirements for the simulated user interaction is a common challenge. Considerations include whether the script is able to click on the correct screen element at the right time or to enter a text into a dialog box only when the box actually appears on the screen. Graphical application elements that are not always displayed at fixed screen coordinates and increasing time delays for the graphical output when the server load increases often take scripts on the clients to their limits. This, in turn, results in a high number of failed simulated user sessions that freeze and thus distort the entire result.
The most successful test approaches based on client platforms involve special client applications or special extensions to standard client software. They concern the presentation layer of an application that is executed on a terminal server. It is therefore necessary either to use or program a modified client component, or to select a test runtime environment that can control a normal client for the most part. Both options incur quite some effort in terms of developing the respective test environments.
Server-Side Testing
An alternative to the client-side test is to use tools that automate user sessions on the server. This shifts the test activities from the presentation layer to the application layer. Macro tools for automating user actions are the best-known members of this product category. The problem with using macro tools is, though, that they change the server constellation. The macro tool itself is an auxiliary program that needs to be run during all tests. It thus distorts the result. Furthermore, events on the client play no role at all, which does not represent normal terminal server environment behavior.
Standardized benchmark tests that physically run on the terminal server are another option. However, for them to be displayed through several clients, they generally need to be launched manually in different terminal server sessions. These tests often contain popular application programs or representative screen output to model reality as closely as possible (such as Ziff-Davis benchmarks). Still, they are not really suitable for objective and individualized testing of a terminal server environment.
The continuous generation of processor load and the targeted consumption of memory resources is another way of testing terminal server performance. This procedure often helps obtain a first impression of the terminal server's behavior when using standard applications under different conditions.
Relevant Test Counters
Objective assessment of the test results can be supported by the following methods:
-
Measuring time using a stop watch or the system timers
-
Comparing the results of the standardized load tests with reference environments
-
Observing system activities with the System Monitor and the Network Monitor
-
Evaluating the test results using statistical and analysis tools
Subsequently, the raw measurement data must be brought into a structured form to determine the utilization of all system resources. In this way, the basic cornerstones can be identified and analyzed to work out the key indicators of satisfactory system performance. Expensive test environments usually include powerful automated interpretation functions. These need to be set up manually when using the more cost-effective tools.
There are a number of products and tools available for the test options described earlier, but using them professionally usually entails relatively high expense. They all have one thing in common, though: they all extensively utilize the Windows Server 2008 System Monitor performance counter objects for evaluations.
Performance counters you should analyze when testing terminal servers:
-
Memory\Available MBytes: Amount of physical memory immediately available for allocation to a process or for system use. It is equal to the sum of memory assigned to the standby (cached), free and zero page lists.
-
Terminal Services\Active Sessions: Total number of Terminal Services sessions.
-
Process(_Total)\Working Set: Current size, in bytes, of all processes in main memory. The Working Set is the set of memory pages touched recently by the threads in each process. If free memory in the computer is above a threshold, pages are left in the Working Set of a process even if they are not in use. When free memory falls below a threshold, pages are trimmed from Working Sets. If they are needed they will then be soft-faulted back into the Working Set before leaving main memory.
-
PhysicalDisk(_Total)\Avg. Disk Queue Length: Determines how many system requests, on average, are waiting for disk access. If values are high, you may want to monitor the Memory\Page Faults/sec counter to make sure that the disk activity is not caused by paging (typically processes configured to use too much memory or file system activity).
-
Processor(_Total)\% Processor Time: Percentage of elapsed time that the processor spends to execute a non-Idle thread. This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval.
-
Paging File(_Total)\% Usage: The percentage of the Page File instance that is in use.
-
System\Context Switches/sec: Combined rate at which all processors on the computer are switched from one thread to another. Context switches occur when a running thread voluntarily relinquishes the processor, is pre-empted by a higher priority ready thread, or switches between user-mode and privileged (kernel) mode to use an Executive or subsystem service. It is the sum of Thread\Context Switches/sec for all threads running on all processors in the computer and is measured in numbers of switches.
-
System\Processor Queue Length: Number of threads in the processor queue. This counter shows ready threads only, not threads that are running. There is a single queue for processor time even on computers with multiple processors or processor cores. Therefore, if a computer has multiple processor cores, you need to divide this value by the number of processor cores servicing the workload.
There are many more performance counters you may want to analyze during your terminal server tests. However, for comparability reasons it is advisable to stick with the same set of performance counters when going through multiple test sequences. This means that you need to make up your mind about which parameters you want to monitor before you start testing.
The goal after interpreting all test results is to be able to make an objective statement on the behavior of terminal server environments under set load conditions, clearly defining the limits pertaining to memory, processors, hard drive systems, and network. Furthermore, it should be possible to give an estimate of the maximum number of simultaneous terminal server users and the scalability of a load-balanced server farm.
| Read in this chapter... | |
| 21 | Testing and Quality Assurance |
| 21.1 | Test Criteria |
| 21.2 | Test Methodology |
| 21.3 | Available Tools and Products |