Understanding the capabilities of Visual Studio 2010 Lab Management
Using Lab Management to run tests, capture bugs, and share snapshots
Configuring end-to-end build-deploy-test workflows with Lab Management
As software development projects become more complex, so do the environments in which that software will run. Such an environment could consist of multiple machines, specific firewall (and other security) settings, databases, and a variety of other configurations that could impact the way in which your software behaves.
To effectively test software, testers must create a test environment that simulates the production environment. Traditionally, this could require securing several dedicated physical machines and developing a potentially labor-intensive process for staging those machines on a regular basis with new builds of your software. And, given the variety of possible configurations, it's usually necessary to have multiple test environments in order to find problems that may arise when you ship your software to customers running different environments, each with his or her own unique configurations.
With the rising popularity and availability of virtualization technology, many testing teams have begun to turn to virtualization to make better use of hardware and to more efficiently stage testing environments. But, despite the advances in virtualization, there are still several challenges related to the process of managing a virtual test lab, which can make this a costly and time-consuming endeavor.
Visual Studio Lab Management 2010 addresses the challenge of working with such virtual test lab environments. Lab Management integrates with Team Foundation Server 2010 to provide the following capabilities:
Creation, management, and teardown of environments consisting of one or more virtual machines (VMs).
Automated deployment of builds into virtual environments.
Execution of manual and automated tests across virtual environments.
Use of snapshots to enable environments to be quickly restored to a given state (such as immediately after a new build of software is deployed or when a new bug is discovered). Snapshots can then be shared between testers and developers to help diagnose and fix bugs.
Network isolation of virtualized environments, allowing clones of environments without fear of IP address collisions or naming conflicts with other machines on your network.
In this chapter, you will learn how Visual Studio Lab Management 2010 can be used to take advantage of these capabilities.
Visual Studio Lab Management 2010 is licensed separately from Team Foundation Server 2010. Once licensed, it can be installed as an additional capability and enabled for use with a Team Foundation Server 2010 instance. Installation and administration of Lab Management is covered extensively in the product documentation and won't be covered in this book, but a few key concepts will be introduced here.
Lab Management includes a license for Microsoft System Center Virtual Machine Manager (SCVMM) 2008 which provides many of the VM administration capabilities required for your test lab. Lab Management includes the SCVMM license you will need for setting up your virtual test lab. SCVMM, in turn, relies on Windows Server 2008 Hyper-V as the low-level virtualization technology.
Note
While SCVMM itself does allow for the management of VMware-based VMs, VMware is not yet supported for use with Lab Management.
SCVMM uses a library server to store copies of VMs, which can then later be deployed to a VM host group (made up of one or more VM hosts). A library server is essentially a file server that SCVMM is aware of and has read/write access to. Each library server can contain one or more library shares, which is basically a shared folder.
A library server can contain VM templates that enable you to customize a VM at the time of deployment. This allows you to specify such settings as machine name, domain or workgroup membership, and product key. VM templates are a powerful tool for building out your test lab, since they provide the most control over how VMs are deployed.
While setting up your test lab, you will need to consider the VM configurations on which you will need to test your software. For example, maybe your software needs to be tested to run in environments containing machines running Windows 7, Windows Server 2008, and Windows Server 2008 R2. You should also consider which other prerequisite software must be installed, such as Internet Information Server (IIS) or database engines such as SQL Server.
The installation guidance for Lab Management refers to the concept of using golden images for populating your library server. A golden image is a VM or VM template that contains all of the prerequisites necessary for testing your software. In the previous example, you might configure a golden image for each operating system version that will eventually be involved in your test environments.
Agents can be installed on VMs to provide additional capabilities that are helpful in deployment, testing, and network isolation with your virtual environments. Three types of agents can be installed:
A build agent allows a VM to participate in Team Foundation Build workflows. This includes the capability to deploy new builds to your VMs and to execute post-build deployment scripts.
A test agent enables manual or automated tests (such as unit tests, coded UI tests, or Web performance tests) to be executed on your VMs.
A lab agent enables network isolation capabilities for a VM environment. With network isolation enabled, you don't have to worry about VMs in your test lab conflicting (name or IP address) with other machines on your network. This makes it possible to have multiple virtual environments with the same IP address and/or machine name without needing to set up dedicated networks for each one.
Note
If you plan on taking advantage of agents in your test lab, you should install those agents on your golden image prior to storing that golden image in your VM library.
The preceding descriptions should provide you with a basic understanding of what is necessary to configure the infrastructure required for taking advantage of Lab Management. But it is by no means a substitute for the detailed product documentation. Your test lab administrator should carefully consult the product documentation for instructions on configuring and optimizing your Lab Management infrastructure. Once configured, you can benefit by using Lab Management as detailed in the remainder of this chapter.
A virtual environment consists of one or more VMs that can be deployed and managed together. An environment usually contains all of the VMs necessary to run a set of test cases. For example, an environment could consist of a database server and a Web server, each running Windows Server 2008. A separate virtual environment might also contain similarly configured database and Web servers, but use Windows Server 2003 to offer expanded test coverage.
The first step in creating a virtual environment with Lab Management is to define the VMs or VM templates that will make up your environment. You will use Microsoft Test Manager, introduced in Chapter 14, to do this. Before completing this step, you must have one or more golden images (VMs or templates) stored in your SCVMM library.
Click Start
From here, you can manage all of the VMs and templates that are available to the team project you are connected to. To add a new VM, click the Import button in the upper-left area of the screen.
Begin by defining the path to your new VM or VM template. This path defines the location within the SCVMM library server where your VM or template is stored. Use the Browse button to explore the library server path(s) defined in SCVMM.
Next, provide a name for your VM or template. You can optionally provide a description (useful for describing what's installed on this VM or VM template), along with a default role (discussed later in this chapter).
The "Machine properties" tab shown allows you to specify default parameters that will be used when your VM is deployed (such as the amount of RAM that should be assigned to your VM when deployed). If you are using a VM template, then the "OS profile" tab is available, allowing you to define additional parameters (such as the machine name, domain or workgroup membership, and product key). You can use the "Machine tags" tab to construct advanced deployment workflows.
Clicking Next will display a summary of your actions. Click Finish and your VM or template will now be listed in the Virtual Machines and Templates activity within Test Manager.
Note
You can repeat this process for defining as many VMs and VM templates as you want. You can even use the same VM template from your SCVMM library as the basis for multiple VMs or templates within Test Manager (such as to specify various default parameters).
Once you have configured one or more VMs or templates, you are ready to define a virtual environment. Click Library
Click New to create a new virtual environment. You can provide a name and description for your environment. You can also specify the location on the SCVMM library server where the environment definition should be stored, along with environment tags that can be used for defining advanced build workflows.
The Machines tab shown in Figure 16-3 is where you can begin constructing your virtual environment based on the VMs or templates you defined earlier. To do this, first select a VM or template from the list of VMs and templates on the right side of the screen. Next, select "Add to environment." This will add the VM to the environment on the left side of the screen. You can add the same VM template multiple times, if desired.
After adding one or more VMs or templates to your environment, you can then specify which role these machines play in your environment (such as a Web server or database server). Roles are used by test settings and build workflows as you will see later on. You can also specify the name that Lab Management will use to refer to the VM within the environment. Note that this name is not necessarily the same as the computer name.
The "Machine properties" tab shown in Figure 16-4 allows you to define the parameters that should be assigned to each of the VMs within your environment. This screen should look similar to the "Machine properties" tab you encountered when defining a VM or template, except that it also includes the capability to select different VMs in your environment by clicking on the role icons across the top of the screen. Any machine properties you defined earlier will be shown here as default values and can be overridden in this step.
The Capabilities tab shown allows you to describe which agents are installed on the VMs within your environment. For build agents and test agents, you will also define the build and test controllers that should be used with those agents. If you are unsure of what values to use here, consult with your Lab Management administrator.
The Summary tab describes your selections. Click Finish to finalize your environment definition.
Once your virtual environment is defined, it will show up in the list of available environments in your library, as shown in Figure 16-2. Once your virtual environment is defined, it is ready to deploy. Select the environment within your library and click the Deploy button in the upper-left of the screen. The "Deploy environment" dialog shown in Figure 16-5 will appear.
From this dialog, you can provide a name and a description for what will become a running instance of your virtual environment. You can also specify the SCVMM VM host group to where you want to deploy your environment. Click "Deploy environment" to begin the virtual environment deployment process.
From the Lab
Once deployed, your virtual environment can be managed from the Lab
You can right-click on a running virtual environment and select Connect to open the Environment Viewer shown in Figure 16-8. The Environment Viewer allows you to interact with the VMs running within your environment.
From the Environment Viewer you can also stop, start, and pause the running environment, see the status of agents running on the VMs (in the upper-left corner), and mark an environment as "In use" (upper-right corner). This signals to other members of your team that you are using the environment and they should not attempt to connect to it.
The "System info" button allows you to view properties of the running VMs (such as the fully qualified machine name). This can be useful information for connecting to the VM from outside of the environment (such as when using a Web browser on a client machine to connect to a Web site running within your virtual environment).
You can also manage snapshots for your environment from here by clicking on the Snapshots tab on the left side of the screen, as shown in Figure 16-9. Snapshots allow you to save the state of an environment at any point in time, and likewise to restore the state of an environment by restoring a snapshot. Lab Management allows you to create a snapshot of an entire environment at once, although you can create a snapshot of individual VMs a few seconds apart, so this may have an impact on transactions that were in process when you started the synchronization.
Snapshots have the following useful applications:
A snapshot can provide a clean "baseline" state that can be used prior to installing each new build.
Snapshots can be created after installing a new build, providing a way to always restore to a known state prior to any tests being executed that may potentially "dirty" an environment.
Snapshots can be created by testers when they find a bug. These snapshots can then be shared with the development team to help them diagnose the bug and deliver a fix.
From the Snapshots tab you can create new snapshots, rename them, delete them, or restore your environment to an existing snapshot.
Now that you understand the basics of creating, deploying, and working with running environments, it's time to explore software testing with virtual environments.
Once you have a running virtual environment, you can use it to run your tests.
In Chapter 14, you configured test settings to define which diagnostics data to collect as you ran your tests (such as video, IntelliTrace files, and action logs). But now that you are going to run tests with an environment, you may want to create a new test setting that specifies how data should be collected from each machine within your environment. This step is optional, but can provide valuable diagnostics data to your developers when bugs are discovered.
Note
To define a test setting that collects data from remote machines, you must have installed and configured test agents on the VMs within your environment.
From within Test Manager, click on Testing Center
Note
The discussion in this section assumes that you have already configured your first test plan as described in Chapter 14.
From within your test plan properties, create a new test settings definition (Manual Runs
The Roles tab shown in Figure 16-11 allows you to select the virtual environment for which you want to define test settings. When defining test settings for automated tests, you can also select the role from where automated tests will be run. When configuring manual tests, the tests are always run from the local machine where Test Manager is running. After selecting your environment, click Next.
The Data and Diagnostics tab shown in Figure 16-12 allows you to define the individual diagnostics data adapters that will be used for each machine within your virtual environment. Data diagnostics adapters were covered in detail in Chapter 14. From here, you can configure the adapters for each machine within your environment.
Run a test case as you normally would by clicking on the Test
Note
The discussion in this section assumes that you have already created one or more test cases as defined in Chapter 14.
Select a test case that you want to run with your environment and click Run
The Run Options dialog shown in Figure 16-13 appears, allowing you to select the test settings and environment with which you want to run your test. If your manual test has associated automated tests (such as a coded UI test) and your test plan is associated with a build definition, then you can also opt to run this test as an automated test. For now, if your test has associated automated tests, just select the "Run all the tests manually" checkbox.
If you defined new test settings for collecting data from your environment, select it here. Also, select your running environment from the Environment drop-down. Click Run and this will launch your test run and the Microsoft Test Runner.
Once the Microsoft Test Runner is open, you can click the "Connect to environment button" (shown in Figure 16-14) to open the Environment Viewer for your environment.
Once the Environment Viewer is open, you can then begin running your test just like you would run any other manual test. You may wish to use the Snapshots tab to restore the environment to a known state (such as immediately after a given build was deployed). As needed, you can even switch among multiple machines within your environment if your test case requires it. Figure 16-15 shows a test case being run with a virtual environment.
If you discover a bug while you are testing, you may wish to create an environment snapshot that can be shared with the development team to help them diagnose the problem. Even though you could do this directly from within the Environment Viewer, a better way is to do so is from within the Microsoft Test Runner. This automatically attaches a pointer to the environment snapshot to the test results.
To create an environment snapshot with Microsoft Test Runner, click the rightmost icon along the Microsoft Test Runner toolbar (shown in the upper-right corner of Figure 16-16). This creates a new snapshot of the environment and saves an .lvr file to your test results. The .lvr file is a pointer to the environment snapshot that can be opened later to restore your environment to this snapshot.
Click the "Create bug" icon within the Microsoft Test Runner to create a new bug along with your test results (hover your mouse over the toolbar icons to discover the Create Bug icon). Figure 16-17 shows the new bug creation form, along with a reference to the .lvr file created earlier.
When reviewing this bug later, a developer can open the .lvr file simply by clicking on it, provided the developer has Test Manager installed. The dialog shown in Figure 16-18 will appear when an .lvr file is opened. This dialog gives you the option of connecting to the running environment as-is, or restoring the environment to the state it was in when the snapshot was created.
Note
You may want to create copies of your running environment so that multiple people can be working with their own copies of a virtual environment. This is especially helpful when a tester finds a bug and wants to create a snapshot for the development team to use in diagnosing the problem.
To do this, the tester should shut down the environment after creating a bug with a snapshot. From the Lab Center
Once a copy of the environment has been stored in the SCVMM library, Figure 16-18 will include an option for the developer to connect to a copy of the environment from where the .lvr file was created.
You have now seen how you can take advantage of virtual environments when running manual tests. You can use a similar process for running manual tests that have associated automation (such as coded UI tests and unit tests). You can also run such tests as part of an automated end-to-end build-deploy-test workflow. You will learn how to configure this next.
The true power of Lab Management comes to life when combined with the automated build, deployment, and testing capabilities of Team Foundation Build. As new builds are produced by the development team, they can be automatically deployed into one or more virtual environments. A snapshot can be created from the environment, thus providing the testing team with a baseline for running any manual tests against an environment with that build. Then, any automated tests can be run automatically, thus providing valuable data about any possible regressions in your test plan. This entire workflow can take place without any manual intervention.
Team Foundation Build is covered in detail in Chapter 21, but this discussion will provide an overview of the settings used when configuring Team Foundation Build for use with a virtual environment. Certain steps within the Team Foundation Build configuration are omitted, since they are covered in Chapter 21. This example assumes that you have a virtual environment with both build agents and test agents installed and configured.
The first step in creating a build definition for use with Lab Management is to select LabDefaultTemplate.xaml as the Build process template. This is configured on the Process tab of your build definition. Selecting this template will change the Build process parameters to those shown in Figure 16-19. Next, you define the Lab Workflow Parameters by clicking on the ellipsis in the Workflow settings row.
As shown in Figure 16-20, the first page of the Lab Workflow Parameters wizard allows you to define which virtual environment should be used as part of your build workflow. You can choose to use an environment that is already running on a VM host group. Keep in mind that deploying a new environment can be a long-running operation and may take an hour or more to complete, whereas using a running lab environment will be much faster.
You can also choose to restore the environment to an environment snapshot prior to proceeding with the workflow. This is useful for establishing a clean baseline for your lab environment before attempting to install a new build or running any tests.
The Build page of the Lab Workflow Parameters wizard defines which build of your software should be used. You can rely on another build definition to create a new build, or you can select an existing build that was generated by another build definition. You can also point to a specific location where your software build resides, even if it wasn't created using Team Foundation Build.
As shown in Figure 16-21, the Deploy page of the Lab Workflow Parameters wizard allows you to specify how a build should be deployed within one or more VMs running in a virtual environment.
The grid allows you to define a sequence of workflow steps that should be executed in order during the build deployment phase. The first column specifies the name of the VM within the lab environment that defines where the given deployment step should be run. Note that this is not the computer name, but is the name of the VM which was provided when you configured the environment.
The second column specifies the command that should be run as part of that workflow step. This might include copying files to a Web server directory, running an .msi file, or even running a batch file. You can use the following variable names here to parameterize your commands.
$(BuildLocation)— This resolves to the location that your build is initially copied to by Team Foundation Build.$(InternalComputerName_VM-name)— This resolves to the hostname of the VM within the environment. For example, this macro would returnmywebserverfor a VM whose fully qualified domain name (FQDN) ismywebserver.contoso.com. To use this command, replaceVM-namewith the name of the VM as defined within your environment. This variable is especially useful when you don't always know the machine name of the VMs within your environment, but your deployment scripts rely on those names. As an example, you might need to update a configuration file in your Web application to use the machine name of the database server in your environment.$(ComputerName_VM-name)— This returns the FQDN of the VM within the environment. To use this command, replaceVM-namewith the name of the VM, as defined within your environment. Typically, the FQDN of a machine is a concatenation of its hostname and its domain suffix. As an example, the FQDN for a VM with a hostname ofmywebserverin thecontoso.comdomain would bemywebserver.contoso.com. Note that when using network isolation,$(InternalComputerName_VM-name)will be the same for a VM in each copy of a given virtual environment but its FQDN will be different. As an example, for a VM with hostnamemywebserverin a network isolated environment, this macro would returnVSLM_<uid>.contoso.comwhere<uid>is a unique alphanumeric identifier. This value can be important when using network isolation, where theInternalComputerNamewill be the same on each copy of a given virtual environment.
Finally, after deploying a build, you can create a new snapshot of the environment by enabling the bottom checkbox and providing a name with which to preface such snapshot names. This will then create new snapshots with names based on the build name and build number such as those in Figure 16-9.
The Tests page of the Lab Workflow Parameters wizard allows you to run any automated tests that you may have in your test plan. Your test cases will need to have associated automation (such as coded UI tests or unit tests). After builds are deployed, these tests will be run automatically, and the test results will be published to your test plan. You will need to specify automated test settings as defined earlier.
The focus of this chapter has been on virtual environments, and virtual environments offer many advantages over physical environments such as using snapshots, as well as better resource utilization. But there may be times when the tests you need to run can't take advantage of virtual environments, such as when your tests rely on specialized hardware that isn't supported by virtual environments.
A physical environment can be defined to run tests remotely and collect diagnostics data. The first step in defining a physical environment is to install a test agent on each machine that will make up your physical environment. Those test agents then must be registered with a test controller associated with your team project collection.
Next, you can define a physical environment from within Test Manager. Open Lab Center
Finally, you can define new test settings for your physical environment by following the same steps you did earlier in this chapter for a virtual environment. This allows you to define which data is collected when running your tests.
Then, run tests by specifying an environment like you did earlier in this chapter with a virtual environment. You won't be able to connect to your physical environments by using the Environment Viewer like you would with tests running in a virtual environment, but you can use a technology such as Remote Desktop to achieve similar results.
In this chapter, you have seen how Lab Management can be used to help create and manage VMs running within a virtual test lab. You learned how to create new environments and define which diagnostic data should be collected on various machines as tests are run on those environments. You learned the benefits of snapshots and how to work with them and share them among team members.
You also learned how an end-to-end workflow can be established to automatically build and deploy your software, then run automated tests within those virtual environments.
Finally, you learned about how physical environments can be used when virtual environments are not an option.
In the next several chapters, you will gain an in-depth understanding of Team Foundation Server, and the central role it plays when performing application lifecycle management with Visual Studio 2010. The examination begins with an overview of Team Foundation Server in Chapter 17.