How to get your remote Coded UI tests running using TFS on prem

We have some internal Coded UI tests for a solution that we have been running manually periodically as part of our development work. We were challenged to try and automate this process so that we could more reliably report of issues for with our UI. We found this a little difficult as there seemed to be many steps to get it up and running and documentation was scattered in different places. Our starting point was that we use Visual Studio 2015 as our main development tool. The code for our solution is checked into TFS on-prem and GitHub.

So, here's some steps that we took to try and get Coded UI or functional testing working with our solution.

Create your initial coded UI tests using Visual Studio.

The step is to create your UI tests. In our case, around 20 tests had already been written with Selenium in our solution and have been run manually to check they work.

Tests

Create your initial coded UI tests using Visual Studio. In this case, around 20 tests are written with Selenium and have been run manually to check they work.

Scripts

Create a script to tear down, drop and recreate databases

TFS Build

Create a TFS build with the following steps:
Build Steps

Steps we added to the build are as follows:

  • Powershell Step to run script that drops and recreate databases
  • Step that builds the Visual solution that contains the source code with the Coded UI tests
  • Copy Files Step that copies file (*\bin\$(BuildConfiguration)*) from $(build.sourcesdirectory) to $(build.artifactstagingdirectory) TFS Build Steps
  • Copy Files Step that copies the assembly that contains the Coded UI tests to the target test machines that will run the Coded UI tests
  • Powershell Script step that runs a script that removes the legal notice that will interfere with Coded UI tests
  • Deploy Test Agent step that installs Microsoft Visual Studio Test Agent 2013 onto the target test machines Run Functional Tests step that actually executes the Coded UI tests.

Gotchas

Ensure Deploy TestAgent step has correct options

As can be seen from this image the following key options are selected:

  • Protocol HTTP. We found that the HTTPS option resulted in some errors during the deployment process. If you are using an on-prem edition of TFS that is on the same internal network as you test machines, HTTPS is not required. We have set the step to us HTTP as it's simpler and less error prone. Obviously, if you are using VSTS, this is required. some further investigation is required.
  • Ensure Interactive Process is checked. This is so that the TestAgent deployment logs in with the required account and has access to the desktop. Here's some more information

Test agent options

Problems with legal notices

A legal notice is a start up message that is displayed as a reminder or any important message every time users login to a Windows computer. it is not always set, but if it is, it is configured in the registry. It tends to be set within organisations by Active Directory Group Policy using the Group Policy Editor for the domain. The group policy/registry changes end up adding values to this location in machine registry.

When the test agent runs the Coded UI tests, it needs to login to the machine and accesses the desktop with the configured user. It doesn't know however, how to deal with the legal notice that appears immediately after it logs in. The simplest solution we found is to remove the notice on the test machine. We use a remote PowerShell script that essentially removes the registry keys. This step is incorporated into the TFS build. There is no risk to anything else happening on the machine/domain as the test user created to run Coded-UI tests and developers are the only people who login to that machine. Here's some more information if you'd like to use Powershell to resolve this issue.

I found this link useful. Microsoft VSTS Tasks Github Rep

Disclaimer please consult with your risk and/or security teams if applicable (or think very carefully) before removing the legal notice. Making changes to the registry is obviously fraught with danger and reader need to ensure they know what they are doing before making such changes.

Here's another slightly different approach - autologon

Have a look at this post.

Which version of the test agent should be installed?

I found it confusing to understand which test agent to use for our solution. Eventually, when I read these links (from Microsoft) the confusion cleared somewhat:

Essentially for TFS 2017, building a solution that is developed using Visual Studio 2015, VS Test Agent for 2013 is most appropriate to use. I.e. this one: https://www.microsoft.com/en-us/download/details.aspx?id=48140

Is any other help available

I found this video on Channel 9 useful as it goes through the whole process end to end.

Troubleshooting
Videos

I found this link from Microsoft useful when investigating why tests didn't run as expected.

Test agent logs

The Visual Studio Test Agent that runs on the target machine generates logs as it runs. If the errors in TFS aren't detailed enough, these logs give more detail:

  • C:\DtaLogs\DtaExecutionHost.exe.log
  • %temp%\TestAgentConfig.Log
Security prompts in Internet Explorer (IE) blocking tests

Once all the above issues are cleared, you may now have an indication from the TFS build that the final Functional Test step is executing, but failing with an error similar to:

'threw exception: Microsoft.VisualStudio.TestTools.UITest.Extension.PlaybackFailureException: Cannot perform 'Click' on the control.'

This issue could be caused by IE security warnings/messages for the logged in test user. These prompt appear for normal human users, who would then click on the appropriate buttons to get rid of the dialogs. Here's an example:

However, the Functional Test step is unable to do this and therefore needs some help to workaround this issue. To do this:

  • login to the test machine with the test user account
  • open IE
  • once the message boxes/warning messages appear, click on the appropriate buttons. In the above example, you can click on either of the two buttons.
  • close the browser and logout.

Normally once this been completed once, the messages will not appear again.

Testing our your process

If all this has worked correctly and all steps in the TFS build are green, check the Tests area under the main TFS menu:

TFS menus

Here you should find a completed test run.

Selecting this should allow you to see the runs the coded UI tests. Double-select one of the items to see a breakdown of an individual test run.

Test run

You may want to run a single test to check everything's working as expected. To do this, use the test case filter. You can either set a particular test category attribute on the single test you want to run and have a filter like TestCategory="MyTestCategory". Alternatively, just use the single test name. In this case, you would have a filter like:

Name="NameOfMyCodedUITest".

If you want to have run two tests via the name filter, add this:

Name="NameOfMyCodedUITest|NameOfMyCodedUITest2".

I.e. something like this:

Test Filter Criteria

Output results

The output results are typically in a TRX file for test runs. This test run is available in the individual test run window in the Summary tab as an attachment:

Test run outputs

Select this file and download. Then double-clicking on this file allows the results to be viewed. By default, this file will be viewable in Visual Studio so ensure you have a viewer installed.

Test run output results

Other general principles

All credentials should be secured correctly in the TFS build

Secure usernames and passwords correctly using TFS secret variable values. Check out this post.

I hope all the information above helps those planning to setup coded UI tests. As specified in the title, this work was carried out specifically against TFS 2017 on-prem. The information may also be useful for other versions of TFS on-prem or VSTS, with some minor tweaks of course.

I've also got to thank those who went before me and struggled to get their remote Coded UI tests working correctly. Have a look at Stackoverflow.com and you will see a few questions that have been asked and answered.

The information on this site is intended to be helpful. We are however not liable if anyone should use any of the information in this blog and inadvertently damage theirs or anyone else's machines.