Loadtest summary shows many URLs for a single page

When you are running a load test against an MVC page you will see an enormous amount of different URLs reported. That’s because dynamic parameters such as IDs are frequently included in the URL of the page that is being accessed.

For example, here is an URL that will update an invoice whose ID is 129101 https://my-system.local/invoice/update/129101?SPHostUrl=..... If we update 5 different invoices, the load test summary will report 5 different URLS. That’s not useful at all.

The solution is to open the .webtest that sends the GET/POST requests and set the “Reporting Name” property. All the calls to that action will then be reported as one URL.There’s no need to have a different “Reporting Name” for the GET and POST requests to this URL, the load test summary will automatically report 2 separate summary lines for the URL.

How to get Selenium to run the browser as a different user

Selenium is great for testing web-apps. One of the challenges that you’ll eventually run into is needing to control which user is connecting to the system under test. Achieving this is far from intuitive. A search for this topic gives many links that don’t solve this problem:

Most of these are attempting to make use of Impersonation. This doesn’t work as the web-browser process ends-up running under your credentials not the credentials that you impersonated into. This is because Selenium’s code on Windows uses the CreateProcess() function to starts its sub-processes. The MSDN page on that says

“If the calling process is impersonating another user, the new process uses the token for the calling process, not the impersonation token”

How do we achieve our objective then? The answer is to start using RemoteWebDriver together with Selenium Grid. Lets assume we want to run testcases under your account and the account of a user I will call user1.

Start a Selenium hub on your local machine under your account.

Start a Selenium node on your local machine under your account.This node will run web-browsers using your credentials. The following command will start a node that can run Firefox and Internet Explorer on Windows: 

java 
-jar Selenium-server-standalone-x.xx.x.jar 
-role node 
-port 55565 
-hub "https://<yourmachine>:4444/grid/register" 
-browser "browserName=firefox" 
-browser "browserName=iexplore"

Start a Selenium node on another machine that is logged on as user1. This node will run web browsers as user1. Make sure that node is starting with different capabilities than your node. For example, here although we start the node on Windows, we manually overrule that and the node tells the hub that its running on Mac.

java 
-jar Selenium-server-standalone-x.xx.x.jar 
-role node 
-port 5556 
-hub "https://<yourmachine>:4444/grid/register" 
-browser "platform=MAC,browserName=firefox,maxinstances=12"

Instead of instantiating the actual implementations such as InternetExplorerDriver or FirefoxDriver, you need to instantiate a RemoteWebDriver. When instantiating the RemoteWebDriver, you can use the DesiredCapabilities object to determine which machine (and thus which user) Selenium will choose to run the browser on. For example:

public IWebDriver CreateNewBrowserFor(string Who)
{
    //Decide which of Selenium nodes we want to connect to 
    string CapabilitiesOfTargetUser;
    if(Who.Equals("user1"))
    {
        //We need to control a browser for user1 on his node
        CapabilitiesOfTargetUser = "platform=Mac;browserName=firefox";
    }
    else if(Who.Equals("me"))
    {
        //We need to control a browser for me on my node
        CapabilitiesOfTargetUser = "platform=WINDOWS";
    }
    else
    {
        throw new ArgumentException();
    }
    
    //Create a Selenium DesiredCapabilities object that contains our choosen capabilities
    Dictionary<string, object> RequestedCapabilities = new Dictionary<string, object> ();
    string[] CapabilitiesArray = CapabilitiesOfTargetUser.Split(';');
    foreach(string KeyValuePair in CapabilitiesArray)
    {
        string key = KeyValuePair.Split('=').First().Trim();
        string value = KeyValuePair.Split('=').Last().Trim();
        RequestedCapabilities[key] = value;
    }
    DesiredCapabilities Capabilities = new DesiredCapabilities(RequestedCapabilities);
    
    //Create the RemoteWebDriver. Selenium's hub will ensure that this RemoteWebDriver is
    //actually controlling a new browser on the correct machine
    return  new RemoteWebDriver
    (
         new Uri("https://localhost:4444/wd/hub")
       , Capabilities
       , new TimeSpan(0, 0, 50)
    );
}

Running your testcases concurrently

System- and integration tests can take a lot of time to complete. If you want to speed this up then you can choose to run multiple test cases at the same time in parallel.

Unit Testcases in Visual Studio


You can configure Visual Studio to run unit concurrent testcases on your local machine using the following procedure:

  1. Ensure your project is using a .testsettings file
  2. Open it in an XML editor (Notepad++ or visual Studio’s “Open With” command)
  3. Set the parallelTestCount attribute on the Execution entity some some integer value

This will only work for situations where all the following is true:

  • The testcases and related frameworks are thread-safe
  • The testcases are of type unit testcase. It really won’t work for coded-ui tests
  • Its NOT a Data-Driven Unit Test
  • There are 0 diagnostic adapters included in the .testsettings file
  • Your machine has multiple cores

Controller and Agents with Visual Studio


If you have enough Physical or Virtual Machines available, then its always possible to configure a Controller and multiple agents. That will allow you to distribute your testcases over the machines. There’s no limitation on the type of testcases here. The only thing to take into account that running testcases that interact with a User Interface will require specific configuration on part of the agent machines. I use this setup mainly for running a lot of performance-tests against my client’s SharePoint farm.

See Configuring Test Controllers and Test Agents for Load Testing for details.

SpecFlow / SpecRun


SpecRun is a test-runner from the makers of SpecFlow that will allow you to run many concurrent SpecFlow scenarios. Its easy to setup and doesn’t really have any significant limitations. I use a combination of SpecFlow, SpecRun and an on-premise Selenium Grid to run my 250+ automated regression tests.

Starting a process in PowerShell with dynamic command-line parameters

Starting a command-line process from PowerShell is very easy. A simple java -jar helloworld.jar works just fine. However when I’m starting nodes in my Selenium Grid I need to dynamically create a different number of parameters. The following code will fail because the various strings wont be correctly mapped to the usual argv[] input parameters for the java.exe process:

$arguments = ''
$arguments += ' ' + '-jar xxxx.jar'
$arguments += ' ' + '-browser'
$arguments += ' ' + 'browserName=firefox,version=3.6,platform=WINDOWS'
$arguments += ' ' + '-browser'
$arguments += ' ' + 'browserName=internet explorer,version=10,platform=WINDOWS'
java $arguments

Instead, just create an array of command-line arguments like this (assume that $Capabilities is an array of hash tables)

        $arguments = @()
        $arguments += '-jar'
        $arguments += $Jar
        $arguments += '-role'
        $arguments += 'node'
        $arguments += '-port'
        $arguments += 5555
        $arguments += '-hub'
        $arguments += '"' + 'https://127.0.0.1:4444/grid/register' + '"'

        foreach ($hashTable in $Capabilities)
        {
            $arguments += '-browser'
            $strCaps = ($hashTable.GetEnumerator() | ForEach-Object { '$($_.Key)=$($_.Value)' }) -join ','
            $strCaps = '"' + $strCaps + '"'
            $arguments += $strCaps
        }
        java $arguments

What to do when SpecRun doesn’t find any testcases to execute

So you’ve installed SpecRun, updated app.config, started a testrun….but SpecRun reports that it found 0 testcases. There’s two fixes for this.

Regenerate the feature files


Usually the *.feature.cs code-behinds aren’t removed when you clean the project. Those files still contain code pertaining to the previous test runner. You need to regenerate the .feature files in the solution before the code-behinds are updated to use the SpecRun test runner.

Clean Visual Studio’s cache for the test runners


Another cause for this unexpected behavior is due due to corruption in Visual Studio’s cache for the test runners. This can be fixed by the following:

  1. Close Visual Studio
  2. navigate to %TEMP%\VisualStudioTestExplorerExtensions\
  3. Delete any SpecRun related folders

How to get video recordings of your testcase

Recording the screen during test execution is very helpful in diagnosing issues. Here are a few ways you can achieve this.

Visual Studio’s Data Diagnostic Adapters

If you’re running your tests with Visual Studio and/or Test Controller/Agents, then you can add a .testsettings file to the test project and configure it to include the “Video Data Diagnostic Adapter” That will record the screen while each testcase is executing and save the results in the output directory specified in the .testsettings file. I find this to be very useful when I’m authoring new testcases on my machine or when I’m running through a Controller/Agent for an app that interacts with the desktop.

Selenium Grid Servlets

If you’re running on a Selenium Grid setup, then there are various ways of getting the nodes to record the screen. Take a look at Tuenti VideoRecorderService or Selenium Video Node.

Screen recording software

You can use some kind of screen recording software like Camstudio or VNC. For me this isn’t preferred as it requires manual action on behalf of the tester.

Single Sign On with Selenium and Firefox in a windows environment

When you run a testcase using Selenium WebDriver against a site like SharePoint, you’ll frequently see that Firefox is blocked waiting for you to enter the username and password of a user. This may seem unexpected because it even happens when your own Firefox window doesn’t need you to enter any credentials.

The reason for this is because the FirefoxDriver creates a new temporary profile and it doesn’t include the settings needed for SSO. There’s 2 ways of dealing with this.

The first way

The first way only works on the local machine that is running the tests. You can configure the FirefoxDriver to use an existing profile instead of creating a new one. The following code snippet shows how to do that in C#.

new FirefoxDriver(new FirefoxProfileManager().GetProfile("default"));

You don’t have to use the “default” profile. You’re free to create a dedicated profile for Selenium tests if you want. Just start Firefox from the command line using firefox -p to start Firefox’s Profile Manager.

The second way

The first way wont work if your tests use RemoteWebDriver to connect to a remote instance of FirefoxDriver. This is due to the fact that identically named profiles, still have a different internal name on each machine.

Instead we will configure Firefox to include the relevant SSO settings into every profile that’s created. You will need administrative credentials during this one-time configuration. Lets assume your site is called “https://myserver.contoso.local/sites/myapp&#8221;

  1. Open Windows Explorer and navigate to the Firefox installation directory. That’s usually “C:\Program Files (x86)\Mozilla Firefox”
  2. Create a file there (in this example, I used “mozilla.cfg”) with the following content: 
    //
lockPref("network.automatic-ntlm-auth.trusted-uris", "contoso.local");
  3. Use Windows Explorer to navigate to Firefox’s directory containing the preferences to be used for new profiles. That’s usually “C:\Program Files (x86)\Mozilla Firefox\defaults\pref”
  4. Create a file there called “local-settings.js” with the following content: 
    //@line 2 "c:\builds\moz2_slave\rel-m-esr31-w32_bld-0000000000\build\browser\app\profile\channel-prefs.js"
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at https://mozilla.org/MPL/2.0/. */
pref("general.config.obscure_value", 0);
pref("general.config.filename", "mozilla.cfg");

See Locking preferences in Firefox for details on this.

Fixing the slow combination of WebDriver and Internet Explorer.

Is your Selenium WebDriver running very slowly with Internet Explorer? Then you’re probably running a 64-bit IEDriverServer.exe with a 32 bit Internet Explorer. Even on 64-bit systems, Windows usually runs the 32-bit iexplore.exe.To solve the problem, just replace your IEDriverServer.exe with the 32 bit one and the speed increase will be enormous.

Describing large forms and data entry in Cucumber / SpecFlow feature files

Did you know that in .feature files you can use the table notation to specify each field and value in rows instead of columns?

This can make the file way more readable when a step needs a lot of different parameters. Here is an example that shows both ways of creating an invoice:

Feature: Demo

Scenario: TwoDifferentStylesOfSteps
	When I create an invoice with number '111' with '119.00' euro total, '19.00' euro VAT from 'ACME Inc' at '1234XYZ 99' in 'USA'
	When I create the following invoice:
		| Field            | Value      |
		| Number           | 222        |
		| TotalAmount      | 238.00     |
		| VATAmount        | 38.00      |
		| Debtor           | ACME Inc   |
		| DebtorAdresLine1 | 1234XYZ 99 |
		| DebtorCountry    | USA        |
	Then The systems invoice store must look like this:
		| Number | TotalAmount | VatAmount |
		| 111    | 119.00      | 19.00     |
		| 222    | 238.00      | 38.00     |

Here is the corresponding binding code:

namespace TableExample
{
    public class Invoice
    {
        public int Number { get; set; }
        public decimal TotalAmount { get; set; }
        public decimal VatAmount { get; set; }
        public string Debtor { get; set; }
        public string DebtorAdresLine1 { get; set; }
        public string DebtorAdresLine2 { get; set; }
        public string DebtorCountry { get; set; }
    }
  
    [Binding]
    public class TablesSteps
    {
        private IList<Invoice> Invoices;

        public TablesSteps()
        {
            this.Invoices = new List<Invoice>();
        }


        [When(@"I create an invoice with number '(.*)' with '(.*)' euro total, '(.*)' euro VAT from '(.*)' at '(.*)' in '(.*)'")]
        public void WhenICreateAnInvoiceWithNumberWithEuroTotalEuroVATFromAtIn(int Number, 
            decimal TotalAmount,
            decimal VatAmount,
            string Debtor,
            string DebtorLine1,
            string DebtorCountry)
        {
            Invoice TestInvoice = new Invoice();
            TestInvoice.Number = Number;
            TestInvoice.TotalAmount = TotalAmount;
            TestInvoice.VatAmount = VatAmount;
            TestInvoice.Debtor = Debtor;
            TestInvoice.DebtorAdresLine1 = DebtorLine1;
            TestInvoice.DebtorCountry = DebtorCountry;
            this.Invoices.Add(TestInvoice);
        }

        [When(@"I create the following invoice:")]
        public void WhenICreateTheFollowingInvoice(Table table)
        {
            Invoice TestInvoice = table.CreateInstance<Invoice>();
            this.Invoices.Add(TestInvoice);
        }

        [Then(@"The systems invoice store must look like this:")]
        public void ThenTheSystemsInvoiceStoreMustLookLike(Table table)
        {
            var rows = table.CreateSet<Invoice>();
            foreach (Invoice test in rows)
            {
                IEnumerable<Invoice> Matches = this.Invoices
                    .Where(invoice => invoice.Number.Equals(test.Number))
                    .Where(invoice => invoice.TotalAmount.Equals(test.TotalAmount))
                    .Where(invoice => invoice.VatAmount.Equals(test.VatAmount));
                Assert.AreEqual(1, Matches.Count(), string.Format("Invoice {0} not correct in invoicestore", test.Number));
            }
        }

    }

}