Before I tidy up everything and publish the source code for the Silverlight Unit Test runner, I thought I'd let you all in on the secret behind it and how it actually works.
The runner is a plugin built for the ReSharper plugin for Visual Studio (a plugin within a plugin – cool). Whenever you open the unit test explorer or do a change and recompile your code, the ReSharper mechanism is to call all plugins and let them gather information about the compiled assemblies and decide if it is a test assembly and then gather the test classes/fixtures and methods for ReSharper. Once this information is gathered, one needs to provide rendering facilities for ReSharper to render the tests in the unit test explorer or other windows it find appropriate.
Then, when you choose to run your tests (either a selection, or all), the plugin starts its trip into some really strange hoops. First we start by copying a specialized test execution engine written in Silverlight and a testpage for it to the destination folder of the test project, typically bindebug under the test projects directory. Then the plugin instantiates Cassini and sets its root path to point to the same location as mentioned before. A windows form is then created with a browser plugin within it and the URL given is the TestPage for the test execution engine previously copied with an additional request string following it with information about what tests to run. The test execution engine hooks up a LogProvider for the test framework by Jeff Willcox and then dynamically loads the XAP for the real test project written by the user and creates a visual for this to be used as the visual root for the application. Now, the tests will run and give feedback to the LogProvider which gathers all the results into an XML. After completing running all the tests, the XML with all the results in it is added to a programatically created DIV object into the DOM of the enclosing testpage.
Back in the plugin we are "listening" to the page and look into the DOM to see if the DIV appears, once this appear we have our result. We parse this result and convert it into ReSharper results and send it back to an internal manager for the plugin. Once ReSharper asks for the result on a callback for all the methods, we ask the manager what the result is and then hand this over.
My original idea was to host a WCF service inside the plugin and let the test execution engine in Silverlight contact the WCF service with results async. This could possibly give a better user experience and give test results as the tests were run. But I guess for now, the way it is implemented works out fine.
Hi Einar!
This is a clever approach, and actually a little bit similar to what Jeff is working on for vNext of the testing framework. I think he’s thinking about a Python launcher with an in-memory web server for communication from the running test in the browser and the Python launcher that grabs the results. So hosting a WCF service in the test launcher and use some WCF REST calls to report progress might be a good idea.
Previously I have blogged about using WatiN to drive IE and the Silverlight Testing Framework to run Silverlight tests from NUnit (and hence being able to integrate it into a CI process). Do you think this approach would be easily transferable to running tests from the command line and in a CI process? (http://jonas.follesoe.no/Silverlight2ContinuousIntegrationTestingUsingWatiN.aspx)
Also, Jamie Cansdale have reported that he have been able to run Silverlight Unit tests through NUnit and TestDriven.Net with out the need for a browser. He basically hosts the Silverlight CLR and executes the test. Similar to what Blend 2.5 does to get design time support. This off course puts some limitations on what you can do in terms of networking and browser interaction, but is interesting nevertheless. Is this something you concider including? Or would this just "work" since R# is able to run NUnit template tests as well? (I’m not familiar with the test runner in R#)
Cheers,
Jonas Follesø
Hi Einar!
This is a clever approach, and actually a little bit similar to what Jeff is working on for vNext of the testing framework. I think he’s thinking about a Python launcher with an in-memory web server for communication from the running test in the browser and the Python launcher that grabs the results. So hosting a WCF service in the test launcher and use some WCF REST calls to report progress might be a good idea.
Previously I have blogged about using WatiN to drive IE and the Silverlight Testing Framework to run Silverlight tests from NUnit (and hence being able to integrate it into a CI process). Do you think this approach would be easily transferable to running tests from the command line and in a CI process? (http://jonas.follesoe.no/Silverlight2ContinuousIntegrationTestingUsingWatiN.aspx)
Also, Jamie Cansdale have reported that he have been able to run Silverlight Unit tests through NUnit and TestDriven.Net with out the need for a browser. He basically hosts the Silverlight CLR and executes the test. Similar to what Blend 2.5 does to get design time support. This off course puts some limitations on what you can do in terms of networking and browser interaction, but is interesting nevertheless. Is this something you concider including? Or would this just "work" since R# is able to run NUnit template tests as well? (I’m not familiar with the test runner in R#)
Cheers,
Jonas Follesø
Hi Einar
This is good stuff! I’m going to check it out when I got resharper installed again. Keep up the good work 🙂
Cheers
Gøran
Hi Einar
This is good stuff! I’m going to check it out when I got resharper installed again. Keep up the good work 🙂
Cheers
Gøran
Hi Jonas,
sorry for the reallllllly delayed answer to your comment.
Thanks for your comment and input.
I loved your WatiN post. I wasn’t aware of that framework.
As far as I can see, using the WatiN and the way I’ve done it would be transferable to command line or as a build task for any build system.
The way I started of my project was actually with the idea of having several runner types were R# is just one of them. In a quest to just get things up and running I took a couple of short cuts though, but will be refactoring my code back to the original idea. If I have the time I also want to enable running unit tests from within Visual Studio without the assistance of an already existing plugin. And I want to create a MSBuild and Nant task for it as well.
Hi Jonas,
sorry for the reallllllly delayed answer to your comment.
Thanks for your comment and input.
I loved your WatiN post. I wasn’t aware of that framework.
As far as I can see, using the WatiN and the way I’ve done it would be transferable to command line or as a build task for any build system.
The way I started of my project was actually with the idea of having several runner types were R# is just one of them. In a quest to just get things up and running I took a couple of short cuts though, but will be refactoring my code back to the original idea. If I have the time I also want to enable running unit tests from within Visual Studio without the assistance of an already existing plugin. And I want to create a MSBuild and Nant task for it as well.