SoftwareEngineering, Thoughts, Uncategorized

Using SpecFlow for WebDriver user acceptance tests - Pros and Cons

Recently in my team we have introduced the use of SpecFlow to held create business driven tests. Specifically we've been refining the way we work with acceptance criteria for some time, and have been pushing towards the Given, When, Then format. This being the obvious precursor for using tools to help the test generation process. There are some challenges in getting a business team to work with this structure, so initially my team has done the work of creating the feature files that contain the scenarios structured as Given When Then (GWT) sets. Ideally over time the business will see how they can write directly in this format and help accelerate the process.

However here in lies an issue... SpecFlow comes in the form of a plugin for Visual Studio, and it aids the writing of tests because it auto-completes based on existing statements in the code base. But the implication of that is having to have your business owners run a visual studio instance, with your code base, in order to make most effective use of the tool. Without that they would not get any of the assistance of existing GWT statements to draw upon. Perhaps there are tools out there to solve this problem, in truth I've not had time to investigate much, but it seems to be truly helpful I need someway of getting the business team able to generate the feature files, whilst getting all the assistance of existing options.

Undeterred by that limitation we cracked on with our test team creating tests this way. I let the team get started and build up a few areas of testing using specflow to generate nunit tests (we use nunit as the test runner for all our tests even though they aren't really unit tests)

This is where things started to look wrong to me... the first hint was trying to navigate from a feature file to the code that implements it, and more importantly, from a method decorated as a Given, When or Then, finding out which tests call it. I could not.

When I had the time I sat down to really understand how things were being integrated and to review the code being produced by my team, and slowly realisation dawned... SpecFlows nunit integration is really kinda hacky.

Why do I say that?, Well it seems that rather than really integrate with nunit, specflow just generates a test which invokes its own custom test runner. So every feature file winds up with a new instance of the spec flow test runner, and every test just invokes that test runner with some strings which are your GWT statements. So you have an nunit test runner, that performs fixture setup, and test setup, then you invoke the specflow test runner which had its own feature setup and test setup, then it invokes your GWT statements, and everything tears down again. It wasn't initially clear to me why specflow didn't just generate straight up c# nunit code with method invocations and associated method stubs waiting to be implemented (or linked to existing ones it can find).

It requires its test runner, because of the way they chose to find your GWT methods. rather than just do so directly to the method, they require that you decorate the method with a could of custom tags, so you end up with [Given ("Something I need")] public void SomethingINeed()

specflow uses this decorator to locate the method you want to call. The 'clever' bit is that it can find these methods in any class, anywhere in your dll completely regardless of the object hierarchy...Now maybe this is fine in the world of real unit tests, but four our code base this is decidedly not cool. That means each statement of a test might be located inside a separate class, in separate parts of a class hierarchy, and each one of them reflectively instantiates the object to call the method.

What is so bad about that? Well in our code base all of our tests inherit within a well defined 'tree' structure, the root of the tree contains all the logic necessary to establish a webdriver session, and anything very common, so all the helper methods likely to be used by every test. below that level are coarse functional level classes, these provide all the things that are quite specific to a functional area, it is unlikely a test of our communications needs access to methods to navigate around our public information pages. Beneath the coarse functional level are the specification specific helpers, things that tend to be very specific use cases and error scenarios associated to a particular specification. And at the leaf nodes we have the tests. Every test has access to every method it needs access to. Now maybe this is just a stupid structure, I'm not going to claim to being a great architect of software, however it works well for our needs and allows us to do some pretty nifty things (for instance a given test run of our 1700 tests reports ~30 hours worth of test nunit test results, but takes just 5.5hours to run...nifty). So in our structure the SpecFlow way of doing things causes each statement to potentially recreate and initialise the entire object tree multiple times including constructors that expect to only run once...

Our nifty test structure makes use of featuresetup to do things like create users for the tests (crazy!) but that relies on nunit feature setup methods being override-able by each feature within our structure and doing various stateful decidedly non-static things. But Specflow feature setup methods must be static, and it seems it had to create a special context class just for the purpose of overcoming the issue of randomly spawning objects using reflection, independent of each other. Basically just using specflow as it expects caused a massive integration headache for me, and the resulting tests performed horribly. It encouraged behaviours where we had nice granular methods to enter different fields on a form, but every one of those methods would reinitialise the webdriver page object! Just filling in a form could spend most of its time loading and reloading the same page for different parts of the same test, mostly because it wasn't clear how to retain interaction state between them.

Now it may seem that I'm pretty down on SpecFlow, but I'm not. I think the idea has a lot going for it, I just don't really agree with the way they've implemented nunit integration. If it were me, I would have generated tests with direct method calls, use the reflection cleverness to go find them, but don't break the object hierarchy, if a method isn't in the right object tree to be called from a test just indicate where there required method currently resides, and let the users refactor. no custom specflow runner to do runtime cleverness, do all of that at generation time and leave users with normal native c# nunit code. Effectively this is what we're now doing. We still write feature files, and those files still benefit from finding the GWT statement in the codebase. However we then take the generated code and turn every testrunner.given("Something i need"); and turn it into the direct call SomethingINeed();. We do all of this in a new class file so that spec flow doesn't just wipe our changes on the next regen. Then we just leave the specflow generated version of the file empty.

With our approach we get the benefits of a business readable feature file, and a clear link between this file and the tests which implement them. But we also get a clear test structure and the ability to use our various nifty optimisations that help us keep things running in a vaguely sane timeframe (I'm still not happy with 5.5 hours I'm sure we can do something about that)

Anyone else out there doing things with SpecFlow and WebDriver? Am I missing something? is my structure crazy and ill advised which leads to all my problems with SpecFlow?

Anyone running a similar amount of webdriver/selenium tests that are prepared to share their execution times/environment details?