Evaluating the success of test automation is a lot like pagan mystic voodoo; there is a lot of activity involved, much chanting and ranting; and a fair amount of ritual, with the end result being completely valid, or highly dubious.

We here would all believe that there is value in automation, otherwise we wouldn’t be part of this group (let alone reading these very wordy forum posts of mine), but how do we know what that value is? Can it be stated in some kind of metrics that make sense to the world at large, or is it still mainly anecdotal?

I have often had to face this scenario of explaining why as a test manager I’ve put resources from my group into test automation. There are the obvious answers such as “it saves us time in regression testing basic scenarios that are manually intensive”. Login and registration functionality being a fine example. But can this be equated in some way with time invested up front VS time saved per test cycle where the automation is used.

Then of course there is the indirect value to consider. Test automation doesn’t find defects in the first instance, that is up to the test techniques that are all manually performed by a tester. So automation isn’t something that contributes immediate value, but value that starts after a certain period of time on a software project. Test automation scripts degrade over time as well, hence they require maintenance. If the maintenance phase is shortened or skipped entirely because of project delivery deadlines, then over time the automation scripts become useless. Hence more time is required to keep scripts valid with every release.

So it seems to me that some form of metric analysis could be performed, but would it really be useful? Is the value of test automation really something that can be quantified, or is it destined to be like Zen; you either get it or you don’t?

Thoughts anyone?

Andy.

How come Ruby ?

 
There are several important Ruby projects, which are also increasing the popularity of the Ruby language in general:

http://www.ruby-lang.org/en/libraries/top-projects/

The most interesting projects from my (and this blog posts) point of view are Watir and Ruby on Rails, the former from a testers point of view, the latter from web developers.

Good projects don’t just come from nowhere, either. The language itself is very elegant and enjoyable to use. Having a programming language that works for you instead of against you is going to leave more time for innovation and thinking about the next big features for your product. The Ruby community is very active and provides very nice modules (=gems), which provide a specific functionality and usually do it very well.

What is Watir?

 
Watir stands for Web Application Testing in Ruby, http://watir.com. It’s a framework, which drives the browser programmatically. Currently the officially supported browsers are IE and Firefox, but gems also exists for Chrome and Safari. Also Opera is working on a port of their own, which they are using in their own development (http://my.opera.com/core/blog/2009/03/06/test-automation-with-operawatir). Unfortunately, it is not available for outsiders yet, which makes guaranteeing websites work on Opera more difficult than it should be.

What about Ruby on Rails?

 
The single most important reason for Ruby gaining a lot of attraction in the recent years is the Ruby on Rails web development framework, http://rubyonrails.org. The framework enables quicker web development with it’s inbuilt support for database access. It’s actually not just support, but ActiveRecord is one of the fundamental technologies behind RoR and makes it especially suited for database driven applications.

How is that related to testing?

 
Watir is great for web automation, and we have used it in extensive automation projects. There is lots of documentation available for the basic usage, but as always, best practices for implementing automation test projects in different types of situations are more difficult to find and require some trial and error to find the best approach.

Typically, one of the main points to consider is how easy the test cases will be to maintain. Writing one or ten test cases is easy without any structure, but scaling the automation suite for hundreds or thousands of test cases requires careful planning. Many of the same rules apply as for any software development project. As most of the testers do not have a developer background, this can be a challenge in the beginning.

There’s is nothing stopping testers taking advantage of the already well thought out features of Ruby on Rails what comes to database access (ActiveRecord) and for sending email (ActionMailer). Database access can be used to verify that a form actually posted the values and they were stored correctly in the database. Email can be, for example, used for sending test reports directly from test automation. There are clear synergies in understanding both of these technologies when aiming the create good test automation projects. And when using the already available gems for the tasks, you’ll save time for actual automation work.

We will cover several of these topics in a series of blog posts.

 
Marko