It feels like ages that I blogged about the place of QA in software development. Granted, in internet years, it has been a while. But the topic never seems to quite go away.

Since I wrote that other blog post, a lot of things have happened here at spriteCloud. In the meantime, we’ve been involved in the testing process with quite a few more customers (yay!). One thing that emerged over time is that many companies don’t exactly come to us for testing alone, but also for answering the question of how to fit testing into the development and release process.

When you try to answer that question, it quite often happens that you discover that the reason these companies didn’t know how to fit testing into their process is that there is not much of a process in place to fit testing into. To us QA-minded people, that may come as a surprise, but it really shouldn’t be.

As one young developer at one of our clients put it to me (slightly paraphrased): “They should really teach release management in university. Programming is easy enough to pick up, but this stuff is hard to figure out on your own.” I sympathize with that, as it mirrors my own experience from some ten years or so ago when I started out on my development career.

Before you can teach anything about development processes, though, it is essential that one understands the software lifecycle.

Continue reading »

Recently I was at a company whose QA effort was done part time by a receptionist, typically just before the release went out. The senior project manager would get her to spend an hour clicking around a pre-production site to see if – in her estimation – it worked sufficiently well to go live to production. Of course the technical team would do QA work, but she was really the person responsible for the UAT effort.

Sounds like she does exploratory testing then and not really UAT, some of you would say.

And you would be right. Any professional tester worth their salt knows that exploratory testing is a valid and very worthwhile testing technique, but the effectiveness varies from individual to individual. This is why testing evolved from the early days of people doing random things in random ways, to later day written test sets which are repeatable, which in turn gave rise to predictable test coverage which could be expressed as a quantifiable number. Everybody loves statistics on your test effort expressed as: total tests run, tests passed, tests failed, critical errors found. Rather than, a gut feeling expressed as, we think it’s okay to go live!

But what’s the real problem here?

Well the problem that some of the company people expressed to me was that more and more bugs were going into production and people were getting unhappy. Rather important people up top, who were expressing their unhappiness by putting heat on the people below. Quite a lot of heat, as it turns out. So I was asked if there was anything I could do to help their receptionist do QA better?

What they were really asking me was, can a receptionist do QA? They didn’t have the money to take on additional resources, so they couldn’t get in a dedicated person to do the QA job. The short of it was, the receptionist was going to continue to do QA for the foreseeable future.

My view on it is, a receptionist is actually an ideal person to do QA when you really think about it. They perfectly represent the great unwashed masses out there in userland who visit consumer sites every day. They will interpret a UI in the same way, they will perform a sequence of actions in more or less the same way, and most importantly they’ll do random things in the same way that royally screw up a website.

This is perfect! Because I believe that QA is only a valuable activity when it’s focused on the consumer. Which means client side QA from a browser is the only kind of useful QA to a company. Why? Because it’s the consumer that determines whether or not your web service will live or die. If they like it, they’ll come back and more than likely bring others with them. If they don’t, they’ll never come back, and more than likely tell the world through conversations, emails, and blogs why your site is rubbish. So is there even a point of doing server side architecture testing? I’m going to really put the cat amongst the pidgeons and say, not that much. Web consumer service QA should face the browser and go forward from there. Look for the next post from me to explain why.

So the receptionist is an ideal person to perform QA for your technical team! The downside however is that when they try to report the error it’s likely to be a description along the lines of:

The site doesn’t work! I wanted to do something on the groups page (not sure which one, but it was definitely a groups page), and then I clicked a button, and then something happened and it stopped working.

Any developer getting this type of bug assigned to them will probably mark it as RESOLVED – WORKS FOR ME. After all, he knows how the groups feature works, and when he goes through the set of actions that any “reasonable” user should go through, it works. Or they’ll write a flaming comment designed to flay the skin off the original author for their epic fail in concise, accurate reporting. Regardless of which way they go, they could be overlooking an important bug that is a bug, but you have to be Average Joe to see it. (Devs in no way represent Average Joe – the user. They shouldn’t ever be allowed to think they can or should, or do. If they were Average Joe, they wouldn’t be working as high end web/internet developers).

Okay, so this is where some training can help. Anyone who gets sent to a beginners testing course will be taught how to write accurate bug reports. Really though, in a couple of hours you can give someone the necessary process for writing bug reports that devs can use in a meaningful way. It’s not rocket science, it’s just a set of principles based around the process of outlining

1. What you were trying achieve.

2. Where you started.

3. What you did.

4. Where you ended.

5. What happened in between as you made each specific action.

Put all this into a bug report without flowery language, and even the most bitterly cynical developer will gruffly acknowledge you did an adequate job describing the problem. Good devs will have enough to go on to track a problem down, without an endless exchange of probing questions designed to extract relevant information out of you, all of which costs time; more and more time for each message in the thread.

I’m hoping to get the opportunity to work with the receptionist in the near future so I can put her through a program that I believe will improve her overall productivity in measurable way, and that will take a couple of hours at most. I’ve never seen a completely non technical user do QA, so I’m looking at it as a challenge; with all the tools available in the professional web testers tool box, just a couple of them should be usable by someone with very few technical bones in their body. I believe they can be imbued with the mental process to learn the art and science of good QA and learn for themselves how to improve with each subsequent cycle they perform.

I think I’ll make a case study out of it as something interesting to discuss in the near future.

Andy.

Agile testing methods

Test driven development

Behaviour driven development

Cucumber

Feature

Feature: Website Navigation
  In order to navigate website pages
  I need to be able to click on menu item

Scenario

Scenario: Going to page Services
  Given that I am on spriteCloud Home
  When I click on link Services
  Then the page title should be "spriteCloud - Services"

Step definitions

require 'spec'
require "watir-webdriver"

SITE = "www.spritecloud.com"
BROWSER = Watir::Browser.start(SITE, :firefox)
PAGES = {
  "spriteCloud Home" => "http://www.spritecloud.com/"
}

The hash PAGES is not really necessary in this case, but can be useful in a case when there is more data and pages involved. It is used for looking up a page URL based on page name in the scenarios.

Given /^that I am on (.*)$/ do |page|
  BROWSER.goto(PAGES[page])
end

Notice that the step syntax is formatted as a regular expression. Given is a keyword but everything else is basically just something that Cucumber can recognize as being part of your test. Let’s break that part down a bit:

• Sentence starts with “Given” keyword. This is mandatory
• What follows is a space and the words “that I am on”
• Another regular expression (.*), which matches everything from “on ” until the end of the line “storing” it for later use for the variable page

Using the capturing group (.*) is important because it allows us to write more generic steps, where for example, the page where we are or the link that we are following is defined in the scenario, not in the step. This again allows for the less technical person to write tests without having to write Ruby code. It is of course also better programming practice not to write the same things multiple times and just changing a value.

When /^I click on link (.*)$/ do |link|
  BROWSER.link(:text, link).click
end

Then /^the page title should be "([^\"]*)"$/ do |title|
  BROWSER.title.eql?(title).should == true
end

Test Execution

This type of test can be run either during the implementation phase, which is the original purpose or later on when the site is fully ready. Even if the test was created during the implementation phase, it should still be run with every new release that goes out. It can even be used as a part of a stand-alone test set for the site.

Part of the power of Cucumber is the nicely formatted output displaying immediately the successful and the failed tests and test steps. Successful tests are colored green on the terminal and failed one, you guessed it, red.

mbp-2:features marko$ cucumber .
Feature: Website Navigation
	In order to navigate website pages
	I need to be able to click on navigation

  Scenario: Going to page Services                        
    Given that I am on spriteCloud Home                   
    When I click on link Services                        
    Then the page title should be "spriteCloud - Services"

1 scenario (1 passed)
3 steps (3 passed)
0m1.261s

Summary

This post only scratches the surface of what can be done with either Watir or Cucumber. They are both powerful test tools by themselves and combining them can be the basis for a successful test automation project. It all depends on the needs in your project.

Marko

Starting to test a project is always a challenge. You don’t necessarily know what to expect of the software. It is supposed to be almost ready, but the definition of ready varies a lot based on the individual developer. And not only of the developer. There are many external factors affecting the project and pushing things along. A project can have, and usually has, schedule pressures and any single person does not want to be the one to blame for the delays. For example, a release or iteration could have 20 new features, which are supposed to be completed on a specific date. The development team does not want to be the one delaying the project and releases the software to QA. As it turns out, only 10 of the features are really implemented and 5 actually work. This is a bit of an extreme scenario, but nonetheless one that many QA people have faced. If external factors have too much affect on the project, the software quality will be a bit of a hit and miss.

Even with all of the problems mentioned here, the initial period on working on the new software is very rewarding. Testers learn how the new software works and find plenty of defects. They look at the product with an open mind and get new ideas and approaches to testing it. A lot of progress can be made in a short time. When the work becomes more routine later on, many times there will also be less innovation.

It’s also a time when test cases evolve rapidly. The initial set is based on the specifications, which don’t necessarily completely reflect the real functionality. The test cases are going to be refined during the initial testing rounds. More test cases will also be added. Very rarely will the initial set of tests have full coverage for the software although it is possible in highly organized development team where quality is what matters most. In practice, most organization can not spend the time it takes to plan for full test coverage, and they probably should not either. Test planning is a continuous process, where both more details to the test cases for current functionality and more test cases for new functionality in new releases are added.

The fact that test cases are in practice a result of a long continuous process makes it even more important to keep track of the early test cases, which can be used to build upon. The cases might not be perfect, but will serve as a foundation for future work.

Marko

Late last week I had the pleasure of visiting a startup that is doing some very interesting things in the social network/online community web space. Like most 2.0 web development shops they consist of a small high performance technical team that work side-by-side a non technical team, each of whom all share equal responsibility for the production of their cloud service. So the developers cut the code that is the running web application, but the ideas that represent the cool innovation in what they are doing – and what gives the company its value – come from everywhere and anybody. As the discussions went on I became intrigued with how they managed all this on-the-fly creativity and how it was fed into the development machinery and eventually turned into production features on their website.

Essentially there was a minimal amount of process as “control” was seen as something that stifles creativity, and slows things down. Given that all the team members sat in the same office there wasn’t a problem with communicating issues so it was easy for them to identify and rectify issues that became problems, and problems that escalated in priority. So it was all good.

Mostly, but it wasn’t all roses and sunshine.

There was some discussions about how they were spending more time now fixing bugs that went into production than there were a few months ago, and the feeling was they didn’t want this trend to continue. In short, they felt the problem was with their QA process, as they all believed that good QA could solve this problem.

Well, yes and no, I thought; a skilled QA person would catch errors during a test cycle, but inevitably testing will only catch 85 – 90 percent of errors in any system. No system is ever error free when it’s put into production. This percentage however is affected by a lot of factors, such as the speed with which a release has to be done, how good the developers are, how conscientious they are with coding standards (which include things like commit reviews, unit testing, adhering to standard conventions), to name just a few. Good development practices always directly influence software quality. Or put another way, even the best QA person can not compensate for sloppy coding from a developer with a poor attitude.

But is there a middle way with process implementation that will support both high innovation and high quality?

I believe there is, and it’s exactly what AGILE tries to do. Remember that AGILE was born out of the need of web developers ten years ago to deliver web services that constantly kept changing from one week to the next. The old traditional methodologies based around waterfall and V model simply didn’t cut the mustard in the new landscape of the web. Instead of trying to control change – and the thing it represented, which was innovation – they embraced it. From that came the evolution of AGILE paradigm and quickly followed by frameworks that supported rapid development.

Really where things go wrong is not when a process isn’t adhered to, because that happens everywhere, but when control is lost. The whole – and arguably only – purpose of a process is to prevent loss of control, and thereby the onset of chaos. While some people I’ve worked with believe that there is such a thing as healthy controlled chaos, I don’t. Any type of chaos means you are not in control of what you’re doing, and at any point in time, inevitably, bad things will happen, and they will impact your ability to deliver work and meet deadlines.

Process really doesn’t have to be restrictive, not at all! It simply has to illustrate clearly how work flows and how problems are handled, both at individual, team and organisational levels. When everyone in a company has exactly the same understanding of who the internal customers are of their work, and how those customers use their work deliverables, then like a water park, everything is going to flow in predictable ways.

Good processes are continually changing as well. The nature of web development is one where what worked today is probably not going to work next month, let alone in six months, but this doesn’t mean your processes should be abandoned; when unpredictable things happen, situations change, or they stop working, then adapt them to something that does work. There isn’t anything wrong with that. In fact it if you’re not continually adjusting your processes they will eventually be ignored, which is worse than not having processes in the first place. Why? Because then it’s all just wasted effort.

In a shop that has adopted them properly, “process” should equal “ubiquitous repeatable success”, which is a very different thing from “stifling control”. Like everything else, it’s the attitude with which it is approached. But that’s something that is outside of the influence of anything other than the individual. Which is why it’s so important to get people with open minds into your teams, and remove those that do not. The quality and success of your software depends on it.

Andy.