Alright, that’s a provocative title right there, I admit it. I couldn’t resist.

A short while ago, I was perusing the July 2011 issue of the Communications of the ACM, when I came across an interview article with some of the people behind a massive retro-documentation project at Microsoft. Microsoft had to document much of the client-server communications of their existing software to allow third parties to implement interoperable software. The article is succinctly called “Microsoft’s Protocol Documentation Program: Interoperability Testing at Scale”. Definitely worth a read.

The sentence that struck me, and that is repeated in sentiment in the remainder of the article is this: “First and foremost, a team would be required to test documentation, not software, which is the inversion of the normal QA process; (…)”.

I’m not sure I agree with that statement. And to be fair, that statement isn’t the point of the CACM article, either. But it presents a good launch pad for something that concerns me a little.

Continue reading »

It feels like ages that I blogged about the place of QA in software development. Granted, in internet years, it has been a while. But the topic never seems to quite go away.

Since I wrote that other blog post, a lot of things have happened here at spriteCloud. In the meantime, we’ve been involved in the testing process with quite a few more customers (yay!). One thing that emerged over time is that many companies don’t exactly come to us for testing alone, but also for answering the question of how to fit testing into the development and release process.

When you try to answer that question, it quite often happens that you discover that the reason these companies didn’t know how to fit testing into their process is that there is not much of a process in place to fit testing into. To us QA-minded people, that may come as a surprise, but it really shouldn’t be.

As one young developer at one of our clients put it to me (slightly paraphrased): “They should really teach release management in university. Programming is easy enough to pick up, but this stuff is hard to figure out on your own.” I sympathize with that, as it mirrors my own experience from some ten years or so ago when I started out on my development career.

Before you can teach anything about development processes, though, it is essential that one understands the software lifecycle.

Continue reading »

Much of the software development in website development is done using agile methods. Automation is usually necessary to keep up with the speed.

It’s not really a question of whether automation is useful. Only a few people would rather repeat the same tasks many times every day then to let a computer do them. The question is whether the benefits are bigger then the costs. There will always be costs; automation is an investment. Businesses expect a return for their investment. There are several advantages:

• Requiring less manual work to reach the same test coverage
• Being able to use testing resources for higher value work improving the user experience
• Happier users -> more users

How to get started?

One of the critical parts in building test automation and automation suites is to have the right people to do it. There are other factors too, which I’ll briefly go through in this post. Tools, planning and working methods are other important factors.

Test automation suite?

What is it?

A test automation suite can be many things. It can and will evolve during its lifetime. The word test automation suite, at least for me, conveys a message of something massive, something that took a long time to build, covering the whole site functionality. That’s one definition, but there is no reason not to use a simpler, more agile approach.

A suite can be as small of automating a specific part of a site, for example, top navigation or a registration form. Anything, which needs a lot of steps to cover with every build, is a good candidate for automation. Can’t these things be tested manually then? Yes, but testers are humans too, you know. They will not be happy clicking the same 100 links or entering the 100 different email addresses required for good coverage. Does it matter that the tester is not happy about something. Well, that’s more of a HR question, but for me, it does matter. People need to be motivated to achieve the best results.

And the main reason for doing any testing is to achieve good quality. Saving the time from manually testing the forms and navigation will allow the tester to really do his/her job and concentrate on the more challenging test areas, for example, making sure the user experience of your site’s users is as good as possible.

And to be honest, no one will test all the links manually with every build. It’s only a matter of time before there is a serious problem when something important is missed in testing.

Okay, okay, we need automation. What next?

Requirements for successful automation project?

Here is a list of things that are important for a successful automation project.

1. Deep understanding of web technologies, HTML page elements. Mind you, this is hopefully something your testing team already understands.
2. Co-operation between development and QA teams in terms of communication about upcoming chances
3. Time. It will take some time to implement the automation. It will still be worth it, in many cases. Just remember to allocate some time for it. It’s important to keep working on automation iteratively, building more coverage as the site and the development work progresses. There will never be a two to four week period in the end of the project where you can just concentrate on building automation. Really. There isn’t.
4. Stability. If the site changes every day, it’s better to not start building an extensive suite. Again, start with the areas that are more stable. If not, you might face the chicken and the egg problem. You need a stable site to build automation but the site does not get stable when you don’t have it.
5. Iterative (agile) approach. Start from a single area in one browser. While doing this, you will see how well the automation works with your site. Once you are comfortable with the automation and how it works with your site, increase the coverage and expand to other browsers and platforms.
6. Tools. Use a tool set that’s easy to maintain. Use modern tools like Cucumber for writing the test cases and combine it with either Watir or Selenium. Whatever tool you use, make sure you can reuse your code. Cucumber is essentially a collection of navigation commands on your site combined with test cases that can be understood by other people, especially business analysts. They could even give you a hand in the test case creation by documenting the requirements in a format that can be used for test cases. Ideally, when using Cucumber, the requirements are the test cases.
7. Test automation is a tool to increase your test coverage, do not make it the software development project. You will not be allowed to keep working on the automation for months and months. While it is important to get the automation done properly, there is a risk that too much effort will be allocated for it. Like in any other form of testing, there is a trade-off between the time spent and the results achieved. 80-20 rule applies here too. Test automation is a tool to make your development project more successful in terms of quality and schedule. When you achieve that, you can be satisfied with the results.

The people

Can your average QA person do automation in a maintainable and reusable way? Most likely not. It requires a different skill set then manual testing. Or should I say, other skills in addition to the functional tester skills. A good solution is to have a team where some of the people are good in planning testing and test cases while others have the coding skills to maintain and build the test automation. If the team does not have the people with coding skills, most people start to automate testing with the help of a tool recording test cases. This will in most cases lead to major difficulties in maintaining the tests in the future. It can also be difficult to customize the tests for data-driven testing.

Required competencies

Automation requires development skills. You don’t need to be a high-level C++ programmer to create automation scripts. However, knowing how to implement code in a managed way will be necessary to get the automation done. And it is not only about completing the initial set of automation tests. The software/website you’re testing will typically receive updates every day/week. Some of those changes are going to break your test automation. Even if all the functionality on the site still works, some elements could have been moved or renamed causing the test automation not to find the element anymore. Elements on the page will need to be identified with the element attributes (for example, link text) or XPATH expressions, which are powerful but rigid. This is really the part where good understanding of HTML is required.

Software development practices

Maintaining any software, also test automation will benefit from good development process. One of the most important ones is using version control systems, for example, Subversion or git. It will enable maintaining the current version of the automation and tracking the changes done to it as well as branching your scripts for the current production system and the development version if there are major differences.

Development guidelines

Having a set of development guidelines to support the automation is needed to minimize unnecessary changes to the site. A big part of the technical implementation of test automation has to do with locating HTML elements on the page. It is good to establish naming conventions for the elements, and then stick to it. While it might be / is a good idea to fix naming of an element from something that’s obviously wrong, it can potentially cause test cases to fail. A typical case of this is could be something like renaming an element to something that’s clearer to other users:


<div>
  <ul class="top-nav">
    <li>Item 1</li>
    <li>Item 2</li>
  </ul>
</div>


Renaming the above unordered list would require the following change to an XPATH expression identifying the element:

//ul[@class='top-nav']  ->  //ul[@class='top-navigation']


While this change does make sense, it can easily cause automation tests to fail. The fact that test cases sometimes fail is not really a problem; it’s a fact of life that needs to be managed by allocating time for maintenance work.

Allocating time for automation

The project manager creates a plan for his project. The plan includes time for QA after the development work is completed.

Project of 6 months:

• 2 month for requirements / user stories
• 3 months for development
• 1 month for QA

Now, what happens when development time is increased by 30%. That’s not at all unrealistic, is it? Even with this relatively low delay percentage, the whole time to QA is basically eaten up by the development work. And there is of course a commitment for a customer to deliver on the agreed date. An additional complication is that there are most likely areas that have not been tested before the QA cycle is supposed to end.

The team will find defects on the last week of the project, things that should have been discovered 3 months earlier. Maybe there are areas that cannot be implemented at all with the current architecture.

This can, especially for a new team, lead to a situation where test automation is never taken into use in a proper way. There is not a lot of time, people are not familiar with the tools and enough progress is not made. In such a situation, management can easily loose faith in the usefulness of automation in general.

  spriteCloud Team

Recently I was at a company whose QA effort was done part time by a receptionist, typically just before the release went out. The senior project manager would get her to spend an hour clicking around a pre-production site to see if – in her estimation – it worked sufficiently well to go live to production. Of course the technical team would do QA work, but she was really the person responsible for the UAT effort.

Sounds like she does exploratory testing then and not really UAT, some of you would say.

And you would be right. Any professional tester worth their salt knows that exploratory testing is a valid and very worthwhile testing technique, but the effectiveness varies from individual to individual. This is why testing evolved from the early days of people doing random things in random ways, to later day written test sets which are repeatable, which in turn gave rise to predictable test coverage which could be expressed as a quantifiable number. Everybody loves statistics on your test effort expressed as: total tests run, tests passed, tests failed, critical errors found. Rather than, a gut feeling expressed as, we think it’s okay to go live!

But what’s the real problem here?

Well the problem that some of the company people expressed to me was that more and more bugs were going into production and people were getting unhappy. Rather important people up top, who were expressing their unhappiness by putting heat on the people below. Quite a lot of heat, as it turns out. So I was asked if there was anything I could do to help their receptionist do QA better?

What they were really asking me was, can a receptionist do QA? They didn’t have the money to take on additional resources, so they couldn’t get in a dedicated person to do the QA job. The short of it was, the receptionist was going to continue to do QA for the foreseeable future.

My view on it is, a receptionist is actually an ideal person to do QA when you really think about it. They perfectly represent the great unwashed masses out there in userland who visit consumer sites every day. They will interpret a UI in the same way, they will perform a sequence of actions in more or less the same way, and most importantly they’ll do random things in the same way that royally screw up a website.

This is perfect! Because I believe that QA is only a valuable activity when it’s focused on the consumer. Which means client side QA from a browser is the only kind of useful QA to a company. Why? Because it’s the consumer that determines whether or not your web service will live or die. If they like it, they’ll come back and more than likely bring others with them. If they don’t, they’ll never come back, and more than likely tell the world through conversations, emails, and blogs why your site is rubbish. So is there even a point of doing server side architecture testing? I’m going to really put the cat amongst the pidgeons and say, not that much. Web consumer service QA should face the browser and go forward from there. Look for the next post from me to explain why.

So the receptionist is an ideal person to perform QA for your technical team! The downside however is that when they try to report the error it’s likely to be a description along the lines of:

The site doesn’t work! I wanted to do something on the groups page (not sure which one, but it was definitely a groups page), and then I clicked a button, and then something happened and it stopped working.

Any developer getting this type of bug assigned to them will probably mark it as RESOLVED – WORKS FOR ME. After all, he knows how the groups feature works, and when he goes through the set of actions that any “reasonable” user should go through, it works. Or they’ll write a flaming comment designed to flay the skin off the original author for their epic fail in concise, accurate reporting. Regardless of which way they go, they could be overlooking an important bug that is a bug, but you have to be Average Joe to see it. (Devs in no way represent Average Joe – the user. They shouldn’t ever be allowed to think they can or should, or do. If they were Average Joe, they wouldn’t be working as high end web/internet developers).

Okay, so this is where some training can help. Anyone who gets sent to a beginners testing course will be taught how to write accurate bug reports. Really though, in a couple of hours you can give someone the necessary process for writing bug reports that devs can use in a meaningful way. It’s not rocket science, it’s just a set of principles based around the process of outlining

1. What you were trying achieve.

2. Where you started.

3. What you did.

4. Where you ended.

5. What happened in between as you made each specific action.

Put all this into a bug report without flowery language, and even the most bitterly cynical developer will gruffly acknowledge you did an adequate job describing the problem. Good devs will have enough to go on to track a problem down, without an endless exchange of probing questions designed to extract relevant information out of you, all of which costs time; more and more time for each message in the thread.

I’m hoping to get the opportunity to work with the receptionist in the near future so I can put her through a program that I believe will improve her overall productivity in measurable way, and that will take a couple of hours at most. I’ve never seen a completely non technical user do QA, so I’m looking at it as a challenge; with all the tools available in the professional web testers tool box, just a couple of them should be usable by someone with very few technical bones in their body. I believe they can be imbued with the mental process to learn the art and science of good QA and learn for themselves how to improve with each subsequent cycle they perform.

I think I’ll make a case study out of it as something interesting to discuss in the near future.

Andy.

Starting to test a project is always a challenge. You don’t necessarily know what to expect of the software. It is supposed to be almost ready, but the definition of ready varies a lot based on the individual developer. And not only of the developer. There are many external factors affecting the project and pushing things along. A project can have, and usually has, schedule pressures and any single person does not want to be the one to blame for the delays. For example, a release or iteration could have 20 new features, which are supposed to be completed on a specific date. The development team does not want to be the one delaying the project and releases the software to QA. As it turns out, only 10 of the features are really implemented and 5 actually work. This is a bit of an extreme scenario, but nonetheless one that many QA people have faced. If external factors have too much affect on the project, the software quality will be a bit of a hit and miss.

Even with all of the problems mentioned here, the initial period on working on the new software is very rewarding. Testers learn how the new software works and find plenty of defects. They look at the product with an open mind and get new ideas and approaches to testing it. A lot of progress can be made in a short time. When the work becomes more routine later on, many times there will also be less innovation.

It’s also a time when test cases evolve rapidly. The initial set is based on the specifications, which don’t necessarily completely reflect the real functionality. The test cases are going to be refined during the initial testing rounds. More test cases will also be added. Very rarely will the initial set of tests have full coverage for the software although it is possible in highly organized development team where quality is what matters most. In practice, most organization can not spend the time it takes to plan for full test coverage, and they probably should not either. Test planning is a continuous process, where both more details to the test cases for current functionality and more test cases for new functionality in new releases are added.

The fact that test cases are in practice a result of a long continuous process makes it even more important to keep track of the early test cases, which can be used to build upon. The cases might not be perfect, but will serve as a foundation for future work.

Marko

Late last week I had the pleasure of visiting a startup that is doing some very interesting things in the social network/online community web space. Like most 2.0 web development shops they consist of a small high performance technical team that work side-by-side a non technical team, each of whom all share equal responsibility for the production of their cloud service. So the developers cut the code that is the running web application, but the ideas that represent the cool innovation in what they are doing – and what gives the company its value – come from everywhere and anybody. As the discussions went on I became intrigued with how they managed all this on-the-fly creativity and how it was fed into the development machinery and eventually turned into production features on their website.

Essentially there was a minimal amount of process as “control” was seen as something that stifles creativity, and slows things down. Given that all the team members sat in the same office there wasn’t a problem with communicating issues so it was easy for them to identify and rectify issues that became problems, and problems that escalated in priority. So it was all good.

Mostly, but it wasn’t all roses and sunshine.

There was some discussions about how they were spending more time now fixing bugs that went into production than there were a few months ago, and the feeling was they didn’t want this trend to continue. In short, they felt the problem was with their QA process, as they all believed that good QA could solve this problem.

Well, yes and no, I thought; a skilled QA person would catch errors during a test cycle, but inevitably testing will only catch 85 – 90 percent of errors in any system. No system is ever error free when it’s put into production. This percentage however is affected by a lot of factors, such as the speed with which a release has to be done, how good the developers are, how conscientious they are with coding standards (which include things like commit reviews, unit testing, adhering to standard conventions), to name just a few. Good development practices always directly influence software quality. Or put another way, even the best QA person can not compensate for sloppy coding from a developer with a poor attitude.

But is there a middle way with process implementation that will support both high innovation and high quality?

I believe there is, and it’s exactly what AGILE tries to do. Remember that AGILE was born out of the need of web developers ten years ago to deliver web services that constantly kept changing from one week to the next. The old traditional methodologies based around waterfall and V model simply didn’t cut the mustard in the new landscape of the web. Instead of trying to control change – and the thing it represented, which was innovation – they embraced it. From that came the evolution of AGILE paradigm and quickly followed by frameworks that supported rapid development.

Really where things go wrong is not when a process isn’t adhered to, because that happens everywhere, but when control is lost. The whole – and arguably only – purpose of a process is to prevent loss of control, and thereby the onset of chaos. While some people I’ve worked with believe that there is such a thing as healthy controlled chaos, I don’t. Any type of chaos means you are not in control of what you’re doing, and at any point in time, inevitably, bad things will happen, and they will impact your ability to deliver work and meet deadlines.

Process really doesn’t have to be restrictive, not at all! It simply has to illustrate clearly how work flows and how problems are handled, both at individual, team and organisational levels. When everyone in a company has exactly the same understanding of who the internal customers are of their work, and how those customers use their work deliverables, then like a water park, everything is going to flow in predictable ways.

Good processes are continually changing as well. The nature of web development is one where what worked today is probably not going to work next month, let alone in six months, but this doesn’t mean your processes should be abandoned; when unpredictable things happen, situations change, or they stop working, then adapt them to something that does work. There isn’t anything wrong with that. In fact it if you’re not continually adjusting your processes they will eventually be ignored, which is worse than not having processes in the first place. Why? Because then it’s all just wasted effort.

In a shop that has adopted them properly, “process” should equal “ubiquitous repeatable success”, which is a very different thing from “stifling control”. Like everything else, it’s the attitude with which it is approached. But that’s something that is outside of the influence of anything other than the individual. Which is why it’s so important to get people with open minds into your teams, and remove those that do not. The quality and success of your software depends on it.

Andy.