More or less by chance I had an opportunity to attend the first day of gamescom this year, Europe’s biggest video game related trade show. The entertainment area where consumers get a glimpse of new games was a blast, as expected. But the business area open only to trade visitors was particularly interesting to me, because there were a fair amount of test companies present.

It seems as if the games industry on the whole takes their testing a lot more seriously than many other sectors of the software world. That shouldn’t be particular surprising, given that games are probably the most abundantly distributed shrink-wrap software still in existence. When you cannot rely on online updates being available to your customers, testing becomes a crucial part of your business — and the stakes are high in video gaming.

Continue reading »

Alright, that’s a provocative title right there, I admit it. I couldn’t resist.

A short while ago, I was perusing the July 2011 issue of the Communications of the ACM, when I came across an interview article with some of the people behind a massive retro-documentation project at Microsoft. Microsoft had to document much of the client-server communications of their existing software to allow third parties to implement interoperable software. The article is succinctly called “Microsoft’s Protocol Documentation Program: Interoperability Testing at Scale”. Definitely worth a read.

The sentence that struck me, and that is repeated in sentiment in the remainder of the article is this: “First and foremost, a team would be required to test documentation, not software, which is the inversion of the normal QA process; (…)”.

I’m not sure I agree with that statement. And to be fair, that statement isn’t the point of the CACM article, either. But it presents a good launch pad for something that concerns me a little.

Continue reading »

Here at spriteCloud we love cucumber. It’s a test framework for behaviour driven development (BDD), that is a development practice that includes testing during development.

BDD is slightly different from other test methodologies in that it’s designed to be used in cross-functional teams. In this post I will briefly touch on these differences, and then proceed to explain how you would change your approach to writing test code in accordance with the BDD philosophy with the help of an example.

The target audience of this blog post is test engineers first and product managers second. Note that I use these terms as roles rather than job descriptions; a test engineer is anyone writing test code, and a product manager is anyone thinking up features for the software. You could be both of them at once.

Continue reading »

Here at spriteCloud HQ, we all share the same background story. Details vary wildly, of course, but some or all of the following holds true for each one of us:

We’ve all worked at companies where software quality assurance didn’t exist. We’ve worked at companies where it didn’t exist, yet some people maintained it did. We’ve worked at companies where SQA was an afterthought, and consequently failed. We’ve worked at companies where that was taken as a reason not to bother with SQA. We’ve worked at companies where people didn’t even know what SQA was. We’ve worked at companies who said “our developers can do QA”, or “we get free QA from our beta users”, and whose software was then ripped apart by users for its bad performance or lack of stability.

We’ve also worked at companies where SQA was taken seriously and executed well, but for most of us those were few and far between.

That begs the question: why is that the case?

More precisely, why is it that a multi-billion dollar industry survives and thrives when it systematically neglects an aspect of product development that no other industry can afford to neglect?

Now let me qualify that previous statement: the big players, the likes of Amazon, Google, Facebook, etc. — they take SQA seriously. I would go as far and say that they are amongst the most powerful drivers of innovation in the field of SQA, both in terms of tools and processes.

With that said, maybe the above question needs to be rephrased: what is it that prevents so many smaller software forges from adopting SQA (properly)?

In obvious answer to that, that these companies lack experience with SQA, is also trite. Experience can be hired.

The real problem appears to be that SQA is not the clearly definable step in the software development life cycle that it appears to be at a glance. You’re all familiar with the waterfall model for software development, and with agile counterparts that iterate over most of the same steps indefinitely.

It’s easy to assume that one of the steps in that famous waterfall model, “verification”, is what SQA is — and that consequently it should happen after (an iteration of) development is done, and that’s the beginning and end of it.

Nothing could be further from the truth.

Properly conducted SQA is not so much verification of your product’s functionality, but verification that your software development process yields high quality at each step.

1. Requirements need to be verified for their suitability to your business. All too often, product requirements appear “magically” out of thin air, maybe copied from some vaguely related competing product. Those aren’t a bad start, mind you — but for the next iteration of your software, these requirements need to change to adapt to what you’ve learned with the previous iteration. SQA can help in that by making the effects of requirements measurable, and comparing those metrics as requirements are adapted.
2. Design needs to be verified; at Joost, back in the days when the company was providing a video desktop client, the button for sharing videos with your friends was changed from white to green in one release. Nothing else in the user interface changed. As a consequence, the number of users who used the button increased manifold. Providing metrics “before” and “after” (A/B testing) gives a sound basis for making design decisions.
   But of course this is not limited to visual design alone; in a similar fashion, metrics can verify that a redesigned algorithm increases the performance of your software, for example.
3. Implementation is the focus of most software testing, and what people usually think of as SQA. The primary focus here is to provide consistently repeatable tests in well-defined environments. Continuous integration provides one of the best methods for ensuring that software works as the developers intended.
4. Verification encompasses not only testing the implementation, but goes beyond that and includes testing that the software works as designed (whether or not the design is good; that question is answered above).
5. Maintenance is not a phase usually found in agile development models, as these methodologies tend to loop back to a new requirements phase once software is verified and published. Nonetheless with regards to SQA, a maintenance phase does exist and should exist in the form of regression testing: ensuring that future versions of the software do not break behaviour that was verified in previous versions.

As you can see, SQA touches on all areas of software development. And you can also see, that SQA is largely concerned with providing metrics in order to gain an insight into your product’s performance (in traditional quality management that’s closer to the realm of quality control).

The role of SQA, then, is not merely to ensure your product works. It is to ensure the business you are building around your product works the way you intended, by ensuring that your product’s target audience is satisfied.

It’s worth noting that because SQA and software product development are so closely intertwined, it follows that there is no silver bullet to SQA processes and tools, just like there is no silver bullet to a software development processes and tools. But as with development, there is a rich pool of techniques, technologies and experience to draw on.

And with that said, how can you afford to neglect SQA if you want your business to thrive?