Alright, that’s a provocative title right there, I admit it. I couldn’t resist.

A short while ago, I was perusing the July 2011 issue of the Communications of the ACM, when I came across an interview article with some of the people behind a massive retro-documentation project at Microsoft. Microsoft had to document much of the client-server communications of their existing software to allow third parties to implement interoperable software. The article is succinctly called “Microsoft’s Protocol Documentation Program: Interoperability Testing at Scale”. Definitely worth a read.

The sentence that struck me, and that is repeated in sentiment in the remainder of the article is this: “First and foremost, a team would be required to test documentation, not software, which is the inversion of the normal QA process; (…)”.

I’m not sure I agree with that statement. And to be fair, that statement isn’t the point of the CACM article, either. But it presents a good launch pad for something that concerns me a little.

Continue reading »

It feels like ages that I blogged about the place of QA in software development. Granted, in internet years, it has been a while. But the topic never seems to quite go away.

Since I wrote that other blog post, a lot of things have happened here at spriteCloud. In the meantime, we’ve been involved in the testing process with quite a few more customers (yay!). One thing that emerged over time is that many companies don’t exactly come to us for testing alone, but also for answering the question of how to fit testing into the development and release process.

When you try to answer that question, it quite often happens that you discover that the reason these companies didn’t know how to fit testing into their process is that there is not much of a process in place to fit testing into. To us QA-minded people, that may come as a surprise, but it really shouldn’t be.

As one young developer at one of our clients put it to me (slightly paraphrased): “They should really teach release management in university. Programming is easy enough to pick up, but this stuff is hard to figure out on your own.” I sympathize with that, as it mirrors my own experience from some ten years or so ago when I started out on my development career.

Before you can teach anything about development processes, though, it is essential that one understands the software lifecycle.

Continue reading »

Here at spriteCloud HQ, we all share the same background story. Details vary wildly, of course, but some or all of the following holds true for each one of us:

We’ve all worked at companies where software quality assurance didn’t exist. We’ve worked at companies where it didn’t exist, yet some people maintained it did. We’ve worked at companies where SQA was an afterthought, and consequently failed. We’ve worked at companies where that was taken as a reason not to bother with SQA. We’ve worked at companies where people didn’t even know what SQA was. We’ve worked at companies who said “our developers can do QA”, or “we get free QA from our beta users”, and whose software was then ripped apart by users for its bad performance or lack of stability.

We’ve also worked at companies where SQA was taken seriously and executed well, but for most of us those were few and far between.

That begs the question: why is that the case?

More precisely, why is it that a multi-billion dollar industry survives and thrives when it systematically neglects an aspect of product development that no other industry can afford to neglect?

Now let me qualify that previous statement: the big players, the likes of Amazon, Google, Facebook, etc. — they take SQA seriously. I would go as far and say that they are amongst the most powerful drivers of innovation in the field of SQA, both in terms of tools and processes.

With that said, maybe the above question needs to be rephrased: what is it that prevents so many smaller software forges from adopting SQA (properly)?

In obvious answer to that, that these companies lack experience with SQA, is also trite. Experience can be hired.

The real problem appears to be that SQA is not the clearly definable step in the software development life cycle that it appears to be at a glance. You’re all familiar with the waterfall model for software development, and with agile counterparts that iterate over most of the same steps indefinitely.

It’s easy to assume that one of the steps in that famous waterfall model, “verification”, is what SQA is — and that consequently it should happen after (an iteration of) development is done, and that’s the beginning and end of it.

Nothing could be further from the truth.

Properly conducted SQA is not so much verification of your product’s functionality, but verification that your software development process yields high quality at each step.

1. Requirements need to be verified for their suitability to your business. All too often, product requirements appear “magically” out of thin air, maybe copied from some vaguely related competing product. Those aren’t a bad start, mind you — but for the next iteration of your software, these requirements need to change to adapt to what you’ve learned with the previous iteration. SQA can help in that by making the effects of requirements measurable, and comparing those metrics as requirements are adapted.
2. Design needs to be verified; at Joost, back in the days when the company was providing a video desktop client, the button for sharing videos with your friends was changed from white to green in one release. Nothing else in the user interface changed. As a consequence, the number of users who used the button increased manifold. Providing metrics “before” and “after” (A/B testing) gives a sound basis for making design decisions.
   But of course this is not limited to visual design alone; in a similar fashion, metrics can verify that a redesigned algorithm increases the performance of your software, for example.
3. Implementation is the focus of most software testing, and what people usually think of as SQA. The primary focus here is to provide consistently repeatable tests in well-defined environments. Continuous integration provides one of the best methods for ensuring that software works as the developers intended.
4. Verification encompasses not only testing the implementation, but goes beyond that and includes testing that the software works as designed (whether or not the design is good; that question is answered above).
5. Maintenance is not a phase usually found in agile development models, as these methodologies tend to loop back to a new requirements phase once software is verified and published. Nonetheless with regards to SQA, a maintenance phase does exist and should exist in the form of regression testing: ensuring that future versions of the software do not break behaviour that was verified in previous versions.

As you can see, SQA touches on all areas of software development. And you can also see, that SQA is largely concerned with providing metrics in order to gain an insight into your product’s performance (in traditional quality management that’s closer to the realm of quality control).

The role of SQA, then, is not merely to ensure your product works. It is to ensure the business you are building around your product works the way you intended, by ensuring that your product’s target audience is satisfied.

It’s worth noting that because SQA and software product development are so closely intertwined, it follows that there is no silver bullet to SQA processes and tools, just like there is no silver bullet to a software development processes and tools. But as with development, there is a rich pool of techniques, technologies and experience to draw on.

And with that said, how can you afford to neglect SQA if you want your business to thrive?

I got to thinking about this – again for the umpteenth time – during an informal lunch conversation with a group of developers and QA’ers discussing challenges in their jobs. This friend of mine, who is a very senior individual at his social network startup, was telling us a story about the problems that he has with a group of external developers who build their service. They tend to be a bit like a group of cowboys that shoot code from the hip without thinking through what they are doing, or with much regard for what effect this has on the production environment. I said to him that it sounded like a recipe for disaster and asked if this resulted in a lot of catastrophic failures like their site going down. Surprisingly he answered, not really.

Their situation is they don’t have a dedicated QA group or tester to run regression tests on builds that come out. This is done by someone on their management team who has an intimate knowledge of their front-end application, and he manages to catch the really bad problems before they make it to production. I asked if this was always the case, to which my friend said, not all the time but generally. He went on to relate to me that one of the good things about them being a cloud service was that they could patch problems on the fly if they had to.

I agreed, in quite a few of the places I worked in, patching-on-the-fly was a standard tactic for fixing post deployment problems. But it’s time consuming for all the individuals that are involved, which if it turns out to be a lengthy issue to fix means time off core development work. If you’re doing releases every week, then this can add up to a significant amount of hours, all of which has to be paid for, either in people having to do more hours to meet development schedules, or development schedules being pushed back. In a worst case scenario, it’s both. The end result, your service delivery starts slowing down.

I’ve mentioned nothing about the aggravation that your users experience, but this is not really as serious I don’t think. After all users are a bit like young kids, they are easily upset, but they are also easily placated. As long as you fix things in a hurry and don’t make them wait too long to be able to get back to doing what they were doing, they will forgive you. It will even give them something to twitter about. I survived when my social network service when down for two hours… I read a BOOK! Which can give you enough infamy to get your name out there if you’re new. To be fair though, there is a line-in-the-sand to a users acceptance of failure, it’s different for each service, but once you cross it, users will start to chant your name in angry denunciation. This is bad because it soon follows that they go somewhere else. Typically once you’ve burned a user badly enough, they are never coming back.

I don’t generally put user satisfaction into a QA benefit for the modern world anymore. Even the most technically minded, and thus socially unaware software developer, keenly understands that user satisfaction is everything. It’s the one true deciding factor in whether an online consumer service lives or dies. This doesn’t need to be explained, everyone just gets it. Time however is something else. While everyone in a development group understands that time is of critical importance, you’d be hard pressed to get agreement on what’s the best way to use it. There are hundreds of documented ways to structure a project to -1- organise teams, and -2- organise development activities to achieve milestones. But the goal is always one thing; to build an application that consumers can use – and someone can make money from – in the shortest possible time.

For development, the real benefit in QA is in making things happen in “the shortest possible time”!

There is a time hit involved with getting a bunch of QA/test people involved on a project as they need time to make test cycles on builds. But if they are involved early enough, it means that they catch the bad problems before they become really bad problems. The worst kind of problems are those that take your site down. Once that happens, every manager and his team gets involved to work the issue until it’s resolved. If you’re lucky it’s a few minutes, if not, then the hours start stacking up.

While there are models for software development that are based around code going straight from development to production, it’s a theory I’ve never seen work in the real world. That’s not to say it can’t, it’s just to say I’ve never seen it happen. Eventually code going from development directly to production breaks production and chaos ensues. People are pulled off their core work to fix issues. The end result being lost time.

The goal then for any QA group is to structure itself in the most effective way to facilitate the rapid flow of work from development and turn it around through a series of test activities (cycles) so it can be deployed to production in the shortest possible time. I’ll be talking about this in more detail the posts to come.

Andy.