Late last week I had the pleasure of visiting a startup that is doing some very interesting things in the social network/online community web space. Like most 2.0 web development shops they consist of a small high performance technical team that work side-by-side a non technical team, each of whom all share equal responsibility for the production of their cloud service. So the developers cut the code that is the running web application, but the ideas that represent the cool innovation in what they are doing – and what gives the company its value – come from everywhere and anybody. As the discussions went on I became intrigued with how they managed all this on-the-fly creativity and how it was fed into the development machinery and eventually turned into production features on their website.

Essentially there was a minimal amount of process as “control” was seen as something that stifles creativity, and slows things down. Given that all the team members sat in the same office there wasn’t a problem with communicating issues so it was easy for them to identify and rectify issues that became problems, and problems that escalated in priority. So it was all good.

Mostly, but it wasn’t all roses and sunshine.

There was some discussions about how they were spending more time now fixing bugs that went into production than there were a few months ago, and the feeling was they didn’t want this trend to continue. In short, they felt the problem was with their QA process, as they all believed that good QA could solve this problem.

Well, yes and no, I thought; a skilled QA person would catch errors during a test cycle, but inevitably testing will only catch 85 – 90 percent of errors in any system. No system is ever error free when it’s put into production. This percentage however is affected by a lot of factors, such as the speed with which a release has to be done, how good the developers are, how conscientious they are with coding standards (which include things like commit reviews, unit testing, adhering to standard conventions), to name just a few. Good development practices always directly influence software quality. Or put another way, even the best QA person can not compensate for sloppy coding from a developer with a poor attitude.

But is there a middle way with process implementation that will support both high innovation and high quality?

I believe there is, and it’s exactly what AGILE tries to do. Remember that AGILE was born out of the need of web developers ten years ago to deliver web services that constantly kept changing from one week to the next. The old traditional methodologies based around waterfall and V model simply didn’t cut the mustard in the new landscape of the web. Instead of trying to control change – and the thing it represented, which was innovation – they embraced it. From that came the evolution of AGILE paradigm and quickly followed by frameworks that supported rapid development.

Really where things go wrong is not when a process isn’t adhered to, because that happens everywhere, but when control is lost. The whole – and arguably only – purpose of a process is to prevent loss of control, and thereby the onset of chaos. While some people I’ve worked with believe that there is such a thing as healthy controlled chaos, I don’t. Any type of chaos means you are not in control of what you’re doing, and at any point in time, inevitably, bad things will happen, and they will impact your ability to deliver work and meet deadlines.

Process really doesn’t have to be restrictive, not at all! It simply has to illustrate clearly how work flows and how problems are handled, both at individual, team and organisational levels. When everyone in a company has exactly the same understanding of who the internal customers are of their work, and how those customers use their work deliverables, then like a water park, everything is going to flow in predictable ways.

Good processes are continually changing as well. The nature of web development is one where what worked today is probably not going to work next month, let alone in six months, but this doesn’t mean your processes should be abandoned; when unpredictable things happen, situations change, or they stop working, then adapt them to something that does work. There isn’t anything wrong with that. In fact it if you’re not continually adjusting your processes they will eventually be ignored, which is worse than not having processes in the first place. Why? Because then it’s all just wasted effort.

In a shop that has adopted them properly, “process” should equal “ubiquitous repeatable success”, which is a very different thing from “stifling control”. Like everything else, it’s the attitude with which it is approached. But that’s something that is outside of the influence of anything other than the individual. Which is why it’s so important to get people with open minds into your teams, and remove those that do not. The quality and success of your software depends on it.

Andy.

There are a good number of different join models with Rails, some more complicated than others.
One of my favorite is has_many :through, it lets you join two models together using a separate join table which can also hold extra attributes for that join.

So… a release has many test cases, but a test case has many releases. A simple HABTM join would almost do here, except, when a test case is part of a release it should have a result (PASS, FAIL, NOT_RUN etc). So a join table is needed to store these extra attributes.
We also want a test case to have the result of NOT_RUN when its add to a release.

So diving into the models while keeping an eye on the diagram above

class Release < ActiveRecord::Base
has_many :cases, :through=>runs
has_many :runs
end

class Case < ActiveRecord::Base
has_many :releases, :through=>runs
has_many :runs
end

class Runs < ActiveRecord::Base
belongs_to :case
belongs_to :release
end

Now for the migrations.
The interesting one is the ‘runs’ table. It contains an identifier to Release and another
one to Case, as well as a third column to store the result. The Release and Case objects dont need any extra identifiers, as its all managed in the Run table.

class SetItUp < ActiveRecord::Migration
def self.up
create_table :releases do |t|
t.string :version
end

create_table :cases do |t|
t.string :title
t.string :steps
t.string :expected
end

create_table :runs do |t|
t.integer :release_id
t.integer :case_id
t.string :result
end
end

def self.down
drop_table :releases
drop_table :cases
drop_table :runs
end
end

Next on the agenda – using them in the console. First lets create a release and two test cases, c1 and c2.

>> r = Release.create(:version=>1)
>> c1 = Case.create(:title=>’A Title’, :steps=>’A step’, :expected=>’Expected Stuff’)
>> c2 = Case.create(:title=>’Another Title’, :steps=>’Another step’, :expected=>’Expected Stuff’)

There are two ways to add a case to a release now. First lets use the << operator, which will automatically create the association between the release and the case.

>> r.cases << c1

Now if you look at the runs table in mysql, you will see release_id and case_id are both filled but the result is empty. Not exactly what we’re after.

+—-+————+———+——–+———————+———————+
| id | release_id | case_id | result | created_at | updated_at |
+—-+————+———+——–+———————+———————+
| 1 | 1 | 1 | NULL | 2009-08-16 15:51:34 | 2009-08-16 15:51:34 |
+—-+————+———+——–+———————+———————+

The next option is to explicitly set the release, the case and the result in the join.

>> Run.create(:release => r, :case => c, :result=>’NOT_RUN’)

Now if we take a look, we can see all the fields we want are filled! Great, now we can go on with testing our release.

+—-+————+———+——–+———————+———————+
| id | release_id | case_id | result | created_at | updated_at |
+—-+————+———+——–+———————+———————+
| 2 | 1 | 2 | NOT_RUN| 2009-08-16 15:53:15 | 2009-08-16 15:53:15 |
+—-+————+———+——–+———————+———————+

However, when we want to change the result in the Run table, we will have to reference the Run object directly.

>> Run.find(1).update_attributes(:result=>’PASS’)

I got to thinking about this – again for the umpteenth time – during an informal lunch conversation with a group of developers and QA’ers discussing challenges in their jobs. This friend of mine, who is a very senior individual at his social network startup, was telling us a story about the problems that he has with a group of external developers who build their service. They tend to be a bit like a group of cowboys that shoot code from the hip without thinking through what they are doing, or with much regard for what effect this has on the production environment. I said to him that it sounded like a recipe for disaster and asked if this resulted in a lot of catastrophic failures like their site going down. Surprisingly he answered, not really.

Their situation is they don’t have a dedicated QA group or tester to run regression tests on builds that come out. This is done by someone on their management team who has an intimate knowledge of their front-end application, and he manages to catch the really bad problems before they make it to production. I asked if this was always the case, to which my friend said, not all the time but generally. He went on to relate to me that one of the good things about them being a cloud service was that they could patch problems on the fly if they had to.

I agreed, in quite a few of the places I worked in, patching-on-the-fly was a standard tactic for fixing post deployment problems. But it’s time consuming for all the individuals that are involved, which if it turns out to be a lengthy issue to fix means time off core development work. If you’re doing releases every week, then this can add up to a significant amount of hours, all of which has to be paid for, either in people having to do more hours to meet development schedules, or development schedules being pushed back. In a worst case scenario, it’s both. The end result, your service delivery starts slowing down.

I’ve mentioned nothing about the aggravation that your users experience, but this is not really as serious I don’t think. After all users are a bit like young kids, they are easily upset, but they are also easily placated. As long as you fix things in a hurry and don’t make them wait too long to be able to get back to doing what they were doing, they will forgive you. It will even give them something to twitter about. I survived when my social network service when down for two hours… I read a BOOK! Which can give you enough infamy to get your name out there if you’re new. To be fair though, there is a line-in-the-sand to a users acceptance of failure, it’s different for each service, but once you cross it, users will start to chant your name in angry denunciation. This is bad because it soon follows that they go somewhere else. Typically once you’ve burned a user badly enough, they are never coming back.

I don’t generally put user satisfaction into a QA benefit for the modern world anymore. Even the most technically minded, and thus socially unaware software developer, keenly understands that user satisfaction is everything. It’s the one true deciding factor in whether an online consumer service lives or dies. This doesn’t need to be explained, everyone just gets it. Time however is something else. While everyone in a development group understands that time is of critical importance, you’d be hard pressed to get agreement on what’s the best way to use it. There are hundreds of documented ways to structure a project to -1- organise teams, and -2- organise development activities to achieve milestones. But the goal is always one thing; to build an application that consumers can use – and someone can make money from – in the shortest possible time.

For development, the real benefit in QA is in making things happen in “the shortest possible time”!

There is a time hit involved with getting a bunch of QA/test people involved on a project as they need time to make test cycles on builds. But if they are involved early enough, it means that they catch the bad problems before they become really bad problems. The worst kind of problems are those that take your site down. Once that happens, every manager and his team gets involved to work the issue until it’s resolved. If you’re lucky it’s a few minutes, if not, then the hours start stacking up.

While there are models for software development that are based around code going straight from development to production, it’s a theory I’ve never seen work in the real world. That’s not to say it can’t, it’s just to say I’ve never seen it happen. Eventually code going from development directly to production breaks production and chaos ensues. People are pulled off their core work to fix issues. The end result being lost time.

The goal then for any QA group is to structure itself in the most effective way to facilitate the rapid flow of work from development and turn it around through a series of test activities (cycles) so it can be deployed to production in the shortest possible time. I’ll be talking about this in more detail the posts to come.

Andy.