Hi All,

In this post we explain how to detect Javascript Errors in firefox using selenium, watir webdriver and cucumber. If you are unfamiliar with these frameworks you can read up on our previous post about Web testing with cucumber.

Get JSErrorCollector

First, you need to get JSErrorCollector.xpi which you can get here. Then, store it under features/support/extensions in your cucumber project location.

Add JSErrorCollector.xpi to your browser profile

require 'rubygems'
require 'selenium-webdriver'
require 'watir-webdriver'
require 'logger'
 
#Start the logger
$log = Logger.new('log/selenium.log')
#Create a profile
profile = Selenium::WebDriver::Firefox::Profile.new
 
#Add JS error detection to the profile!
profile.add_extension "features/support/extensions/JSErrorCollector.xpi" rescue p "Cannot add JSErrorCollector.xpi to profile"

Launch a browser using the profile you created

Now you can launch a new browser instance using:

$BROWSER = Watir::Browser.new 'firefox', :profile => $profile
$BROWSER.goto "http://www.spritecloud.com"

When the JSErrorCollector is installed successfully, you will see the error collector counter in the bottom right of the launched browser as displayed in the image below.

Collect the JS errors

First off, we create a separate method to get the JS errors formatted how you want to display them when one or more JS errors are detected.

#Function that returns a string that presents the details of the occurred JS errors
def get_js_error_feedback()
  jserror_descriptions = ""
  begin
    jserrors = $BROWSER.execute_script("return window.JSErrorCollector_errors.pump()")
    jserrors.each do |jserror|
      $log.debug "ERROR: JS error detected:\n#{jserror["errorMessage"]} (#{jserror["sourceName"]}:#{jserror["lineNumber"]})"
 
      jserror_descriptions += "JS error detected:
   #{jserror["errorMessage"]} (#{jserror["sourceName"]}:#{jserror["lineNumber"]})
"
    end
  rescue Exception => e
    $log.debug "Checking for JS errors failed with: #{e.message}"
  end
  jserror_descriptions
end

Raise exception when JS error is detected

When using cucumber, you can create a hook after every teststep. In this hook we raise an exception when one or more JS errors are detected.

AfterStep do |scenario|
  raise get_js_error_feedback() unless get_js_error_feedback().empty?
end

Note that this is a Collector, so at the end of each execution, the JS errors of all the visited pages are collected. For speeding up the tests, you can do the check after the end of each scenario (use hook AfterScenario) and cleanup the list after that.

Good Luck!

• Our customer base in the UK is growing. We’ve always operated internationally, but after the Netherlands, the UK has been the place we have been most active in. By opening an office here, we will be able to serve local demands much, much better.
• Requests for development consultancy have been growing. While we primarily offer quality assurance services, our skillset includes specialization in mobile software development, high scalability server architecture and process consulting, and increasingly customers have asked for our help in these areas. By starting a new business unit dedicated to these services, we’re improving our focus, and will be able to pursue these activities with the dedication they deserve.
• Synergy is not just a buzzword. When we previously spoke to customers about QA or development services, we did so in isolation, because we were thinking of ourselves as “a QA company that can do development”. Ironically, by giving full credit to each part separately, we’re in a much better position to offer the combined package to our customers.

You can reach the Cambridge office straight away on +44 (0) 1223 853863, or send an email to development@spritecloud.com. Our office/postal address is:

spriteCloud Ltd.
ideaSpace/Entrepreneurship Centre
3 Charles Babbage Road
Cambridge
CB3 0GT
United Kingdom

The rules are simple:

1. Spot any of us at FOSDEM, and take a picture as proof. For the purposes of this competition, the whole of Brussels counts as “at FOSDEM”, at least from 02/02/2012 to 05/02/2012 (CET timezone).
2. Send the image to fosdem@spritecloud.com. No entries arriving past the 29th of February 2012 will be accepted (CET timezone). Please include your name, email address, and whether or not you want either of them published. We reserve the right to publish your image submission anonymously, though.
3. We’ll judge which pictures are best. Bonus points if you include yourself, make it funny, or manage to connect it to a particular FOSS project, etc. Offensive submissions will be disqualified. We’re impervious to bribery, but we may take whatever freebies you offer anyway.
4. The three best entries get a guest blog post here, on any subject related to FOSS and/or testing or writing software. It goes without saying, really, but offensive content will be rejected even if it’s relevant.
5. The best entry also gets a free T-shirt.

We’d love to give away more swag, but we’re unprepared – this whole thing has been somewhat spontaneous. If it’s a success, we’ll do it again next year, but bigger. So be sure to participate!

P.S.: Here’s the boring legalese:

1. The contest is open to anyone except employees and the families of employees of spriteCloud B.V. By entering this contest, contestants agree that spriteCloud B.V. may use all images created for the contest for promotional purposes.
2. All entries and all rights relating there become property of spriteCloud B.V.
3. By entering, all entrants agree to release spriteCloud B.V. and their employees from all liability and claims relating to participation in the contest. No substitutions or transfer of prizes or cash redemptions permitted by winners.
4. spriteCloud B.V. reserves the right, in its sole discretion, to cancel, suspend or alter the rules & parameters of the contest. spriteCloud B.V. also reserves the right to substitute a prize or prizes as it deems suitable in the event that the prize is unavailable for any reason whatsoever.
5. Winners under the age of 18 (as of win announcement) must have their parents’ and/or legal guardians’ consent to accept prizes. Winners, by acceptance of prizes, agree to release spriteCloud B.V. from any and all liability claims or actions of any kind whatsoever for damages.
6. Offer void where prohibited, regulated or restricted by law in a manner inconsistent with the purposes and rules hereof.

Interviews at The Power of One (1 of 2) from BlueVia on Vimeo.

Jump to ca. 1:45 if you want to see my ugly mug. Unfortunately they didn’t show the main bits when I was talking about QA, and got my name wrong. But it’s all good in the end!

Here you can find the pictures of the event. The winner of the one and only ‘Website of the Year 2011′ Award was Tweakers.net.

Below you will find the results of all the sub categories:

Even though errors like this don’t occur that much anymore most Chrome and Safari users know that there are still quite some annoying bugs when surfing the internet.

The truth is in the numbers
Does this means that developers still can’t be bothered to do cross browser testing? Does it means that companies don’t demand that their web applications work on all browsers? The truth is in the numbers;

Source: StatCounter Global Stats – Browser Market Share

IE 6.0, 7.0, 8.0, 9.0, Firefox 3.6, 4.0, 3.5, 5.0, Chrome 12.0, 8.0, 10.0, Safari 5.0 and so on. What works great in IE 9.0 doesn’t automatically work in IE7. Ok. That shouldn’t be hard to find out.

Source: StatCounter Global Stats – Browser Version Market Share

The top 4 is: Chrome 13, IE 8.0, Firefox 6, IE9. Only thing is, they add up to about 70%.

One hundred percent
In testing 100% is impossible. For cross browser testing this is no different. Let’s say you want to test on the browsers used by at least 85% of all people. This means you’d have to have at least 12 dedicated systems running, and being maintained. Not everyone wants this, which is why we have it. As well as a hosted test environment. We have everything running here in our test lab. Almost everything. From IE x.x, Chrome x, Mozilla, and up

Mobile
When testing mobile web applications you need to be able to test on mobile handsets. We have a wide coverage of handsets for the big mobile operating systems popular today.

Pleased
Having all the handsets; hosted test environments; a test lab based on an official Microsoft Developers Network (MSDN) subscription are just some of the reasons why our customer base, including high profile Digital Agencies, are very pleased with the testing we currently perform for them.

At spriteCloud we are domain experts in AGILE software Quality Assurance and software testing, particularly for web consumer service companies and digital agencies.

It seems as if the games industry on the whole takes their testing a lot more seriously than many other sectors of the software world. That shouldn’t be particular surprising, given that games are probably the most abundantly distributed shrink-wrap software still in existence. When you cannot rely on online updates being available to your customers, testing becomes a crucial part of your business — and the stakes are high in video gaming.

What’s all the more interesting, then, is that so many of the testing companies I’ve seen don’t seem to have all that much of an expertise with testing networking related things. Few if any advertise test labs for massively multiplayer online games. I suppose this is only surprising to industry outsiders such as ourselves…

… however, it means we shouldn’t be industry outsiders. Normal web-related functional, performance and load testing aside, our founders are people who have helped build and test on-demand and live video streaming code at Joost. Online gaming has different demands on network performance than video streaming, granted, but in terms of the two major test variables in networking — bandwidth and latency — video streaming places some of the toughest demands on systems.

Chances are, if you’re an online game company caring about the performance of your servers, you’ll find few testing services providers better suited to your needs than us. So contact us, by all means!

(Also, when we find the time, we’re partial to some instance runs in LotRO. You can ping us just for that, too.)

A short while ago, I was perusing the July 2011 issue of the Communications of the ACM, when I came across an interview article with some of the people behind a massive retro-documentation project at Microsoft. Microsoft had to document much of the client-server communications of their existing software to allow third parties to implement interoperable software. The article is succinctly called “Microsoft’s Protocol Documentation Program: Interoperability Testing at Scale”. Definitely worth a read.

The sentence that struck me, and that is repeated in sentiment in the remainder of the article is this: “First and foremost, a team would be required to test documentation, not software, which is the inversion of the normal QA process; (…)”.

I’m not sure I agree with that statement. And to be fair, that statement isn’t the point of the CACM article, either. But it presents a good launch pad for something that concerns me a little.

What the Microsoft guy is effectively saying is that the normal QA process is to take some documentation and some software, test the software, and if it doesn’t match up with the documentation, then reject it. While that sounds like a good description of the QA process as it exists in many people’s mind, the implication of it must be that the software is an implementation of some strictly documented specification, and that in turn strikes me as unrealistic.

My experience is primarily as a software engineer, and while I have occasionally had to implement some RFC or similar specification, in the vast majority of cases, that’s not happening. In the rare cases that software is implemented from a specification, it almost invariably deviates from the specification, too — and often intentionally so. Think of how browsers deviate in how they render documents as an example; some of these deviations are due to the way specs are interpreted by different parties, but some are deliberate.

What happens instead is that software is vaguely specced out, written, unit tested (if you’re lucky), and if developers have a good enough incentive, documentation of what the software de-facto does is done last. Now whether or not that’s a good thing is up for debate, but in this post I don’t want to focus on passing judgement on existing software engineering practices.

Based on such experience, I think it’s a slightly dangerous misunderstanding of the QA process to assume that it treats documentation as the standard against which to test software. A more realistic and valuable understanding would be that the QA process attempts to detect where documentation and software behaviour are in conflict.

The rationale is centered around what in the scientific method is called confirmation bias; it describes the tendency of people to find evidence to support their hypothesis.

In terms of testing, confirmation bias is most easily evident when tests are written to only test known good parameters instead of also testing known bad combinations. If you take the example of testing a login form that require you to enter an email address and a password, the truly interesting tests are those that test without the email, or without the password, or with a string in the email field that doesn’t conform to email address syntax. It’s more obvious to confirm the hypothesis that the login form should work than to disprove it.

Testing a login form is a relatively simple example, though. Confirmation bias in testing can occur in more subtle forms. If you’re testing a network protocol, for example, a lot of the goings on on the wire will be defined as requests and responses of some sort. It is usually implied — and most likely documented — that each request to a machine should result in a response being sent back from that machine. What if the machine sends multiple responses? What if it sends a response packet without a request being made? What if a different machine responds than the one the request was sent to?

In many protocols, such behaviour is so outlandish that it’s not even documented that, for example, such extra responses should be discarded (as one may somewhat reasonably assume). Yet it’s the existence of such responses that can throw off server software quite badly. The problem here isn’t that documentation describes behaviour that deviates from what the software does, it’s that the documentation omits behaviour entirely. It’s entirely reasonable to assume that even seasoned QA engineers will therefore also omit testing this crucial aspect of the software and/or documentation.

Given all the above, I do agree strongly with the Microsoft team that a change of perspective on QA may be required and useful. Where they speak about a specific goal, however, I think the change in perspective should be more general. In order to avoid all of the consequences of confirmation bias, QA should not be viewed as testing whether software conforms to specification. Instead, that should be viewed as a rare case that applies only in the special circumstances when software is also implemented from such specification.

In the real world (to get back to the title), QA tests documentation, software, and the ability of engineers to fill the gaps left between the two alike.

Since I wrote that other blog post, a lot of things have happened here at spriteCloud. In the meantime, we’ve been involved in the testing process with quite a few more customers (yay!). One thing that emerged over time is that many companies don’t exactly come to us for testing alone, but also for answering the question of how to fit testing into the development and release process.

When you try to answer that question, it quite often happens that you discover that the reason these companies didn’t know how to fit testing into their process is that there is not much of a process in place to fit testing into. To us QA-minded people, that may come as a surprise, but it really shouldn’t be.

As one young developer at one of our clients put it to me (slightly paraphrased): “They should really teach release management in university. Programming is easy enough to pick up, but this stuff is hard to figure out on your own.” I sympathize with that, as it mirrors my own experience from some ten years or so ago when I started out on my development career.

Before you can teach anything about development processes, though, it is essential that one understands the software lifecycle.

Let’s start with the basics: it used to be taught (usually as a bad example) that the software lifecycle works a bit like a waterfall: Software is completely specified, before it is completely implemented, before it is completely tested, before it is released.

The model deliberately simplifies things to the point of being fantastically incorrect in describing the real world. It does, however, describe accurately that some tasks within the development process by necessity depend upon the completion of other tasks. Where it fails is that it assumes these tasks can be grouped into five completely separate stages. In this model, testing falls solely into the fourth stage, that of verification. (You can click on the images to enlarge them).

When agile development methodologies sprung up, they usually started by pointing out the flawed assumptions behind the waterfall model. The most glaring flaw to many is that software rarely goes into maintenance mode — usually it is developed further, and newer versions simply supersede older versions. That last stage of the waterfall model, then, should really be got rid of.

What happens instead is that a whole new design-build-release cycle gets started that works out bugs in the previous release and adds new features, all depending of course on the overall product plan. The best model of describing things, then, should circle back into itself, as any good lifecycle would.

One thing that’s slightly alarming to quality-minded people is that a “verification” stage is usually dropped from this model. Depending on which flavour of agile proponent you ask about this, they’ll either state that testing should drive development, or that testing is done in production by end-users. The latter group usually legitimize their stance by also demanding very short iterations on this circular model, such that end-users would not be inconvenienced by spurious bugs for a long time anyway.

There is an argument to be made for both, of course, but both miss critical points about testing:

1. Test-driven development misses the fact that testing isn’t just about testing code. Behaviour driven development is a much saner approach, as it focuses more on the end-user experience than anything else. As such, it implies that whoever thinks up what a piece of software should do should be involved in defining what behaviour needs to be tested for, which in larger teams means there’s now a dependency between several people’s deliverables.
2. On the other hand, production testing misses the fact that end-users aren’t necessarily product designers. Of course it’s important to take their feedback into account, but different users have different acceptance criteria. It requires careful design to figure out how to best serve the majority of users, and it’s that design that needs to be fulfilled, not the user’s expectations.

Put differently, you now need a separate stage in the process again at which acceptance test criteria are verified.

If you’ve followed my line of argumentation closely, you may have noticed that I’m edging towards a statement about testing that is often slightly misunderstood: of course testing serves to eliminate bugs, but when bugs come in the form of undesirable software behaviour rather than crashes or spurious errors, someone really needs to sit down and describe what the desired software behaviour should be. In other words, there is no testing without a plan. And plans, at least traditionally, happen close to the start of any project (or iteration).

Once you’ve internalized that, it becomes quite clear that putting a neat “verification” box into either the waterfall or agile model cannot really reflect the reality of the software lifecycle. So let’s draw a picture that’s closer to reality.

That looks a bit more daunting, doesn’t it? It should, because it gives up on the whole “stages” thing and breaks down the lifecycle into roles and the things these roles produce (lined arrows). Dotted arrows describe which artefacts serve as input into which other artefacts.

Once we started describing these roles and their interactions within the lifecycle to customers, things tended to become a lot clearer for them — they started seeing how some form of test planning needs to happen even before development starts properly.

But surprisingly, things became clearer to us, too. You see, the green blobs in the image cover spriteCloud’s core competencies. We’re a QA company, sure… but by being able to explain and conduct processes and best practices, we cover the release manager and (to an extent) lead developer roles as well, whether we plan to do so or not.

Put into numbers, we provide expertise for four out of eight distinct roles (50%) and produce six out of ten artefacts (60%). That’s over half of the stuff that happens in software development!

If we tried to push it a little, we could go even further: we all have, to a lesser or greater degree, software development experience. Writing test code is not at all the same as writing functional code, but it’s more experience than many newbie developers sport. We all have, to a lesser or greater degree, operations experience. Setting up test environments does that to you.

By contrast, many young companies out there have great product managers, UX designers and developers, but not much of the green glue that holds them together. No wonder do we often start talking about things other than testing!

There’s only one sensible conclusion from all this: we’ve decided that we’re “officially” shifting our company focus away from pure testing services, and now offer consultancy along the whole software lifecycle.