When you are using watir-webdriver for web testautomation, you might encounter problems that are not be easy to debug. For example interaction with page elements such as links or buttons that are hidden. Or locating elements in other iframes. One easy way to see what is going on is by interacting with your web browser through the command line!

To make this happen, make sure you have;

1. a recent version of ruby installed (1.9.3 or later)
2. a recent version of firefox installed
3. the ruby gem ‘watir-webdriver’ installed;

gem install 'watir-webdriver'

Now, start the Interactive Ruby Shell…
In this irb, after loading ‘watir-webdriver’, you can start firefox with one command and interact with it right away!

sC-MBP:~ sCblogger$ irb
irb(main):001:0> require 'watir-webdriver'
=> true
irb(main):002:0> b = Watir::Browser.new
=> #
irb(main):003:0> b.goto "www.google.com"
=> "http://www.google.nl/"
irb(main):004:0> b.button(:text => /Google zoeken/).id
=> "gbqfba"
irb(main):005:0>

NOTE: with ruby 1.8.7 or earlier, you’d have to require ‘rubygems’ first before require ‘watir-webdriver’

The following post makes recommendations for platforms, operating systems and browsers to use when testing commercial marketing web sites aimed at the general public in Europe.

Our recommendations are based on usage figures widely available on the Internet, our experience and analysis of clients needs. Since modern browser versions change frequently, we review and update these recommendations regularly.

Assumptions

• For the majority of users, Chrome and Firefox silently auto-update in the background.
• Each new version cannibalises the market share of previous versions quickly, i.e. within a quarter.
• Significant differences between versions are few and far between.

List of latest stable releases

Desktop browsers

• Microsoft Internet Explorer (MS-IE) 10 for Windows 7 released to general public (February 26, 2013)
• Chrome Stable release 25.0 (February 21, 2013)
• Firefox (FF) Stable release 19.0 (February 19, 2013)
• Opera Stable release 12.14 (February 5, 2013)
• Safari Stable release 6.1.2 (February 19, 2013)

Mobile and tablet browsers

• Android 4.2.2 “Jelly Bean” (February 11, 2013)
• iOS 6.1.2 (February 19, 2013)
• Windows Phone 8.0.10211.204 (Portico) (after October 29, 2012)

Recommendations

General

• Invest in the future, not the past. Focus project time and effort on optimising web sites for future-facing operating system and browser combinations while making sure that popular older combinations are still viewable.
  • We define ‘the future’ as the next 3-6 months, i.e. 2 x internet lifetimes of three months each.
• Use a range of mainstream devices to ensure coverage of most regular users everyday experience.
• Avoid testing with specific minor releases unless there are compelling reasons to do so.
  • Example of a compelling reason: a third-party augmented reality (AR) component is known to work with Safari 6.0.1 but not with Safari 6.0.2.
• Do not support a browser if it is not supported by the manufacturer.
  • E.g. Mozilla discontinued support for Firefox 3.6 on 24 April 2012. Firefox 3.6 is therefore defunct.
• Do not support beta-release operating systems and browsers. Allow them time to be released and become mainstream first.
  • We define ‘mainstream’ as above 5% market share.

PC platform recommendations

• Test with Windows XP.
• Test with Windows Vista.
• Test with Windows 7.
• Do not test with Windows 8 – too early to adopt and not yet mainstream
• Do not test with Linux. Market penetration is too low.

• Do not test with MS-IE 6. The version is defunct plus market penetration is low and getting smaller.
  • Exception: Web sites targeted at China.
• Do not test with MS-IE 7. The version is practically defunct plus market penetration is low and getting smaller.
• Test with MS-IE 8.
• Test with MS-IE 9.
• Do not test with MS-IE 10 – too early to adopt and not yet mainstream.
• Test with Chrome latest release only.
• Test with Firefox (FF) latest release only.
• Do not test with Safari on the PC platform. The PC version is defunct plus market penetration is low and getting smaller.
• Do not test with Opera. Market penetration is too low.
  • Exception: Web sites targeted at Eastern Europe, especially Belarus.

Mac platform recommendations

• Test with OS/X 10.6 ‘Snow Leopard’ using Safari 5.1.7.
• Test with OS/X 10.7 ‘Lion’ using the latest stable release of Safari.
• Test with OS/X 10.8 ‘Mountain Lion’ using the latest stable release of Safari.

• Test with the latest release of OS/X using Firefox (FF) latest release only.
• Test with the latest release of OS/X using Chrome latest release only.
• Do not test with Opera on the Mac platform. Market penetration is too low.

Mobile and tablet platform recommendations

• Test in both landscape & portrait modes.
• Test with the latest release of iOS 6 using the native Safari browser only.
• Test with a version of iOS 5 using the native Safari browser only.

• Test with Android 2.3 ‘Gingerbread’, using the native browser only.
• Test with Android 3.x ‘Honeycomb’, tablets only, using the native browser only.
• Test with Android 4.0 ‘Ice Cream Sandwich’, using the native browser only.
• Test with Android 4.1 ‘Jelly Bean’, using the native browser only.
• Do not test with Android 4.2 ‘Jelly Bean’ – too early to adopt and not yet mainstream.

• Do not test with Windows Phone 7.5. Market penetration is too small and will decline further.
• Do not test with Windows Phone 8 – too early to adopt and not yet mainstream.
• Do not test with Blackberry. Market penetration is too small.
  • Do not test with Blackberry 10 (Initial release: January 30, 2013) – too early to adopt and not yet mainstream.
• Do not test with Firefox OS smartphones and tablet computers – too early to adopt and not yet mainstream.
• Do not test with Firefox, Chrome, Opera or any other non-native third-party mobile browser variants on any mobile or tablet platform. Market penetration is too small.

Flash recommendations

• Test with no Flash. At the very least, ‘no Flash’ should prompt the site visitor to download Flash.
• Test with Flash 10.3.
• Test with Flash 11.x (latest version).

Sources

• https://blog.mozilla.org/futurereleases/2012/03/23/upcoming-firefox-support-changes/
• http://www.ie6countdown.com/
• http://en.wikipedia.org/wiki/Usage_share_of_web_browsers
• http://en.wikipedia.org/wiki/Android_version_history
• http://developer.android.com/about/dashboards/index.html
• http://www.adobe.com/products/player_census/flashplayer/enterprise_penetration.html
• http://www.statowl.com
• spriteCloud staff

071-ICT is the platform for the Leiden ICT sector and ICT professionals. 071-IT organizes free network meetings in Leiden four times a year. The network has been active for several years on the road, but an official status was never reached, until now.

Jeroen Monkhorst (director of Spreadit online results), the first president of the new foundation. Mark Barzilay (co-founder of spriteCloud), takes the treasurer and Natascha from Duuren (IP / IT lawyer and partner), by De Clercq Lawyers Notaries is the secretary of the foundation. Monkhorst: “The 071-ICT Foundation has set itself the goal of stimulating innovation, cooperation and knowledge in the region ‘Rijnland’. 071-ICT actively promotes the sharing of the Leiden ICT knowledge and experience through regular network meetings and other knowledge moments”.

The entire board consists of 071-ICT consists of:
Jeroen Monk Horst (Spreadit online results)
Mark Barzilay (spriteCloud)
Natascha van Duuren (De Clercq lawyers and notaries)
Martijn van Pelt (Interpulse)
Erik Knol (Qeam)

From left to right: Jeroen Monkhorst, Martijn van Pelt, Janbert Heemstra, Natascha van Duuren, Mark Barzilay (afwezig Erik Knol).

Hi Again,

In this post we would like to explain a bit more about the mobile test automation that we are using here at spriteCloud. The past years we have seen that the mix of cucumber with selenium webdriver has been extremely powerful. This setup allows for test automation that is, besides easy to setup and maintain once you get the hang of it, understandable and easy to expand for any stakeholder in your project. We are using this same setup now for testing native mobile applications and it works like a charm!

We will start with mobile test automation for android. For this test automation we are using calabash-android which is basically a mix of cucumber with robotium under the hood…

Step 1 – Setup your environment

First, you need to get all your libraries sorted. This should be fairly easy to achieve since you just need to;
- Install ruby if you dont have it installed already
- install calabash-android:

gem install calabash-android

Depending on your testscenarios you might need some more libraries but you’ll get there along the way.

Now, make a root folder where you want to store your test automation files and create the skeleton files for your test automation by running the following command:

calabash-android gen

This step is also well explained at the installation section of the calabash-android project

Step 2 – Use the default debug build key and signature

Well done, you’ve got all the requirements to build your test automation suite. Well, you’d need an app as well. Plus you’d need to sign it and use the exact same key and credentials for your calabash testserver, else it wont be able to install and run it on the emulator or device. Although creating a key and use it to sign your apk is all explained on the android developers pages, the easiest would be to just have the test automation use the default debug key that android uses (which is being generated when you build your application). The only thing you need todo for that is to create/edit a .calabash_settings file in the root directory of your test automation folder and point to that keystore file. For example:

{
  "package_name":"debug.keystore",
  "api_level":"14",
  "keystore_location":"./features/support/debug.keystore",
  "keystore_password":"android",
  "keystore_alias":"androiddebugkey",
  "keystore_alias_password":"android"
}

NOTE: api_level obviously depends on what android platform you are building your app on so make sure to have that set correctly.

Step 3 – Test the test setup

There you go. Now your test setup should be all set. You dont have any test scenarios yet, but running calabash should show you if you have everything set up properly!

calabash-android run <link_to_your_android_debug_build.apk>

In the features folder you can place your scenarios. Information about creating feature files can be found in our blog about cucumber and web testing.

For iOS you say?…

Yes, there is a gem called calabash-ios. This is working somewhat different since there is no robotium under the hood. Its too much work to cover that here, so hopefully we are able to give some more information for you at a later stage.

‘Happy test-automationing’!

Hi All,

In this post we explain how to detect Javascript Errors in firefox using selenium, watir webdriver and cucumber. If you are unfamiliar with these frameworks you can read up on our previous post about Web testing with cucumber.

Get JSErrorCollector

First, you need to get JSErrorCollector.xpi which you can get here. Then, store it under features/support/extensions in your cucumber project location.

Add JSErrorCollector.xpi to your browser profile

require 'rubygems'
require 'selenium-webdriver'
require 'watir-webdriver'
require 'logger'
 
#Start the logger
$log = Logger.new('log/selenium.log')
#Create a profile
profile = Selenium::WebDriver::Firefox::Profile.new
 
#Add JS error detection to the profile!
profile.add_extension "features/support/extensions/JSErrorCollector.xpi" rescue p "Cannot add JSErrorCollector.xpi to profile"

Launch a browser using the profile you created

Now you can launch a new browser instance using:

$BROWSER = Watir::Browser.new 'firefox', :profile => profile
$BROWSER.goto "http://www.spritecloud.com"

When the JSErrorCollector is installed successfully, you will see the error collector counter in the bottom right of the launched browser as displayed in the image below.

Collect the JS errors

First off, we create a separate method to get the JS errors formatted how you want to display them when one or more JS errors are detected.

#Function that returns a string that presents the details of the occurred JS errors
def get_js_error_feedback()
  jserror_descriptions = ""
  begin
    jserrors = $BROWSER.execute_script("return window.JSErrorCollector_errors.pump()")
    jserrors.each do |jserror|
      $log.debug "ERROR: JS error detected:\n#{jserror["errorMessage"]} (#{jserror["sourceName"]}:#{jserror["lineNumber"]})"
 
      jserror_descriptions += "JS error detected:
   #{jserror["errorMessage"]} (#{jserror["sourceName"]}:#{jserror["lineNumber"]})
"
    end
  rescue Exception => e
    $log.debug "Checking for JS errors failed with: #{e.message}"
  end
  jserror_descriptions
end

Raise exception when JS error is detected

When using cucumber, you can create a hook after every teststep. In this hook we raise an exception when one or more JS errors are detected.

AfterStep do |scenario|
  raise get_js_error_feedback() unless get_js_error_feedback().empty?
end

Note that this is a Collector, so at the end of each execution, the JS errors of all the visited pages are collected. For speeding up the tests, you can do the check after the end of each scenario (use hook AfterScenario) and cleanup the list after that.

Good Luck!

• Our customer base in the UK is growing. We’ve always operated internationally, but after the Netherlands, the UK has been the place we have been most active in. By opening an office here, we will be able to serve local demands much, much better.
• Requests for development consultancy have been growing. While we primarily offer quality assurance services, our skillset includes specialization in mobile software development, high scalability server architecture and process consulting, and increasingly customers have asked for our help in these areas. By starting a new business unit dedicated to these services, we’re improving our focus, and will be able to pursue these activities with the dedication they deserve.
• Synergy is not just a buzzword. When we previously spoke to customers about QA or development services, we did so in isolation, because we were thinking of ourselves as “a QA company that can do development”. Ironically, by giving full credit to each part separately, we’re in a much better position to offer the combined package to our customers.

You can reach the Cambridge office straight away on +44 (0) 1223 853863, or send an email to development@spritecloud.com. Our office/postal address is:

spriteCloud Ltd.
ideaSpace/Entrepreneurship Centre
3 Charles Babbage Road
Cambridge
CB3 0GT
United Kingdom

The rules are simple:

1. Spot any of us at FOSDEM, and take a picture as proof. For the purposes of this competition, the whole of Brussels counts as “at FOSDEM”, at least from 02/02/2012 to 05/02/2012 (CET timezone).
2. Send the image to fosdem@spritecloud.com. No entries arriving past the 29th of February 2012 will be accepted (CET timezone). Please include your name, email address, and whether or not you want either of them published. We reserve the right to publish your image submission anonymously, though.
3. We’ll judge which pictures are best. Bonus points if you include yourself, make it funny, or manage to connect it to a particular FOSS project, etc. Offensive submissions will be disqualified. We’re impervious to bribery, but we may take whatever freebies you offer anyway.
4. The three best entries get a guest blog post here, on any subject related to FOSS and/or testing or writing software. It goes without saying, really, but offensive content will be rejected even if it’s relevant.
5. The best entry also gets a free T-shirt.

We’d love to give away more swag, but we’re unprepared – this whole thing has been somewhat spontaneous. If it’s a success, we’ll do it again next year, but bigger. So be sure to participate!

P.S.: Here’s the boring legalese:

1. The contest is open to anyone except employees and the families of employees of spriteCloud B.V. By entering this contest, contestants agree that spriteCloud B.V. may use all images created for the contest for promotional purposes.
2. All entries and all rights relating there become property of spriteCloud B.V.
3. By entering, all entrants agree to release spriteCloud B.V. and their employees from all liability and claims relating to participation in the contest. No substitutions or transfer of prizes or cash redemptions permitted by winners.
4. spriteCloud B.V. reserves the right, in its sole discretion, to cancel, suspend or alter the rules & parameters of the contest. spriteCloud B.V. also reserves the right to substitute a prize or prizes as it deems suitable in the event that the prize is unavailable for any reason whatsoever.
5. Winners under the age of 18 (as of win announcement) must have their parents’ and/or legal guardians’ consent to accept prizes. Winners, by acceptance of prizes, agree to release spriteCloud B.V. from any and all liability claims or actions of any kind whatsoever for damages.
6. Offer void where prohibited, regulated or restricted by law in a manner inconsistent with the purposes and rules hereof.

Interviews at The Power of One (1 of 2) from BlueVia on Vimeo.

Jump to ca. 1:45 if you want to see my ugly mug. Unfortunately they didn’t show the main bits when I was talking about QA, and got my name wrong. But it’s all good in the end!

Here you can find the pictures of the event. The winner of the one and only ‘Website of the Year 2011′ Award was Tweakers.net.

Below you will find the results of all the sub categories:

Even though errors like this don’t occur that much anymore most Chrome and Safari users know that there are still quite some annoying bugs when surfing the internet.

The truth is in the numbers
Does this means that developers still can’t be bothered to do cross browser testing? Does it means that companies don’t demand that their web applications work on all browsers? The truth is in the numbers;

Source: StatCounter Global Stats – Browser Market Share

IE 6.0, 7.0, 8.0, 9.0, Firefox 3.6, 4.0, 3.5, 5.0, Chrome 12.0, 8.0, 10.0, Safari 5.0 and so on. What works great in IE 9.0 doesn’t automatically work in IE7. Ok. That shouldn’t be hard to find out.

Source: StatCounter Global Stats – Browser Market Share

The top at time of writing was: Chrome 13, IE 8.0, Firefox 6, IE9. Only thing is, they add up to about 70%.

One hundred percent
In testing 100% is impossible. For cross browser testing this is no different. Let’s say you want to test on the browsers used by at least 85% of all people. This means you’d have to have at least 12 dedicated systems running, and being maintained. Not everyone wants this, which is why we have it. As well as a hosted test environment. We have everything running here in our test lab. Almost everything. From IE x.x, Chrome x, Mozilla, and up

Mobile
When testing mobile web applications you need to be able to test on mobile handsets. We have a wide coverage of handsets for the big mobile operating systems popular today.

Pleased
Having all the handsets; hosted test environments; a test lab based on an official Microsoft Developers Network (MSDN) subscription are just some of the reasons why our customer base, including high profile Digital Agencies, are very pleased with the testing we currently perform for them.

At spriteCloud we are domain experts in AGILE software Quality Assurance and software testing, particularly for web consumer service companies and digital agencies.