Agile testing methods

Test driven development

Behaviour driven development

Cucumber

Feature

Feature: Website Navigation
  In order to navigate website pages
  I need to be able to click on menu item

Scenario

Scenario: Going to page Services
  Given that I am on spriteCloud Home
  When I click on link Services
  Then the page title should be "spriteCloud - Services"

Step definitions

require 'spec'
require "watir-webdriver"

SITE = "www.spritecloud.com"
BROWSER = Watir::Browser.start(SITE, :firefox)
PAGES = {
  "spriteCloud Home" = "http://www.spritecloud.com/"
}

The hash PAGES is not really necessary in this case, but can be useful in a case when there is more data and pages involved. It is used for looking up a page URL based on page name in the scenarios.

Given /^that I am on (.*)$/ do |page|
  BROWSER.goto(PAGES[page])
end

Notice that the step syntax is formatted as a regular expression. Given is a keyword but everything else is basically just something that Cucumber can recognize as being part of your test. Let’s break that part down a bit:

• Sentence starts with “Given” keyword. This is mandatory
• What follows is a space and the words “that I am on”
• Another regular expression (.*), which matches everything from “on ” until the end of the line “storing” it for later use for the variable page

Using the capturing group (.*) is important because it allows us to write more generic steps, where for example, the page where we are or the link that we are following is defined in the scenario, not in the step. This again allows for the less technical person to write tests without having to write Ruby code. It is of course also better programming practice not to write the same things multiple times and just changing a value.

When /^I click on link (.*)$/ do |link|
  BROWSER.link(:text, link).click
end

Then /^the page title should be "([^\"]*)"$/ do |title|
  BROWSER.title.eql?(title).should == true
end

Test Execution

This type of test can be run either during the implementation phase, which is the original purpose or later on when the site is fully ready. Even if the test was created during the implementation phase, it should still be run with every new release that goes out. It can even be used as a part of a stand-alone test set for the site.

Part of the power of Cucumber is the nicely formatted output displaying immediately the successful and the failed tests and test steps. Successful tests are colored green on the terminal and failed one, you guessed it, red.

mbp-2:features marko$ cucumber .
Feature: Website Navigation
	In order to navigate website pages
	I need to be able to click on navigation

  Scenario: Going to page Services
    Given that I am on spriteCloud Home
    When I click on link Services
    Then the page title should be "spriteCloud - Services"

1 scenario (1 passed)
3 steps (3 passed)
0m1.261s

Summary

This post only scratches the surface of what can be done with either Watir or Cucumber. They are both powerful test tools by themselves and combining them can be the basis for a successful test automation project. It all depends on the needs in your project.

Marko

When you want to send tidy e-mails for marketing or other purposes you quickly run into the solution of using html within e-mails. In this post we will show you a code snippet that allows you to send such a html e-mail using standard python libraries.

However, before we start you should know that there are quite some articles and posts mentioning that creating html e-mails is a (very) bad idea. I believe it is ok sending out html e-mails these days; loads of support within e-mail clients has been added and you can see the number of html e-mails growing every day. However, there are a few caveats that you should be aware of! Lets list some of them first!..

CSS support

Obviously you want to use some CSS styles in your HTML e-mail. There are some differences with the support of CSS in an e-mail, support per css property can really differ per e-mail client/browser. Since it is really hard and not too much fun to try out all the different e-mail clients/portals you’d be better off to first see what html tags and css properties are actually supported. The best guide to this CSS Support maze in e-mail clients is probably this one.

When you are using CSS in your html e-mail, the best thing to do is to just put the CSS in a <style> tag in the <head>. Another possibility is to put a link to the CSS file but since that needs to be retrieved and might be blocked upon first view, so that is not the best solution. A problem here is that, as of writing, google gmail does not support a <style> tag or <link> to your CSS file in the <head>! Thus, if you want your style to be processed and visible in gmail, you have to include it in the html tags itself. Now, this is not the most ideal situation, but one thing you can do is to just put some minimal style in the html tags and put the rest in the CSS style tag. This approach has been used in the table below. This way, even when style tags are not supported you would still get the minimal ‘display tidyness’.

Catch e-mail clients where html e-mail is unsupported

With an html e-mail you have to change the “MIME type” of the email. Nearly all email clients support this nowadays, but it can still happen that you come across a client that doesnt. In the example you can see that, with a very simple setting, you can prevent the user from not receiving any feedback.

Always give a reference to the source/online example

Perhaps obvious, but it is advised to always have an example or reference to the content in the email online, and point to that. This way the receiver will not only be able to see the content in any situation, but you also have another option to drive the users to your website or service.

Reason and Un-subscribe

Another slightly obvious point; you should always give a reference to why someone receives the HTML e-mail (etc, subscribed to a newsletter..) so the receiver doesnt feel like he just received spam/unwanted marketing. Also an option to unsubscribe/stop receiving the emails should be included.

The example

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
 
def py_mail(SUBJECT, BODY, TO, FROM):
    """With this function we send out our html email"""
 
    # Create message container - the correct MIME type is multipart/alternative here!
    MESSAGE = MIMEMultipart('alternative')
    MESSAGE['subject'] = SUBJECT
    MESSAGE['To'] = TO
    MESSAGE['From'] = FROM
    MESSAGE.preamble = """
Your mail reader does not support the report format.
Please visit us <a href="http://www.mysite.com">online</a>!"""
 
    # Record the MIME type text/html.
    HTML_BODY = MIMEText(BODY, 'html')
 
    # Attach parts into message container.
    # According to RFC 2046, the last part of a multipart message, in this case
    # the HTML message, is best and preferred.
    MESSAGE.attach(HTML_BODY)
 
    # The actual sending of the e-mail
    server = smtplib.SMTP('smtp.gmail.com:587')
 
    # Print debugging output when testing
    if __name__ == "__main__":
        server.set_debuglevel(1)
 
    # Credentials (if needed) for sending the mail
    password = "mypassword"
 
    server.starttls()
    server.login(FROM,password)
    server.sendmail(FROM, [TO], MESSAGE.as_string())
    server.quit()
 
if __name__ == "__main__":
    """Executes if the script is run as main script (for testing purposes)"""
 
    email_content = """
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <title>html title</title>
  <style type="text/css" media="screen">
    table{
        background-color: #AAD373;
        empty-cells:hide;
    }
    td.cell{
        background-color: white;
    }
  </style>
</head>
<body>
  <table style="border: blue 1px solid;">
    <tr><td class="cell">Cell 1.1</td><td class="cell">Cell 1.2</td></tr>
    <tr><td class="cell">Cell 2.1</td><td class="cell"></td></tr>
  </table>
</body>
"""
 
    TO = 'receiver@email.com'
    FROM ='sender@mysite.com'
 
    py_mail("Test email subject", email_content, TO, FROM)

The example aftermath

The example requires you to change the FROM and TO email addresses and the password to send the email. If you do not use gmail to send the e-mail, you can probably find your smtp server settings here. The debugging option should give you a clear idea what the problem is, if something goes wrong regarding the sending process. There are also a lot of posts online about troubleshooting with python’s smtp lib.

When you execute the example and view the email in your gmail browser client you will see that the ‘in-tag-html’ styles are processed but not the style defined in the style tag. In other clients however all the styles in the example are processed.

When you are automating your website tests, you will probably run into forms that need testing sooner or later. Even better (or worse), you wont be able to automate your tests at all if a simple login post is needed to access your site and you are unable to do so.

In python there are several approaches to post web-forms. In fact, there are too many to choose from. Therefore we created a list of the most used approaches, mention their differences and speak our preference. We dont go into much depth, but just show a simple example (and, if able, also prints the forms found on the page) and discuss the pro’s and cons.

The discussed libraries/approaches are:

• basic urllib2 post support
• Mechanize
• Twill
• ClientForm

There is also a library called Zope , able to post forms. The approach is much like the one from Mechanize, so I will not discuss that one here. When you are using pycurl, you probably end up using a different approach much like the urllib2 post support. An example for posting forms with pycurl can be found here.

import urllib, urllib2
page = 'http://www.mywebsite.com/myformpage.html'
raw_params = {'usern':'test_name','passw':'test_pass'}
params = urllib.urlencode(raw_params)
request = urllib2.Request(page, params)
page = urllib2.urlopen(request)
info = page.info()

If this works for you, and you dont find yourself using alot of different forms, this simple approach is the way to go, and there is no need to start using another library.

Pros: Simple, no other library needed

Cons: Very limited flexibility and support. Unable to detect any forms on a requested page. No success or failure feedback of a post.

import mechanize
br = mechanize.Browser()
 
try:
	br.open(page)
	forms = br.forms()
	print 'Forms:'
 
	for form in forms:
		print 'form', form
		# Force try using the expected form
		br.select_form(name="form")
		br["usern"] = "test_name"
		br["passw"] = "test_pass"
		response = br.submit()
		content = response.get_data()
		br.close()
 
	except urllib2.HTTPError, e:
		sys.exit("%d: %s" % (e.code, e.msg))
	except IOError, e:
		print e
	except:
		pass

Pros: Easy to use,  Can detect forms on page.

Cons: Limited flexibility, Can choke on bad html.

from twill.commands import go, showforms, formclear, fv, show, submit
 
go(page)
print "Forms:"
showforms()
 
try:
	# Force try using the first form found on a page.
	formclear('1')
	fv("1", "usern", "test_name")
	fv("1", "passw", "test_pass")
	submit('0')
	content = show()
	print 'debug twill post content:', content
 
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e
except:
	pass

I have found that this library is the most robust against incorrect formatted html. It supports several ‘html cleanup libraries’ and is tidying up html by default. In some cases it was able to detect forms where others could not. The flexibility is still limited though.

Pros: Easy to use. Can detect forms on page. Robust against bad formatted html.

Cons: Limited flexibility, Not the fastest.

import ClientForm, urllib2
request = urllib2.Request(page)
response = urllib2.urlopen(request)
forms = ClientForm.ParseResponse(response, backwards_compat=False)
print 'Forms:'
for form in forms:
	print form
try:
	# Force try using the expected form
	form = forms[0]
	form["usern"] = "test_name"
	form["passw"] = "test_pass"
	content = urllib2.urlopen(form.click()).read()
	check_success(content)
 
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e
except:
	pass

As you can see, ClientForm can detect forms on a page and specifically select a form. Trying to set a value will only work if it that operation is really possible, so it tests itself already basically.

Pros: Great flexibility and support. Most comprehensive documentation and examples.

Cons: Html cleanup advised before form detection.

I have found that most of the libraries are not very robust against incorrect html format. At time of writing I have found for example that a simple incorrect hr tag (<hr/> instead of <hr /> or <hr>) can break the detection of the form. a study with the incorrect hr tag showed that only twill was able to successfully detect the form, and none were able to do a successful post. It is therefore recommended to prettify your markup before ‘feeding’ it to your library. Although not very fast, the library you want to have a look at to do this is: BeautifulSoup.

Concluding I can say that, if urllib2 post support is sufficient, there is no need using other libraries. However, once the forms you start testing get more divers and the html pages contain incorrect html markup, you will find yourself struggling with posting forms where ClientForm gives a clear overview of what forms are available and knows all the possible fields and options.
So, due to the clear examples and documentation for ClientForm, and the biggest support and freedom for form field settings, this is definately my preferred library for posting forms.

Finally, make sure you support cookies, or e.g. trying to log in wont have the desired effect. The way to go here is using cookielib. An old but still useful example on usage can be found here.

How come Ruby ?

 
There are several important Ruby projects, which are also increasing the popularity of the Ruby language in general:

http://www.ruby-lang.org/en/libraries/top-projects/

The most interesting projects from my (and this blog posts) point of view are Watir and Ruby on Rails, the former from a testers point of view, the latter from web developers.

Good projects don’t just come from nowhere, either. The language itself is very elegant and enjoyable to use. Having a programming language that works for you instead of against you is going to leave more time for innovation and thinking about the next big features for your product. The Ruby community is very active and provides very nice modules (=gems), which provide a specific functionality and usually do it very well.

What is Watir?

 
Watir stands for Web Application Testing in Ruby, http://watir.com. It’s a framework, which drives the browser programmatically. Currently the officially supported browsers are IE and Firefox, but gems also exists for Chrome and Safari. Also Opera is working on a port of their own, which they are using in their own development (http://my.opera.com/core/blog/2009/03/06/test-automation-with-operawatir). Unfortunately, it is not available for outsiders yet, which makes guaranteeing websites work on Opera more difficult than it should be.

What about Ruby on Rails?

 
The single most important reason for Ruby gaining a lot of attraction in the recent years is the Ruby on Rails web development framework, http://rubyonrails.org. The framework enables quicker web development with it’s inbuilt support for database access. It’s actually not just support, but ActiveRecord is one of the fundamental technologies behind RoR and makes it especially suited for database driven applications.

How is that related to testing?

 
Watir is great for web automation, and we have used it in extensive automation projects. There is lots of documentation available for the basic usage, but as always, best practices for implementing automation test projects in different types of situations are more difficult to find and require some trial and error to find the best approach.

Typically, one of the main points to consider is how easy the test cases will be to maintain. Writing one or ten test cases is easy without any structure, but scaling the automation suite for hundreds or thousands of test cases requires careful planning. Many of the same rules apply as for any software development project. As most of the testers do not have a developer background, this can be a challenge in the beginning.

There’s is nothing stopping testers taking advantage of the already well thought out features of Ruby on Rails what comes to database access (ActiveRecord) and for sending email (ActionMailer). Database access can be used to verify that a form actually posted the values and they were stored correctly in the database. Email can be, for example, used for sending test reports directly from test automation. There are clear synergies in understanding both of these technologies when aiming the create good test automation projects. And when using the already available gems for the tasks, you’ll save time for actual automation work.

We will cover several of these topics in a series of blog posts.

 
Marko

Building a Ruby on Rails test server on FreeBSD 8.0 RC1

We’re a Ruby on Rails shop at our place and we work on a number of different platforms for developing and testing our stuff; the devs normally work on macs, while the staging environment – this week – is Ubuntu’s Jaunty Jackalope (9.04) distribution. However I’m a big FreeBSD fan, and have been for a number of years. Speaking personally for myself, FreeBSD makes sense in every aspect of its operation and configuration, and the ports collection is second to none for all aspects of application management. (disclaimer – *nix distributions are more like pairs of jeans than they are operating systems in the hearts and minds of regular users; what fits well on one user, doesn’t fit at all on another. So I’m absolutely not saying FreeBSD is best, it’s just best for me). When the FreeBSD 8.0 RC1 build was made available I decided to put together a test server for deploying new builds of our apps for first round QA work. More or less from a standing start this is the road I travelled, which at the end resulted in a fully operational test server.

First off was the installation of the 8.0-RC1 build itself on the test server machine, which was my Lenovo X60 laptop with 2GB of RAM. I got a hold of a build ISO image from a mirror detailed on the main “Get FreeBSD” site page, and took disc 1 only. The idea was to make a minimal install which meant only the first disc was necessary. With a minimal install you don’t get add the ports collection by default, so the first post installation task was getting this added. At this point, I was ready to install Rails, but as I also wanted a desktop I went ahead and installed KDE4. I’m not going to go into the details here, as there are other fine posts about this easily found with a Google search, but the process is fairly straight forward with no major gotchas yet takes around 10 – 12 hours to complete.