Mar 10

Creating and sending HTML e-mails with python

Automation, Presentation, Python, Tools 2 Comments »

When you want to send tidy e-mails for marketing or other purposes you quickly run into the solution of using html within e-mails. In this post we will show you a code snippet that allows you to send such a html e-mail using standard python libraries.

However, before we start you should know that there are quite some articles and posts mentioning that creating html e-mails is a (very) bad idea. I believe it is ok sending out html e-mails these days; loads of support within e-mail clients has been added and you can see the number of html e-mails growing every day. However, there are a few caveats that you should be aware of! Lets list some of them first!..

CSS support

Obviously you want to use some CSS styles in your HTML e-mail. There are some differences with the support of CSS in an e-mail, support per css property can really differ per e-mail client/browser. Since it is really hard and not too much fun to try out all the different e-mail clients/portals you’d be better off to first see what html tags and css properties are actually supported. The best guide to this CSS Support maze in e-mail clients is probably this one.

When you are using CSS in your html e-mail, the best thing to do is to just put the CSS in a <style> tag in the <head>. Another possibility is to put a link to the CSS file but since that needs to be retrieved and might be blocked upon first view, so that is not the best solution. A problem here is that, as of writing, google gmail does not support a <style> tag or <link> to your CSS file in the <head>! Thus, if you want your style to be processed and visible in gmail, you have to include it in the html tags itself. Now, this is not the most ideal situation, but one thing you can do is to just put some minimal style in the html tags and put the rest in the CSS style tag. This approach has been used in the table below. This way, even when style tags are not supported you would still get the minimal ‘display tidyness’.

Catch e-mail clients where html e-mail is unsupported

With an html e-mail you have to change the “MIME type” of the email. Nearly all email clients support this nowadays, but it can still happen that you come across a client that doesnt. In the example you can see that, with a very simple setting, you can prevent the user from not receiving any feedback.

Always give a reference to the source/online example

Perhaps obvious, but it is advised to always have an example or reference to the content in the email online, and point to that. This way the receiver will not only be able to see the content in any situation, but you also have another option to drive the users to your website or service.

Reason and Un-subscribe

Another slightly obvious point; you should always give a reference to why someone receives the HTML e-mail (etc, subscribed to a newsletter..) so the receiver doesnt feel like he just received spam/unwanted marketing. Also an option to unsubscribe/stop receiving the emails should be included.

The example

import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
 
def py_mail(SUBJECT, BODY, TO, FROM):
    """With this function we send out our html email"""
 
    # Create message container - the correct MIME type is multipart/alternative here!
    MESSAGE = MIMEMultipart('alternative')
    MESSAGE['subject'] = SUBJECT
    MESSAGE['To'] = TO
    MESSAGE['From'] = FROM
    MESSAGE.preamble = """
Your mail reader does not support the report format.
Please visit us <a href="http://www.mysite.com">online</a>!"""
 
    # Record the MIME type text/html.
    HTML_BODY = MIMEText(BODY, 'html')
 
    # Attach parts into message container.
    # According to RFC 2046, the last part of a multipart message, in this case
    # the HTML message, is best and preferred.
    MESSAGE.attach(HTML_BODY)
 
    # The actual sending of the e-mail
    server = smtplib.SMTP('smtp.gmail.com:587')
 
    # Print debugging output when testing
    if __name__ == "__main__":
        server.set_debuglevel(1)
 
    # Credentials (if needed) for sending the mail
    password = "mypassword"
 
    server.starttls()
    server.login(FROM,password)
    server.sendmail(FROM, [TO], MESSAGE.as_string())
    server.quit()
 
if __name__ == "__main__":
    """Executes if the script is run as main script (for testing purposes)"""
 
    email_content = """
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  <title>html title</title>
  <style type="text/css" media="screen">
    table{
        background-color: #AAD373;
        empty-cells:hide;
    }
    td.cell{
        background-color: white;
    }
  </style>
</head>
<body>
  <table style="border: blue 1px solid;">
    <tr><td class="cell">Cell 1.1</td><td class="cell">Cell 1.2</td></tr>
    <tr><td class="cell">Cell 2.1</td><td class="cell"></td></tr>
  </table>
</body>
"""
 
    TO = 'receiver@email.com'
    FROM ='sender@mysite.com'
 
    py_mail("Test email subject", email_content, TO, FROM)

The example aftermath

The example requires you to change the FROM and TO email addresses and the password to send the email. If you do not use gmail to send the e-mail, you can probably find your smtp server settings here. The debugging option should give you a clear idea what the problem is, if something goes wrong regarding the sending process. There are also a lot of posts online about troubleshooting with python’s smtp lib.

When you execute the example and view the email in your gmail browser client you will see that the ‘in-tag-html’ styles are processed but not the style defined in the style tag. In other clients however all the styles in the example are processed.

Jan 08

Posting forms with Python

Automation, Python, Tools 3 Comments »

When you are automating your website tests, you will probably run into forms that need testing sooner or later. Even better (or worse), you wont be able to automate your tests at all if a simple login post is needed to access your site and you are unable to do so.

In python there are several approaches to post web-forms. In fact, there are too many to choose from. Therefore we created a list of the most used approaches, mention their differences and speak our preference. We dont go into much depth, but just show a simple example (and, if able, also prints the forms found on the page) and discuss the pro’s and cons.

The discussed libraries/approaches are:

  • basic urllib2 post support
  • Mechanize
  • Twill
  • ClientForm

There is also a library called Zope , able to post forms. The approach is much like the one from Mechanize, so I will not discuss that one here. When you are using pycurl, you probably end up using a different approach much like the urllib2 post support. An example for posting forms with pycurl can be found here.

Urllib2 post support:

import urllib, urllib2
page = 'http://www.mywebsite.com/myformpage.html'
raw_params = {'usern':'test_name','passw':'test_pass'}
params = urllib.urlencode(raw_params)
request = urllib2.Request(page, params)
page = urllib2.urlopen(request)
info = page.info()

If this works for you, and you dont find yourself using alot of different forms, this simple approach is the way to go, and there is no need to start using another library.

Pros: Simple, no other library needed

Cons: Very limited flexibility and support. Unable to detect any forms on a requested page. No success or failure feedback of a post.

Mechanize:

import mechanize
br = mechanize.Browser()
 
try:
	br.open(page)
	forms = br.forms()
	print 'Forms:'
 
	for form in forms:
		print 'form', form
		# Force try using the expected form
		br.select_form(name="form")
		br["usern"] = "test_name"
		br["passw"] = "test_pass"
		response = br.submit()
		content = response.get_data()
		br.close()
 
	except urllib2.HTTPError, e:
		sys.exit("%d: %s" % (e.code, e.msg))
	except IOError, e:
		print e
	except:
		pass

Pros: Easy to use,  Can detect forms on page.

Cons: Limited flexibility, Can choke on bad html.

Twill:

from twill.commands import go, showforms, formclear, fv, show, submit
 
go(page)
print "Forms:"
showforms()
 
try:
	# Force try using the first form found on a page.
	formclear('1')
	fv("1", "usern", "test_name")
	fv("1", "passw", "test_pass")
	submit('0')
	content = show()
	print 'debug twill post content:', content
 
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e
except:
	pass

I have found that this library is the most robust against incorrect formatted html. It supports several ‘html cleanup libraries’ and is tidying up html by default. In some cases it was able to detect forms where others could not. The flexibility is still limited though.

Pros: Easy to use. Can detect forms on page. Robust against bad formatted html.

Cons: Limited flexibility, Not the fastest.

ClientForm:

import ClientForm, urllib2
request = urllib2.Request(page)
response = urllib2.urlopen(request)
forms = ClientForm.ParseResponse(response, backwards_compat=False)
print 'Forms:'
for form in forms:
	print form
try:
	# Force try using the expected form
	form = forms[0]
	form["usern"] = "test_name"
	form["passw"] = "test_pass"
	content = urllib2.urlopen(form.click()).read()
	check_success(content)
 
except urllib2.HTTPError, e:
	sys.exit("%d: %s" % (e.code, e.msg))
except IOError, e:
	print e
except:
	pass

As you can see, ClientForm can detect forms on a page and specifically select a form. Trying to set a value will only work if it that operation is really possible, so it tests itself already basically.

Pros: Great flexibility and support. Most comprehensive documentation and examples.

Cons: Html cleanup advised before form detection.

Last words (Conclusion, if you will):

I have found that most of the libraries are not very robust against incorrect html format. At time of writing I have found for example that a simple incorrect hr tag (<hr/> instead of <hr /> or <hr>) can break the detection of the form. a study with the incorrect hr tag showed that only twill was able to successfully detect the form, and none were able to do a successful post. It is therefore recommended to prettify your markup before ‘feeding’ it to your library. Although not very fast, the library you want to have a look at to do this is: BeautifulSoup.

Concluding I can say that, if urllib2 post support is sufficient, there is no need using other libraries. However, once the forms you start testing get more divers and the html pages contain incorrect html markup, you will find yourself struggling with posting forms where ClientForm gives a clear overview of what forms are available and knows all the possible fields and options.
So, due to the clear examples and documentation for ClientForm, and the biggest support and freedom for form field settings, this is definately my preferred library for posting forms.

Finally, make sure you support cookies, or e.g. trying to log in wont have the desired effect. The way to go here is using cookielib. An old but still useful example on usage can be found here.

Tagged with: • • • •
preload preload preload