Fwd: [Work] Automated testing -- Tools discussion
michelle.dsouza at utoronto.ca
Wed Apr 7 13:55:46 UTC 2010
An interesting summary of the research Kasper has done into automated
Begin forwarded message:
> From: Kasper Galschiot Markus <kasper2 at markus.dk>
> Date: April 7, 2010 7:39:53 AM GMT-04:00
> To: CollectionSpace Work list <work at lists.collectionspace.org>, obara.justin at gmail.com
> Subject: [Work] Automated testing -- Tools discussion
> I have been browsing around in the quite extensive amount of free
> web test automation tools around. Everyone seems to have their own
> preferences, and there are pros and cons for each tool. Below I'll
> try summarizing my observations (not experience, but what I have
> read). In the very bottom, I posted my personal opinion of choice.
> I'm sending this mail to the work list, because any experience,
> rumors, comments, suggestions, etc. would be highly appreciated.
> Also comments on the importance of: speed of tests, reality of
> tests, ease of programming, etc, as well as suggestions for other
> tools I should look into would be appreciated.
> Selenium: Probably the best known tool.
> It automates/scripts real browsers, which is an advantage as you get
> close to reality.
> Well established and documented
> GUI record tool for firefox, making it easy to write test (for all
> Support multiple programming languages (for writing the tests)
> Has been praised for being the easiest to use/get into
> Supports ant
> Has grid functionality to distribute the tests (to a certain extend)
> Events are simulated, which is sometimes buggy (based on rather old
> articles, and justins (fluid) experiences X months ago, might have
> Needs actual running windows+mac machines to run against
> Slow compared to emulated browser approach (can be somewhat remedied
> by running tests in parallel)
> Disregards JS errors (unless they affect the asserts)!!!!
> Potential problem with waiting for pages to load (think this has
> Canoo Webtest: Also widely known tool.
> Do not need specific OS, as browsers are emulated (uses HtmlUnit)
> Record tool for writing easy tests
> Well established/documented
> Supports ant
> Easy to integrate/setup continuous testing
> Very good reports
> Does not use real browsers, but emulates them using HtmlUnit. This
> means some errors might potentially be not be caught.
> A bit steeper learning curve than Selenium
> Doesn't accept (too) badly formed html (Dont think this will be a
> Doh.Robot: A smaller project, brought to my attention by Justin.
> Does not simulate mouse/keys, but makes actual system calls using
> java's robot
> As close as can be to real users (except for embedded scripts)
> Not too well documented/established
> Unsure if updated or discontinued
> requires code to be inserted on the page
> GUI record for writing tests
> Small community
> Confusing Interface
> Multi browser (& OS) support
> Has a rich API
> Has a ‘Simple’ class (for non-tech users)
> Watij & Watin (Java & .NET)
> Have to learn Ruby (unless you choose Watij or Watin)
> Every browser requires a different library
> Selenium 2:
> Has both standard selenium and HtmlUnit available for testing
> Circumvent browser restrictions by using whichever mechanism is most
> appropriate to control the browser. For Firefox, this means that
> WebDriver is implemented as an extension. For IE, WebDriver makes
> use of IE's Automation controls, etc.
> WebDriver can make use of facilities offered by the OS (allows to
> get closer to a real user, similar to robot i guess?!)
> Alpha state
> Sparse documentation, and probably somewhat buggy
> Steeper learning curve due to amount of features and lack of docs
> From what I can tell, Selenium and Canoo Webtest seems to be the
> most appealing choices, despite their shortcommings. It seems that
> Webtest receives less criticism than Selenium, but it is hard to
> tell whether this is due to Selenium being used/compared to more
> often. Generally, both people using Selenium and Webtest are
> satisfied. The others; Doh Robot, Sahi and Watir all have their
> advantages (except for Sahi :( ), but for various reasons didn't
> appeal to me (too little documentation, steep learning curve,
> lousier version of Webtest/Selenium). There is also the option of
> Selenium 2, but the problem with this is that it is still in alpha,
> and the documentation is rather sparse. Also, it might be kind of
> In many ways I am split between Selenium and Webtest, but I guess I
> am leaning towards Selenium. There are two main reasons for this:
> Using real browsers is a lot closer to reality, which I weigh high
> (some of) the shortcommings of Selenium will be fixed in Selenium 2.
> I take it that our tests written for Selenium will be compatible
> with Selenium 2. When S2 will be at a decent stage it is easy to
> make the switch.
> The major problem with Selenium is the need of access to windows/mac
> machines (windows is doable, using Vms, I hate mac :).
> Another major problem is the speed. This can hopefully be remedied
> using Selenium Grid.
> As a final note, I guess I should mention that I might be a bit
> biased towards Selenium, since I knew that first, have read more of
> the documentation, etc.
> Work mailing list
> Work at lists.collectionspace.org
Software Developer, Fluid Project
Adaptive Technology Resource Centre
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the fluid-work