Selenium IDE vs Selenium Webdriver vs CasperJS

Or more specifically: Selenium IDE (Firefox plugin) vs Selenium Webdriver (Python and other languages) vs CasperJS (and PhantomJS or SlimerJS)

Selenium allows you, a programmer or non-programmer, to control a web browser and make it do things that you would otherwise do manually. With that ability, you can test your website over and over (and automatically from cron), similate users, or visit any number of web pages and read data (web scraping) on them and save to a file for processing.

If you go to the Selenium website you will actually see a number of Selenium projects. The two main ways you'll want to use Selenium is via Selenium IDE or Selenium Webdriver.

1. Selenium IDE

"Selenium IDE is a Firefox plugin which records and plays back user interactions with the browser. Use this to either create simple scripts or assist in exploratory testing. It can also export Remote Control or WebDriver scripts, though they tend to be somewhat brittle and should be overhauled into some sort of Page Object-y structure for any kind of resiliency."

Selenium IDE is only for Firefox as it's a Firefox plugin. While plugins for Firefox and extensions for Chrome are similar in their APIs and are written in JavaScript, a Firefox plugin won't automatically work in Chrome. The Selenium plugin has other Firefox-specific hooks which make it harder to port to other browsers. There are Chrome extensions for scripting and automating Chrome in similar ways (Scirocco, iMacros) but are not compatible with Selenium nor do all the things Selenium can do. For full Chrome automation, check out the Chrome WebDriver which will be explained below.

Selenium IDE has the advantage of being usable without programming knowledge, although it is still rather technical. You can simply press the record button and take some actions in the browser and then replay those actions later. Selenium doesn't always record what you expect, but you can view and edit the steps in the saved script. This will require some knowledge of the structure of HTML and of CSS selectors. The use case for IDE is to record and replay single macros. Creating and managing and running collections of many such macros or using macros to crawl and export and data (including verifying externally if seen test data is correct) is outside the scope of Selenium IDE and so for all but the simplest users and use cases you should start with Selenium Webdriver instead of Selenium IDE.

2. Selenium WebDriver

Selenium 2.0 is Selenium Webdriver. Selenium 1.0 and Selenium-RC are outdated so just use Webdriver.

"WebDriver is a tool for automating web application testing, and in particular to verify that they work as expected."

First, to use WebDriver you need to write code. You can choose from a number of languages: C#, Python, Ruby, Perl, PHP, and JavaScript.

It's not clear from the official documentation that Javascript is supported for the Webdriver as it's not listed there. Selenium officially supports JavaScript (node.js) as a client language and Webdriver. There are also 3rd-party wrappers for Node to connect to Selenium. So Selenium with JavaScript is an alternative to JS guys currently using CasperJS or PhantomJS for web scraping or testing. There's some documentation about the new JavaScript/node.js bindings for WebDriver here: https://code.google.com/p/selenium/wiki/WebDriverJs

The way coding for WebDriver works is simple. WebDriver has APIs for opening the browser, then waiting for the DOM to finish loading (and other forms of waiting), then various methods of finding elements (findElement..By..ID, by selector, by xpath, etc.). Then you can click on links (whether they navigate to a new url or just trigger some JavaScript in the page) or submit forms. You can also execute your own custom JavaScript in the context of the page in the browser, e.g. running a bit of jQuery code. And since, unlike curl, you're controlling an actual browser instance, you can also virtually hit the back button and go click on every other link on the page, one at a time.

You can run WebDriver on your personal laptop or on a headless remote Linux server. In the latter case, you don't actually need to have X11 running but you do need Xvfb, a virtual frame buffer which is where Firefox will think it's running.

One huge advantage of Selenium is that it can drive just about any browser, as most of them now have support, from IE to Chrome, as well as Safari on iOS and Chrome on Android. So while PhantomJS/CasperJS can pretend to be a WebKit browser (meaning Safari), it's not actually truly one of those browsers plus it can't pretend to be Internet Explorer or Firefox or Chrome or all those other browsers.

3. CasperJS

If all you care about is browsing web pages but not testing for their correct functionality in different browsers, then the above statement about Selenium doesn't matter. And if you don't need to do your scripting in all those Selenium-supported languages above like Python, C#, PHP, etc. and you are happy writing server-side JavaScript then CasperJS http://casperjs.org could be for you.

PhantomJS is a JavaScript-based headless browser, similar to Selenium WebDriver except that there's no browser running at all. Unlike the use case with Selenium and Xvfb above, all the graphical browsing is just being rendered via PhantomJS's own WebKit implementation. [Actually, PhantomJS 1.4 and earlier did require X11+Xvfb.]

One should note that while Phantom/Casper are server-side JavaScript, they are not node.js and you can't just automatically include and run it together with node code. And some of the JavaScript apis are different. I find that Selenium WebDriver exposes the common functionality you would want when browsing around web pages and selecting content within them. PhantomJS has a much larger and more complicated set of APIs while not making it so easy to do the common things and for doing testing. Casper is meant to wrap Phantom and make it easier to do those common things. And for doing filesystem IO you have to use Phantom's own fs module, which differs from node.js. With Python WebDriver, you just do I/O in Python as you normally would. In fact, if you want to do anything fancy besides crawling websites, you should do it in another language instead of PhantomJS's unique and poorly documented JavaScript API.

SlimerJS: Now Casper supports the Gecko rendering engine which Firefox uses, so you get both a Safari-like browser and a Firefox-like one. The support is new as of SlimerJS 0.8 and CasperJS 1.1beta1 and you switch to Slimer by running Casper with --engine=slimerjs. It might not be as good as a real instance of Firefox. And I've had to switch out different versions of Firefox with Selenium (on Linux) before which would be more of a chore to do with Slimer.