Thursday, March 12, 2015

Does Protractor + Selenium WebDriver Sound "Promising?"

I have been diving into JavaScript and AngularJS heavily over the past few weeks, pushing the capabilities of my organization's application testing in the process.  Despite having written quite a bit of JavaScript in my past for a number of award-winning Web applications (in hackathons), some of the latest trends in that language had bypassed me completely.  In coming up to speed on JavaScript Promises, here is some code that has proven very useful in my activities.

Typically, doing Web page testing requires interacting with the user interface, then waiting for something to happen (you logged in, paid your bill, ordered food, wrote a review, etc.).  There are three ways to wait for such UI interactions to complete in JavaScript:

  • Pure asynchronous callbacks (nested and ugly)
  • Unchained promises (still can be nested, and has potential to get ugly)
  • Chained promises (Pretty straightforward)
The essence of these three methodologies is described succinctly and effectively on the GitHub page for WD.js (a popular WebDriver for Selenium automated GUI tests).  My organization has elected to use Protractor as its framework of choice for our AngularJS tests, plus Gulp as our build manager, and this tends to force the use of "selenium-webdriver" as opposed to WD.js.  The constructs in WD.js are mostly dissimilar and require conversion in order to be compatible with selenium-webdriver (perhaps an open-source hero could be made if someone wrote a converter!).  For instance, selenium-webdriver only seems to be happy with unchained promises, where WD.js seems to accept all types of promises.  Thus, it is important to note the following code pertains to Protractor and selenium-webdriver.

Waiting On External Events

In some cases, you may not wish to wait for changes to a UI element on the browser, but instead for a connection and query to a database or perhaps a call to an external API.  For this, most people use the "Q" Node module; however, selenium-webdriver already includes the "Q" framework through browser.controlFlow().  Specifically, browser.controlFlow().execute(<some function defined as a variable>).  The function you name must return a promise.

Here's a quick example, including a hypothetical function to connect to a database, and the test code that would wait for this function to return the connection object:

connectToDB = function() {
    var promise = protractor.promise.defer();
    var mydb = require('dbhelper');
        // some hard-coded settings
    }, function(err, connection) {
        if (err)
    return promise.promise;

describe('a test suite', function() {
    it('should connect to the DB', function() {
        browser.controlFlow().execute(connectToDB).then(function(conn) {
            console.log('I now have a connection!');

Side Note: Yes, you can use JavaScript to connect to databases directly, even Oracle databases!  Just check out the recent (and official) node-oracle Node module if you need connectivity to Oracle DBs.  I'm sure there are other great ones for MySQL, Microsoft SQL Server, MariaDB, Cassandra, etc.

This is all fine and dandy, but note that connectToDB is not set up in such a way as to take arguments.  First, the function needs to return promise.promise to execute() rather than the connection object that connectToDB() makes.  Second, due to the way control flows work, nothing is really supposed to get run right when tasks are defined, thus the parentheses must be omitted from connectToDB when it is passed as an argument to execute(), because otherwise it will get run right away.  Only when the entire test flow described inside it() has been parsed may functions returning promises get run.

This makes it rather difficult to pass arguments to functions that return promises!  Does this mean you'll have to handle making your database query in the same function, and lose the capability to abstract that into a generic query function?  No!  And, in fact, it's not difficult at all to pass arguments to such functions.

Passing Arguments to Promise Functions

Given a function select(conn, query) that runs a select-type query specified by query on the database connection conn, you can pass those arguments and still wait for the promise as such:

// code from before, plus definition of "query"...
browser.controlFlow().execute(connectToDB).then(function(conn) {
    // simply elaborating on the contents of this same function from above
    var qr = function() {    // qr = query request ;)
        return select(conn, query);
    browser.controlFlow().execute(qr).then(function(rows) {

Voila, a control flow that can wait on a database connection and query!

Help!  I can't see my code anymore!

The astute observer may have noticed that with enough of these .then() calls, your code will be so far to the right that you will constantly have to scroll in order to see it.  In this sense, it's no better than using callbacks.  However, other mechanisms in Protractor allow you to remedy this.  You can run promises all at once, and then wait for them all to complete, by using this construct:

var ar = function() {    // hypothetical API request
    httpRequest(settings);    // settings = what you use for http.request(settings, ...) where http = require('http')
var ar2 = function() {    // hpothetical request #2
protractor.promise.all([browser.controlFlow().execute(ar), browser.controlFlow().execute(ar2)].then(function(responseArr) {
    // Assuming httpRequest() fulfills the promise with the raw HTTP response:
    console.log(responseArr[0]);    // response from ar
    console.log(responseArr[1]);    // response from ar2

Just use protractor.promise.all([<array>]).then, and you can greatly cut down on indentations required.  This even works, of course, on native selenium-webdriver functions that don't require being wrapped in execute().  Unfortunately, when making HTTP calls, my experiments showed that these calls were not run in parallel.  Then again, the calls I made returned very quickly; perhaps if I made a request involving more latency, I might see more parallel behavior.

A Word Of Caution Regarding Loops

If you have code inside loops that calls other functions based on the loop iterator's value, it's easy to run into a situation where the function uses the final value of the loop iterator each (or most of the) time it runs.  For instance, if inside the second .then() function from the database example above, you wanted to call external APIs based on data in each row retrieved, you could end up executing that API call using the last row as the parameter each time unless you heed some basic JavaScript syntax advice.

browser.controlFlow().execute(qr).then(function(rows) {
    for (r in rows) {
        (function(row) {
            // Make the HTTP request based on each row in here

This essentially creates a whole bunch of function instances where row is defined within the scope of each instance, not globally; then these functions get executed in series using the data you expect.  I bet that by assigning each function as an array element, you could possibly kick off all these functions in parallel with some other clever constructs!

Thursday, March 5, 2015

Angular + Protractor + Sauce Connect, launched from Gulp, all behind a corporate firewall!

You didn't think it could be done, did you?

Well, let me prove you wrong!  First, some terms:
  • AngularJS: An MVC framework for JavaScript, allowing you to write web apps without relying on JQuery.
  • Protractor: A test harness for Angular apps.
  • Sauce Labs: A company providing cloud services to help you run E2E testing on your web app in any environment combination.
  • Node.js: Package manager for JavaScript.  You'll need this for installing all the dependencies to get this tool set working.
  • Gulp: A build manager and mundane-task automator.  Competitor with the older, well-entrenched Grunt, but gaining popularity by the hour.  Uses JavaScript syntax, but could theoretically be used as a Makefile or shell script replacement.

The Basic Premise

My organization is writing web apps using Angular.  Long before I joined, they selected Gulp to manage application tasks such as allowing it to run on localhost at port 8888 for development & unit test purposes.  They also selected Protractor as a test harness to interact with the web app.  Protractor depends on the presence of Angular in order to work properly, and provides the use of Selenium WebDriver (for interacting with browsers) and unchained promises (a JavaScript construct to avoid callback functions).

Sauce Labs has been selected as the testing tool of choice because it saves us from having to set aside a massive amount of infrastructure to run tests on multiple platforms.  Through the configuration file for Protractor, I can specify exactly what OS platform & browser combination I want the test to run on.  Of course, being an organization such as it is, they also have a corporate firewall in place that will prevent the VMs at Sauce Labs from accessing development & test deployments of our web apps under construction under normal circumstances.  This is where Sauce Connect comes in: it provides a secure mechanism for the external Sauce Labs VMs to acquire the data that the server would serve to you as if you were inside the corporate firewall.  Winful for everybody!  The best part is that Sauce Labs is free for open-source projects.

Journey Through the Forest: Wiring All This Together

It is, truthfully, "stupid simple" to set up a Gulp task that will run Protractor tests through the Sauce Connect mechanism.  All you need in your Protractor configuration file is:

exports.config = {
    sauceUser: "your login name",
    sauceKey: "the GUID provided to you on your dashboard on Sauce's site",
    specs: ["the files to run as your test"],
    sauceSeleniumAddress: "this is optional: default is, but localhost:4445/wd/hub is also valid for when you're running sc locally and ondemand doesn't work",
    capabilities: {
        'tunnel-identifier': 'I will explain this later',
        'browserName': "enter your browser of choice here"

(Note that where it says ":4445" above should be replaced by the port number specified by the sc binary if it says anything different.)  It's so simple that you don't even need any "require()"s in the config file.  And in your Gulpfile, all you need is this:

gulp.task('sauce-test', function() {
    gulp.src('same as your "specs" from above, for the most part (unless your working directory is different)')
        configFile: 'path to the config file I described above'
    })).on('error', function (e) {
        throw e;
    }).on('end', function() {
        // anything you want to run after the Sauce tests finish

Then, of course, you can run your tests by writing "gulp sauce-test" on the command line set to the same directory as the Gulpfile.  However, proper functioning of this configuration eluded me for a long time because I did not know the Sauce Connect binary ("sc" / "sc.exe") was supposed to be running on my machine.  I thought the binary was running on another machine in the organization, or on, and all I needed to do was set the settings in the Gulpfile to the instance of sc that's remote (with the SauceSeleniumAddress entry).  While I could point the SauceSeleniumAddress to a different host, it was a flawed assumption on my part that anyone else in my organization was running "sc" already.  Also, might not answer the problem because it doesn't provide the services in "sc" by itself.  It is most convenient to run sc on your own system.

This configuration issue stymied me so much that I actually played with Grunt and several plugins therein before realizing that running tests through Sauce Connect was even possible through JavaScript to any extent.  Ultimately, I found a Node plugin for Grunt called "grunt-mocha-webdriver" that proved to me this was possible, and even doable in Gulp with Protractor and Selenium-WebDriver like I want, as opposed to Grunt/Mocha/WD.js.  (By the way, blessings to jmreidy, since he also wrote the sauce-tunnel which is relied upon heavily in this tutorial.)

Nevertheless, the easiest way to run Sauce Connect on your own system is to install the "sauce-tunnel" package through npm, the Node Package Manager (visit for other hilarious things "npm" could stand for :-P).  This is, of course, achievable by running the following on the command line:

npm install sauce-tunnel

If sauce-tunnel is already in your node_modules directory, then good for you!  Otherwise, you could run this in any directory that "npm" is recognized as a valid command, but you might want to place this module strategically; the best place to put it will be revealed below.  Nevertheless, you need to traverse to the directory where sc is located; this depends on what OS you are running, as the sauce-connect package contains binaries for Mac OSX (Darwin), Linux 32/64-bit, and Windows.  So, run the "sc" executable for your given platform before you run the Gulp task specified above, or else Gulp will appear to time out (ETIMEDOUT) when it's trying to get into Sauce Connect.

The minimum options you need for sc are your Sauce login name and your Sauce key (the GUID as specified above).  There are more options you can include, such as proxy configurations, as specified in the Sauce Connect documentation.  (Note that the tunnel-identifier, as called out in the Protractor config file, can be specified as an argument to sc.)

In simple terms, here's what we have thus far:

[assuming you've set up all the Node packages]:

vendor/[platform]/bin$ sc -u <your user name> -k <your GUID key> [-i <tunnel name>] [other options]

gulp-workingdir$ gulp sauce-test

This will set up "sc" for as long as your computer is hooked up to the Internet, and will run the Sauce tests on the existing tunnel.  The tunnel will remain active until you disconnect your computer from the Internet or end the sc process, but the tests running through Gulp will set up & tear down a Selenium WebDriver that'll drive the UI on your web app.

Help!  The test did not see a new command for 90 seconds, and is timing out!!!

If you are seeing this message, you might be behind a corporate proxy that is not letting your request go straight through to the Sauce servers.  Protractor has in its "runner.js" file a section where it will pick a specific DriverProvider based on certain settings you provide in the configuration file, and by providing the "sauceUser" and "sauceKey" values, it will pick the "sauce" DriverProvider.  The sauce DriverProvider provides an "updateJob" function that communicates with Sauce Labs (via an HTTP PUT request) on the status of the job.  This function is supposed to run after the tests conclude, and if that HTTP request fails, then the Gulp task will not end properly; thus, you will see this message.  Your list of tests in your Sauce Connect dashboard will look like this:

This message is so severe in Sauce that it doesn't just show up as "Fail," it shows up as "Error".  It also wastes a bunch of execution time, as seen in the picture above, and will obscure the fact that all the test cases actually passed (as they did in the pictured case above).  If you see this message after it is apparent that there are no more commands to be run as part of the test, then it is probably a proxy issue which is easy to resolve.

Here's how:

In your Protractor configuration file, add the following lines:

var HttpsProxyAgent = require("https-proxy-agent");

var agent = new HttpsProxyAgent('http://<user>:<password>@<proxy host>:<port>');

exports.config = {
    agent: agent,
    // things you had in there before

Then, in your node_modules/protractor/lib/driverProviders/sauce.js file (i.e. the DriverProvider for Sauce Labs in Protractor), add this:

this.sauceServer_ = new SauceLabs({
    username: this.config_.sauceUser,
    password: this.config_.sauceKey,
    agent: this.config_.agent    // this is the line you add

Once you have your https-proxy-agent in place as specified, your PUT request should go through, and your tests should pass (as seen in the Sane jobs).

The whole process, end-to-end, running in Gulp

If it does not satisfy you to simply run the "sc" binary from the command line and then kick off a Gulp task that relies on the tunnel already existing, you can get everything to run in Gulp from end to end.  To do this, you need to require sauce-tunnel in your Gulpfile (thus you might as well run npm install sauce-tunnel from the same directory that your Gulpfile exists).  Then, you need to make some changes to the Gulpfile: add some additional tasks for tunnel setup & teardown, and some special provisions so these tasks are executed in series rather than in parallel.

var SauceTunnel = require('sauce-tunnel');
var tunnel;

gulp.task('sauce-start', function(cb) {
    tunnel = new SauceTunnel("<your Sauce ID>", "<Your Sauce Key>", "<Sauce tunnel name -- this must be specified and match the tunnel-identifier name specified in the Protractor conf file>");
    // >>>> Enhance logging - this function was adapted from that Node plugin for Grunt, which runs grunt-mocha-wd.js
    var methods = ['write', 'writeln', 'error', 'ok', 'debug'];
    methods.forEach(function (method) {
        tunnel.on('log:'+method, function (text) {
            console.log(method + ": " + text);
        tunnel.on('verbose:'+method, function (text) {
            console.log(method + ": " + text);
    // <<<< End enhance logging

    tunnel.start(function(isCreated) {
        if (!isCreated) {
            cb('Failed to create Sauce tunnel.');
        console.log("Connected to Sauce Labs.");

gulp.task('sauce-end', function(cb) {
    tunnel.stop(function() {

gulp.task('sauce-test', ['sauce-start'], function () {
    gulp.src('<path to your Protractor spec file(s)>')
        configFile: '<path to your Protractor conf file>'
    })).on('error', function (e) {
        throw e;
    }).on('end', function() {
        console.log('Stopping the server.');'sauce-end');

Note here that the cb() function is new to Gulp, yet the "" construct mentioned toward the bottom of the code snippet above is actually deprecated.  I will get around to fixing that once it stops working, but I think that in the grand scheme of priorities, I'd rather clean the second-story gutter with only a plastic Spork first before fixing that deprecated line. :-P

At this point, you should be able to run a test with Sauce Connect from end to end in Gulp without any extra intervention.  However, if Gulp is failing because it can't write to a file in a temporary folder pertaining to the tunnel (whose name you picked), then you can always run gulp as root find a way to have it save to a different temporary location that you have access to, since it's always good to minimize running things as root.

One Brief Important Interruption about Lingering sc Instances...

If these instructions haven't worked out 100% for you, or you are me and spent a great deal of time exploring this, you may be frustrated with how many times Sauce Connect hangs around when there's been a problem.  You can't start the Sauce Connect binary again if it's already running, yet if you try to do this, it gives you an esoteric error message that does not make it apparent that this is indeed what happened.  To remedy this in a *nix operating system, simply write "pkill sc", as long as you don't have other critical processes that have "sc" in their name.  In my case, the other processes with "sc" in the name are running under a different user, and I don't have privileges to kill them (I'm not logged in as root nor running "sudo pkill sc"), so it doesn't do anything harmful to the system.

Shutting It Down Cleanly

In order to properly shut down sc, you may have noticed one final Gulp task in the code snippet above -- "sauce-end".  This task, in the background, runs an HTTP DELETE operation on, and is subject to corporate proxy rules once again.  To circumvent this, you can simply require https-proxy-agent in node_modules/sauce-tunnel/index.js (like we did in the Protractor configuration file), and set up the agent in the same way.  In this case, you will edit the code in node_modules/sauce-tunnel/index.js as such:

// other pre-existing requires
var HttpsProxyAgent = require("https-proxy-agent");

var agent = new HttpsProxyAgent('http://<user>:<password>@<proxy host>:<port>');

// other existing code
this.emit('verbose:debug', 'Trying to kill tunnel');
  method: "DELETE",
  url: this.baseUrl + "/tunnels/" +,
  json: true,
  agent: agent    // this is the line you add
}, // ... etc

Now, obviously, this is not sustainable if you wish to ever upgrade sauce-tunnel or wish not to include a proxy agent.  For this, I will be submitting "less hacky" fixes to the respective GitHub repositories for these open-source Node modules in order to make it easier for all users in the future to use Sauce Connect with Protractor through their corporate proxies.

Nevertheless, there's no harm in this DELETE call failing, other than it makes the Gulp task stall another minute or so, which is annoying when you're at work late trying to learn how all this stuff works in order to finish off some polishing touches on your big project.

To recap running everything from end to end in Gulp:

[Assuming you've set up all your Node packages to run a Protractor script with the conf file set up for Sauce Labs, as described above]:
  • In the same directory as your Gulpfile, run:
    npm install sauce-tunnel
  • Set up your Gulpfile in the manner I described above, with the sauce-tunnel require, and the "sauce-start", "sauce-end", and "sauce-test" tasks, and with the "Sauce tunnel name" (3rd argument in new SauceTunnel()) set to the same value as the Protractor config file "tunnel-identifier" value.  Be sure to study all the possible values that "new SauceTunnel()" takes, as you can pass in options to the sc binary if you need them.
  • If you are behind a corporate proxy or firewall, make the recommended edits to the Sauce DriverProvider at node_modules/protractor/lib/driverProviders/sauce.js, and to the sauce-tunnel module at node_modules/sauce-tunnel/index.js.
  • Run the Gulp task.
    gulp sauce-test
    sudo gulp sauce-test
Once again, I plan to check in "more sustainable" and "less hacky" code to help you deal with corporate proxies in the future without making temporary workarounds to downloaded modules.