that time when our mocha tests passed consistently... every second time that we ran them

We had a thorough suite of tests for our node/express server and mongodb test database, and we were delighted that they were putting our app through the paces. We were even happier that we'd tweaked our app's code until all of them passed. But we were befuddled when the tests consistently failed every second time that we ran them.

A peak inside our code

describe('Video Routes API', function () {

  var server;

  before(function (done) {
    process.env['MONGOHQ_URL']="mongodb://ourApp:ourPWD@subdomain.mongohq.com:10042/ourTestDB";
    server = child.exec('node web.js &', { killSignal: "SIGTERM"}, function(err, stdout, stderr){
      console.log('something', err, stdout,stderr);
      done();
    });
  });
  after(function(done){
    server.kill();
    done();
  });
}

What we were hoping to achieve here was the ability to spin up our server instance from inside of the test suite. This way we could set our environmental variable for the database to direct connections to our remote test database. Also, this would provide for a less granular set of integration tests.

We used the child.exec() method because we expected it to asynchronously start our server from the shell, and return a callback when it was ready.

The results:

First time: All tests pass

Second time:
0 passing (8s) 6 failing

1) User API Video Routes API GET /clips should return 200:
   Error: timeout of 2000ms exceeded
    at null.<anonymous> (/usr/local/lib/node_modules/mocha/lib/runnable.js:175:14)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
...

6) User API Video Routes API should return 200 upon reciept of favorite of a video:
   Error: connect ECONNREFUSED
    at errnoException (net.js:901:11)
    at Object.afterConnect [as oncomplete] (net.js:892:19)

We were wrong. A few things were fairly obvious:

1. the callback was never being called!

2. our tests were unable to connect to our server.

Fine, but why only every other time? We hypothetsized that our server was not hanging up after our tests were done running, but it didn't make a whole lot of sense that it would be unavailable for reconnections.

Resolution

After a long, painful exploration of the native child process functions in nodejs (really, I am still quite grateful for the opportunity to have become familiarized), we understood that our callback wasn't being called because there was never any "finish" to the buffered output of a still-running server. child.exec() was not the method we needed. Sure enough, we verified that our server process was still running after the tests ran, and on the next subsequent attempt, our tests connected and passed. Apparently our server.kill() method wasn't getting the job done for us. As it turns out, child.exec() isn't going to be our friend for that method either.

The answer was two-fold:

1. substitute child.exec() with the synchronous child.spawn().

2. implement a timeout before the tests run, just to give the server some time to get up and running before we try to test it.

It's kind of a hacky solution, we know. In a world where we had more time with this project, we would have explored having our server emit a 'ready' message for our tests to listen to. For now, though, this worked just fine:

describe('Video Routes API', function () {  
  var server;

  before(function (done) {
    process.env['MONGOHQ_URL']="mongodb://ourApp:ourPWD@subdomain.mongohq.com:10042/ourTestDB";
    server = child.spawn('/usr/local/bin/node', ['/Users/path/to/ourApp/web.js'],  {detached: true, stdio: [ 0, out, err ]});
    setTimeout(done, 1000);
  });
  after(function(done){
    server.kill();
    done();
  });
}

Now our tests run properly every single time, and they pass to-boot!

TL;DR

Consider child.spawn() when you need to start a different program from inside of your running, node program. And don't presume that your server is ready for connections immediately after instantiating it!