Running multiple NodeJs versions in TeamCity on Windows

If you’re building Web applications using TeamCity, you’re probably going to want to execute your Front-End pipeline as part of the build. You’ve worked hard to perfect all that lovely unit testing, linting, CSS precompilation and so on in your Gulp* build, so this part of your software packaging process should be a first-class citizen in your automated build process.

*Most of what’s in this post probably applies equally to Grunt.

Don’t have a centralised build server set up? I strongly recommend you buy a copy of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation to understand not only build automation, but further techniques to improve both speed and quality of your software releases.

In theory, this should be straightforward if you’re using TeamCity – it’s a powerful and flexible product. However, the first issue to understand (and one of the primary factors in what’s to follow) is that all of our back-end work is done on the Microsoft stack, and so our TeamCity Agents are Windows Server VM’s. Layered on top of that you then have Node, NPM, Gulp and all the various package dependencies that your particular pipeline and solution brings in.

The reality is that getting this all working definitely isn’t the well-trodden path that I expected. I’ve lost countless hours Googling, reading StackOverflow posts and mailing list discussions to get to the bottom of all manner of bugs, caveats and gotchas, so it felt right that I share my experiences.

Welcome to the Inferno…

Beginner Level: Fixed Node Version

So in the early days of our build process, things were nice and simple. We installed a particular version of NodeJs on all our TeamCity build agents. To accomplish this, we have a batch script which uses Chocolatey to install dependencies on agents. The following line from this script installs NodeJs v4.5.0, which in turn brings Node package manager (NPM) v2.15.9 along for the ride:

cinst nodejs -version 4.5.0 -y

With NodeJs and NPM installed as pre-requisites on all the agents, it was then fairly straightforward to add steps to our build configuration which invoke the Front-End pipeline. I’ve previously written about how we setup our TeamCity instance, and mentioned Eugene Petrenko’s TeamCity plugin for running Node.js, NPM, Gulp and Grunt.

So the front-end elements of our build process followed this sequence:

1. Runner type: Node.js NPM. Command: “install”
2. Runner type: Gulp*. Task command: “default”

*Again could be Grunt if that’s what you use. Ain’t nobody be judgin’ here. But be aware that if you are using Grunt, you’ll need the Grunt command line to be available to your build, either by adding an initial NPM runner step which does an “install grunt-cli”, or by installing it globally (Warren Buckley covered this approach in a helpful post whilst at Cogworks)

Note that the runner types used are generally the ones from Eugene Petrenko’s plugin that I mentioned – these worked well enough at this stage.

Intermediate Level: Variable Node Version

So after we’d been running many builds happily using the approach described above, we came to realise that NodeJs 4.5.0, released in August 2016, was getting rather out of date. (version 9.x had been released in late 2017). Such is the pace of the front-end development ecosystem, there were new front-end libraries being adopted within the industry which required more recent versions of NodeJs, and yet we were tied to a hard dependency on 4.5.0 which was installed on our build agents.

Time for a rethink.

My front-end colleagues pointed me in the direction of Node Version Manager (NVM) which they would use during local development in order to avoid a fixed NodeJs dependency. NVM comes in 2 flavours: the original NVM bash script which runs on Unix variants (Mac/Linux), and then there is NVM for Windows, developed using the Go language. One thing to mention is that there is not feature parity between these 2 tools. More on this later.

So we created a new TeamCity agent and installed the Chocolatey NVM for Windows package, so that our builds would be freed from the shackles of NodeJs v4.5.0:

cinst nvm -version 1.1.5 -y

With NVM installed, I attempted (sensibly, one might conclude) to use the NVM runner from Eugene Petrenko’s plugin, but was a little confused to find my builds wouldn’t run on the new agent and was greeted with the explanation that NVM was an ‘unmet requirement’ not just within my default pool of 2 existing agents (as expected), but also on my new agent shown at the top of this image:

Only after digging further did I then discover that this plugin has a known limitation which means it only detects NVM properly when running on Linux (at the time of writing there are currently several open issues on the GitHub repo about this).

Pro tip!

Here’s my top tip if you’re going through this same setup: make the first step in your build process a Command Line runner, and add the following commands into the “Custom script” box:

set /p nodeversion=<.nvmrc
nvm install %%nodeversion%%
nvm use %%nodeversion%%

Then, within your solution, include a .nvmrc file containing your target NodeJs version. Another crucial fact I learnt along this journey is that the Windows version of NVM doesn’t support the .nvmrc feature. However the command line script above is a “poor man’s” emulation of this capability. There are a few advantages to this approach. Firstly, you source control your required NodeJs version alongside the code, and can version it accordingly in different branches etc. This means no TeamCity parameters are required. Secondly, your front-end build is one step more compatible between Windows and Mac.

After having problems with it, I decided to avoid the TeamCity Node plugin altogether for the other front-end steps, and switch to simply calling npm and Gulp from command-line runners. With the NVM switch successfully configured, our builds were no longer blocked by the false-negative issue. At each build execution they would adopt the correct NodeJs version, run npm install and Gulp steps, and finally call a Nuget Pack step in order to package up the software into something that could be distributed.

I thought I was home and dry. If you’ve made it with me this far, well done.

WTF Level: Am I the only one?

It wasn’t over yet. I was still trapped in the bloody battle between the Windows beast and the Node monster.

I kinda got a bit lost in the pits of gotchas of various NPM package dependencies running on Windows at this point. To be honest, I don’t think I kept a record of every problem I had to resolve – I certainly lost count of the various different blog posts I read and hours I burned. However to get an idea of the scale of the problems you’re facing here, it’s worth taking a look through the comments in this post.

This comment summarises pretty much sums it up:

Zen Level: It’s not just me

Just when you’re about to claw your eyes out with a blunt pencil and wondering whether your entire career has been a mistake, you hit the GitHub post titled “Windows users are not happy”. Stumbling across this blog post was the equivalent of being on the side of an icy mountain for 2 weeks, running out of supplies, getting frostbite … and then falling through the door of a warm tavern full of welcoming and equally lost travellers, each with similar stories to tell.

The last pieces of the puzzle to get the whole process up and running were:

1. Ensure the C++ build tools module has been installed as part of the Visual Studio installation you have on your build agent.
2. Install Python 2 on your build agent. As per the linked post, it’s a dependency of node-gyp:
   

   choco install python2

3. I got a very intermittent error message in my build logs which said Error: EPERM: operation not permitted, scandir. From all the comments left in the thread of this particular post, the fix which worked for me was to run an extra command line step of “npm cache verify” just before the npm install.

And that was that. Our builds went green, consistently.

Of course if you’re very lucky you may not encounter as many issues as I did, depending on the exact nature of what you’re building, how many front-end packages you’re making use of etc. But I hope this narrative serves as a useful map [here be dragons!] of some potential pain-points to shortcut the journey of anyone else setting up similar build pipelines.