4

I'm working in a web application (JavaScript/C#, version controlled by TFS) and our team wants to start using Visual Studio 2015. Microsoft is moving developers to use existing popular tools like Gulp for automated tasks, so I've written a few Gulp tasks that will run on the server.

My problem is that our automated builds generate new project folders on the build server, so I can't run gulp myBuildTask without first running npm install. The npm install adds over 2 minutes to the build process, and it seems very inefficient to download the same dependencies for every build (since they will change rarely).

Is there anyway I can run a Gulp task on a new project folder without first running npm install?

Options I've considered:

  1. Include node_modules in TFS. I couldn't add the node_modules folder to TFS (which would cause it to exist in each new build folder) because bower's nested dependencies have file paths that are too long for Windows. I could go this route without bower, but I'm not certain I want all those files in my solution (much of which is not needed, like readme's and test files).

  2. Run npm install after each automated build. As already mentioned, I don't want to do this because it adds several minutes to the build process.

  3. Install NPM modules globally. I'm not sure if this is even possible, but I'm wondering if I can install all project dependencies globally on the build server (avoiding having to install at the project level). My concern with an approach like this is that I don't want to have to manually update the build server's globally installed NPM modules every time we add a gulp plugin.

Ideally, the solution would be something like #3. The modules would install globally, but every build could run an npm install which would verify every module is installed. If a new npm module was added to the package.json, it would be downloaded. This npm install would be pretty fast since in most cases, all modules would already exist (globally installed on the build server).

Improve this question

1 Answer 1

Reset to default
2

There are a few things you might do:

  • Make npm install run faster. For this purpose, use newest npm (if possible) or use npm dedupe. Running dedupe may result in having less dependencies than with plain npm install. Then run npm shrinkwrap which creates npm-shrinkwrap.json file which contain 'freezed' info about what exactly gets installed (and in which version) during npm install.

  • Remember, node_modules is just a directory, if you can copy / rsync it to your installation, you can skip the npm install phase altogether

  • Node package resolution approach is to first try local node_modules directory and if not successful, (node_modules not there or dependency missing in node_modules) check out node_modules of the parent directory, then grandparent directory and so on. This means, you don't have to install packages globally, semi-global installation is quite sufficient

:

my_project
    node_modules/
        dependency1
        dependency2
    build_001/ 
    build_002/
    build_00x/
        no node_modules here, 
        no deps here

Note however, that this, naturally, works only if your dependencies are really not changing. Since in real life you install something new from time to time, slightly enhanced approach might be helpful: organize your directories as follows:

my_project
    ver_af729b
        node_modules
        build_001
        build_002
    ver_82b5f3
        node_modules
        build_003
        build_004

af729b and 82b5f3 being (prefixes of) sha hashes of your npm-shrinkwrap.json file. If you then add new dependency, shrinkwrap file gets updated, build script creates new ver_something directory and executes npm install in it. Doing all this would naturally require extra work, but it should work great.

------------------ EDIT -------------------

If you are not trying to avoid npm install completely (you just want it to be quick) you can stick to the typical scenario: you checkout the sources always to the same directory, and let npm install re-use the old node_modules as much as possible.

If you want always to create a new directory for your build, you may still create a node_modules symlink to the older version of node_modules - also in this scenario, npm will reuse as much as possible from symlinked folder.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Excellent suggestion using a parent directory, I didn't realize that's how npm worked. A folder structure of my_project/node_modules and my_project/builds should work out nicely. Then I could run npm install with every build and it would be quick since everything would already be installed (except when new modules are added by devs). I'll give it a shot Monday and update here (and accept an answer).
wait, it's not that easy. If you run npm install it ignores the fact that there is ../node_modules directory and just runs full install, creating node_modules in the same dir where package.json is present. However, you could do another thing: symlink the parents node_modules to the right place and then run npm install. I've tried this and it works like a charm. I'll edit the answer and mention this possibility.
So, did you try this / does it work for you? If so, could you please accept the answer? Thanks.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.