npm install dependencies of a git submodule

Question

I've got a git project with handlebars.js as a submodule. I will provide here a bare-bones example - the full project is more complicated with dojo tasks and deployment steps that aren't relevant here.

$ mkdir handlebarstest ; cd handlebarstest ; git init
$ mkdir src ; git add src
$ git submodule add https://github.com/wycats/handlebars.js.git src/handlebars.js

Our package.json doesn't mention handlebars.js, just grunt:

{
  "name": "handlebarstest",
  "version": "1.0.0",
  "dependencies": {},
  "devDependencies": {
    "grunt": "~0.4.1",
    "grunt-contrib-clean": "~0.4.1",
    "grunt-run-grunt": "latest"
  }
}

In this project's history, our install and build procedure has been:

$ npm install   # install dependencies in package.json
$ grunt init    # run the 'init' task which initialises git submodules
$ grunt build   # run the 'build' task which compiles dojo etc.

Having done a fresh clone of our project, say on our build server, git submodules need to be initialised, and we control that from grunt, so we must install grunt first, then run a task to init submodules, including the troublesome handlebars.js.

We do npm install and install grunt, and grunt init which fetches handlebars.js including its package.json. So the root of our problem is that the package.json is not available when the top level npm install is run.

Our Gruntfile.js knows how to call the Gruntfile.js in handlebars.js:

/*jshint node:true */
module.exports = function (grunt) {
    /*jshint camelcase: false */
    var path = require('path');

    grunt.initConfig({
        clean: [ 'dist' ],
        run_grunt: {
            options: {

            },
            handlebars_build: {
                options: {
                    log: true
                },
                src: [ 'src/handlebars.js/Gruntfile.js' ],
                task: 'build'
            },
            handlebars_amd: {
                options: {
                    log: true
                },
                src: [ 'src/handlebars.js/Gruntfile.js' ],
                task: 'amd'
            }
        }
    });

    // var handlebarsLink= grunt.registerTask('handlebarsLink', function () {
    //  var done = this.async(),
    //      child = grunt.util.spawn({
    //          cmd: 'npm',
    //          args: [ 'link', 'src/handlebars.js' ]
    //      }, function (error) {
    //          if (error) {
    //              grunt.warn(error);
    //              done(false);
    //              return;
    //          }
    //          done();
    //      });

    //  child.stdout.on('data', function (data) {
    //      grunt.log.write(data);
    //  });
    //  child.stderr.on('data', function (data) {
    //      grunt.log.error(data);
    //  });
    // });
    var submodules;
    if (grunt.file.exists(__dirname, '.git')) {
        submodules = grunt.registerTask('submodules', function () {
            var done = this.async(),
                child = grunt.util.spawn({
                    cmd: 'git',
                    args: [ 'submodule', 'update', '--init', '--recursive' ]
                }, function (error) {
                    if (error) {
                        grunt.warn(error);
                        done(false);
                        return;
                    }
                    done();
                });

            child.stdout.on('data', function (data) {
                grunt.log.write(data);
            });
            child.stderr.on('data', function (data) {
                grunt.log.error(data);
            });
        });
    }
    var init = submodules ? [ 'submodules'/*, 'handlebarsLink'*/ ] : [];
    grunt.registerTask('init', init);
    grunt.registerTask('default', 'build');
    grunt.registerTask('build', init.concat([ 'clean', 'run_grunt:handlebars_build', 'run_grunt:handlebars_amd' ]));

    grunt.loadTasks(path.join(__dirname, 'grunt'));
    grunt.loadTasks(path.join(__dirname, 'src', 'intern', 'tasks'));
    grunt.loadNpmTasks('grunt-contrib-clean');
    grunt.loadNpmTasks('grunt-run-grunt');
};

Running grunt fails because it recurses into handlebars.js but the module dependencies in its package.json have not been installed.

Running "run_grunt:handlebars_build" (run_grunt) task
--> starting  "src/handlebars.js/Gruntfile.js"
--> reporting "src/handlebars.js/Gruntfile.js"
  |  >> Local Npm module "grunt-contrib-clean" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-concat" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-connect" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-copy" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-requirejs" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-uglify" not found. Is it installed?
  |  >> Local Npm module "grunt-contrib-watch" not found. Is it installed?
  |  >> Local Npm module "grunt-saucelabs" not found. Is it installed?
  |  >> Local Npm module "es6-module-packager" not found. Is it installed?
  |  Loading "metrics.js" tasks...ERROR
  |  >> Error: Cannot find module 'underscore'
  |  Loading "publish.js" tasks...ERROR
  |  >> Error: Cannot find module 'underscore'
  |  Loading "version.js" tasks...ERROR
  |  >> Error: Cannot find module 'async'
  |  Warning: Task "clean" not found. Use --force to continue.
  |  
  |  Aborted due to warnings.
  |  
--> failed "src/handlebars.js/Gruntfile.js" (304ms)
--> failed handlebars_build @ "src/handlebars.js/Gruntfile.js"

Warning: 1 gruntfile failed and completed 0 (308ms)
 Use --force to continue.

Solutions might be:

Use npm link somehow in our top level package.json to do something clever with a script hook after npm install has finished. This seems impossible, because the handlebars.js/package.json isn't going to be available until well after npm install has finished.
Put a manual step before run_grunt:handlebars_build to recurse into src/handlebars.js and run npm install in that directory. This seems more manual than I would expect.
Put a step before run_grunt:handlebars_build that installs dependencies from src/handlebars.js/package.json into the project's top level node_modules directory, which might then be picked up when running Gruntfile.js inside src/handlebars.js (this may require that there is no node_modules directory in src/handlebars.js - my knowledge of npm and node_modules is not great).
Give up on trying to build handlebars.js as part of our project's build, remove the src/handlebars.js submodule from the project, and just add built files as plain files in its place. This would be a shame because you lose the benefits of tracking a project's versions intelligently via the git submodule approach.

Some advice would be appreciated on the best way forward.

I guess I'm missing why you are "building" handlebars in the first place? You say to "track a project's versions intelligently", but I'm not sure what you mean by that. How are you tracking the versions? And why? Separately from that, are you using Handlebars only on the client, or in a precompilation step? If only on the client, perhaps switching that dependency to something like Bower would be better? — Jordan Kasper
– Jordan Kasper, Commented Jan 26, 2014 at 17:22
@jakerella by 'tracking versions' I simply mean that we can have a specific version of the project available at a time, no other reason.. basically what the package.json 'dependency' line would give us, but managed in this case via the git submodule revision. It is for a precompilation step.. our grunt build needs to run the Dojo build process, which collects files in the build directory (including handlebars and its AMD pieces) and prepares them as a 'dist' web app. Perhaps Bower is the right way forward for us, I will try it. Thank you for the tip. — Neek
– Neek, Commented Jan 27, 2014 at 3:13
Hmm... ok, so if it's necessary as a build dependency, then I guess I don't follow why you can't include it using npm and the package.json file. Sorry, I'm sure I'm missing something, just not sure what. — Jordan Kasper
– Jordan Kasper, Commented Jan 27, 2014 at 14:14

nazreen · Accepted Answer · 2021-01-04 07:20:58Z

51

To ensure your submodule's dependencies are installed whenever you install your main repo's dependencies, you need to modify your main package.json.

Assuming your submodule is called 'common' and sits under src/common and its package.json is in src/common, then,

You should run npm install --save file:src/common and it will add the dependency to your package.json (or you can add by yourself, just append to your dependencies key the key:value of "common": "file:src/common",

Running npm install afterwards, will install all dependencies in the main module and submodule.

These steps will also help solve the issue in continuous deployment environments.

edited Jan 4, 2021 at 7:20

nazreen

2,1282 gold badges22 silver badges32 bronze badges

answered May 7, 2020 at 16:02

Yarh

1,1188 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

nazreen Over a year ago

this answer is the more useful one and should be the accepted answer

kkokki Over a year ago

this answer is really helpful!

Vivek Patel Over a year ago

I have been looking for this answer for the past 6 hours. Thanks, it helped. This answer should have been accepted.

VitoCK Oct 8 at 13:05

Hey, @Yarh, have you ever hit a module not found when building an image and running as a container? I have problem with one specific package node-cron and just cannot understand what is the problem, and why it isn't hoisted properly.

Yarh Oct 12 at 12:14

Not sure it's directly related to the original question/answer. Maybe it's an issue with peer-dependencies (although it's solved with newer npm versions): stackoverflow.com/questions/35207380/…

VitoCK Oct 14 at 9:06

Heya, the problem was that for some reason, npm didn't hoist one of the submodules' packages. I assume it was because it was imported using CommonJS instead of ES module style. The second problem was with my submodule package name. I renamed it from "common": "file:submodules/commons-submodule", to "commons-submodule": "file:submodules/commons-submodule",, and it was building and running it without problems.

bvfnbk · Accepted Answer · 2015-02-10 14:28:05Z

this might be of some help to someone stumbling over this question like i did during my research regarding a similar question:

My solution uses the grunt plugin grunt-run to run npm install in the submodule directory (installing its dependencies into its own node_modules directory: that would be src/handlebars.js/node_modules/ in your case).

In my case i needed to compile the TypeScript files using a grunt task inside the sub-module afterwards. I solved this using the plugin grunt-submodule.

The

>> Local Npm module "grunt-contrib-clean" not found. Is it installed?`

errors seem to appear if grunt is being executed in the wrong working directory - if you've managed to install the dependencies before.

Example:

Only the sub-module uses grunt-contrib-clean to clean its build artifacts. You've installed the sub-modules requirements (inside its directory) but you run grunt clean in the parents directory. The node module is not known in this place.

Hope that helps,

Markus

These tasks are of course run from within grunt, integrated into my build/development life-cycle.

References:

Collectives™ on Stack Overflow

npm install dependencies of a git submodule

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related