132

I'm trying to execute curl using node child_process to get a JSON file (about 220Ko) from a shared folder in a local network. But it actually returns a buffer problem that I can't get throught. Here is my code:

var exec = require('child_process').exec;

var execute = function(command, callback){
    exec(command, function(error, stdout, stderr){ callback(error, stdout); });
};

execute("curl http://" + ip + "/file.json", function(err, json, outerr) {
    if(err) throw err;
    console.log(json);
})

And here is the error I get:

if(err) throw err;
          ^
Error: stdout maxBuffer exceeded.
    at Socket.<anonymous> (child_process.js:678:13)
    at Socket.EventEmitter.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:746:14)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
    at Pipe.onread (net.js:526:21)
Improve this question

4 Answers 4

Reset to default
214

You need to use and set the maxBuffer option when using child_process.exec. From the documentation:

maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

The documentation also states that the default value of maxBuffer is 200KB.

As an example, the maximum buffer size is increased to 500KB in the following code:

var execute = function(command, callback){
    exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ callback(error, stdout); });
};

Additionally, you may want to read about http.get to see if it is capable of achieving what you are trying to do.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

This solved my problem, thanks! The shared folder is actually under webdav protocol that requires a digest authentication, that's why i'm using curl which handles it very easily with curl --digest http://login:password@" + ip + "/webdav/file.json
This default is ridiculously small. This is the second time I was bitten by this in a hard-to-find way.
The default is now 1MB @jlh, nodejs.org/api/…
A little trick. If you set {maxBuffer: undefined} then there will no limit to the buffer size. Internally child_process allocates small buffer chunks as the command produces output, so it never preallocate a maxBuffer-sized buffer, but rather grows it as needed. maxBuffer is meant only as a fail-safe to stop a process that's outputting too much. undefined happens to slips through the validation checks and also circumvents the fail safe. It's not documented anywhere and may stop working at any time. Use with care.
to be clear: maxBuffer is a copy. OS' allocate an internal buffer for stdout per their own rules. If proc would overfill that buffer, it's suspended. Assuming the emit requirements by nodeJS are met (which I believe is a simple newline character anywhere in the buffer?), subprocess.stdout.on will fetch a chunk of the buffer, fire it at the handler, and free that chunk, letting the OS resume the producing subprocess. I believe the C API's for stdout require that it be simply a cbuf[] with some indicies, universally exposed as a circular array. Is this all correct?
69

I had a similar issue and I fixed it moving from exec to spawn:

var child = process.spawn('<process>', [<arg1>, <arg2>]);

child.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

child.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

child.on('close', function (code) {
    console.log('child process exited with code ' + code);
});
Improve this answer

4 Comments

nodejs.org/api/…
This answer is not necessarily the most appropriate one. I think console output in the question could be just an example. Hardly someone would fetch a 200KB file to throw it to console. However, if process.exec is used in things like CLI tools, then yes, switching to spawn should be the way to go.
wow... spawn is cool. It's not even using call backs or promises... just events. That could be really useful for streaming stdout to the console. @Pavel Gatilov, that is exactly what we're doing. FFMpeg likes to show progress every second... which takes a toll on the buffer
This event listener pattern can be used in exactly the same way with exec instead of spawn (just don't use a callback in exec). I prefer using exec since it does not require splitting up arguments into a separate array.
9

Adding some explanation to the answers.

The exec command buffers the data before sending it to the parent process. It is generally suitable for the commands producing the smaller output. The above error occurred because the output generated by the execution of the command was larger than the max buffer size. One way to solve the above issue is to specify the buffer size as answered by Tim Cooper.

var execute = function(command, callback){
exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ 
 callback(error, stdout); });
};

Another solution is to use the spawn method which is generally faster compared to the exec and it does not buffer the data before sending. It sends the data as a stream hence the problem of buffer size never occurs. The code snippet used by Isampaio.

var child = process.spawn('<process>', [<arg1>, <arg2>]);
child.stdout.on('data', function (data) {
 console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
 console.log('stderr: ' + data);
});
child.on('close', function (code) {
 console.log('child process exited with code ' + code);
});
Improve this answer

Comments

0

Fastest solution ever:

If this line outputs a buffer:

childProc.stdout.on('data', (data) => console.log(data)); // <Buffer 1b 5b 34 6d>

You just change it to:

childProc.stdout.on('data', (data) => console.log(`${data}`)); // bla bla bla

JS console will encode your buffer.

Or we can simply use: console.log(data.toString()).

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.