2

NodeJS is asynchronous, so for example if running an Express server, one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.

Since these are log files, it's not a simple write. Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.

There is a synchronous append function (fs.appendFile) but this would require us to delay servicing a request to wait for a log file write to complete and I'm still not sure if that guarantees an atomic append. What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?

Improve this question
2
  • There are a couple of atomic write packages on npm, just FYI (not sure if they'd solve your problem). npmjs.com/search?q=atomic%20write Commented Nov 2, 2016 at 14:45
  • Generate a guid at the start of a request and write it out at the start of every log entry. This means passing context around throughout your app, but it's not that hard really. In java this is called a Mapped Diagnostic Context (MDC). Commented Nov 2, 2016 at 14:49

1 Answer 1

Reset to default
2

one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.

The individual write calls will be atomic, so as long as you make a single log write call per request, you won't have any corruption of log messages. It is normal, however if you log multiple messages while processing a request, for those to be interleaved between many different concurrent requests. Each message is intact, but they are in the log file in chronological order, not grouped by request. That is fine. You can filter on a request UUID if you want to follow a single request in isolation.

Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.

Don't allow multiple processes to write to the same file or log. Use process.stdout and all will be fine. Or if you really want to log directly to the filesystem, use an exclusive lock mechanism.

What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?

process.stdout, one write call per coherent log message. You can let your process supervisor (systemd or upstart) write your logs for you, or use a log manager such as multilog, sysvlogd and pipe your stdout to them and let them handle writing to disk.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.