NodeJS Buffer Size Logic: why 8KB as a default and not just the data size?

Question

If you read a file smaller than 8KB in size NodeJS allocates 8 KB for it.

This isn't intuitive or expected, we are getting "more" bytes than we put. But it isn't a bug either at least in the sense that it is documented.

Minimal NodeJS Example

const { Buffer } = require("node:buffer");

// Create a buffer smaller than `Buffer.poolSize`.
const nodeBuffer = Buffer.from(new Uint8Array([0, 1, 2, 3, 4]));

// size is 8KB, not 5 bytes.
console.log(nodeBuffer.buffer);

Question

Why would 8KB buffer size be preferred over what we expect to have (the size of the data in the file.) ?

Files over 8KB would have an allocated space that matches the number of bytes in the file.

Other Details

I'm aware that we can use the offsets for reading:

// 1 KB
const buffer = fs.readFileSync(fileName, buffer.byteOffset, buffer.byteLength)

but the question isn't this.

AKX · Accepted Answer · 2024-03-14 12:03:25Z

0

The clue is in the name of the property you linked, poolSize.

Elsewhere on the same page:

The Buffer module pre-allocates an internal Buffer instance of size Buffer.poolSize that is used as a pool for the fast allocation of new Buffer instances created using Buffer.allocUnsafe(), Buffer.from(array), and Buffer.concat() only when size is less than Buffer.poolSize >>> 1 (floor of Buffer.poolSize divided by two).

In other words, it's for performance reasons, since allocating and deallocating memory can be slow; instead, there's a pool of memory the module dips into for suitably small allocations. If you want a smaller buffer, it'd probably still be less efficient to allocate it from out of the pool than to have a buffer that's a bit bigger than what you need.

(At the time of writing) you can use the buf.buffer and buf.byteOffset properties to see that two small allocations were indeed done off the same ArrayBuffer:

> const { Buffer } = require("node:buffer");
> a = Buffer.from([1,2,3,4])
<Buffer 01 02 03 04>
> a.buffer
ArrayBuffer {
  [Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 00 00 00 00 00 ...>,
  byteLength: 8192
}
> b = Buffer.from([5,6,7,8])
<Buffer 05 06 07 08>

> b.buffer
ArrayBuffer {
  [Uint8Contents]: <2f 00 00 00 00 00 00 00 01 02 03 04 00 00 00 00 05 06 07 08 00 ...>,
  byteLength: 8192
}
> a.byteOffset
8
> b.byteOffset
16
>

edited Mar 14, 2024 at 12:03

answered Mar 14, 2024 at 11:28

AKX

171k16 gold badges147 silver badges229 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user23422735 Over a year ago

That's interesting. So reading a file < 8KB is actually putting the content in an already existing buffer. I wonder if they could overwrite data by mistake, but I guess they fill it up sequentially and keep an offset.

AKX Over a year ago

You're free to look for bugs in the implementation.

Collectives™ on Stack Overflow

NodeJS Buffer Size Logic: why 8KB as a default and not just the data size?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related