The fastest way to work with a large array of data in node

Question

I have some db with 4 000 000 numbers and I need to check if some number is exist in db

Database exapmle: [177,219,245,309,348,436,...]

I tried use mysql table but it takes 1300ms for check if some 1 number is exist

It very long time and I`ve check some another ways:

1) Push all numbers to array and check it with indexOf

2) Push all numbers to string and check it with indexOf

3) Push all numbers to object and check it with hasOwnProperty

4) Push all numbers to object and check value

I`ve made 100 checks in all of this ways and measured the time:

try [...].indexOf(...) === 0
Total: 726ms

try String(...).indexOf(...) === 0
Total: 1915ms

try {...}.hasOwnProperty(...) === true
Total: 4ms

try {...}[...] === true === true
Total: 4ms

The code of tests:

// Reads db json with ~4 000 000 elements
var arr = utils.readJSON('members.json');
// Converts db to string type
var arr_string = arr.toString();
// Makes empty object
var arr_obj = {};
// Converts db to object type
for(var i in arr){
    arr_obj[arr[i]] = true;
}
// Makes empty array
var checkList = [];
// Gets 100 different values from different places of db
for(var i = 0; i < 100; i++){
    checkList.push(arr[Math.floor(arr.length / 100 * i)]);
}
// Main time variable
var time;

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try [...].indexOf('+ checkList[i]+') === ' + arr.indexOf(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try String(...).indexOf(' + checkList[i] + ') === ' + arr_string.indexOf(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try {...}.hasOwnProperty(' + checkList[i] + ') === ' + arr_obj.hasOwnProperty(checkList[i]));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

// test
time = new Date().getTime();
for (var i in checkList) {
    console.log('try {...}[' + checkList[i] + '] === true === ' + (arr_obj[checkList[i]] === true));
}
console.log('Total: ' + (new Date().getTime() - time) + 'ms');

I believe that the best way to perform this task is Object with hasOwnProperty, but what different ways do you know?

Maybe I should try using other databases?

Have you defined an Index at the column which stores the number what you are searching for, in your table on MySQL? — Diego ZoracKy
– Diego ZoracKy, Commented Mar 30, 2017 at 14:58
Well if you work with a list, the time complexity is O(n) whereas the time complexity for an object is O(1). — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Mar 30, 2017 at 15:00

rsp · Accepted Answer · 2017-03-30 15:55:04Z

3

It started out as a comments but it didn't fit so I post it as an answer. As other has already pointed out, use an index on the column that you want to search or sort anything by. Always. Not only in this example.

If it isn't enough then consider using some other data store like Redis that may be more efficient to handle your particular use case.

answered Mar 30, 2017 at 15:55

rsp

112k31 gold badges210 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mixalbl4 Over a year ago

Yes, ALTER TABLE members ADD INDEX member_id (member_id); Accelerated the search to 0.0003 seconds for 1 element! Thanks!

Collectives™ on Stack Overflow

The fastest way to work with a large array of data in node

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related