Filter large arrays using node js

Question

I have 2 large arrays populated with strings both containing > 150.000 elements

const allNew = allArrayOneValues.filter(val => !allArrayTwoValues.includes(val));

I need to compare the two arrays like this to find out which elements are not in ArrayTwo yet or to find out which elements to delete from ArrayTwo as they are no longer in ArrayOne.

Filtering here takes around 3 to 5 minutes... is there a way to do a far more efficient compared to find out which values in ArrayOne are not yet in ArrayTwo OR which values are in ArrayTwo which are not in ArrayOne...

Thanks Thomas

are the arrays sorted? have you tried to use Set or an object? — Nina Scholz
– Nina Scholz, Commented Apr 7, 2021 at 8:32
No, they contain random values like 1#4#5 and 7#9#100 etc... I have not looked at set yet — thomas
– thomas, Commented Apr 7, 2021 at 11:45
sorted meand ascending or descending value, not necessarily random values. — Nina Scholz
– Nina Scholz, Commented Apr 7, 2021 at 11:48
This is an algorithm question and since you're using the brute-force approach, it's taking a very long time. Please describe if data has any special properties - (sorted values, all unique values/has duplicates values, min and max value possible, etc) for us to provide the best possible solution. — Sachin
– Sachin, Commented Apr 8, 2021 at 4:19

Sachin · Accepted Answer · 2021-04-08 04:14:05Z

2

Your current algorithm is O(m*n) (where m= length of first array, n= length of second array)

Using an efficient data-structure that can do sub-linear lookup, it's possible to this in atleast O(m*lg(n))

So for 150,000 elements, it would be 10 thousand times faster and should take a few milliseconds instead of minutes.

let allArrayOneValues = [1, 2, 4]
let allArrayTwoValues = [3, 9, 2]
let hash = new Set();
allArrayTwoValues.forEach((value) => {
  hash.add(value)
})

const allNew = allArrayOneValues.filter(val => !hash.has(val));
console.log(allNew)

edited Apr 8, 2021 at 4:14

answered Apr 8, 2021 at 4:06

Sachin

3,5704 gold badges21 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

thomas Over a year ago

Awesome, thanks @Sachin, this is definitively the accepted answers, works in just a second instead of mins!

Debug Anywhere · Accepted Answer · 2021-04-08 03:01:07Z

1

use Set or Object may be a good choice. Here is an example:

// convert allArrayTwoValues to Object.
const tmp = allArrayTwoValues.reduce((prev, item) => { prev[item] = true; return prev; }, {});
// filter values not in allArrayTwoValues
const allNew = allArrayOneValues.filter(item => !tmp[item]);

edited Apr 8, 2021 at 3:01

answered Apr 7, 2021 at 8:38

Debug Anywhere

113 bronze badges

3 Comments

user115014 Over a year ago

Could you add more detail about how/why this works

thomas Over a year ago

I also do not understand how this would work. In ArrayTwo, I need to add the values from ArrayOne which are missing. Then on top of that, I need to find out which values exist in ArrayTwo which are no longer in ArrayOne (elements that may have been removed from ArrayOne) - so that after these 2 comparisons, I can add elements to ArrayTwo and remove certain ones and then ArrayTwo will match ArrayOne

Debug Anywhere Over a year ago

@thomas Hi, i have updated the code. the code only filter the values are not in arrayTwo. There is need to do more thing like that.

Collectives™ on Stack Overflow

Filter large arrays using node js

2 Answers 2

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related