Reduce inside map inside map not efficient

Question

I am trying to run a series of keywords against a series of categories and then within those categories there are some options. So I have ended up doing a map over map over a reduce and when dealing with a lot of entries node consumes way too much memory.

I got this, what it does is not really the problem, but how to make it not crave for such amounts of memory?

const keywords = [
  {
    Keyword: 'foo',
    URL: 'https://www.facebook.co.uk'
  },
  {
    Keyword: 'foo',
    URL: 'https://www.twitter.co.uk/blue'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.facebook.co.uk/twitter'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.apple.co.uk/green'
  }
]

const categories = [
  {
    name: 'Tech',
    options: [
      {
        method: 'include',
        regex: 'facebook'
      },
      {
        method: 'exclude',
        regex: 'twitter'
      }
    ]
  },
  {
    name: 'Green',
    options: [
      {
        method: 'include',
        regex: 'green'
      }
    ]
  }
]

const result = keywords.map((obj) => {
  categories.map(({ name, options }) => {
    const option = options.reduce((acc, { method, regex }) => {
      acc.push(method === 'include' ? obj.URL.includes(regex) : !obj.URL.includes(regex))
      return acc
    }, [])

    obj[name] = !option.includes(false)
  })

  return obj
})

console.log(result)

.as-console-wrapper { max-height: 100% !important; top: 0; }

If you can add some information about, how are you preparing to get the expected o/p? — Apoorva Chikara
– Apoorva Chikara, Commented Jan 6, 2022 at 17:10
You don't use the result of the inner map(), so it should be forEach(). — Barmar
– Barmar, Commented Jan 6, 2022 at 17:46
@ApoorvaChikara The question says that the output is correct, the problem is that it uses too much memory producing it. — Barmar
– Barmar, Commented Jan 6, 2022 at 17:47

Barmar · Accepted Answer · 2022-01-06 17:57:45Z

1

You're creating lots of arrays that aren't needed.

If you only care if option includes false(), you can replace the reduce() call with every().

categories.map() should be categories.forEach(), since you don't use the resulting array.

Since you're modifying obj in place and returning this, the objects in the result array will be the same as in the keywords array, so there's no need to create a new array there, either. So you could use forEach() as well.

const keywords = [{
    Keyword: 'foo',
    URL: 'https://www.facebook.co.uk'
  },
  {
    Keyword: 'foo',
    URL: 'https://www.twitter.co.uk/blue'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.facebook.co.uk/twitter'
  },
  {
    Keyword: 'faa',
    URL: 'https://www.apple.co.uk/green'
  }
]

const categories = [{
    name: 'Tech',
    options: [{
        method: 'include',
        regex: 'facebook'
      },
      {
        method: 'exclude',
        regex: 'twitter'
      }
    ]
  },
  {
    name: 'Green',
    options: [{
      method: 'include',
      regex: 'green'
    }]
  }
]

keywords.forEach((obj) =>
  categories.forEach(({name, options}) => 
    obj[name] = options.every(({method, regex}) => 
      method === 'include' ? obj.URL.includes(regex) : !obj.URL.includes(regex))))

console.log(keywords)

.as-console-wrapper {
  max-height: 100% !important;
  top: 0;
}

answered Jan 6, 2022 at 17:57

Barmar

789k57 gold badges555 silver badges669 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Álvaro Over a year ago

Thanks, I tried it but when dealing with 300K keywords I get CPU 99% and memory quite high too around 2GB

Barmar Over a year ago

No matter what you do, it has to iterate #keywords x #options times, so it's doing lots of work. The only memory it's allocating is the added category properties in each keyword, that can't be avoided.

Barmar Over a year ago

You could try changing the forEach() functions into for loops, that may be more efficient.

Álvaro Over a year ago

I actually had changed it to for of, well I guess it is what it is! Thanks!

Collectives™ on Stack Overflow

Reduce inside map inside map not efficient

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related