20

I have Redis with a lot of keys in some format and I want to get keys that match some pattern and do some operations on them. I don't use KEYS method since it's not recommend in production. Using SCAN I'm wondering what is the best way to write it in code. I have to do something like a while loop but using promises, my current solution looks like this (code is simplified a little):

'use strict'
const Promise = require('bluebird');
const config = require('./config');
const client = require('./clinet');

let iterator = 0;
Promise.coroutine(function* () {
  do {
    iterator = yield clinet.scanAsync(iterator, 'myQuery', 'COUNT', config.scanChunkSize)
      .then(data => {
        let nextIterator = data[0];
        let values = data[1];
        //do some magic with values
        return nextIterator;
      })
  } while (iterator !== '0');
})();

Is there a better way to do it that I'm missing?

8 Answers 8

27

I realize this is a really old question, but I found all of the other answers very unsatisfying. Here is yet another attempt to scan in a relatively clean way using async await (WITHOUT the use of yet another external dependency). You can easily modify this to continuously delete each set of found keys (you would want to tackle them in batches like this in case there are LOTS). Pushing them into an array just demonstrates one very basic thing you could do with them during this stage.

const redis = require('redis');
const { promisify } = require('util');

const client = redis.createClient({...opts});
const scan = promisify(client.scan).bind(client);

const scanAll = async (pattern) => {
  const found = [];
  let cursor = '0';

  do {
    const reply = await scan(cursor, 'MATCH', pattern);

    cursor = reply[0];
    found.push(...reply[1]);
  } while (cursor !== '0');

  return found;
}
Sign up to request clarification or add additional context in comments.

Comments

22

You can use recursion to keep calling scan until done.

function scanAsync(cursor, pattern, returnSet){

    return redisClient.scanAsync(cursor, "MATCH", pattern, "COUNT", "100").then(
        function (reply) {

            cursor = reply[0];
            var keys = reply[1];
            keys.forEach(function(key,i){
                returnSet.add(key);
            });

            if( cursor === '0' ){
                return Array.from(returnSet);
            }else{
                return scanAsync(cursor, pattern, returnSet)
            }

    });
}

Pass in a Set() to make sure keys aren't duplicated

myResults = new Set();

scanAsync('0', "NOC-*[^listen]*", myResults).map( 
    function( myResults ){ console.log( myResults); }
);

2 Comments

I am passing a variable in sscan but it is not working properly is any reason for this?
to get async methods in your client instances, use bluebird.promisifyAll(redis.RedisClient.prototype);
12

You can try this snippet to scan (1000) keys per iteration and 'delete`.

var cursor = '0';
function scan(pattern,callback){

  redisClient.scan(cursor, 'MATCH',pattern,'COUNT', '1000', function(err, reply){
    if(err){
        throw err;
    }
    cursor = reply[0];
    if(cursor === '0'){
        return callback();
    }else{

        var keys = reply[1];
        keys.forEach(function(key,i){                   
            redisClient.del(key, function(deleteErr, deleteSuccess){
                console.log(key);
            });
        });


        return scan(pattern,callback);
    }
  });
}

scan(strkey,function(){
    console.log('Scan Complete');
});

1 Comment

if there is anything in reply[1] i.e. [] this code will result in an infinite loop!
10

Nice option for node-redis module is to use scan iterators. Example:

const redis = require("redis");
const client = redis.createClient();

async function getKeys(pattern="*", count=10) {
    const results = [];
    const iteratorParams = {
        MATCH: pattern,
        COUNT: count
    }
    for await (const key of client.scanIterator(iteratorParams)) {
        results.push(key);
    }
    return results;
}

(Of course you can also process your keys on the fly in for await loop without storing them in additional array if that's enough for you).

If you do not want to override scan parameters (MATCH/COUNT) you can just skip them and execute client.scanIterator() without parameter (defaults will be used then, MATCH="*", COUNT=10).

Comments

9

node redis doc

for await (const key of redisclient.scanIterator()) {
  // use the key!
  const k = await redisclient.get(key);
  console.log(k);
}

This is simplest way to do it in 2023. Some other answers can result on the freezing depending on what kind of promise dependency you are using. For node js app using express this optimal check documents on link. So if you are reading this in 2023 this is my answer.

1 Comment

This is the correct answer in 2024. Please upvote.
2

I think the node bindings for Redis are pushing too much responsibility to the caller here. So I created my own library for scanning as well, using generators in node:

const redis = require('redis')
const client = redis.createClient(…)
const generators = require('redis-async-gen')
const { keysMatching } = generators.using(client)

…

for await (const key of keysMatching('test*')) {
  console.info(key)
}

It's the last bit that obviously is the thing that you should care about. Instead of having to carefully control an iterator yourself, all you need to do is use a for comprehension.

I wrote more about it here.

1 Comment

after finding 3 keys, your library return this error D:\web\node_modules\redis-async-gen\dist\redis-async-gen.js:33 return new Promise(function (resolve, reject) { ^ RangeError: Maximum call stack size exceeded
0

Now there is a scanIterator() function in the redis client library that returns an async generator.

This makes it a lot easier to iterate over records using the for await syntax.

for await (const key of client.scanIterator({ MATCH: 'key-prefix:*' }) {
    console.log(key);
}

Comments

-1

Go through this, it may help.

https://github.com/fritzy/node-redisscan

do not use the library as it, go through the code available at https://github.com/fritzy/node-redisscan/blob/master/index.js

1 Comment

Since in 'do magic line' I have a lot of operations based on promises I wanted to avoid passing callbacks and mixing it but do it in more promise friendly way.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.