Many thanks Tim, I’m sure that will be useful to others!

And thirdly – because SCAN is non-deterministic, you won’t be able to mix it with commands that write (e.g. EXPIRE) in a Lua script – Redis will throw an error to that effect :)

Yes, each SCAN is a round trip and incurs some overhead. I didn’t do it in lua for a couple of reasons.

Firstly, we need a way to limit the amount of work being done in each round trip to avoid blocking redis. You could reduce the number of trips to one by scanning the entire keyspace in one go, but this solution seeks to avoid doing that.

Secondly, the number of round trips for SCAN is very small compared to the EXPIRE commands, because we’re only doing one SCAN per 500 keys (in my example above). So where expiring 500 keys wold previously have required 501 round trips, it now requires 1. Thus, the benefit of doing the scans in lua is 500x less than doing the expires (depending on the scan size). If you want to do less rounds trips, you can increase the size of each scan but this means doing more work in each iteration, and thus blocking redis for longer.

I think what you might be getting at though, is whether it’s possible to do the scan in lua, while yielding to other clients on each iteration (to avoid blocking). You could implement the bash logic above in a language with a native redis client (python, ruby, lua), which would enable you to reuse the same connection and thus reduce tcp overhead. The benefit would be relatively small though, and this was done for a one-off batch operation, so that level of optimisation would have been somewhat premature.

Note that you shouldn’t do the scan within the EVAL, because, as redis is single threaded, it will be blocked for the duration of the EVAL’s execution. Ref: http://redis.io/commands/eval#atomicity-of-scripts
Also, a script can not run for longer than 5 seconds: http://redis.io/commands/eval#sandbox-and-maximum-execution-time

If you do come up with a pure lua solution or reimplement the bash logic please link to it here, I’d be very interested to see it!

Thank you Mike, that’s a subtle bug but an important one, I will update the post!

keys=$(echo $reply | awk '{for (i=2; i<=NF; i++) print $i}')

I was testing it with just one key in my keyspace and nothing was happening :)

Cheers Kostas!

And, ouch :(

Remembered it again today while reviewing code submitted by our dev team I found usage of keys in a key service that will frequently do that against a redis deployment used by a distributed system with several thousand clients.

You’d think the warning on the redis docs would be sufficient deterrent, but apparently it’s not.