22

I have a Redis cluster of 6 instances, 3 master and 3 slaves. My ASP .NET Core application uses it as a cache. Sometimes I get such an error:

StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=5KiB, 5504ms elapsed, timeout is 5000ms), command=GET, next: GET [email protected], inst: 0, qu: 0, qs: 6, aw: False, rs: DequeueResult, ws: Idle, in: 0, in-pipe: 5831, out-pipe: 0, serverEndpoint: 5.178.85.30:7002, mgr: 9 of 10 available, clientName: f0b7b81f5ce5, PerfCounterHelperkeyHashSlot: 9236, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=10,Free=32757,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) (Most recent call last)

Improve this question
7
  • Have you read the article that is recommended in the error message yet? stackexchange.github.io/StackExchange.Redis/Timeouts Commented Aug 27, 2019 at 23:12
  • @howcheng Yes, I tried to do what is described there and it does not help Commented Dec 4, 2019 at 6:39
  • Whenever it happens, run SLOWLOG GET 10 to see slow queries which might cause it. Commented Dec 9, 2019 at 8:31
  • 1
    what operation are you trying to do ? If its get or set what is the size of the data pushed/stored and also the mention the datastructure being used like String,HashTable or SET? Commented Dec 9, 2019 at 15:06
  • 1
    One of the possible cause is thread exhaustion, check this video to debug. youtube.com/watch?v=isK8Cel3HP0 Commented Aug 9, 2023 at 4:23

1 Answer 1

Reset to default
40

As I can see from your exception message, your minimum worker process count is too low for the traffic you have.

WORKER: (Busy=10,Free=32757,Min=2,Max=32767)

You had 10 busy worker threads when this exception happened, while you had 2 worker threads for start.

When your application runs out of available threads to complete an operation, .NET starts a new one (until maximum value, of course). And waits a bit to see if an additional worker thread is needed. If your application still needs worker threads, then .NET starts another one. Then another, then another... But this requires time. It doesn't occur in 0 ms. By looking your exception message, we can see that .NET had created 8 additional worker threads (10 - 2 = 8). While the creation process, this particular Redis operation had waited and eventually timed out.

You could use ThreadPool.SetMinThreads(Int32, Int32) method at the beginning of your application to set minimum thread count. I suggest you to start with ThreadPool.SetMinThreads(10, 10) and tweak it as you test it.

Additional Read:

https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadpool.setminthreads https://stackexchange.github.io/StackExchange.Redis/Timeouts.html

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

This worked for me but it wasn't enough, we are going to scale our app too. Thank you for your answer!
Thanks, great point, it did help to some extent! However when load is increasing, even setting minThread to 500 did not help and the command is failing with 222 workers out of 500, which is weird. Any ideas what to tweak next? Timeout performing GET (5000ms) + IOCP: (Busy=0,Free=1000,Min=500,Max=1000), WORKER: (Busy=222,Free=32545,Min=500,Max=32767), POOL: (Threads=222,QueuedItems=75,CompletedItems=198), v: 2.6.66.47313
@adamsfamily you have a different situation now. You have enough worker threads ready. However, some of them are waiting on the queue. This made me believe that the actual Redis Server may not handle all of the incoming requests (due to e.g. the complexity of requests). Another possibility may be that the network bandwidth between the server containing the .NET application and the server containing the Redis Server is not enough.
I have some insights. The above error was generated on a Heroku app with 4 concurrent processes. I gradually increased the number of processes (8, 16, 32) and the error went away at 32 processes (dynos). The Redis server (add-on) seems to be able to handle the number of connections. Could it be a memory issue in .NET?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.