Performance issues on startup when Redis is down right from the start

[mkrzem]

Hi Everyone, for starters really appreciate the project and job it's doing!
I have an issue when testing resiliency against backing cache being down, in my case redis. I've created a proxy that simulates redis not responding which results in RedisConnectionExceptions in my application. Reproduction steps are plain easy:

• Just start the application, redis being down right from the start, no values stored yet.
• Have anything reading from cache GetOrSetAsync, with hardcoded fallback, so no external calls
• Hardcoded fallback is being returned after set HardTimeout
• Application isn't blocked but is dramatically underperforming. Preflight request take up to 4 minutes if first cache hit is after startup and listeners starting. If I put cache read during startup application never starts but also is not being blocked but extremely slow.

Context:

• There's obviously a lot of business logic going on, filters, middlewares, distributed locks (RedLock) all checking the cache using redisConnectorMultiplexers (not fusion cache).
• I'm removing that cache read when redis is down and app is behaving perfectly fine.
• Switched to class using RedisConnectorMultiplexers directly and everything works as expected
• SoftTimeout, HardTimeout, CircuitBreakerDuration all set and work, can see them in logs
• EnableAutoRecovery true/false makes no difference

That is really really strange am I completely misconfiguring/misusing fusioncache or this is real thing?
Thanks in advance for any help!

[jodydonetti]

Hi @mkrzem , have you already looked into distributed components timeouts or background distributed operations?

Hope this helps, let me know.

[mkrzem]

Yes I did. I have timeouts set and they work. Also I was trying to enable both distributed cache and backplane operations in background, also without any difference. Turning off auto recovery didn't make a difference. The behavior is really strange. FusionCache behaves according to configured properties but after that application is basically unresponsive. I've checked if it's not thread pool starvation, that it's spawning new threads with idle CPU but it's not that I've prepopulated the thread pool with 200 threads on startup and nothing changed. Claude and copilot got stuck on it too. Despite having continuous dumps of running app. But it's not gradual slowdown. Immediate after first call.
The difference from transient error is that it's redis is unreachable from start so no connection multiplexer got ever connected.
I'm using a custom proxy to simulate redis failing it's listening on ports but never accepting the connection. Which shouldn't really matter. My app didn't get a response and it threw RedisConnectionException

[jodydonetti]

Ok, I'll look more into this then.

Meanwhile, would you be able to provide an MRE?

Based on your description I suppose it's something like:

• specify a Redis L2 (and backplane?)
• Redis totally down (like: a local Redis container not even started)
• code being a couple of GetOrSetAsync() in sequence

Is that it?

Hi Everyone, for starters really appreciate the project and job it's doing!

I almost forgot: thanks!

[mkrzem]

MRE will be a problem because it's a typical, not ideal monolith, so lots of moving parts middlewares, filters etc. I didn't try bare bone console app with redis connection string simply not responding, but I doubt it will reproduce the problem, that would be reported long time ago.
Your reproduction steps are correct:

• specify redis L2 and backplane
• redis totally down
• start the app
• exactly 4 GetOrSetAsync calls during startup. Maybe the problem was that it was from open telemetry entity framework dB command enricher. But it didn't block there (2 seconds of hard iltimeout) and it was literally couple of calls

That was also reproducible when I started the app with redis online, turned off redis and tried calling an endpoint. Preflight requests started taking 4 minutes, endpoint response never got it.
With the complexity of the system, I'd say it's a bug, hazard, deadlocks somewhere on my side. But the moment I remove call to FusionCache and just return an object everything starts working again.
I've tried debugging FusionCache to locate if something isn't blocking on connection creation, in loop trying to retry, consuming too many resources etc, but couldn't find it.

[mkrzem]

If you planned to take a look, hold on for a second, we're still on version 1.3.0 because of compatibility issues. I'll try to udpate it to latest and will let you know whether it helped. In the meantime I'm directly using connection multiplexers to get the values from cache and everything works so I have a workaround for now.

[mkrzem]

Still the same on 2.7.2 unfortunately.