Auth server Production issue with Redis CLUSTERDOWN MIcroservices K8s #3141

• ABP Framework version: v5.2.0
• UI type: Angular
• DB provider: EF Core
• Tiered (MVC) or Identity Server Separated (Angular): yes
• Exception message and stack trace: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.<>c__DisplayClass50_0.<Set>g__SetRealCache|0() 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Set(String key, Byte[] value, DistributedCacheEntryOptions options) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 1189 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.<>c__DisplayClass50_0.<Set>g__SetRealCache|0() 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Set(String key, Byte[] value, DistributedCacheEntryOptions options) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1189 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 [16:16:14 WRN] CLUSTERDOWN The cluster is down 2022-05-25 21:46:14 at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable1 hideErrors, Boolean considerUow) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.Get(String key) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) 2022-05-25 21:46:14 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) 2022-05-25 21:46:14 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 324 2022-05-25 21:46:14 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54 2022-05-25 21:46:14 at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2817 2022-05-25 21:46:14 StackExchange.Redis.RedisServerException: CLUSTERDOWN The cluster is down 2022-05-25 20:54:40 [15:24:40 ERR] An error occurred while reading the key ring. Log labels

app authserver

container authserver

filename /var/log/pods/{appName}-stag_authserver-7494b9846f-brl9d_ff534c8b-8c20-4340-87b8-f1652332979a/authserver/0.log

job {appName}-stag/authserver

namespace {appName}-stag

pod authserver-7494b9846f-brl9d

pod_template_hash 7494b9846f

stream stdout Detected fields

ts 2022-05-25T15:24:40.647Z

tsNs 1653492280647510856 2022-05-25 20:54:35 [15:24:35 INF] Starting {appName}.AuthServer. 2022-05-25 20:52:36 [15:22:36 INF] Application is shutting down... 2022-05-25 20:52:35 [15:22:35 INF] Disposed RabbitMQ Channel Pool (1 channels in 928.38 ms). 2022-05-25 20:52:34 [15:22:34 INF] Disposing channel pool (1 channels).

• Steps to reproduce the issue:" facing an issue on Auth server