error 499 , and redis distributed cache error #1909

• ABP Framework version: v4.3.2
• UI type: Angular
• DB provider: EF Core / PostgreSQL
• Exception message and stack trace: 2021-09-27 12:21:58.904 +03:00 [INF] Executed action Zenithr.Intelligence.DataApis.ContractTypes.ContractTypeController.GetListStreamAsync (Zenithr.Intelligence.HttpApi) in 135740.9566ms 2021-09-27 12:21:58.904 +03:00 [INF] Executed endpoint 'Zenithr.Intelligence.DataApis.ContractTypes.ContractTypeController.GetListStreamAsync (Zenithr.Intelligence.HttpApi)' 2021-09-27 12:21:58.906 +03:00 [WRN] The operation was canceled. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token) at Volo.Abp.Caching.DistributedCache2.GetAsync(TCacheKey key, Nullable1 hideErrors, Boolean considerUow, CancellationToken token) 2021-09-27 12:21:58.909 +03:00 [ERR] Failed executing DbCommand (1ms) [Parameters=[@__ef_filter__p_0='?' (DbType = Boolean), @__id_0='?' (DbType = Guid)], CommandType='"Text"', CommandTimeout='30'] SELECT t."Id", t."ConcurrencyStamp", t."CreationTime", t."CreatorId", t."DeleterId", t."DeletionTime", t."ExtraProperties", t."FromExperience", t."IsActive", t."IsDeleted", t."LastModificationTime", t."LastModifierId", t."Name", t."ResellerId" FROM ( SELECT a."Id", a."ConcurrencyStamp", a."CreationTime", a."CreatorId", a."DeleterId", a."DeletionTime", a."ExtraProperties", a."FromExperience", a."IsActive", a."IsDeleted", a."LastModificationTime", a."LastModifierId", a."Name", a."ResellerId" FROM "AbpTenants" AS a WHERE (@__ef_filter__p_0 OR NOT (a."IsDeleted")) AND (a."Id" = @__id_0) ORDER BY a."Id" LIMIT 1 ) AS t ORDER BY t."Id" 2021-09-27 12:21:58.911 +03:00 [ERR] An exception occurred while iterating over the results of a query for context type 'Volo.Abp.TenantManagement.EntityFrameworkCore.TenantManagementDbContext'. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() 2021-09-27 12:21:59.146 +03:00 [WRN] Could not save the audit log object: AUDIT LOG: [200: GET ] /api/intelligence/data-apis/contracttype/streamall

• UserName - UserId : -
• ClientIpAddress : ::1
• ExecutionDuration : 135742
• Actions:
  • Zenithr.Intelligence.DataApis.ContractTypes.ContractTypeAppService.GetListAsync (75916 ms.) {}
  • Zenithr.Intelligence.DataApis.ContractTypes.ContractTypeController.GetListStreamAsync (135740 ms.) {}

2021-09-27 12:21:59.146 +03:00 [ERR] The operation was canceled. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository3.FindAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Abp.TenantManagement.TenantStore.GetCacheItemAsync(Nullable1 id, String name) at Volo.Abp.TenantManagement.TenantStore.FindAsync(Guid id) at IntelligenceService.Host.MultiTenantConnectionStringResolver.FindTenantConfigurationAsync(Guid tenantId) in C:\Users\LENOVO\source\repos\ZENITHR 3.0\microservices\IntelligenceService.Host\MultiTenantConnectionStringResolver.cs:line 155 at IntelligenceService.Host.MultiTenantConnectionStringResolver.ResolveAsync(String connectionStringName) in C:\Users\LENOVO\source\repos\ZENITHR 3.0\microservices\IntelligenceService.Host\MultiTenantConnectionStringResolver.cs:line 37 at Volo.Abp.Uow.EntityFrameworkCore.UnitOfWorkDbContextProvider1.ResolveConnectionStringAsync(String connectionStringName) at Volo.Abp.Uow.EntityFrameworkCore.UnitOfWorkDbContextProvider1.GetDbContextAsync() at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository2.InsertAsync(TEntity entity, Boolean autoSave, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed) at Volo.Abp.AuditLogging.AuditingStore.SaveLogAsync(AuditLogInfo auditInfo) at Volo.Abp.AuditLogging.AuditingStore.SaveAsync(AuditLogInfo auditInfo) 2021-09-27 12:21:59.148 +03:00 [INF] Request finished HTTP/1.1 GET https://localhost:44373/api/intelligence/data-apis/contracttype/streamall - 0 - 200 - application/octet-stream 136189.0078ms 2021-09-27 12:21:59.349 +03:00 [DBG] Added 0 entity changes to the current audit log 2021-09-27 12:21:59.351 +03:00 [DBG] Added 0 entity changes to the current audit log

So i have a MicroServices Archeticture Application , and what iam trying to do here is inserting and reading data for MultiTenants at the same time.

i tried clearing redis cache from docker Cli , but still the same problem , sometimes i get this error , and sometimes i get error 499 , and sometimes it works normally. knowing that there is 1 million records in this table for each one of the Tenants.