Error making requests to /api/abp/application-localization and /api/abp/application-configuration #8723

• ABP Framework version: v9.0
• UI Type: MVC
• Database System: EF Core (SQL Server)
• Tiered (for MVC) or Auth Server Separated (for Angular): yes
• Exception message and full stack trace:

2025-01-30 00:07:47.653 +00:00 [INF] Request starting HTTP/1.1 GET https://api.myapp.com/api/abp/application-configuration?IncludeLocalizationResources=False&api-version=1.0 - null null 2025-01-30 00:07:47.654 +00:00 [INF] Executing endpoint 'Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc)' 2025-01-30 00:07:47.654 +00:00 [INF] Route matched with {area = "abp", action = "Get", controller = "AbpApplicationConfiguration", page = ""}. Executing controller action with signature System.Threading.Tasks.Task1[Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationDto] GetAsync(Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationRequestOptions) on controller Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController (Volo.Abp.AspNetCore.Mvc). 2025-01-30 00:07:47.657 +00:00 [INF] Executing action method Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc) - Validation state: "Valid" 2025-01-30 00:07:47.662 +00:00 [INF] Authorization failed. These requirements were not met: PermissionRequirement: Project.Project.Edit 2025-01-30 00:07:47.662 +00:00 [INF] Authorization failed. These requirements were not met: PermissionRequirement: Project.Project.ScheduleEdit 2025-01-30 00:07:47.662 +00:00 [INF] Authorization failed. Fail() was explicitly called. 2025-01-30 00:07:47.662 +00:00 [WRN] The operation was canceled. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token) at Volo.Abp.Caching.DistributedCache2.GetAsync(TCacheKey key, Nullable1 hideErrors, Boolean considerUow, CancellationToken token) 2025-01-30 00:07:47.664 +00:00 [ERR] An error occurred using the connection to database 'sqldb-myapp' on server 'sql-myserver.database.windows.net'. 2025-01-30 00:07:48.172 +00:00 [ERR] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "An internal error occurred during your request!", "details": null, "data": null, "validationErrors": null }

2025-01-30 00:07:48.172 +00:00 [ERR] A task was canceled. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable1 source, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable1 source, CancellationToken cancellationToken) at Volo.Abp.FeatureManagement.EntityFrameworkCore.EfCoreFeatureValueRepository.GetListAsync(String providerName, String providerKey, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Abp.FeatureManagement.FeatureManagementStore.SetCacheItemsAsync(String providerName, String providerKey, String currentName, FeatureValueCacheItem currentCacheItem) at Volo.Abp.FeatureManagement.FeatureManagementStore.GetCacheItemAsync(String name, String providerName, String providerKey) at Volo.Abp.FeatureManagement.FeatureManagementStore.GetOrNullAsync(String name, String providerName, String providerKey) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Abp.Features.TenantFeatureValueProvider.GetOrNullAsync(FeatureDefinition feature) at Volo.Abp.Features.FeatureChecker.GetOrNullValueFromProvidersAsync(IEnumerable1 providers, FeatureDefinition feature) at Volo.Abp.Features.FeatureChecker.GetOrNullAsync(String name) at Volo.Abp.Features.FeatureCheckerBase.IsEnabledAsync(String name) at Volo.Abp.Features.FeatureCheckerExtensions.IsEnabledAsync(IFeatureChecker featureChecker, Boolean requiresAll, String[] featureNames) at Volo.Abp.Features.RequireFeaturesSimpleStateChecker1.IsEnabledAsync(SimpleStateCheckerContext1 context) at Volo.Abp.SimpleStateChecking.SimpleStateCheckerManager1.InternalIsEnabledAsync(TState state, Boolean useBatchChecker) at Volo.Abp.SimpleStateChecking.SimpleStateCheckerManager1.IsEnabledAsync(TState state) at Volo.Abp.Authorization.Permissions.PermissionChecker.IsGrantedAsync(ClaimsPrincipal claimsPrincipal, String[] names) at Volo.Abp.Authorization.Permissions.PermissionChecker.IsGrantedAsync(String[] names) at Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationAppService.GetAuthConfigAsync() at Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationAppService.GetAsync(ApplicationConfigurationRequestOptions options) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.GlobalFeatures.GlobalFeatureInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Auditing.AuditingInterceptor.ProceedByLoggingAsync(IAbpMethodInvocation invocation, AbpAuditingOptions options, IAuditingHelper auditingHelper, IAuditLogScope auditLogScope) at Volo.Abp.Auditing.AuditingInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Validation.ValidationInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync(ApplicationConfigurationRequestOptions options) at lambda_method2653(Closure, Object) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2025-01-30 00:07:48.172 +00:00 [INF] Executing ObjectResult, writing value of type 'Volo.Abp.Http.RemoteServiceErrorResponse'. 2025-01-30 00:07:48.173 +00:00 [INF] Executed action Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc) in 518.3966ms 2025-01-30 00:07:48.173 +00:00 [INF] Executed endpoint 'Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc)' 2025-01-30 00:07:48.200 +00:00 [INF] Request finished HTTP/1.1 GET https://api.myapp.com/api/abp/application-configuration?IncludeLocalizationResources=False&api-version=1.0 - 500 null application/json; charset=utf-8 546.3452ms

• Steps to reproduce the issue:

We have just upgraded from ABP v4.4 to v9.0, deployed to production and monitored the logs. In the log for the API the request above was made every 5 mins since deployment and then after 8+ hours became less frequent. We can see in the logs for our test environment that the same requests were made at 5 minute intervals. This also happens when no users are in the app.

I have seen some other users with the same or similar issues, but they are closed without any resolution noted.

Are there any processes that run on a 5 minute interval that could cause this? How can we debug this to find the root cause? It doesn't seem to happen when running the app locally.