An exception occurred while iterating over the results of a query for context type 'Volo.Saas.EntityFrameworkCore.SaasDbContex #2084

• ABP Framework version: v4.4.3
• UI type: MVC
• DB provider: EF Core
• Tiered (MVC) or Identity Server Separated (Angular): yes
• Exception message and stack trace:

2021-11-03 21:21:43.774 -05:00 [ERR] An exception occurred while iterating over the results of a query for context type 'Volo.Saas.EntityFrameworkCore.SaasDbContext'. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() 2021-11-03 21:21:43.838 -05:00 [ERR] A task was canceled. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.BufferedDataRecord.InitializeAsync(DbDataReader reader, IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.BufferedDataReader.InitializeAsync(IReadOnlyList1 columns, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.InitializeReaderAsync(DbContext _, Boolean result, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[TSource](IAsyncEnumerable1 asyncEnumerable, CancellationToken cancellationToken) at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository3.FindAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Volo.Saas.Tenants.TenantStore.GetCacheItemAsync(Nullable1 id, String name) at Volo.Saas.Tenants.TenantStore.FindAsync(Guid id) at ClientsLink.Data.ClientsLinkTenantDatabaseMigrationHandler.MigrateAndSeedForTenantAsync(Guid tenantId, String adminEmail, String adminPassword) in D:\Century\Internal\Clients.Link\Clients.Link\src\ClientsLink.Domain\Data\ClientsLinkTenantDatabaseMigrationHandler.cs:line 98

This is occurring on the highlighted line of code below:

private async Task MigrateAndSeedForTenantAsync(
    Guid tenantId,
    string adminEmail,
    string adminPassword)
{
  try
  {
    using (_currentTenant.Change(tenantId))
    {
      // Create database tables if needed
      using (var uow = _unitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
      {
        var tenantConfiguration = await _tenantStore.FindAsync(tenantId); // EXCEPTION THROWN HERE
        if (tenantConfiguration?.ConnectionStrings != null &&
            !tenantConfiguration.ConnectionStrings.Default.IsNullOrWhiteSpace())
        {
          foreach (var migrator in _dbSchemaMigrators)
          {
            _logger.LogDebug($"Running migration {migrator.ToString()}");
            await migrator.MigrateAsync();
          }
        }

        await uow.CompleteAsync();
      }

• Steps to reproduce the issue:

1. Use ABP Commercial Suite v 4.4.3 to generate a Sample Tiered MVC application
2. Add a couple of entities - one global and one multitenant with a navigation property to the global one
3. Create a Sample database
4. Run the DbMigrator - host database will be migrated successfully
5. Run the web application, login as admin, create a new Test tenant - after about a minute, the UI reports an Internal Error and the exception above is logged

It seems fairly obvious after working this for two weeks, there is something preventing the execution of the _tenantStore.FindAsync(tenantId) call.

What possible things could cause that call to timeout or be canceled?

Please think about your response and be as verbose as possible.

I have swapped out every component. I have built and rebuilt on two separate machines. I have rewritten my data seed contributors. I have swapped out System.Data.SqlClient for Microsoft.Data.SqlClient and set the Command Timeout connection string argument two 3600. Nothing I have tried in two weeks is resolving this issue.