Possibly related to this? https://github.com/abpframework/abp/issues/4272
I don't think this is related to user requests cancelling, because this happens consistently every few hours even in the test project running locally with no user interaction and the API idling.
I now believe there is an impact on users when this happens, as I was on our site and got a service unavailable error for a few seconds at the same time these errors were logged. The errors are always in SettingManagement so it seems like there's an issue in that module handling cancellation or cache refreshes.
After configuring our project to use the Redis Cache, we are getting occasional errors (once or twice a day) logged in the background. Based on the traces, it appears this may be happening in the SettingManagement module after the cache tries to refresh.
Usually 1 warning and 3 errors get logged at the same time:
Warning: The operation was canceled.
Exception:
System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Volo.Abp.Caching.StackExchangeRedis.AbpRedisCache.GetAndRefreshManyAsync(IEnumerable`1 keys, Boolean getData, CancellationToken token) at Volo.Abp.Caching.StackExchangeRedis.AbpRedisCache.GetManyAsync(IEnumerable`1 keys, CancellationToken token) at Volo.Abp.Caching.DistributedCache`2.GetManyAsync(IEnumerable`1 keys, Nullable`1 hideErrors, Boolean considerUow, CancellationToken token)
Properties:
<properties><property key='SourceContext'>Volo.Abp.Caching.DistributedCache</property><property key='ActionId'>15fe26d2-905e-451e-956e-9273e2ee2f29</property><property key='ActionName'>Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationScriptController.Get (Volo.Abp.AspNetCore.Mvc)</property><property key='RequestId'>0HMJE86AB1IHV:00000063</property><property key='RequestPath'>/Abp/ApplicationConfigurationScript</property><property key='ConnectionId'>0HMJE86AB1IHV</property><property key='CorrelationId'>50d2b898ed494793b23a9d6d5f6f5da3</property></properties>
Error: An error occurred using the connection to database '{database}' on server '{server}'.
Exception: NULL
Properties:
<properties><property key='database'>test-web-db</property><property key='server'>tcp:redis-test.database.windows.net,1433</property><property key='EventId'><structure type=''><property key='Id'>20004</property><property key='Name'>Microsoft.EntityFrameworkCore.Database.Connection.ConnectionError</property></structure></property><property key='SourceContext'>Microsoft.EntityFrameworkCore.Database.Connection</property><property key='ActionId'>15fe26d2-905e-451e-956e-9273e2ee2f29</property><property key='ActionName'>Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationScriptController.Get (Volo.Abp.AspNetCore.Mvc)</property><property key='RequestId'>0HMJE86AB1IHV:00000063</property><property key='RequestPath'>/Abp/ApplicationConfigurationScript</property><property key='ConnectionId'>0HMJE86AB1IHV</property><property key='CorrelationId'>50d2b898ed494793b23a9d6d5f6f5da3</property></properties>
Error: An exception occurred while iterating over the results of a query for context type '{contextType}'.{newline}{error}
Exception:
System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
Properties:
<properties><property key='contextType'>Volo.Abp.SettingManagement.EntityFrameworkCore.SettingManagementDbContext</property><property key='newline'> </property><property key='error'>System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()</property><property key='EventId'><structure type=''><property key='Id'>10100</property><property key='Name'>Microsoft.EntityFrameworkCore.Query.QueryIterationFailed</property></structure></property><property key='SourceContext'>Microsoft.EntityFrameworkCore.Query</property><property key='ActionId'>15fe26d2-905e-451e-956e-9273e2ee2f29</property><property key='ActionName'>Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationScriptController.Get (Volo.Abp.AspNetCore.Mvc)</property><property key='RequestId'>0HMJE86AB1IHV:00000063</property><property key='RequestPath'>/Abp/ApplicationConfigurationScript</property><property key='ConnectionId'>0HMJE86AB1IHV</property><property key='CorrelationId'>50d2b898ed494793b23a9d6d5f6f5da3</property></properties>
Error: An unhandled exception has occurred while executing the request.
Exception:
System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternalAsync(Boolean errorsExpected, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenAsync(CancellationToken cancellationToken, Boolean errorsExpected) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Query.Internal.SplitQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync() at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken) at Volo.Abp.SettingManagement.EntityFrameworkCore.EfCoreSettingRepository.GetListAsync(String[] names, String providerName, String providerKey, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at ...
Properties:
<properties><property key='EventId'><structure type=''><property key='Id'>1</property><property key='Name'>UnhandledException</property></structure></property><property key='SourceContext'>Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware</property><property key='RequestId'>0HMJE86AB1IHV:00000063</property><property key='RequestPath'>/Abp/ApplicationConfigurationScript</property><property key='ConnectionId'>0HMJE86AB1IHV</property></properties>
I am able to reproduce this issue in a fresh project:
[DependsOn(
...
typeof(AbpCachingModule),
typeof(AbpCachingStackExchangeRedisModule)
)]
...
Configure<AbpDistributedCacheOptions>(options =>
{
//options.KeyPrefix = configuration["Redis:KeyPrefix"];
});
Domain object:
public class Log : Entity<int>
{
public string Message { get; set; }
public string MessageTemplate { get; set; }
public string Level { get; set; }
public DateTime? Timestamp { get; set; }
public string Exception { get; set; }
public string Properties { get; set; }
}
DbContext:
public DbSet<Log> Logs { get; set; }
...
builder.Entity<Log>(b =>
{
b.ToTable(RedisTestConsts.DbTablePrefix + "Logs", RedisTestConsts.DbSchema);
b.ConfigureByConvention();
b.Property(x => x.Message);
b.Property(x => x.MessageTemplate);
b.Property(x => x.Level);
b.Property(x => x.Timestamp);
b.Property(x => x.Exception);
b.Property(x => x.Properties);
});
Program.cs
...
builder.Host
.AddAppSettingsSecretsJson()
.UseAutofac()
.UseSerilog(
(hostingContext, services, loggerConfiguration) => loggerConfiguration
.MinimumLevel.Debug()
.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
.Enrich.FromLogContext()
.WriteTo.Async(c => c.File("Logs/logs.txt"))
.AuditTo.MSSqlServer(
connectionString: hostingContext.Configuration["ConnectionStrings:Default"],
sinkOptions: new MSSqlServerSinkOptions()
{
TableName = "AppLogs"
},
restrictedToMinimumLevel: LogEventLevel.Warning)
);
...
Other information:
I'm not sure if there is impact on the user when this happens, but we have alerting on errors so we would like to resolve these even if there is no user impact. Please let me know if you have any ideas or need more information.
Received. Thanks!
After running the OWASP ZAP penetration test tool, the report shows 2 Cross Site Scripting (Reflected) warnings and 1 SQL Injection warning on the log-out page. These may be false warnings, but can these be resolved as this issue was?
Still experiencing this issue after removing skipIssuerCheck: true. I've done some more testing and can now consistently cause the issue in my project:
Please also note we are using the subdomain tenant resolver. Let me know if you have any other suggestions. Thanks!
Thanks for the suggestion. I have removed the skipIssuerCheck: true. I'm still not sure what causes this error to happen so I'll watch it the next few days and will let you know if it happens again.
Our app is not currently public, but I've included the information below. Let me know if there's anything else that could be helpful.
IdentityTokenLifetime: 300
AccessTokenLifetime: 3600
AuthorizationCodeLifetime: 300
AbsoluteRefreshTokenLifetime: 608400
SlidingRefreshTokenLifetime: 304200
Here is our environment.ts in case that helps as well:
Occasionally when visiting the app after the credentials expire, we get Error refreshing token in the console. The app then immediately navigates to the login page. After entering credentials, verifying with two factor, and redirecting to the Angular home page, the user is still not logged in. They click login again and must enter credentials and two factor again, and then they are logged in.
Here is the console throughout this flow when this happens:
Any suggestions for a workaround for this?
We're just visually hiding the concept of a UserName by setting UserName = EmailAddress. That way users are created and sign in using only their email address, with no option for a different username.
I ended up solving my issue by adding a hidden userName property with a default value.
export function removeUserNameContributor(
propList: FormPropList<IdentityUserDto>
) {
let previousUserNameProp = propList.get(propList.findIndex(p => p.value.name == 'userName')).value
propList.dropByValue(
'userName',
(prop, text) => prop.name === text
)
propList.addHead({
...previousUserNameProp,
visible: c => false,
defaultValue: 'DEFAULTVALUE'
} as FormProp<IdentityUserDto>);
let emailPropIndex = propList.findIndex(p => p.value.name == 'email');
propList.addHead(propList.get(emailPropIndex).value);
propList.dropByIndex(emailPropIndex + 1);
}
Then in CreateAsync I ignore the default value in the username and only use the email to create the user.
var user = new Volo.Abp.Identity.IdentityUser(
GuidGenerator.Create(),
input.Email,
input.Email,
CurrentTenant.Id
);
Now the username for all users will be their email address.
