- ABP Framework version: v7.1.0
- UI type: Angular
- DB provider: EF Core
- Tiered (MVC) or Identity Server Separated (Angular): yes
- Exception message and stack trace:
System.InvalidOperationException: Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary2.set_Item(TKey key, TValue value)
at Volo.Abp.Uow.UnitOfWorkExtensions.GetOrAddItem[TValue](IUnitOfWork unitOfWork, String key, Func2 factory) at Volo.Abp.Caching.DistributedCache2.GetAsync(TCacheKey key, Nullable1 hideErrors, Boolean considerUow, CancellationToken token) at Volo.Saas.Tenants.TenantStore.GetCacheItemAsync(Nullable1 id, String name)
at Volo.Saas.Tenants.TenantStore.FindAsync(Guid id)
at Volo.Abp.MultiTenancy.MultiTenantConnectionStringResolver.FindTenantConfigurationAsync(Guid tenantId)
at Volo.Abp.MultiTenancy.MultiTenantConnectionStringResolver.ResolveAsync(String connectionStringName)
at Volo.Abp.Uow.EntityFrameworkCore.UnitOfWorkDbContextProvider1.ResolveConnectionStringAsync(String connectionStringName) at Volo.Abp.Uow.EntityFrameworkCore.UnitOfWorkDbContextProvider1.GetDbContextAsync()
at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository2.GetDbSetAsync() at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository2.GetQueryableAsync()
at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository2.WithDetailsAsync() at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository3.FindAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken)
at Volo.Abp.Domain.Repositories.EntityFrameworkCore.EfCoreRepository3.GetAsync(TKey id, Boolean includeDetails, CancellationToken cancellationToken) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync()
at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed)
at Doohlink.CampaignManagement.Screens.ScreenAppService.GetAsync(Guid id) in C:\Development\Projects\Examples\Doohlink\aspnet-core\modules\Doohlink.CampaignManagement\src\Doohlink.CampaignManagement.Application\Screens\ScreenAppService.cs:line 22
at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Authorization.AuthorizationInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync()
at Volo.Abp.GlobalFeatures.GlobalFeatureInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed)
at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Auditing.AuditingInterceptor.ProceedByLoggingAsync(IAbpMethodInvocation invocation, AbpAuditingOptions options, IAuditingHelper auditingHelper, IAuditLogScope auditLogScope) at Volo.Abp.Auditing.AuditingInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed) at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo) at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync()
at Volo.Abp.Validation.ValidationInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func3 proceed)
at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue1.ProceedAsync() at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation) at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
at Doohlink.Screens.ScreenAppService.GetAsync(Guid id) in C:\Development\Projects\Examples\Doohlink\aspnet-core\src\Doohlink.Application\Screens\ScreenAppService.cs:line 23
Hello, I have an application which i created two different modules and from the main app i am calling application services of those modules async methods in parallel. sth like this.
public async Task<ScreenDto> GetAsync(Guid id)
{
var inventory = _inventoryScreenAppService.GetAsync(id);
var campaign = _campaignScreenAppService.GetAsync(id);
await Task.WhenAll(inventory, campaign);
var screenDto = new ScreenDto();
ObjectMapper.Map(inventory.Result, screenDto);
ObjectMapper.Map(campaign.Result, screenDto);
return screenDto;
}
not always but sometimes I am getting this exception on runtime. It is very straightforward what happens inside those 2 appservices.
Inventory ScreenAppService
public async Task<ScreenDto> GetAsync(Guid id)
{
var screen = await _screenRepository.GetAsync(id);
return ObjectMapper.Map<Screen, ScreenDto>(screen);
}
Campaign ScreenAppService
public async Task<ScreenDto> GetAsync(Guid id)
{
var screen = await _screenRepository.GetAsync(id);
return ObjectMapper.Map<Screen, ScreenDto>(screen);
}
two modules have different dbcontexts(2 seperate databases). and storing different information about screen. it seems it is the same method but they are in different namespaces and different properties they hold for screen and screendto.
Campaign ScreenDto
using System;
using System.Collections.Generic;
using Volo.Abp.Application.Dtos;
namespace Doohlink.CampaignManagement.Screens;
public class ScreenDto : EntityDto<Guid>
{
public ResolutionDto Resolution { get; set; }
public List<OpeningHoursDto> OpeningHours { get; set; }
public List<double> Durations { get; set; }
public List<string> MimeTypes { get; set; }
}
Inventory ScreenDto
using System;
using Volo.Abp.Application.Dtos;
namespace Doohlink.InventoryManagement.Screens;
public class ScreenDto : EntityDto<Guid>
{
public string Name { get; set; }
public string MacAddress { get; set; }
public LocationDto Location { get; set; }
}
as I see from the exception it is trying to get the tenant and resolve the configuration
at System.Collections.Generic.Dictionary2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary2.set_Item(TKey key, TValue value)
at Volo.Abp.Uow.UnitOfWorkExtensions.GetOrAddItem[TValue](IUnitOfWork unitOfWork, String key, Func2 factory) at Volo.Abp.Caching.DistributedCache2.GetAsync(TCacheKey key, Nullable1 hideErrors, Boolean considerUow, CancellationToken token) at Volo.Saas.Tenants.TenantStore.GetCacheItemAsync(Nullable1 id, String name)
at Volo.Saas.Tenants.TenantStore.FindAsync(Guid id)
at Volo.Abp.MultiTenancy.MultiTenantConnectionStringResolver.FindTenantConfigurationAsync(Guid tenantId)
it seems it is a concurrency issue while tenant has been fetched from the cache or db. Any suggestion to fix this issue?
Ps: I have tried to change the method in main app
public async Task<ScreenDto> GetAsync(Guid id)
{
var inventory = _inventoryScreenAppService.GetAsync(id);
var campaign = _campaignScreenAppService.GetAsync(id);
await Task.WhenAll(inventory, campaign);
var screenDto = new ScreenDto();
ObjectMapper.Map(inventory.Result, screenDto);
ObjectMapper.Map(campaign.Result, screenDto);
return screenDto;
}
to
public async Task<ScreenDto> GetAsync(Guid id)
{
var inventory =await _inventoryScreenAppService.GetAsync(id);
var campaign =await _campaignScreenAppService.GetAsync(id);
var screenDto = new ScreenDto();
ObjectMapper.Map(inventory, screenDto);
ObjectMapper.Map(campaign, screenDto);
return screenDto;
}
it didn't fix the issue.
9 Answer(s)
-
0
HI,
Could you share a minimal reproducible example? shiwei.liang@volosoft.com thanks.
-
0
Hello @liangshiwei, is it okay if i share the original project? it is not a super complicated project i think you will understand.
-
0
Hello again, you can find the project in this github repo. https://github.com/cangunaydin/abpConcurrencyIssue to produce the same error you have to do the followings.
1- Clone the project.
2- Go to aspnet-core\etc\docker
3- execute run-docker-sideapps.ps1 this will create the redis and postgres(postgis) container.(stick to these images since appsettings has been configured according to those docker images and geolocation is involved)
4- Build project and run dbmigrator. This will seed a new tenant named "Tribulus" and one screen belongs to this tenant with id: "ad16f9e2-bf39-43cf-af69-47b7388ee9cd". You can check TenantDataSeeder class if you want.
5- abp install-libs inside AuthServer folder then Run Doohlink.AuthServer and Doohlink.HttpApi.Host project
6- Run angular app and get bearer token for Tribulus tenant with username: admin, password: 123qwe, note the bearer token and tenantid to somewhere, you will use them on step 8
7- Install Jmeter to do multiple http requests async. If you want you can use another tool but important part is you need bunch of async calls at the same time.
https://jmeter.apache.org/download_jmeter.cgi
Download zip file, unzip it then run jmeter.bat
8- After running the Jmeter, you need to open the TestJmeter.jmx file on the root folder of the project. This is a script that will do the call to the endpoint.
9- Change the bearer token and tenantid to what you got from step 6. You need to do that from Http Header Manager
10-Run the jmeter, it is gonna take 10 seconds, you can see if http requests are successful on treeview. Most of them will succeed around 5 of them will fail. you can look at the logs afterwards.
-
0
ok, I will check it
-
0
Hi,
You can see the test result:
This way should work:
public async Task<ScreenDto> GetAsync(Guid id) { var inventory = await _inventoryScreenAppService.GetAsync(id); var campaign = await _campaignScreenAppService.GetAsync(id); var screenDto = new ScreenDto(); ObjectMapper.Map(inventory, screenDto); ObjectMapper.Map(campaign, screenDto); return screenDto; } -
0
But it's still a problem, I'll consider enhancing it: https://github.com/abpframework/abp/issues/16323
-
0
Hello @liangshiwei. As you mention it works when you await, probably when i try to await i did sth wrong. It was late at night :) Thank you for sharing the issue link with me So it seems, collection has been updated from two threads. And collection that is updated in Volo.Abp.Uow.UnitOfWorkExtensions.GetOrAddItem is not thread safe. Maybe changing that to ConcurrentDictionary can help. https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2?view=net-5.0
-
0
Hi,
Yes,we will fix it: https://github.com/abpframework/abp/pull/16324
-
0
Thanks a lot, i have seen the fix now, I am closing the issue then.
