Hi all, I recently upgraded my project from 9.0.4 to 9.2.2. I have the following entities:
Partner - has a List<PartnerStudent> PartnerStudent - Entity with PartnerId, Partner, StudentId, Student. Key being [PartnerId,StudentId] Student - has a List<PartnerStudent>
It's a very standard many-to-many relationship. One of my partners has 2000 students. When I do .Include(x => x.PartnerStudents)
when querying for that specific partner, the app slows to a crawl. This is ONLY if I have tracking on. If I do .AsNoTracking()
, it is fine. Something you guys changed recently in AbpEfCoreNavigationHelper, or somewhere near it, has caused this performance regression. It used to be immediate, and now it takes several minutes, or it just crashes the api. I replaced the service locally with the 9.0 versions I found on github and it is normal. I then use the 9.2 versions and it is incredibly slow, again, if it even manages to finish at all. I'm talking in the realm of 5-10+ minutes, or again, just crashes the app. Please investigate and fix. This is not the first time we have had issues with the AbpDbContext-related code and performance issues - please add more thorough testing to your release processes as well. I want to stay on the latest versions of ABP, but it is getting scarier to upgrade when stuff like this happens.
I've gotta assume it's this: https://github.com/abpframework/abp/commit/17c4e6be5364c13fe788216d68dbcab8a5b63fac
24 Answer(s)
-
0
There is a confirmed performance regression in ABP 9.2.x related to tracking many-to-many navigation properties with large collections, as you described. The issue is specifically triggered when using
.Include(x => x.PartnerStudents)
with tracking enabled. This is likely due to recent changes in ABP's internal entity tracking and navigation property handling (notably inAbpEfCoreNavigationHelper
andAbpEntityEntry
).As you observed, using
.AsNoTracking()
avoids the problem, but this is not always a viable workaround if you require change tracking.The regression does not exist in 9.0.x, and replacing the affected services with 9.0 versions restores normal performance, confirming the root cause is in the ABP 9.2 changes.
Recommended Actions:
Review the official ABP 9.2 migration guide and release notes for any related breaking changes or known issues:
If you require tracking and cannot use
.AsNoTracking()
, consider temporarily reverting to ABP 9.0.x for affected DbContext/services until a fix is released.Report the issue with a minimal reproducible example to the ABP GitHub repository if you have not already, to help the team prioritize a fix.
Monitor the ABP release notes and GitHub issues for updates regarding this regression.
Sources:
This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.
-
0
hi
This problem was addressed in 9.2.2. Can you share a project that reproduces the problem? I will add a test to prevent this from happening in the future.
liming.ma@volosoft.com https://github.com/abpframework/abp/pull/23188
Thanks
-
0
hi
I added a simple unit test, and there were no performance problems. https://github.com/abpframework/abp/pull/23316/files
-
0
Hi, that is not my experience. I'm on 9.2.2. When I revert to 9.0.4, the performance is back to normal. When I use 9.2.2, but the 9.0.4's version of the abp ef core navigation helper, the performance is back to normal. When I simply use 9.2.2, the performance is horrible.
Additionally, your unit test is not accurate to my problem. Please write a test where there is a single blog with several thousand posts, then try to load that blog including those posts. Your test as it stands is 5000 blogs, each with one post.
-
0
hi
I will work on a minimal repro tomorrow.
Thank you very much. I will check your project as soon as possible once I receive it.
-
0
I edited my response after I took a closer look at your test. Please see my edit where I point out the difference in the scenarios.
-
0
I will add more test cases. Waiting for your real project.
Thanks.
-
0
Thank you. I'm confident you'll see the performance issue if you do a test with one blog that has several thousand posts.
If you don't see the performance issue with that test, I'll produce a minimal repro in the morning. Thanks.
-
0
ok, I'm working on it.
-
0
Thank you, I appreciate it
-
0
hi
I've reproduced your problem; I will fix it as soon as possible.
Thanks.
-
0
-
0
Thanks. Is there anything I can do until 9.2.3 is released? When will that be? I saw your unit test - you have reproduced the problem I was having. Any minimal repro I send you would look exactly like that test.
-
0
hi
Can you try to override the
AbpEfCoreNavigationHelper
in your project?using System.Collections; using System.Collections.Generic; using System.Linq; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.ChangeTracking; using Microsoft.EntityFrameworkCore.Internal; using Microsoft.EntityFrameworkCore.Metadata; using Volo.Abp.DependencyInjection; using Volo.Abp.Domain.Entities; namespace Volo.Abp.EntityFrameworkCore.ChangeTrackers; public class MyAbpEntityEntry { public string Id { get; set; } public EntityEntry EntityEntry { get; set; } public List<AbpNavigationEntry> NavigationEntries { get; set; } private bool _isModified; public bool IsModified { get { return _isModified || EntityEntry.State == EntityState.Modified || NavigationEntries.Any(n => n.IsModified); } set => _isModified = value; } public MyAbpEntityEntry(string id, EntityEntry entityEntry) { Id = id; EntityEntry = entityEntry; NavigationEntries = EntityEntry.Navigations.Select(x => new AbpNavigationEntry(x, x.Metadata.Name)).ToList(); } public void UpdateNavigationEntries() { foreach (var navigationEntry in NavigationEntries) { if (IsModified || EntityEntry.State == EntityState.Modified || navigationEntry.IsModified || navigationEntry.NavigationEntry.IsModified) { continue; } var currentValue = AbpNavigationEntry.GetOriginalValue(navigationEntry.NavigationEntry.CurrentValue); if (currentValue == null) { continue; } switch (navigationEntry.OriginalValue) { case null: navigationEntry.OriginalValue = currentValue; break; case IEnumerable originalValueCollection when currentValue is IEnumerable currentValueCollection: { var existingList = originalValueCollection.Cast<object?>().ToList(); var newList = currentValueCollection.Cast<object?>().ToList(); if (newList.Count > existingList.Count) { navigationEntry.OriginalValue = currentValue; } break; } default: navigationEntry.OriginalValue = currentValue; break; } } } } [Dependency(ReplaceServices = true)] [ExposeServices(typeof(AbpEfCoreNavigationHelper))] public class MyAbpEfCoreNavigationHelper : AbpEfCoreNavigationHelper { protected Dictionary<string, MyAbpEntityEntry> MyEntityEntries { get; } = new(); public override void ChangeTracker_Tracked(object? sender, EntityTrackedEventArgs e) { EntityEntryTrackedOrStateChanged(e.Entry); DetectChanges(e.Entry); } public override void ChangeTracker_StateChanged(object? sender, EntityStateChangedEventArgs e) { EntityEntryTrackedOrStateChanged(e.Entry); DetectChanges(e.Entry); } protected override void EntityEntryTrackedOrStateChanged(EntityEntry entityEntry) { if (entityEntry.State != EntityState.Unchanged) { return; } var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return; } if (MyEntityEntries.ContainsKey(entryId)) { return; } MyEntityEntries.Add(entryId, new MyAbpEntityEntry(entryId, entityEntry)); } protected override void DetectChanges(EntityEntry entityEntry, bool checkEntityEntryState = true) { #pragma warning disable EF1001 var stateManager = entityEntry.Context.GetDependencies().StateManager; var internalEntityEntityEntry = stateManager.TryGetEntry(entityEntry.Entity, throwOnNonUniqueness: false); if (internalEntityEntityEntry == null) { return; } var foreignKeys = entityEntry.Metadata.GetForeignKeys().ToList(); foreach (var foreignKey in foreignKeys) { var principal = stateManager.FindPrincipal(internalEntityEntityEntry, foreignKey); if (principal == null) { continue; } var entryId = GetEntityEntryIdentity(principal.ToEntityEntry()); if (entryId == null || !MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { continue; } myAbpEntityEntry.UpdateNavigationEntries(); if (!myAbpEntityEntry.IsModified && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { myAbpEntityEntry.IsModified = true; DetectChanges(myAbpEntityEntry.EntityEntry, false); } var navigationEntry = myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is INavigation navigationMetadata && navigationMetadata.ForeignKey == foreignKey) ?? myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is ISkipNavigation skipNavigationMetadata && skipNavigationMetadata.ForeignKey == foreignKey); if (navigationEntry != null && IsEntityEntryChanged(entityEntry)) { navigationEntry.IsModified = true; } } var skipNavigations = entityEntry.Metadata.GetSkipNavigations().ToList(); foreach (var skipNavigation in skipNavigations) { var joinEntityType = skipNavigation.JoinEntityType; var foreignKey = skipNavigation.ForeignKey; var inverseForeignKey = skipNavigation.Inverse.ForeignKey; foreach (var joinEntry in stateManager.Entries) { if (joinEntry.EntityType != joinEntityType || stateManager.FindPrincipal(joinEntry, foreignKey) != internalEntityEntityEntry) { continue; } var principal = stateManager.FindPrincipal(joinEntry, inverseForeignKey); if (principal == null) { continue; } var entryId = GetEntityEntryIdentity(principal.ToEntityEntry()); if (entryId == null || !MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { continue; } myAbpEntityEntry.UpdateNavigationEntries(); if (!myAbpEntityEntry.IsModified && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { myAbpEntityEntry.IsModified = true; DetectChanges(myAbpEntityEntry.EntityEntry, false); } var navigationEntry = myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is INavigation navigationMetadata && navigationMetadata.ForeignKey == inverseForeignKey) ?? myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is ISkipNavigation skipNavigationMetadata && skipNavigationMetadata.ForeignKey == inverseForeignKey); if (navigationEntry != null && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { navigationEntry.IsModified = true; } } } #pragma warning restore EF1001 } protected override bool IsEntityEntryChanged(EntityEntry entityEntry) { return entityEntry.State == EntityState.Added || entityEntry.State == EntityState.Deleted || entityEntry.State == EntityState.Modified; } public override List<EntityEntry> GetChangedEntityEntries() { return MyEntityEntries .Where(x => x.Value.IsModified) .Select(x => x.Value.EntityEntry) .ToList(); } public override bool IsEntityEntryModified(EntityEntry entityEntry) { if (entityEntry.State == EntityState.Modified) { return true; } var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return false; } return MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry) && myAbpEntityEntry.IsModified; } public override bool IsNavigationEntryModified(EntityEntry entityEntry, int? navigationEntryIndex = null) { var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return false; } if (!MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { return false; } if (navigationEntryIndex == null) { return myAbpEntityEntry.NavigationEntries.Any(x => x.IsModified); } var navigationEntryProperty = myAbpEntityEntry.NavigationEntries.ElementAtOrDefault(navigationEntryIndex.Value); return navigationEntryProperty != null && navigationEntryProperty.IsModified; } public override AbpNavigationEntry? GetNavigationEntry(EntityEntry entityEntry, int navigationEntryIndex) { var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return null; } if (!MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { return null; } return myAbpEntityEntry.NavigationEntries.ElementAtOrDefault(navigationEntryIndex); } protected override string? GetEntityEntryIdentity(EntityEntry entityEntry) { if (entityEntry.Entity is IEntity entryEntity && entryEntity.GetKeys().Length == 1) { return $"{entityEntry.Metadata.ClrType.FullName}:{entryEntity.GetKeys().FirstOrDefault()}"; } return null; } public override void RemoveChangedEntityEntries() { MyEntityEntries.RemoveAll(x => x.Value.IsModified); } public override void Clear() { MyEntityEntries.Clear(); } }
-
0
Hi, that is definitely a lot better - now on my local machine (very powerful), the query takes 1.36s instead of several minutes. That still feels pretty slow given how simple the query is, but it is ok for now. I would like to see some work done in the next major version of ABP around performance.
Thank you.
-
0
Hi
If you can share a minimal project I can check it, maybe we can improve more performance.
Thanks
-
0
I will try today. For what it's worth, the query in 9.0.4 took ~600ms. If I add in this code to EfCoreNavigationHelper in ChangeTracker_Tracked:
if (e.Entry.Entity is Partner or PartnerStudent or Student) return;
Effectively ignoring everything that happens in EntityEntryTrackedOrStateChanged and DetectChanges, it takes ~300ms. That is the baseline I'd expect performance to be at, or close to. Not double.
I appreciate your assistance.
-
0
Hi
I will check this case with your project.
Thanks
-
0
Hi, the slowdown appears to be in calling the UpdateNavigationEntries() function. With my changes below, the performance is back to 650ms or so. It seems to call that function many times per singular entity which I think is the real problem.
I have this at the top of the ef core navigation helper class:
protected Dictionary<string, MyAbpEntityEntry> MyEntityEntriesNavUpdated { get; } = new();
Then this in
DetectChanges
:if (!MyEntityEntriesNavUpdated.ContainsKey(entryId)) { myAbpEntityEntry.UpdateNavigationEntries(); MyEntityEntriesNavUpdated.Add(entryId, myAbpEntityEntry); }
So it only runs that function once per entity entry, and that also fixed the performance back to 650ms. But I do not know if this would cause any bugs. I don't know the use case of all of this navigation helper code, nor do I have access to your guys' tests.
-
0
Hi
I will check if we can refactor this function.
Thanks
-
0
Thank you. I did not have time today to produce a minimal repro, but your unit test basically covers my scenario. I appreciate you adding performance regression tests.
-
0
Hi
I will create a test project and check the results if you don’t have time to prepare the demo.
Thanks
-
0
hi
The new version
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.ChangeTracking; using Microsoft.EntityFrameworkCore.Internal; using Microsoft.EntityFrameworkCore.Metadata; using Volo.Abp.DependencyInjection; using Volo.Abp.Domain.Entities; namespace Volo.Abp.EntityFrameworkCore.ChangeTrackers; public class MyAbpEntityEntry { public string Id { get; set; } public EntityEntry EntityEntry { get; set; } public List<MyAbpNavigationEntry> NavigationEntries { get; set; } private bool _isModified; public bool IsModified { get { return _isModified || EntityEntry.State == EntityState.Modified || NavigationEntries.Any(n => n.IsModified); } set => _isModified = value; } public MyAbpEntityEntry(string id, EntityEntry entityEntry) { Id = id; EntityEntry = entityEntry; NavigationEntries = EntityEntry.Navigations.Select(x => new MyAbpNavigationEntry(x, x.Metadata.Name)).ToList(); } public void UpdateNavigation(EntityEntry entityEntry, MyAbpNavigationEntry navigationEntry) { if (IsModified || EntityEntry.State == EntityState.Modified || navigationEntry.IsModified) { return; } var currentValue = navigationEntry.NavigationEntry.CurrentValue; if (currentValue == null) { return; } if (navigationEntry.NavigationEntry is CollectionEntry) { navigationEntry.OriginalValue!.As<List<object>>().Add(entityEntry.Entity); } else { navigationEntry.OriginalValue = currentValue; } } } public class MyAbpNavigationEntry { public NavigationEntry NavigationEntry { get; set; } public string Name { get; set; } public bool IsModified { get; set; } public object? OriginalValue { get; set; } public object? CurrentValue => NavigationEntry.CurrentValue; public MyAbpNavigationEntry(NavigationEntry navigationEntry, string name) { NavigationEntry = navigationEntry; Name = name; if (navigationEntry.CurrentValue != null ) { OriginalValue = navigationEntry is CollectionEntry ? new List<object>() : navigationEntry.CurrentValue; } } } [Dependency(ReplaceServices = true)] [ExposeServices(typeof(AbpEfCoreNavigationHelper))] public class MyAbpEfCoreNavigationHelper : AbpEfCoreNavigationHelper { protected Dictionary<string, MyAbpEntityEntry> MyEntityEntries { get; } = new(); public override void ChangeTracker_Tracked(object? sender, EntityTrackedEventArgs e) { EntityEntryTrackedOrStateChanged(e.Entry); DetectChanges(e.Entry); } public override void ChangeTracker_StateChanged(object? sender, EntityStateChangedEventArgs e) { EntityEntryTrackedOrStateChanged(e.Entry); DetectChanges(e.Entry); } protected override void EntityEntryTrackedOrStateChanged(EntityEntry entityEntry) { if (entityEntry.State != EntityState.Unchanged) { return; } var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return; } if (MyEntityEntries.ContainsKey(entryId)) { return; } MyEntityEntries.Add(entryId, new MyAbpEntityEntry(entryId, entityEntry)); } protected override void DetectChanges(EntityEntry entityEntry, bool checkEntityEntryState = true) { #pragma warning disable EF1001 var stateManager = entityEntry.Context.GetDependencies().StateManager; var internalEntityEntityEntry = stateManager.TryGetEntry(entityEntry.Entity, throwOnNonUniqueness: false); if (internalEntityEntityEntry == null) { return; } var foreignKeys = entityEntry.Metadata.GetForeignKeys().ToList(); foreach (var foreignKey in foreignKeys) { var principal = stateManager.FindPrincipal(internalEntityEntityEntry, foreignKey); if (principal == null) { continue; } var entryId = GetEntityEntryIdentity(principal.ToEntityEntry()); if (entryId == null || !MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { continue; } var navigationEntry = myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is INavigation navigationMetadata && navigationMetadata.ForeignKey == foreignKey) ?? myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is ISkipNavigation skipNavigationMetadata && skipNavigationMetadata.ForeignKey == foreignKey); if (navigationEntry != null && checkEntityEntryState && entityEntry.State == EntityState.Unchanged) { myAbpEntityEntry.UpdateNavigation(entityEntry, navigationEntry); } if (!myAbpEntityEntry.IsModified && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { myAbpEntityEntry.IsModified = true; DetectChanges(myAbpEntityEntry.EntityEntry, false); } if (navigationEntry != null && IsEntityEntryChanged(entityEntry)) { navigationEntry.IsModified = true; } } var skipNavigations = entityEntry.Metadata.GetSkipNavigations().ToList(); foreach (var skipNavigation in skipNavigations) { var joinEntityType = skipNavigation.JoinEntityType; var foreignKey = skipNavigation.ForeignKey; var inverseForeignKey = skipNavigation.Inverse.ForeignKey; foreach (var joinEntry in stateManager.Entries) { if (joinEntry.EntityType != joinEntityType || stateManager.FindPrincipal(joinEntry, foreignKey) != internalEntityEntityEntry) { continue; } var principal = stateManager.FindPrincipal(joinEntry, inverseForeignKey); if (principal == null) { continue; } var entryId = GetEntityEntryIdentity(principal.ToEntityEntry()); if (entryId == null || !MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { continue; } var navigationEntry = myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is INavigation navigationMetadata && navigationMetadata.ForeignKey == inverseForeignKey) ?? myAbpEntityEntry.NavigationEntries.FirstOrDefault(x => x.NavigationEntry.Metadata is ISkipNavigation skipNavigationMetadata && skipNavigationMetadata.ForeignKey == inverseForeignKey); if (navigationEntry != null && checkEntityEntryState && entityEntry.State == EntityState.Unchanged) { myAbpEntityEntry.UpdateNavigation(entityEntry, navigationEntry); } if (!myAbpEntityEntry.IsModified && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { myAbpEntityEntry.IsModified = true; DetectChanges(myAbpEntityEntry.EntityEntry, false); } if (navigationEntry != null && (!checkEntityEntryState || IsEntityEntryChanged(entityEntry))) { navigationEntry.IsModified = true; } } } #pragma warning restore EF1001 } protected override bool IsEntityEntryChanged(EntityEntry entityEntry) { return entityEntry.State == EntityState.Added || entityEntry.State == EntityState.Deleted || entityEntry.State == EntityState.Modified; } public override List<EntityEntry> GetChangedEntityEntries() { return MyEntityEntries .Where(x => x.Value.IsModified) .Select(x => x.Value.EntityEntry) .ToList(); } public override bool IsEntityEntryModified(EntityEntry entityEntry) { if (entityEntry.State == EntityState.Modified) { return true; } var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return false; } return MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry) && myAbpEntityEntry.IsModified; } public override bool IsNavigationEntryModified(EntityEntry entityEntry, int? navigationEntryIndex = null) { var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return false; } if (!MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { return false; } if (navigationEntryIndex == null) { return myAbpEntityEntry.NavigationEntries.Any(x => x.IsModified); } var navigationEntryProperty = myAbpEntityEntry.NavigationEntries.ElementAtOrDefault(navigationEntryIndex.Value); return navigationEntryProperty != null && navigationEntryProperty.IsModified; } public override AbpNavigationEntry? GetNavigationEntry(EntityEntry entityEntry, int navigationEntryIndex) { var entryId = GetEntityEntryIdentity(entityEntry); if (entryId == null) { return null; } if (!MyEntityEntries.TryGetValue(entryId, out var myAbpEntityEntry)) { return null; } var element = myAbpEntityEntry.NavigationEntries.ElementAtOrDefault(navigationEntryIndex); if (element == null) { return null; } return new AbpNavigationEntry(element.NavigationEntry, element.Name) { IsModified = element.IsModified, OriginalValue = element.OriginalValue }; } protected override string? GetEntityEntryIdentity(EntityEntry entityEntry) { if (entityEntry.Entity is IEntity entryEntity && entryEntity.GetKeys().Length == 1) { return $"{entityEntry.Metadata.ClrType.FullName}:{entryEntity.GetKeys().FirstOrDefault()}"; } return null; } public override void RemoveChangedEntityEntries() { MyEntityEntries.RemoveAll(x => x.Value.IsModified); } public override void Clear() { MyEntityEntries.Clear(); } }
-
0
hi
9.2.3 is released. You can test it.
Thanks.