A lot of inserts take an extremely long time #10748

6 Answer(s)

• 

  0

  AI-Bot created 3 months ago

  Support Team ABP Agent

  • uow.CompleteAsync() in ABP does more than “commit”. Even when isTransactional: false, it still runs the Unit Of Work completion pipeline, which includes (among other things) publishing/processing “auto events” (domain/entity change events) inside the same UoW. With ~1,000,000 inserts, any “per-entity” event collection/publication/handler execution can dominate the runtime and look like “hangs for hours”.

  • The fix is to avoid accumulating 1M changes in a single UoW and avoid/disable auto-event generation/handling for bulk imports.

  What to do (practical, ABP-friendly)

  1. Split the import into batches and complete multiple UoWs
     • E.g. process 5k–50k entities, then CompleteAsync(), then start a new UoW.
     • This prevents ABP from holding a gigantic “changed entity / event” list until the very end.

  const int batchSize = 10000;
  
     foreach (var batch in data.Chunk(batchSize))
     {
         using var uow = _unitOfWorkManager.Begin(requiresNew: true, isTransactional: false);
  
         await _installer.InstallBatchAsync(batch);
  
         await uow.CompleteAsync();
     }
  

  2. Disable/avoid auto-event publishing for this import path

     • Your suspicion is very likely correct: ABP’s “auto events” are handled in the same unit of work (so they run during CompleteAsync()).
     • If you have any event handlers (local/distributed) hooked to entity changes (directly or indirectly), they can be triggered massively.

     The exact way to disable auto-event publishing depends on what you’re using (domain events, entity change events, distributed events, etc.). However, the most robust approach for “1M inserts import” is:

     • Do not use regular repository insert/update APIs that trigger the ABP entity change/event pipeline, or
     • Use a bulk import approach (EFCore.BulkExtensions / SqlBulkCopy) in a separate, dedicated path (so ABP’s UoW/event mechanisms are bypassed).

  3. If you keep EF Core inserts: flush periodically

     • Even within one UoW, forcing periodic SaveChangesAsync() can reduce memory/event buildup, but it still leaves the final CompleteAsync() pipeline cost. Batching UoWs is usually better.

  Why [DisableAuditing] didn’t help

  • Auditing is only one part of the pipeline. CompleteAsync() can still be slow due to auto-event processing and other UoW completion tasks, even if auditing is disabled and the entities aren’t auditable.

  ---

  Sources

  • https://abp.io/docs/latest/framework/architecture/domain-driven-design/unit-of-work
  • https://abp.io/docs/latest/release-info/migration-guides/abp-5-0
  • https://abp.io/community/articles/0r248xsr

  ---

  This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 3 months ago

  Support Team Fullstack Developer

  Hi,

  Could you share _installer.InstallAsync? We'd like to see exactly how the rows are being written — IRepository<T>.InsertAsync in a loop, InsertManyAsync, _dbContext.AddRange + SaveChangesAsync, or something else.

  One thing to flag up front — [DisableEntityChangeTracking] and repository.DisableTracking() only affect read queries, not InsertAsync / UpdateAsync. The docs are explicit about this:

  Change tracking behavior doesn't affect tracking entity objects returned from InsertAsync and UpdateAsync methods.

  So those attributes won't change anything on the insert path.

  Here's an approach you can try — a small replacement for IEntityChangeEventHelper that suppresses entity events for an explicit using scope. Could you drop this into your project, wrap the import call site like below, and let us know whether CompleteAsync drops to something reasonable on your 1M case?

  [Dependency(ReplaceServices = true)]
  [ExposeServices(typeof(IEntityChangeEventHelper), typeof(EntityChangeEventHelper))]
  public class ScopedEntityChangeEventHelper : EntityChangeEventHelper
  {
      private static readonly AsyncLocal<bool> _suppressed = new();
  
      public ScopedEntityChangeEventHelper(
          IUnitOfWorkManager uowManager,
          IEntityToEtoMapper etoMapper,
          IOptions<AbpDistributedEntityEventOptions> options)
          : base(uowManager, etoMapper, options) { }
  
      public static IDisposable Suppress()
      {
          var prev = _suppressed.Value;
          _suppressed.Value = true;
          return new DisposeAction(() => _suppressed.Value = prev);
      }
  
      public override void PublishEntityCreatedEvent(object entity)
      {
          if (!_suppressed.Value) base.PublishEntityCreatedEvent(entity);
      }
  
      public override void PublishEntityUpdatedEvent(object entity)
      {
          if (!_suppressed.Value) base.PublishEntityUpdatedEvent(entity);
      }
  
      public override void PublishEntityDeletedEvent(object entity)
      {
          if (!_suppressed.Value) base.PublishEntityDeletedEvent(entity);
      }
  }
  

  Usage at your import call site — _installer.InstallAsync itself stays unchanged:

  using var _ = ScopedEntityChangeEventHelper.Suppress();
  using var uow = _unitOfWorkManager.Begin(requiresNew: true, isTransactional: false);
  await _installer.InstallAsync(data);
  await uow.CompleteAsync();
  

  The effect is scoped to the using block, so the rest of your app keeps behaving normally.

  To be clear about what this does and doesn't touch — it only intercepts the per-entity broadcast events (EntityCreatedEventData<T> / EntityUpdatedEventData<T> / EntityDeletedEventData<T> and their distributed Eto counterparts). Everything else ABP normally does for tracked entities keeps running: CreationTime / CreatorId / LastModificationTime / ConcurrencyStamp are still set automatically, multi-tenancy and soft-delete still work, and any domain events you raise yourself with entity.AddLocalEvent(...) / AddDistributedEvent(...) go through a different pipeline and are not affected.

  If you do have any local or distributed event handlers subscribed to EntityCreatedEventData<T> / EntityCreatedEto<T> for the entities being inserted, those will be skipped during the import — usually fine for a bulk install, but worth confirming on your side.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  ageiter created 3 months ago

  Hi, thanks for this detailed answer. This is exactly the kind of solution I was looking for. I’ll be happy to test it out, and I think this is what I need.

  _installer.InstallAsync() is relatively complex (recursive traversal and inserts across multiple repositories), so I can’t share it here. But if your approach doesn’t work, I’ll send you the code via email.

  By the way: after 1000 inserts, I call await _unitOfWorkManager.Current!.SaveChangesAsync(). I also clear the DbContext’s ChangeTracker. The data is also fully present in the database. That’s why I suspect the problem might be in postprocessing (event pipeline).

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 3 months ago

  Support Team Fullstack Developer

  Hi,

  Sounds good. Once you've had a chance to try it, let us know how it goes. If anything doesn't work as expected, share whatever details you can — timings, a small repro, or send the code via email if that's easier — and we'll keep digging on our side.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  ageiter created 3 months ago

  Great, it works :-D

  Thank you so much for the quick help! I wish I'd asked much sooner ;-)

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 3 months ago

  Support Team Fullstack Developer

  : )

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel