Distributed Event Inbox — a failing handler blocks the entire inbox (head-of-line blocking) #10761

4 Answer(s)

• 

  0

  AI-Bot created 2 months ago

  Support Team ABP Agent

  • In ABP 9.3.x, yes—this head-of-line blocking is the effective/expected behavior of the default InboxProcessor: it processes inbox records in order, and when a handler throws, the processor rethrows and the same earliest failing inbox item is retried on every cycle, preventing later items from being processed.

  • In ABP 9.3.5 there is no built-in configuration to “skip the failing one and continue” (so the behavior you see is consistent with the framework behavior in that version).

  • Built-in max-retry / discard/dead-letter for inbox: this was added in ABP 10.0 via a new InboxProcessor failure retry policy.

    • InboxProcessorFailurePolicy options:
      • Retry (default): keeps strict order; retries from the failing event next cycle (head-of-line blocking remains).
      • RetryLater: skips the failing event, continues with following events, and retries the failed one later with exponential backoff; default max retry is 10 (configurable); discards after max retry.
      • Discard: discards the failing event immediately.
    • InboxProcessorRetryBackoffFactor: initial backoff factor (default 10), used by RetryLater with exponential backoff.

  • Recommended pattern for “Event A depends on data produced by Event B” in a microservice/inbox world:

    1. Do not rely on arrival order between independent publishers/services; design handlers to tolerate reordering.
    2. Prefer making handlers idempotent and order-insensitive by persisting local state (e.g., “pending” / “already applied” / versioning) so that processing A before B is safe and eventually consistent.
    3. If you must use “throw-to-retry”, then on ABP 9.3.5 you risk permanent head-of-line blocking like you described; the practical fix is upgrade to ABP 10+ and use RetryLater (or redesign to remove the causal dependency / add a local projection/state machine).
    4. ABP also has a community article discussing disordering in distributed events and patterns to handle it (idempotency, local state, disentangling causality).

  Actionable fix (framework-level): upgrade to ABP 10+

  • Upgrade to ABP 10.x, add the required EF Core migration (ABP 10.0 notes mention IncomingEventRecord schema changes), then configure:
    • InboxProcessorFailurePolicy = RetryLater (to prevent head-of-line blocking)
    • adjust max retry / backoff as desired (per ABP 10.0 migration guide).

  Sources (verified):

  • https://abp.io/docs/10.0//release-info/migration-guides/abp-10-0
  • https://github.com/abpframework/abp/pull/23563
  • https://abp.io/community/articles/yi9vq3p4

  ---

  This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 2 months ago

  Support Team Fullstack Developer

  Hi,

  Quick answers to your questions:

  1. Yes, on 9.3.x this head-of-line blocking is the expected behavior. The inbox is processed strictly in CreationTime order, and when a handler throws, the default policy rethrows so the same earliest event is picked up again on the next cycle. Events that were processed before the failing one are already committed (each runs in its own transaction), so it's the failing event and everything after it that stays stuck.

  2 & 3. There was no built-in "skip and continue" or max-retry/dead-letter for the inbox in 9.3.5. We added exactly this in ABP 10.0 via InboxProcessorFailurePolicy (abpframework/abp#23563). You configure it on AbpEventBusBoxesOptions:

  Configure<AbpEventBusBoxesOptions>(options =>
  {
      options.InboxProcessorFailurePolicy = InboxProcessorFailurePolicy.RetryLater;
      options.InboxProcessorMaxRetryCount = 10;       // default
      options.InboxProcessorRetryBackoffFactor = 10;  // default
  });
  

  The three policies:

  • Retry (default) — keeps strict order, retries the failing event next cycle. This is the head-of-line blocking you're seeing.
  • RetryLater — skips the failing event and keeps processing the rest, then retries the failed one after a delay. The delay is factor * 2^retryCount (so 10s, 20s, 40s … with the defaults). After InboxProcessorMaxRetryCount it's discarded. This is the one that fixes your A-blocks-B case: A is pushed out to a later retry, B gets processed in the meantime, and when A comes back its data is ready.
  • Discard — drops the failing event immediately.

  This needs an upgrade to ABP 10.x. The inbox table schema changed in 10.0 (the old Processed/ProcessedTime columns became Status/HandledTime plus new RetryCount/NextRetryTime), so after upgrading you'll add an EF Core migration for it. If you have any custom IEventInbox implementation, note that the interface also got RetryLaterAsync and MarkAsDiscardAsync in 10.0, so that would need updating too.

  4. RetryLater solves the inbox blocking, but I wouldn't lean on throw-to-retry as the main mechanism for a real causal dependency between events from two independent services. With the defaults the total retry window is bounded (roughly a couple of hours before discard), so if B can ever be delayed longer than that, A gets dropped. The more robust pattern is to make the dependent handler order-insensitive: when A arrives and its data isn't there yet, persist a small "pending" record and return normally instead of throwing, then complete that work when B's handler runs (or have a background worker reconcile pending items). Keep RetryLater as a safety net for genuinely transient failures, not as the way to order causally-dependent events.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  yazilim.ithinka created 2 months ago

  Hello,

  We are currently unable to address the ABP 10.x version update, but we can consider modifying the ‘pending’ structure or the event flow as you have suggested.

  Thanks.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 2 months ago

  Support Team Fullstack Developer

  Hi,

  That redesign is the durable fix. If you need something for 9.3.5 right now without upgrading, you can replace the inbox processor so a failing handler no longer blocks the events behind it. It skips the failing event for the current run, keeps processing the rest, and retries the failed one on the next cycle (by which point the event it depends on may already be processed):

  using Microsoft.Extensions.Logging;
  using Microsoft.Extensions.Options;
  using Volo.Abp.DependencyInjection;
  using Volo.Abp.DistributedLocking;
  using Volo.Abp.EventBus.Distributed;
  using Volo.Abp.Threading;
  using Volo.Abp.Timing;
  using Volo.Abp.Uow;
  
  [Dependency(ReplaceServices = true)]
  [ExposeServices(typeof(IInboxProcessor))]
  public class SkipFailingInboxProcessor : InboxProcessor
  {
      public SkipFailingInboxProcessor(
          IServiceProvider serviceProvider,
          AbpAsyncTimer timer,
          IDistributedEventBus distributedEventBus,
          IAbpDistributedLock distributedLock,
          IUnitOfWorkManager unitOfWorkManager,
          IClock clock,
          IOptions<AbpEventBusBoxesOptions> eventBusBoxesOptions)
          : base(serviceProvider, timer, distributedEventBus, distributedLock,
                 unitOfWorkManager, clock, eventBusBoxesOptions)
      {
      }
  
      protected override async Task RunAsync()
      {
          if (StoppingToken.IsCancellationRequested)
          {
              return;
          }
  
          await using (var handle = await DistributedLock.TryAcquireAsync(DistributedLockName, cancellationToken: StoppingToken))
          {
              if (handle == null)
              {
                  Logger.LogDebug("Could not obtain the distributed lock: " + DistributedLockName);
                  try
                  {
                      await Task.Delay(EventBusBoxesOptions.DistributedLockWaitDuration, StoppingToken);
                  }
                  catch (TaskCanceledException) { }
                  return;
              }
  
              await DeleteOldEventsAsync();
  
              // Events that failed in this run. Skipping them keeps one bad event
              // from blocking the rest, and prevents a tight loop on the while below.
              var failedEventIds = new HashSet<Guid>();
  
              while (true)
              {
                  var pendingEvents = (await GetWaitingEventsAsync())
                      .Where(x => !failedEventIds.Contains(x.Id))
                      .ToList();
  
                  if (pendingEvents.Count <= 0)
                  {
                      break;
                  }
  
                  foreach (var waitingEvent in pendingEvents)
                  {
                      try
                      {
                          using (var uow = UnitOfWorkManager.Begin(isTransactional: true, requiresNew: true))
                          {
                              await DistributedEventBus
                                  .AsSupportsEventBoxes()
                                  .ProcessFromInboxAsync(waitingEvent, InboxConfig);
  
                              await Inbox.MarkAsProcessedAsync(waitingEvent.Id);
  
                              await uow.CompleteAsync(StoppingToken);
                          }
                      }
                      catch (OperationCanceledException) when (StoppingToken.IsCancellationRequested)
                      {
                          throw; // shutting down: don't treat cancellation as an event failure
                      }
                      catch (Exception ex)
                      {
                          failedEventIds.Add(waitingEvent.Id);
                          Logger.LogWarning(ex, $"Inbox event {waitingEvent.Id:N} failed; skipping it for this run, will retry next cycle.");
                      }
                  }
              }
          }
      }
  }
  

  A few things to keep in mind, this is a stopgap, not a backport of the 10.0 policy:

  • It skips failed events within the current processor run; it has no backoff and no max-retry/discard. A failing event keeps retrying every cycle (with a warning log each time) until it succeeds, it just no longer blocks the others. The bounded retry-with-backoff and discard is what InboxProcessorFailurePolicy.RetryLater adds in 10.0.
  • The processor reads a page of waiting events (InboxWaitingEventMaxCount, default 1000) ordered by CreationTime. As long as the failing events are a small part of that page, the rest keep flowing. If failing events ever pile up to the full page size, the later events would be blocked again, so you can raise InboxWaitingEventMaxCount if needed.

  The durable fix is still the order-insensitive design, this just keeps the inbox moving in the meantime.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel