ABP Framework version: 9.1.1
Template: Microservice (with multi-tenancy)
When creating a new tenant in a microservice template, the admin user is sometimes created and sometimes not. The behavior is non-deterministic.
Root cause analysis:
In the Identity Service, there are ~7 registered IDistributedEventHandler<TenantCreatedEto> implementations — one from our custom IdentityServiceDatabaseMigrationEventHandler (which seeds the admin user) and 6 from ABP's built-in EF Core modules (SaasEntityFrameworkCoreModule, AbpPermissionManagementEntityFrameworkCoreModule, AbpFeatureManagementEntityFrameworkCoreModule, AbpSettingManagementEntityFrameworkCoreModule, AbpIdentityProEntityFrameworkCoreModule, AbpOpenIddictProEntityFrameworkCoreModule).
When a TenantCreatedEto event is received, all 7 handlers are dispatched nearly simultaneously (within milliseconds). Each handler calls MigrateDatabaseSchemaAsync(), which attempts to acquire a distributed lock via Redis. Only one handler can acquire the lock at a time — the remaining 6 time out and throw an exception.
In the custom handler's HandleEventAsync override, both MigrateDatabaseSchemaAsync() and the admin seeding (_dataSeeder.SeedAsync()) are wrapped in a single try/catch. If the distributed lock times out during migration, the entire block fails — including the admin user seeding. The exception is caught by HandleErrorTenantCreatedAsync which does not rethrow, so the inbox processor considers the event "processed" and does not retry.
Evidence from AbpEventInbox table:
Each tenant creation generates 7 inbox records with the same MessageId, all marked as Processed = true. Example for tenant "lbb":
- 7 records, same MessageId 3a1fe4020958a96fbf4bcf7f97ce85e3
- CreationTime spread: 08:02:16.189 to 08:02:16.196 (7ms total)
- ProcessedTime spread: 08:02:16.650 to 08:02:16.857 (207ms total)
Impact: In a shared-database scenario, the admin user is intermittently not created for new tenants, making the tenant unusable.
Expected behavior: The admin user should always be created regardless of distributed lock contention, since for shared-database tenants the schema migration is unnecessary (the schema already exists from the host migration).
4 Answer(s)
-
0
hi
I will check this case asap.
Thanks.
-
0
hi
The
HandleErrorTenantCreatedAsyncwill try to publish the event again, and your event handler will execute again.BTW, it will delay random
5-15s.protected virtual async Task HandleErrorTenantCreatedAsync( TenantCreatedEto eventData, Exception exception) { var tryCount = IncrementEventTryCount(eventData); if (tryCount <= MaxEventTryCount) { Logger.LogWarning($"Could not perform tenant created event. Re-queueing the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}."); Logger.LogException(exception, LogLevel.Warning); await Task.Delay(RandomHelper.GetRandom(5000, 15000)); await DistributedEventBus.PublishAsync(eventData); } else { Logger.LogError($"Could not perform tenant created event. Canceling the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}."); Logger.LogException(exception); } }Thanks,
-
0
Hi, but when I create a new tenant, 50% of the time it doesn't create the admin user. I have event management with In/Outbox, and even though the database is single, I have different DbContexts for the various services: SaaS, Identity, Administration, etc
-
0
hi
Do your logs contain
Could not perform tenant created event. Re-queueing the operation. TenantIdmessages?Let's make sure the retry mechanism is working.
Thanks
