Issue creating new Tenants in production #10384

5 Answer(s)

• 

  0

  AI-Bot created 15 days ago

  Support Team ABP Agent

  • In the microservice template, creating a tenant in SaaS does not directly create the tenant’s Identity admin user/role in the Identity DB. Instead, SaaS publishes a distributed event (outbox), and the Identity service is expected to consume it and then run the tenant migration + data seeding for that new tenant (admin role/user creation happens during that seed).
  • Your SaaS pod log shows: “Found 1 events in the outbox. Sent 1 events to message broker”. That means the SaaS side is doing its part. The missing users/roles in CloverleafCMS_PR_Identity indicates the consumer side (Identity service) is not processing the tenant-created event, or it processes it but the seed/migration fails (often with a DB connection/timeout issue in production).

  What to do (production-accurate checklist)

  1. Check Identity service logs right after tenant creation

     • You should see logs about handling the tenant-created event and running migration/seeding for the new tenant.
     • If you see nothing, the Identity service is not subscribed/consuming (message broker / consumer / inbox issue).

  2. Verify the distributed event infrastructure is healthy

     • Since SaaS says it sent the event, check:
       • Message broker connectivity/credentials for Identity service (RabbitMQ/Kafka, etc.).
       • Consumer is enabled and running in the Identity pod.
     • If you use inbox/outbox tables, also check Identity’s inbox background worker health (if enabled).

  3. Most common real cause in production: seeding/migration fails due to DB connection/timeout

     • When seeding cannot complete, ABP won’t create the tenant admin user.
     • Check production error logs around tenant creation time for SQL timeouts, connection string resolution issues, or migration failures.

  4. If you are using “shared database” (no per-tenant connection string)

     • There is a known/previously-reported issue pattern where tenant seeding can fail depending on how the unit of work is started during tenant migration/seed flow (especially around connection string resolution / timeouts).
     • See the discussion and workaround notes in the support thread below.

  5. Confirm tenant DB (or schemas) are created/migrated and seed completed

     • ABP’s KB explicitly points out that inability to login / missing tenant admin typically means tenant seeding didn’t complete; you must check the logs for the seeding process.

  Sources

  • https://abp.io/docs/latest/kb/can-not-login-with-admin-user
  • https://abp.io/support/questions/2240

  ---

  This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  Save Cancel
• 

  0

  rogercprops created 15 days ago

  I've checked the logs for the Authserver, Identity service, Administration service and Saas service. The only thing that generated a log was the tenant create (POST) in the Saas service.

  I checked the rabbitmq log and there was nothing there.

  I then logged into the rabbitmq management console and didn't see any outstanding messages in the queues

  Save Cancel
• 

  0

  maliming created 15 days ago

  Support Team Fullstack Developer

  hi

  This is most likely because the distributed event bus isn't working properly.

  SaasService will publish a TenantCreatedEto event, and IdentityService will create the admin for the new tenant.

  Can you check the logs of IdentityService?

  Thanks.

  Save Cancel
• 

  0

  rogercprops created 14 days ago

  Hi,

  I just added a new tenant displayed the pod log for the Identity service and there was nothing there at the time I created the tenant. However I did see this in the log at 02:47 UTC

  2026-02-03T02:47:36.683515240Z [02:47:36 WRN] None of the specified endpoints were reachable
  2026-02-03T02:47:36.683541140Z RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
  2026-02-03T02:47:36.683565841Z  ---> System.AggregateException: One or more errors occurred. (Connection failed)
  2026-02-03T02:47:36.683591141Z  ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed
  2026-02-03T02:47:36.683597542Z  ---> System.Net.Sockets.SocketException (111): Connection refused
  2026-02-03T02:47:36.683602742Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
  2026-02-03T02:47:36.683607242Z    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
  2026-02-03T02:47:36.683611442Z    at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
  2026-02-03T02:47:36.683616042Z --- End of stack trace from previous location ---
  2026-02-03T02:47:36.683620442Z    at RabbitMQ.Client.TcpClientAdapter.ConnectAsync(String host, Int32 port)
  2026-02-03T02:47:36.683624942Z    at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout)
  2026-02-03T02:47:36.683638043Z    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
  2026-02-03T02:47:36.683644043Z    --- End of inner exception stack trace ---
  2026-02-03T02:47:36.683648743Z    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
  2026-02-03T02:47:36.683665643Z    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout, AddressFamily family)
  2026-02-03T02:47:36.683675744Z    at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout)
  2026-02-03T02:47:36.683681144Z    at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
  2026-02-03T02:47:36.683707844Z    at RabbitMQ.Client.Framing.Impl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, ArrayPool`1 pool, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
  2026-02-03T02:47:36.683732245Z    at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)
  2026-02-03T02:47:36.683744445Z    at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
  2026-02-03T02:47:36.683749445Z    --- End of inner exception stack trace ---
  2026-02-03T02:47:36.683753645Z    at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
  2026-02-03T02:47:36.683775946Z    at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
  2026-02-03T02:47:36.683780846Z    --- End of inner exception stack trace ---
  2026-02-03T02:47:36.683785046Z    at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
  2026-02-03T02:47:36.683789146Z    at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
  2026-02-03T02:47:36.683793646Z    at RabbitMQ.Client.ConnectionFactory.CreateConnection()
  2026-02-03T02:47:36.683798447Z    at Volo.Abp.RabbitMQ.ConnectionPool.&lt;&gt;c__DisplayClass9_0.&lt;GetConnection&gt;b__1()
  2026-02-03T02:47:36.683807147Z    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
  2026-02-03T02:47:36.683811447Z    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
  2026-02-03T02:47:36.683815547Z    at System.Lazy`1.CreateValue()
  2026-02-03T02:47:36.683869748Z    at Volo.Abp.RabbitMQ.ConnectionPool.GetConnection(String connectionName, ConnectionFactory connectionFactory)
  2026-02-03T02:47:36.683892549Z    at Volo.Abp.RabbitMQ.ConnectionPool.Get(String connectionName)
  2026-02-03T02:47:36.683897449Z    at Volo.Abp.RabbitMQ.RabbitMqMessageConsumer.TryCreateChannelAsync()
  
  

  Save Cancel
• 

  0

  maliming created 14 days ago

  Support Team Fullstack Developer

  hi

  Can you try enabling debug logs and check again?

  The event will be added to the SaaS server Outboxes and sent to the identity server Inboxes. Can you also check the tables in the database?

  https://abp.io/support/questions/8622/How-to-enable-Debug-logs-for-troubleshoot-problems

  Thanks.

  Save Cancel