Hi,
I just added a new tenant displayed the pod log for the Identity service and there was nothing there at the time I created the tenant. However I did see this in the log at 02:47 UTC
2026-02-03T02:47:36.683515240Z [02:47:36 WRN] None of the specified endpoints were reachable
2026-02-03T02:47:36.683541140Z RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
2026-02-03T02:47:36.683565841Z ---> System.AggregateException: One or more errors occurred. (Connection failed)
2026-02-03T02:47:36.683591141Z ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed
2026-02-03T02:47:36.683597542Z ---> System.Net.Sockets.SocketException (111): Connection refused
2026-02-03T02:47:36.683602742Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2026-02-03T02:47:36.683607242Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2026-02-03T02:47:36.683611442Z at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2026-02-03T02:47:36.683616042Z --- End of stack trace from previous location ---
2026-02-03T02:47:36.683620442Z at RabbitMQ.Client.TcpClientAdapter.ConnectAsync(String host, Int32 port)
2026-02-03T02:47:36.683624942Z at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout)
2026-02-03T02:47:36.683638043Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
2026-02-03T02:47:36.683644043Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683648743Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
2026-02-03T02:47:36.683665643Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout, AddressFamily family)
2026-02-03T02:47:36.683675744Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout)
2026-02-03T02:47:36.683681144Z at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
2026-02-03T02:47:36.683707844Z at RabbitMQ.Client.Framing.Impl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, ArrayPool`1 pool, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
2026-02-03T02:47:36.683732245Z at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)
2026-02-03T02:47:36.683744445Z at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
2026-02-03T02:47:36.683749445Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683753645Z at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
2026-02-03T02:47:36.683775946Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
2026-02-03T02:47:36.683780846Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683785046Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
2026-02-03T02:47:36.683789146Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
2026-02-03T02:47:36.683793646Z at RabbitMQ.Client.ConnectionFactory.CreateConnection()
2026-02-03T02:47:36.683798447Z at Volo.Abp.RabbitMQ.ConnectionPool.<>c__DisplayClass9_0.<GetConnection>b__1()
2026-02-03T02:47:36.683807147Z at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
2026-02-03T02:47:36.683811447Z at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
2026-02-03T02:47:36.683815547Z at System.Lazy`1.CreateValue()
2026-02-03T02:47:36.683869748Z at Volo.Abp.RabbitMQ.ConnectionPool.GetConnection(String connectionName, ConnectionFactory connectionFactory)
2026-02-03T02:47:36.683892549Z at Volo.Abp.RabbitMQ.ConnectionPool.Get(String connectionName)
2026-02-03T02:47:36.683897449Z at Volo.Abp.RabbitMQ.RabbitMqMessageConsumer.TryCreateChannelAsync()
I've checked the logs for the Authserver, Identity service, Administration service and Saas service. The only thing that generated a log was the tenant create (POST) in the Saas service.
I checked the rabbitmq log and there was nothing there.
I then logged into the rabbitmq management console and didn't see any outstanding messages in the queues
When creating a new Tenant in our Production environment, the Identity admin role and user are not being created. We last created a new tenant in December with no problems. We are not having this issue in our development environment.
This is all that's showing in the Saas service pod log:
[20:46:25 INF] Request starting HTTP/1.1 POST http://cloverleafcms-pr-apps-saas/api/saas/tenants?api-version=1.0 - application/json; charset=utf-8 221
[20:46:25 INF] Executing endpoint 'Volo.Saas.Host.TenantController.CreateAsync (Volo.Saas.Host.HttpApi)'
[20:46:25 INF] Route matched with {area = "saas", controller = "Tenant", action = "Create"}. Executing controller action with signature System.Threading.Tasks.Task1[Volo.Saas.Host.Dtos.SaasTenantDto] CreateAsync(Volo.Saas.Host.Dtos.SaasTenantCreateDto) on controller Volo.Saas.Host.TenantController (Volo.Saas.Host.HttpApi). [20:46:25 INF] Executing ObjectResult, writing value of type 'Volo.Saas.Host.Dtos.SaasTenantDto'. [20:46:25 INF] Executed action Volo.Saas.Host.TenantController.CreateAsync (Volo.Saas.Host.HttpApi) in 23.8808ms [20:46:25 INF] Executed endpoint 'Volo.Saas.Host.TenantController.CreateAsync (Volo.Saas.Host.HttpApi)' [20:46:25 INF] Request finished HTTP/1.1 POST http://cloverleafcms-pr-apps-saas/api/saas/tenants?api-version=1.0 - 200 null application/json; charset=utf-8 81.0558ms [20:46:25 INF] Request starting HTTP/1.1 GET http://cloverleafcms-pr-apps-saas/api/saas/tenants?GetEditionNames=True&SkipCount=0&MaxResultCount=10&api-version=1.0 - null null [20:46:25 INF] Executing endpoint 'Volo.Saas.Host.TenantController.GetListAsync (Volo.Saas.Host.HttpApi)' [20:46:25 INF] Route matched with {area = "saas", controller = "Tenant", action = "GetList"}. Executing controller action with signature System.Threading.Tasks.Task1[Volo.Abp.Application.Dtos.PagedResultDto1[Volo.Saas.Host.Dtos.SaasTenantDto]] GetListAsync(Volo.Saas.Host.Dtos.GetTenantsInput) on controller Volo.Saas.Host.TenantController (Volo.Saas.Host.HttpApi). [20:46:25 INF] Executing ObjectResult, writing value of type 'Volo.Abp.Application.Dtos.PagedResultDto1[[Volo.Saas.Host.Dtos.SaasTenantDto, Volo.Saas.Host.Application.Contracts, Version=9.2.0.0, Culture=neutral, PublicKeyToken=null]]'.
[20:46:25 INF] Executed action Volo.Saas.Host.TenantController.GetListAsync (Volo.Saas.Host.HttpApi) in 32.3796ms
[20:46:25 INF] Executed endpoint 'Volo.Saas.Host.TenantController.GetListAsync (Volo.Saas.Host.HttpApi)'
[20:46:25 INF] Request finished HTTP/1.1 GET http://cloverleafcms-pr-apps-saas/api/saas/tenants?GetEditionNames=True&SkipCount=0&MaxResultCount=10&api-version=1.0 - 200 null application/json; charset=utf-8 38.8072ms
[20:46:26 INF] Found 1 events in the outbox.
[20:46:26 INF] Sent 1 events to message broker
Steps to reproduce the issue:
SELECT *
FROM [CloverleafCMS_PR_Identity].[dbo].[AbpRoles]
-- WHERE [TenantId] is not null
WHERE
[TenantId] = '994e1b70-468f-84c4-987f-3a1f324821cd' -- New tenant Id
ORDER BY [TenantId], [Name]
SELECT
[Id]
,[UserName]
,[NormalizedUserName]
,[Name]
,[Surname]
,[Email]
FROM [CloverleafCMS_PR_Identity].[dbo].[AbpUsers]
WHERE [TenantId] = '994e1b70-468f-84c4-987f-3a1f324821cd'
The queries don't return the admin user or admin role (or any user/role).
We can close this for now. I ran the queries above and saw that the authserver had suspended transactions. I restarted the pod and the problem went away.
[20:28:35 INF] Request starting HTTP/1.1 POST http://cloverleafcms-pr-apps-identity/api/identity/users - application/json 519
[20:28:35 INF] Executing endpoint 'Volo.Abp.Identity.IdentityUserController.CreateAsync (Volo.Abp.Identity.Pro.HttpApi)'
[20:28:35 INF] Route matched with {area = "identity", controller = "User", action = "Create"}. Executing controller action with signature System.Threading.Tasks.Task`1[Volo.Abp.Identity.IdentityUserDto] CreateAsync(Volo.Abp.Identity.IdentityUserCreateDto) on controller Volo.Abp.Identity.IdentityUserController (Volo.Abp.Identity.Pro.HttpApi).
[20:29:05 ERR] Failed executing DbCommand (30,010ms) [Parameters=[@p2='?' (DbType = Guid), @p0='?' (Size = 40), @p3='?' (Size = 40), @p1='?' (DbType = Int32), @p6='?' (DbType = Guid), @p4='?' (Size = 40), @p7='?' (Size = 40), @p5='?' (DbType = Int32), @p8='?' (DbType = Guid), @p9='?' (DbType = Guid), @p10='?' (DbType = Guid), @p11='?' (DbType = Guid), @p12='?' (DbType = Guid), @p13='?' (DbType = Guid), @p18='?' (DbType = Guid), @p14='?' (Size = 40), @p19='?' (Size = 40), @p15='?' (DbType = Int32), @p16='?' (DbType = DateTime2), @p17='?' (DbType = Guid)], CommandType='Text', CommandTimeout='30']
SET NOCOUNT ON;
UPDATE [AbpRoles] SET [ConcurrencyStamp] = @p0, [EntityVersion] = @p1
OUTPUT 1
WHERE [Id] = @p2 AND [ConcurrencyStamp] = @p3;
UPDATE [AbpRoles] SET [ConcurrencyStamp] = @p4, [EntityVersion] = @p5
OUTPUT 1
WHERE [Id] = @p6 AND [ConcurrencyStamp] = @p7;
INSERT INTO [AbpUserRoles] ([RoleId], [UserId], [TenantId])
VALUES (@p8, @p9, @p10),
(@p11, @p12, @p13);
UPDATE [AbpUsers] SET [ConcurrencyStamp] = @p14, [EntityVersion] = @p15, [LastModificationTime] = @p16, [LastModifierId] = @p17
OUTPUT 1
WHERE [Id] = @p18 AND [ConcurrencyStamp] = @p19;
[20:29:05 ERR] An exception occurred in the database while saving changes for context type 'CloverleafCMS.IdentityService.Data.IdentityServiceDbContext'.
Microsoft.EntityFrameworkCore.DbUpdateException: An error occurred while saving the entity changes. See the inner exception for details.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
ClientConnectionId:25031d3f-7e9e-4a45-b673-e954e0acbcfd
Error Number:-2,State:0,Class:11
--- End of inner exception stack trace ---
This seems to be the only service that's having the issue.
This is impacting our production users so immediate attention would be appreciated.
Thank you
We're considering deploying our microservices solution to Azure Container Apps vs Kubernetes. Has anyone done this before and if so, are there documents / templates to follow?
Hi,
In the Identity service OpenIddictDataSeeder on this block of code:
if (!Uri.TryCreate(redirectUri, UriKind.Absolute, out var uri) || !uri.IsWellFormedOriginalString())
{
throw new ApplicationException("Invalid redirect URI: " + redirectUri);
}
We get:
Exception has occurred: CLR/System.ApplicationException
An exception of type 'System.ApplicationException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'Invalid redirect URI: https://{0}.mydomain.dev/signin-oidc'
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<CreateOrUpdateApplicationAsync>d__15.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 627
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<CreateClientsAsync>d__14.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 354
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<SeedAsync>d__9.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 57
at Castle.DynamicProxy.AsyncInterceptorBase.<ProceedAsynchronous>d__13.MoveNext()
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapter.<ProceedAsync>d__7.MoveNext()
at Volo.Abp.Uow.UnitOfWorkInterceptor.<InterceptAsync>d__2.MoveNext()
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.<InterceptAsync>d__2.MoveNext()
at CloverleafCMS.IdentityService.Data.IdentityServiceDataSeeder.<SeedOpenIddictAsync>d__12.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceDataSeeder.cs:line 105
at CloverleafCMS.IdentityService.Data.IdentityServiceDataSeeder.<SeedAsync>d__9.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceDataSeeder.cs:line 52
at CloverleafCMS.IdentityService.Data.IdentityServiceRuntimeDatabaseMigrator.<SeedAsync>d__2.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceRuntimeDatabaseMigrator.cs:line 35
at Volo.Abp.EntityFrameworkCore.Migrations.EfCoreRuntimeDatabaseMigratorBase`1.<LockAndApplyDatabaseMigrationsAsync>d__35.MoveNext()
We've already done the above. We added "https://{0}.mydomain.dev/" to the appsettings in the Identity service and it threw an error that it's a malformed url in the CreateOrUpdateApplicationAsync of OpenIddictDataSeeder.cs.
We have a Saas web application written in Vue JS that uses our Abp Microservice solution for the backend. Each tenant will have their own domain like https://client1.mydomain, https://client2.mydomain.dev, etc.
We added this to the auth server module PreconfigureServices PreConfigure(options => { options.EnableWildcardDomainSupport = true; // Development environment domains options.WildcardDomainsFormat.Add("https://{0}.mydomain.dev/signin-oidc"); options.WildcardDomainsFormat.Add("https://{0}.mydomain.dev/silent-refresh"); options.WildcardDomainsFormat.Add("https://{0}.mydomain.dev/auth/login"); options.WildcardDomainsFormat.Add("https://{0}.mydomain.dev/auth/signout-callback-oidc"); });
In the Identity database OpenIddictApplications table we manually updated the RedirectUris and PostLogoutRedirectUris with the wildcard character '*'
[
{
"id": "9335f213-8a0d-7ce3-f069-3a1b786183db",
"ClientId": "CloverleafCMS",
"RedirectUris": "["https://*.mydomain.dev/signin-oidc","https://*.mydomain.dev/auth/silent-refresh"]",
"PostLogoutRedirectUris": "["https://.cloverleafcms.us/auth/login"]"
}
]
The issue is that the OpenIddictApplications table is reseeded every time the Identity service is restarted. So we have to manually update the redirect Uris with the wildcard manually
I tried adding the wildcard in the Identity data seeder but it through a malformed url error.
I read this article https://abp.io/docs/latest/guides/ms-multi-tenant-domain-resolving but not sure that will fix our problem since our client application is written in Vue.
How do we get around this? <br>
I think this fixed it.
Thank you.
