Hi,
I just added a new tenant displayed the pod log for the Identity service and there was nothing there at the time I created the tenant. However I did see this in the log at 02:47 UTC
2026-02-03T02:47:36.683515240Z [02:47:36 WRN] None of the specified endpoints were reachable
2026-02-03T02:47:36.683541140Z RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
2026-02-03T02:47:36.683565841Z ---> System.AggregateException: One or more errors occurred. (Connection failed)
2026-02-03T02:47:36.683591141Z ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed
2026-02-03T02:47:36.683597542Z ---> System.Net.Sockets.SocketException (111): Connection refused
2026-02-03T02:47:36.683602742Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2026-02-03T02:47:36.683607242Z at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2026-02-03T02:47:36.683611442Z at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2026-02-03T02:47:36.683616042Z --- End of stack trace from previous location ---
2026-02-03T02:47:36.683620442Z at RabbitMQ.Client.TcpClientAdapter.ConnectAsync(String host, Int32 port)
2026-02-03T02:47:36.683624942Z at RabbitMQ.Client.Impl.TaskExtensions.TimeoutAfter(Task task, TimeSpan timeout)
2026-02-03T02:47:36.683638043Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
2026-02-03T02:47:36.683644043Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683648743Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, TimeSpan timeout)
2026-02-03T02:47:36.683665643Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout, AddressFamily family)
2026-02-03T02:47:36.683675744Z at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingIPv4(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan timeout)
2026-02-03T02:47:36.683681144Z at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
2026-02-03T02:47:36.683707844Z at RabbitMQ.Client.Framing.Impl.IProtocolExtensions.CreateFrameHandler(IProtocol protocol, AmqpTcpEndpoint endpoint, ArrayPool`1 pool, Func`2 socketFactory, TimeSpan connectionTimeout, TimeSpan readTimeout, TimeSpan writeTimeout)
2026-02-03T02:47:36.683732245Z at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint)
2026-02-03T02:47:36.683744445Z at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
2026-02-03T02:47:36.683749445Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683753645Z at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func`2 selector)
2026-02-03T02:47:36.683775946Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
2026-02-03T02:47:36.683780846Z --- End of inner exception stack trace ---
2026-02-03T02:47:36.683785046Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
2026-02-03T02:47:36.683789146Z at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
2026-02-03T02:47:36.683793646Z at RabbitMQ.Client.ConnectionFactory.CreateConnection()
2026-02-03T02:47:36.683798447Z at Volo.Abp.RabbitMQ.ConnectionPool.<>c__DisplayClass9_0.<GetConnection>b__1()
2026-02-03T02:47:36.683807147Z at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
2026-02-03T02:47:36.683811447Z at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
2026-02-03T02:47:36.683815547Z at System.Lazy`1.CreateValue()
2026-02-03T02:47:36.683869748Z at Volo.Abp.RabbitMQ.ConnectionPool.GetConnection(String connectionName, ConnectionFactory connectionFactory)
2026-02-03T02:47:36.683892549Z at Volo.Abp.RabbitMQ.ConnectionPool.Get(String connectionName)
2026-02-03T02:47:36.683897449Z at Volo.Abp.RabbitMQ.RabbitMqMessageConsumer.TryCreateChannelAsync()
I've checked the logs for the Authserver, Identity service, Administration service and Saas service. The only thing that generated a log was the tenant create (POST) in the Saas service.
I checked the rabbitmq log and there was nothing there.
I then logged into the rabbitmq management console and didn't see any outstanding messages in the queues
We can close this for now. I ran the queries above and saw that the authserver had suspended transactions. I restarted the pod and the problem went away.
Hi,
In the Identity service OpenIddictDataSeeder on this block of code:
if (!Uri.TryCreate(redirectUri, UriKind.Absolute, out var uri) || !uri.IsWellFormedOriginalString())
{
throw new ApplicationException("Invalid redirect URI: " + redirectUri);
}
We get:
Exception has occurred: CLR/System.ApplicationException
An exception of type 'System.ApplicationException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'Invalid redirect URI: https://{0}.mydomain.dev/signin-oidc'
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<CreateOrUpdateApplicationAsync>d__15.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 627
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<CreateClientsAsync>d__14.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 354
at CloverleafCMS.IdentityService.Data.OpenIddictDataSeeder.<SeedAsync>d__9.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/OpenIddictDataSeeder.cs:line 57
at Castle.DynamicProxy.AsyncInterceptorBase.<ProceedAsynchronous>d__13.MoveNext()
at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapter.<ProceedAsync>d__7.MoveNext()
at Volo.Abp.Uow.UnitOfWorkInterceptor.<InterceptAsync>d__2.MoveNext()
at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.<InterceptAsync>d__2.MoveNext()
at CloverleafCMS.IdentityService.Data.IdentityServiceDataSeeder.<SeedOpenIddictAsync>d__12.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceDataSeeder.cs:line 105
at CloverleafCMS.IdentityService.Data.IdentityServiceDataSeeder.<SeedAsync>d__9.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceDataSeeder.cs:line 52
at CloverleafCMS.IdentityService.Data.IdentityServiceRuntimeDatabaseMigrator.<SeedAsync>d__2.MoveNext() in /Users/rogerhopkins/App_Development/CodeRepositories/CloverleafCMS-Microservices/services/identity/CloverleafCMS.IdentityService/Data/IdentityServiceRuntimeDatabaseMigrator.cs:line 35
at Volo.Abp.EntityFrameworkCore.Migrations.EfCoreRuntimeDatabaseMigratorBase`1.<LockAndApplyDatabaseMigrationsAsync>d__35.MoveNext()
We've already done the above. We added "https://{0}.mydomain.dev/" to the appsettings in the Identity service and it threw an error that it's a malformed url in the CreateOrUpdateApplicationAsync of OpenIddictDataSeeder.cs.
I think this fixed it.
Thank you.
I just granted you access to the Token service and auth-server GitHub repositories
I made the change to the token service passing var policy = new DiscoveryPolicy
{
RequireHttps = false
};
and am now getting this error
{
"error": {
"code": null,
"message": "Error retrieving discovery document: Invalid base address for endpoint http://authserver.mydomain.dev/connect/authorize. Valid base addresses: https://authserver.mydomain.dev.",
"details": null,
"data": {},
"validationErrors": null
}
}
This answer still doesn't make sense. We have not touched the Token service since 8/27. But I just made a change to the authserver and redeployed. It's now working again (not throwing a http error and successfully returning a token).
So something else must be happening.