Hi again,
One more thing. This might be a kubernetes issue. I noticed our AKS cluster was at 75% memory utilization so I deleted all of the applications and then re-deployed them.
The member-config-service is running without error.
Could there be a memory leak in the application? And could a lack of available memory be causing this error and just not catching it so that it shows that error message?
Hi,
I sent you an invite to our microservices shared repository and member-config-service repositories.
The classes you requested are in the shared/CprOnAbp.Shared.Hosting.Microservices/DbMigrations/MongoDb folder.
Please let me know if you need anything else.
Thanks
Another piece of information. I applied the same docker image to our production AKS cluster and it's running fine.
So it's clearly an issue running in AKS. Could there be something in cache or some other AKS condition that's causing the microservice error?
Hi,
I looked at the GitHub issue and it doesn't seem to apply. I'm not having an issue with seeding the database and I can't find RuntimeDatabaseMigrator anywhere.
Here are the 3 classes
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Driver;
using Volo.Abp;
using Volo.Abp.Data;
using Volo.Abp.DependencyInjection;
using Volo.Abp.Domain.Entities.Events.Distributed;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.MongoDB;
using Volo.Abp.MultiTenancy;
using Volo.Abp.Uow;
using Volo.Saas.Tenants;
namespace CprOnAbp.Shared.Hosting.Microservices.DbMigrations.MongoDb;
public abstract class MongoDBMigrationEventHandlerBase<TDbContext> :
ITransientDependency
where TDbContext : AbpMongoDbContext, IAbpMongoDbContext
{
protected const string TryCountPropertyName = "TryCount";
protected const int MaxEventTryCount = 3;
protected ICurrentTenant CurrentTenant { get; }
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected ITenantStore TenantStore { get; }
protected ITenantRepository TenantRepository { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected ILogger<MongoDBMigrationEventHandlerBase<TDbContext>> Logger { get; set; }
protected IServiceProvider ServiceProvider { get; }
protected string DatabaseName { get; }
protected MongoDBMigrationEventHandlerBase(
ICurrentTenant currentTenant,
IUnitOfWorkManager unitOfWorkManager,
ITenantStore tenantStore,
ITenantRepository tenantRepository,
IDistributedEventBus distributedEventBus,
string databaseName, IServiceProvider serviceProvider)
{
CurrentTenant = currentTenant;
UnitOfWorkManager = unitOfWorkManager;
TenantStore = tenantStore;
DatabaseName = databaseName;
ServiceProvider = serviceProvider;
DistributedEventBus = distributedEventBus;
TenantRepository = tenantRepository;
Logger = NullLogger<MongoDBMigrationEventHandlerBase<TDbContext>>.Instance;
}
/// <summary>
/// Apply pending EF Core schema migrations to the database.
/// Returns true if any migration has applied.
/// </summary>
protected virtual async Task<bool> MigrateDatabaseSchemaAsync(Guid? tenantId)
{
var result = false;
using (CurrentTenant.Change(tenantId))
{
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
async Task<bool> MigrateDatabaseSchemaWithDbContextAsync()
{
var dbContexts = ServiceProvider.GetServices<IAbpMongoDbContext>();
var connectionStringResolver = ServiceProvider.GetRequiredService<IConnectionStringResolver>();
if (ServiceProvider.GetRequiredService<ICurrentTenant>().IsAvailable)
{
dbContexts = dbContexts.Where(x =>
!x.GetType().IsDefined(typeof(IgnoreMultiTenancyAttribute)));
}
foreach (var dbContext in dbContexts)
{
var connectionString =
await connectionStringResolver.ResolveAsync(
ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType()));
if (connectionString.IsNullOrWhiteSpace())
{
continue;
}
var mongoUrl = new MongoUrl(connectionString);
var databaseName = mongoUrl.DatabaseName;
var client = new MongoClient(mongoUrl);
if (databaseName.IsNullOrWhiteSpace())
{
databaseName = ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType());
}
(dbContext as AbpMongoDbContext)?.InitializeCollections(client.GetDatabase(databaseName));
}
return true;
}
if (tenantId == null)
{
//Migrating the host database
result = await MigrateDatabaseSchemaWithDbContextAsync();
}
else
{
var tenantConfiguration = await TenantStore.FindAsync(tenantId.Value);
if (!tenantConfiguration.ConnectionStrings.Default.IsNullOrWhiteSpace() ||
!tenantConfiguration.ConnectionStrings.GetOrDefault(DatabaseName).IsNullOrWhiteSpace())
{
//Migrating the tenant database (only if tenant has a separate database)
result = await MigrateDatabaseSchemaWithDbContextAsync();
}
}
await uow.CompleteAsync();
}
}
return result;
}
protected virtual async Task HandleErrorOnApplyDatabaseMigrationAsync(
ApplyDatabaseMigrationsEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not apply database migrations. Re-queueing the operation. TenantId = {eventData.TenantId}, Database Name = {eventData.DatabaseName}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not apply database migrations. Canceling the operation. TenantId = {eventData.TenantId}, DatabaseName = {eventData.DatabaseName}.");
Logger.LogException(exception);
}
}
protected virtual async Task HandleErrorTenantCreatedAsync(
TenantCreatedEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not perform tenant created event. Re-queueing the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not perform tenant created event. Canceling the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception);
}
}
protected virtual async Task HandleErrorTenantConnectionStringUpdatedAsync(
TenantConnectionStringUpdatedEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not perform tenant connection string updated event. Re-queueing the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not perform tenant connection string updated event. Canceling the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception);
}
}
protected virtual async Task QueueTenantMigrationsAsync()
{
var tenants = await TenantRepository.GetListWithSeparateConnectionStringAsync();
foreach (var tenant in tenants)
{
await DistributedEventBus.PublishAsync(
new ApplyDatabaseMigrationsEto
{
DatabaseName = DatabaseName,
TenantId = tenant.Id
}
);
}
}
private static int GetEventTryCount(EtoBase eventData)
{
var tryCountAsString = eventData.Properties.GetOrDefault(TryCountPropertyName);
if (tryCountAsString.IsNullOrEmpty())
{
return 0;
}
return int.Parse(tryCountAsString);
}
private static void SetEventTryCount(EtoBase eventData, int count)
{
eventData.Properties[TryCountPropertyName] = count.ToString();
}
private static int IncrementEventTryCount(EtoBase eventData)
{
var count = GetEventTryCount(eventData) + 1;
SetEventTryCount(eventData, count);
return count;
}
}
public abstract class PendingMongoDbMigrationsChecker<TDbContext> : PendingMigrationsCheckerBase where TDbContext : AbpMongoDbContext
{
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected IServiceProvider ServiceProvider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected IAbpDistributedLock DistributedLockProvider { get; }
protected string DatabaseName { get; }
protected IDataSeeder DataSeeder { get; }
protected PendingMongoDbMigrationsChecker(
ILoggerFactory loggerFactory,
IUnitOfWorkManager unitOfWorkManager,
IServiceProvider serviceProvider,
ICurrentTenant currentTenant,
IDistributedEventBus distributedEventBus,
IAbpDistributedLock abpDistributedLock,
string databaseName,
IDataSeeder dataSeeder) : base(loggerFactory)
{
UnitOfWorkManager = unitOfWorkManager;
ServiceProvider = serviceProvider;
CurrentTenant = currentTenant;
DistributedEventBus = distributedEventBus;
DistributedLockProvider = abpDistributedLock;
DatabaseName = databaseName;
DataSeeder = dataSeeder;
}
public virtual async Task CheckAndApplyDatabaseMigrationsAsync()
{
await TryAsync(async () =>
{
using (CurrentTenant.Change(null))
{
// Create database tables if needed
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
await MigrateDatabaseSchemaAsync();
await DataSeeder.SeedAsync();
await uow.CompleteAsync();
}
}
});
}
/// <summary>
/// Apply scheme update for MongoDB Database.
/// </summary>
protected virtual async Task<bool> MigrateDatabaseSchemaAsync()
{
var result = false;
await using (var handle = await DistributedLockProvider.TryAcquireAsync("Migration_" + DatabaseName))
{
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
Log.Information($"Lock is acquired for db migration and seeding on database named: {DatabaseName}...");
if (handle is null)
{
Log.Information($"Handle is null because of the locking for : {DatabaseName}");
return false;
}
async Task<bool> MigrateDatabaseSchemaWithDbContextAsync()
{
var dbContexts = ServiceProvider.GetServices<IAbpMongoDbContext>();
var connectionStringResolver = ServiceProvider.GetRequiredService<IConnectionStringResolver>();
foreach (var dbContext in dbContexts)
{
var connectionString =
await connectionStringResolver.ResolveAsync(
ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType()));
if (connectionString.IsNullOrWhiteSpace())
{
continue;
}
var mongoUrl = new MongoUrl(connectionString);
var databaseName = mongoUrl.DatabaseName;
var client = new MongoClient(mongoUrl);
if (databaseName.IsNullOrWhiteSpace())
{
databaseName = ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType());
}
(dbContext as AbpMongoDbContext)?.InitializeCollections(client.GetDatabase(databaseName));
}
return true;
}
//Migrating the host database
result = await MigrateDatabaseSchemaWithDbContextAsync();
await uow.CompleteAsync();
}
return result;
}
}
}
using System;
using System.Threading.Tasks;
using Volo.Abp.Data;
using Volo.Abp.DependencyInjection;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.MongoDB;
using Volo.Abp.MultiTenancy;
using Volo.Abp.Uow;
namespace CprOnAbp.Shared.Hosting.Microservices.DbMigrations.MongoDb;
public class PendingMongoDBMigrationsCheckerBase<TDbContext> : ITransientDependency
where TDbContext : AbpMongoDbContext
{
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected IServiceProvider ServiceProvider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected string DatabaseName { get; }
protected PendingMongoDBMigrationsCheckerBase(
IUnitOfWorkManager unitOfWorkManager,
IServiceProvider serviceProvider,
ICurrentTenant currentTenant,
IDistributedEventBus distributedEventBus,
string databaseName)
{
UnitOfWorkManager = unitOfWorkManager;
ServiceProvider = serviceProvider;
CurrentTenant = currentTenant;
DistributedEventBus = distributedEventBus;
DatabaseName = databaseName;
}
public virtual async Task CheckAsync()
{
using (CurrentTenant.Change(null))
{
// Create database tables if needed
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
await DistributedEventBus.PublishAsync(
new ApplyDatabaseMigrationsEto
{
DatabaseName = DatabaseName
}
);
await uow.CompleteAsync();
}
}
}
}
I made some changes to the MemberConfigService, tested the service on localhost successfully but when I deploy to Kubernetes I get the following error that's just running in an infinite loop until I kill the pod. I deployed the previous image that was running in in AKS and it too is getting he same error.
An exception was thrown while activating CprOnAbp.MemberConfigService.DbMigrations.MemberConfigServiceDatabaseMigrationEventHandler.
Autofac.Core.DependencyResolutionException: An exception was thrown while activating CprOnAbp.MemberConfigService.DbMigrations.MemberConfigServiceDatabaseMigrationEventHandler.
---> Autofac.Core.DependencyResolutionException: None of the constructors found on type 'CprOnAbp.MemberConfigService.DbMigrations.MemberConfigServiceDatabaseMigrationEventHandler' can be invoked with the available services and parameters:
Cannot resolve parameter 'System.String databaseName' of constructor 'Void .ctor(Volo.Abp.MultiTenancy.ICurrentTenant, Volo.Abp.Uow.IUnitOfWorkManager, Volo.Abp.MultiTenancy.ITenantStore, Volo.Saas.Tenants.ITenantRepository, Volo.Abp.EventBus.Distributed.IDistributedEventBus, System.String, System.IServiceProvider)'.
See https://autofac.rtfd.io/help/no-constructors-bindable for more info.
at Autofac.Core.Activators.Reflection.ReflectionActivator.<>c__DisplayClass14_0.<UseSingleConstructorActivation>b__0(ResolveRequestContext ctxt, Action`1 next)
at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.DisposalTrackingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Builder.RegistrationBuilder`3.<>c__DisplayClass41_0.<PropertiesAutowired>b__0(ResolveRequestContext ctxt, Action`1 next)
at Autofac.Core.Resolving.Middleware.DelegateMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
--- End of inner exception stack trace ---
at Autofac.Core.Resolving.Middleware.ActivatorErrorHandlingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.RegistrationPipelineInvokeMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.SharingMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.Middleware.CircularDependencyDetectorMiddleware.Execute(ResolveRequestContext context, Action`1 next)
at Autofac.Core.Resolving.Pipeline.ResolvePipelineBuilder.<>c__DisplayClass14_0.<BuildPipeline>b__1(ResolveRequestContext ctxt)
at Autofac.Core.Pipeline.ResolvePipeline.Invoke(ResolveRequestContext ctxt)
at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, ResolveRequest request)
at Autofac.Core.Resolving.ResolveOperation.ExecuteOperation(ResolveRequest request)
at Autofac.Core.Resolving.ResolveOperation.Execute(ResolveRequest request)
at Autofac.Core.Lifetime.LifetimeScope.ResolveComponent(ResolveRequest request)
at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType, IEnumerable`1 parameters)
at Autofac.ResolutionExtensions.Resolve(IComponentContext context, Type serviceType)
at Autofac.Extensions.DependencyInjection.AutofacServiceProvider.GetRequiredService(Type serviceType)
at Volo.Abp.EventBus.IocEventHandlerFactory.GetHandler()
at Volo.Abp.EventBus.EventBusBase.TriggerHandlerAsync(IEventHandlerFactory asyncHandlerFactory, Type eventType, Object eventData, List`1 exceptions, InboxConfig inboxConfig)
at Volo.Abp.EventBus.EventBusBase.TriggerHandlersAsync(Type eventType, Object eventData, List`1 exceptions, InboxConfig inboxConfig)
at Volo.Abp.EventBus.EventBusBase.TriggerHandlersAsync(Type eventType, Object eventData)
at Volo.Abp.EventBus.RabbitMq.RabbitMqDistributedEventBus.ProcessEventAsync(IModel channel, BasicDeliverEventArgs ea)
at Volo.Abp.RabbitMQ.RabbitMqMessageConsumer.HandleIncomingMessageAsync(Object sender, BasicDeliverEventArgs basicDeliverEventArgs)
[20:56:07 ERR] ---------- Exception Data ----------
ActivatorChain = CprOnAbp.MemberConfigService.DbMigrations.MemberConfigServiceDatabaseMigrationEventHandler
I can give whoever access to our GitHub repository if necessary.
Please see closed quest #7304 which is now closed and locked.
We applied the suggested code changes to the ClientServiceHttpApiHostModule class in the ClientService.HttpApi.Host project but the IP address in the audit logs is still showing the IP address of the web gateway.
This is pretty important to us as a SaaS provider for audit and training purposes.
Thank you.
If you read this after I deleted the answer please disregard.
I determined that the message was in fact not being scheduled in the future by inspecting the message properties in ASB.
We do have the app running in a production environment but it has a different problem which is what drove me to look at the logs in the test environment.
In the production environment we get a 502 Nginx error after logging in with the authserver. There's nothing in any of the logs indicating why that's happening.
Hi,
I updated the program.cs in the web-gateway and in the HttpApi.Host program.cs in one of the microservices. The log is still showing the ip address of the web-gateway pod.
This is the code in the HttpApi.Host program.cs
\`\*\*using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.AspNetCore.HttpOverrides;
using CprOnAbp.Shared.Hosting.AspNetCore;
using Serilog;
namespace CprOnAbp.ClientService;
public class Program
{
public async static Task Main(string[] args)
{
var assemblyName = typeof(Program).Assembly.GetName().Name;
SerilogConfigurationHelper.Configure(assemblyName);
try
{
Log.Information($"Starting {assemblyName}.");
var builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders =
ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
});
builder.Host
.AddAppSettingsSecretsJson()
.UseAutofac()
.UseSerilog();
await builder.AddApplicationAsync<ClientServiceHttpApiHostModule>();
var app = builder.Build();
app.UseForwardedHeaders();
await app.InitializeApplicationAsync();
await app.RunAsync();
return 0;
}
catch (Exception ex)
{
Log.Fatal(ex, $"{assemblyName} terminated unexpectedly!");
return 1;
}
finally
{
Log.CloseAndFlush();
}
}
}
\*\*\`
Hi,
I've provide you read access to the repository.
https://github.com/cpr-ops/web-app