Hi again,
One more thing. This might be a kubernetes issue. I noticed our AKS cluster was at 75% memory utilization so I deleted all of the applications and then re-deployed them.
The member-config-service is running without error.
Could there be a memory leak in the application? And could a lack of available memory be causing this error and just not catching it so that it shows that error message?
Hi,
I sent you an invite to our microservices shared repository and member-config-service repositories.
The classes you requested are in the shared/CprOnAbp.Shared.Hosting.Microservices/DbMigrations/MongoDb folder.
Please let me know if you need anything else.
Thanks
Another piece of information. I applied the same docker image to our production AKS cluster and it's running fine.
So it's clearly an issue running in AKS. Could there be something in cache or some other AKS condition that's causing the microservice error?
Hi,
I looked at the GitHub issue and it doesn't seem to apply. I'm not having an issue with seeding the database and I can't find RuntimeDatabaseMigrator anywhere.
Here are the 3 classes
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Driver;
using Volo.Abp;
using Volo.Abp.Data;
using Volo.Abp.DependencyInjection;
using Volo.Abp.Domain.Entities.Events.Distributed;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.MongoDB;
using Volo.Abp.MultiTenancy;
using Volo.Abp.Uow;
using Volo.Saas.Tenants;
namespace CprOnAbp.Shared.Hosting.Microservices.DbMigrations.MongoDb;
public abstract class MongoDBMigrationEventHandlerBase<TDbContext> :
ITransientDependency
where TDbContext : AbpMongoDbContext, IAbpMongoDbContext
{
protected const string TryCountPropertyName = "TryCount";
protected const int MaxEventTryCount = 3;
protected ICurrentTenant CurrentTenant { get; }
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected ITenantStore TenantStore { get; }
protected ITenantRepository TenantRepository { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected ILogger<MongoDBMigrationEventHandlerBase<TDbContext>> Logger { get; set; }
protected IServiceProvider ServiceProvider { get; }
protected string DatabaseName { get; }
protected MongoDBMigrationEventHandlerBase(
ICurrentTenant currentTenant,
IUnitOfWorkManager unitOfWorkManager,
ITenantStore tenantStore,
ITenantRepository tenantRepository,
IDistributedEventBus distributedEventBus,
string databaseName, IServiceProvider serviceProvider)
{
CurrentTenant = currentTenant;
UnitOfWorkManager = unitOfWorkManager;
TenantStore = tenantStore;
DatabaseName = databaseName;
ServiceProvider = serviceProvider;
DistributedEventBus = distributedEventBus;
TenantRepository = tenantRepository;
Logger = NullLogger<MongoDBMigrationEventHandlerBase<TDbContext>>.Instance;
}
/// <summary>
/// Apply pending EF Core schema migrations to the database.
/// Returns true if any migration has applied.
/// </summary>
protected virtual async Task<bool> MigrateDatabaseSchemaAsync(Guid? tenantId)
{
var result = false;
using (CurrentTenant.Change(tenantId))
{
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
async Task<bool> MigrateDatabaseSchemaWithDbContextAsync()
{
var dbContexts = ServiceProvider.GetServices<IAbpMongoDbContext>();
var connectionStringResolver = ServiceProvider.GetRequiredService<IConnectionStringResolver>();
if (ServiceProvider.GetRequiredService<ICurrentTenant>().IsAvailable)
{
dbContexts = dbContexts.Where(x =>
!x.GetType().IsDefined(typeof(IgnoreMultiTenancyAttribute)));
}
foreach (var dbContext in dbContexts)
{
var connectionString =
await connectionStringResolver.ResolveAsync(
ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType()));
if (connectionString.IsNullOrWhiteSpace())
{
continue;
}
var mongoUrl = new MongoUrl(connectionString);
var databaseName = mongoUrl.DatabaseName;
var client = new MongoClient(mongoUrl);
if (databaseName.IsNullOrWhiteSpace())
{
databaseName = ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType());
}
(dbContext as AbpMongoDbContext)?.InitializeCollections(client.GetDatabase(databaseName));
}
return true;
}
if (tenantId == null)
{
//Migrating the host database
result = await MigrateDatabaseSchemaWithDbContextAsync();
}
else
{
var tenantConfiguration = await TenantStore.FindAsync(tenantId.Value);
if (!tenantConfiguration.ConnectionStrings.Default.IsNullOrWhiteSpace() ||
!tenantConfiguration.ConnectionStrings.GetOrDefault(DatabaseName).IsNullOrWhiteSpace())
{
//Migrating the tenant database (only if tenant has a separate database)
result = await MigrateDatabaseSchemaWithDbContextAsync();
}
}
await uow.CompleteAsync();
}
}
return result;
}
protected virtual async Task HandleErrorOnApplyDatabaseMigrationAsync(
ApplyDatabaseMigrationsEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not apply database migrations. Re-queueing the operation. TenantId = {eventData.TenantId}, Database Name = {eventData.DatabaseName}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not apply database migrations. Canceling the operation. TenantId = {eventData.TenantId}, DatabaseName = {eventData.DatabaseName}.");
Logger.LogException(exception);
}
}
protected virtual async Task HandleErrorTenantCreatedAsync(
TenantCreatedEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not perform tenant created event. Re-queueing the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not perform tenant created event. Canceling the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception);
}
}
protected virtual async Task HandleErrorTenantConnectionStringUpdatedAsync(
TenantConnectionStringUpdatedEto eventData,
Exception exception)
{
var tryCount = IncrementEventTryCount(eventData);
if (tryCount <= MaxEventTryCount)
{
Logger.LogWarning(
$"Could not perform tenant connection string updated event. Re-queueing the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception, LogLevel.Warning);
await Task.Delay(RandomHelper.GetRandom(5000, 15000));
await DistributedEventBus.PublishAsync(eventData);
}
else
{
Logger.LogError(
$"Could not perform tenant connection string updated event. Canceling the operation. TenantId = {eventData.Id}, TenantName = {eventData.Name}.");
Logger.LogException(exception);
}
}
protected virtual async Task QueueTenantMigrationsAsync()
{
var tenants = await TenantRepository.GetListWithSeparateConnectionStringAsync();
foreach (var tenant in tenants)
{
await DistributedEventBus.PublishAsync(
new ApplyDatabaseMigrationsEto
{
DatabaseName = DatabaseName,
TenantId = tenant.Id
}
);
}
}
private static int GetEventTryCount(EtoBase eventData)
{
var tryCountAsString = eventData.Properties.GetOrDefault(TryCountPropertyName);
if (tryCountAsString.IsNullOrEmpty())
{
return 0;
}
return int.Parse(tryCountAsString);
}
private static void SetEventTryCount(EtoBase eventData, int count)
{
eventData.Properties[TryCountPropertyName] = count.ToString();
}
private static int IncrementEventTryCount(EtoBase eventData)
{
var count = GetEventTryCount(eventData) + 1;
SetEventTryCount(eventData, count);
return count;
}
}
public abstract class PendingMongoDbMigrationsChecker<TDbContext> : PendingMigrationsCheckerBase where TDbContext : AbpMongoDbContext
{
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected IServiceProvider ServiceProvider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected IAbpDistributedLock DistributedLockProvider { get; }
protected string DatabaseName { get; }
protected IDataSeeder DataSeeder { get; }
protected PendingMongoDbMigrationsChecker(
ILoggerFactory loggerFactory,
IUnitOfWorkManager unitOfWorkManager,
IServiceProvider serviceProvider,
ICurrentTenant currentTenant,
IDistributedEventBus distributedEventBus,
IAbpDistributedLock abpDistributedLock,
string databaseName,
IDataSeeder dataSeeder) : base(loggerFactory)
{
UnitOfWorkManager = unitOfWorkManager;
ServiceProvider = serviceProvider;
CurrentTenant = currentTenant;
DistributedEventBus = distributedEventBus;
DistributedLockProvider = abpDistributedLock;
DatabaseName = databaseName;
DataSeeder = dataSeeder;
}
public virtual async Task CheckAndApplyDatabaseMigrationsAsync()
{
await TryAsync(async () =>
{
using (CurrentTenant.Change(null))
{
// Create database tables if needed
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
await MigrateDatabaseSchemaAsync();
await DataSeeder.SeedAsync();
await uow.CompleteAsync();
}
}
});
}
/// <summary>
/// Apply scheme update for MongoDB Database.
/// </summary>
protected virtual async Task<bool> MigrateDatabaseSchemaAsync()
{
var result = false;
await using (var handle = await DistributedLockProvider.TryAcquireAsync("Migration_" + DatabaseName))
{
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
Log.Information($"Lock is acquired for db migration and seeding on database named: {DatabaseName}...");
if (handle is null)
{
Log.Information($"Handle is null because of the locking for : {DatabaseName}");
return false;
}
async Task<bool> MigrateDatabaseSchemaWithDbContextAsync()
{
var dbContexts = ServiceProvider.GetServices<IAbpMongoDbContext>();
var connectionStringResolver = ServiceProvider.GetRequiredService<IConnectionStringResolver>();
foreach (var dbContext in dbContexts)
{
var connectionString =
await connectionStringResolver.ResolveAsync(
ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType()));
if (connectionString.IsNullOrWhiteSpace())
{
continue;
}
var mongoUrl = new MongoUrl(connectionString);
var databaseName = mongoUrl.DatabaseName;
var client = new MongoClient(mongoUrl);
if (databaseName.IsNullOrWhiteSpace())
{
databaseName = ConnectionStringNameAttribute.GetConnStringName(dbContext.GetType());
}
(dbContext as AbpMongoDbContext)?.InitializeCollections(client.GetDatabase(databaseName));
}
return true;
}
//Migrating the host database
result = await MigrateDatabaseSchemaWithDbContextAsync();
await uow.CompleteAsync();
}
return result;
}
}
}
using System;
using System.Threading.Tasks;
using Volo.Abp.Data;
using Volo.Abp.DependencyInjection;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.MongoDB;
using Volo.Abp.MultiTenancy;
using Volo.Abp.Uow;
namespace CprOnAbp.Shared.Hosting.Microservices.DbMigrations.MongoDb;
public class PendingMongoDBMigrationsCheckerBase<TDbContext> : ITransientDependency
where TDbContext : AbpMongoDbContext
{
protected IUnitOfWorkManager UnitOfWorkManager { get; }
protected IServiceProvider ServiceProvider { get; }
protected ICurrentTenant CurrentTenant { get; }
protected IDistributedEventBus DistributedEventBus { get; }
protected string DatabaseName { get; }
protected PendingMongoDBMigrationsCheckerBase(
IUnitOfWorkManager unitOfWorkManager,
IServiceProvider serviceProvider,
ICurrentTenant currentTenant,
IDistributedEventBus distributedEventBus,
string databaseName)
{
UnitOfWorkManager = unitOfWorkManager;
ServiceProvider = serviceProvider;
CurrentTenant = currentTenant;
DistributedEventBus = distributedEventBus;
DatabaseName = databaseName;
}
public virtual async Task CheckAsync()
{
using (CurrentTenant.Change(null))
{
// Create database tables if needed
using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: false))
{
await DistributedEventBus.PublishAsync(
new ApplyDatabaseMigrationsEto
{
DatabaseName = DatabaseName
}
);
await uow.CompleteAsync();
}
}
}
}
If you read this after I deleted the answer please disregard.
I determined that the message was in fact not being scheduled in the future by inspecting the message properties in ASB.
We do have the app running in a production environment but it has a different problem which is what drove me to look at the logs in the test environment.
In the production environment we get a 502 Nginx error after logging in with the authserver. There's nothing in any of the logs indicating why that's happening.
Hi,
I updated the program.cs in the web-gateway and in the HttpApi.Host program.cs in one of the microservices. The log is still showing the ip address of the web-gateway pod.
This is the code in the HttpApi.Host program.cs
\`\*\*using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.AspNetCore.HttpOverrides;
using CprOnAbp.Shared.Hosting.AspNetCore;
using Serilog;
namespace CprOnAbp.ClientService;
public class Program
{
public async static Task Main(string[] args)
{
var assemblyName = typeof(Program).Assembly.GetName().Name;
SerilogConfigurationHelper.Configure(assemblyName);
try
{
Log.Information($"Starting {assemblyName}.");
var builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<ForwardedHeadersOptions>(options =>
{
options.ForwardedHeaders =
ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
});
builder.Host
.AddAppSettingsSecretsJson()
.UseAutofac()
.UseSerilog();
await builder.AddApplicationAsync<ClientServiceHttpApiHostModule>();
var app = builder.Build();
app.UseForwardedHeaders();
await app.InitializeApplicationAsync();
await app.RunAsync();
return 0;
}
catch (Exception ex)
{
Log.Fatal(ex, $"{assemblyName} terminated unexpectedly!");
return 1;
}
finally
{
Log.CloseAndFlush();
}
}
}
\*\*\`
Hi,
I've provide you read access to the repository.
https://github.com/cpr-ops/web-app
Since this is C# code it's obvious in goes in a .net application. Is that in the front-end application, the ocelot gateway application or the microservice.
If it's in the front end application that won't work for us since we're using Vue for the front-end and using microservices for the backend.
The MVC app is the one that's created with your Microservice Template and we haven't modified it.
The application is running in AKS. I updated the original question with the output from the pod log. The error just repeats infinitely until I delete the application