Microservice Startup Template: Database Migrations

Running Sql-Server infrastructural service is required since default connection strings for microservices use Sql-Server running on container.

There are two ways to migrate databases and seed data for microservices. Either one of them or both of them can be used as long as data seeders are synced.

Auto Migration (On-the-fly Migration)

RabbitMq infrastructural service is also required for auto migration along with Sql-Server.

The aspect of auto migration is, each microservice migrating its own database and seeding its own data. To achieve this, each microservice checks pending migration and handles it on its own.

Microservices has DbMigrations folder that contains DatabaseMigrationChecker and DatabaseMigrationEventHandler specific to that microservice. Since this behavior is shared across all the microservices, it has been defined as PendingMigrationsCheckerBase and DatabaseMigrationEventHandlerBase under shared Hosting Microservices project DbMigrations folder.

Microservices runs the DatabaseMigrationChecker at OnPostApplicationInitialization method. If any pending migration exists, new ApplyDatabaseMigrationsEto event is published to DistributedEventBus to queue migration.

public virtual async Task CheckAsync()
{
    ...
    
    var pendingMigrations = await ServiceProvider
        .GetRequiredService<TDbContext>()
        .Database
        .GetPendingMigrationsAsync();

    if (pendingMigrations.Any())
    {
        await DistributedEventBus.PublishAsync(
            new ApplyDatabaseMigrationsEto
            {
                DatabaseName = DatabaseName
            }
        );
    }
    
    ...
}

Each migration handles its own database migration under HandleEventAsync(ApplyDatabaseMigrationsEto eventData) method of DatabaseMigrationEventHandler.

public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
{
    ...
        
    var schemaMigrated = await MigrateDatabaseSchemaAsync(eventData.TenantId);

    if (eventData.TenantId == null && schemaMigrated)
    {
        /* Migrate tenant databases after host migration */
        await QueueTenantMigrationsAsync();
    }

    ...
}

Tenant Migrations are also handled after host database migrations by queueing a new ApplyDatabaseMigrationsEto event passing TenantId argument.

protected virtual async Task QueueTenantMigrationsAsync()
{
    var tenants = await TenantRepository.GetListWithSeparateConnectionStringAsync();
    foreach (var tenant in tenants)
    {
        await DistributedEventBus.PublishAsync(
            new ApplyDatabaseMigrationsEto
            {
                DatabaseName = DatabaseName,
                TenantId = tenant.Id
            }
        );
    }
}

This is handled under MigrateDatabaseSchemaAsync(Guid? tenantId) method of DatabaseMigrationEventHandlerBase.

protected virtual async Task<bool> MigrateDatabaseSchemaAsync(Guid? tenantId)
{
    var result = false;

    using (CurrentTenant.Change(tenantId))
    {        
        ...
    }

    return result;
}

With this feature; newly created tenant with a specified connection string will have it's new database automatically created with all the tables and the initial seed data if available. Tenant management UI also has manually trigger for database creation & migration in case any problem occurs with automatic migration.

apply-database-migrations

If you don't want to use Auto Migration for any microservice, delete the OnPostApplicationInitialization method of the HttpApiHost module and microservice specific DatabaseMigrationChecker file along with related DataSeeder.

Database Migrator

If you want to use only Auto Migration (On-the-fly-Migration), it is safe to delete this project completely.

DbMigrator is a console application and used for database migrations and seeding data. It is located under the shared folder in the microservice template solution. DbMigratorModule depends on SharedHostingModule to use mapped database configurations. Since it will be migrating the databases of microservices; it also depends on each microservice's EntityFrameworkCoreModule and ApplicationContractsModule modules.

DbMigratorService migrates the host first then tenants.

public async Task MigrateAsync(CancellationToken cancellationToken)
{
    await MigrateHostAsync(cancellationToken);
    await MigrateTenantsAsync(cancellationToken);
    _logger.LogInformation("Migration completed!");
}

MigrateHostAsync method migrates all the databases.

private async Task MigrateAllDatabasesAsync(Guid? tenantId, CancellationToken cancellationToken)
{
   ...
       
   if (tenantId == null)
   {
      /* SaaS schema should only be available in the host side */
      await MigrateDatabaseAsync<SaasServiceDbContext>(cancellationToken);
   }

    await MigrateDatabaseAsync<AdministrationServiceDbContext>(cancellationToken);
    await MigrateDatabaseAsync<IdentityServiceDbContext>(cancellationToken);
    await MigrateDatabaseAsync<ProductServiceDbContext>(cancellationToken);

    await uow.CompleteAsync(cancellationToken);
    
	...
}

MigrateTenantsAsync method migrates each tenant based on their connection string

private async Task MigrateTenantsAsync(CancellationToken cancellationToken)
{
    ...

    var tenants = await _tenantRepository.GetListAsync(includeDetails: true, cancellationToken: cancellationToken);
    var migratedDatabaseSchemas = new HashSet<string>();
    foreach (var tenant in tenants)
    {
        using (_currentTenant.Change(tenant.Id))
        {
            // Database schema migration
            var connectionString = tenant.FindDefaultConnectionString();
            if (!connectionString.IsNullOrWhiteSpace() && //tenant has a separate database
                !migratedDatabaseSchemas.Contains(connectionString)) //the database was not migrated yet
            {
                _logger.LogInformation($"Migrating tenant database: {tenant.Name} ({tenant.Id})");
                await MigrateAllDatabasesAsync(tenant.Id, cancellationToken);
                migratedDatabaseSchemas.AddIfNotContains(connectionString);
            }

            //Seed data
            ...
        }
	}
}

If you decide to use both DbMigrator and Auto Migration approaches, you need to keep duplicate dataseeder files; one under your microservice DbMigrations folder for auto migration and other under shared DbMigrator project. You need to keep both of your data seeder files synchonized.

IdentityService Data Seeding

IdentityService uses three different mapped database configurations; IdentityService, AdministrationService and SaasService which are located under appsettings.json file.

IdentityService seeding is required for AuthServer since it seeds the admin user/password (identity data) and initial identity server data (clients, api resources, scopes).

IdentityServer Data Seeding

Both Auto Migration Data Seeding and DbMigrator Data Seeding uses IdentityServerDataSeeder. The seeder uses IdentityServer section in appsettings.json file to seed client and resource cors and redirectUri data.

"IdentityServer": {
    "Clients": {
      "MyProjectName_Web": {
        "RootUrl": "https://localhost:44321/"
      },
      "MyProjectName_Blazor": {
        "RootUrl": "https://localhost:44307/"
      },
      "MyProjectName_BlazorServer": {
        "RootUrl": "https://localhost:44314/"
      },
      "MyProjectName_PublicWeb": {
        "RootUrl": "https://localhost:44335/"
      },
      "MyProjectName_Angular": {
        "RootUrl": "http://localhost:4200"
      },
      "WebGateway": {
        "RootUrl": "https://localhost:44325"
      },
      "PublicWebGateway": {
        "RootUrl": "https://localhost:44353"
      }
    },
    "Resources": {
      "AccountService": {
        "RootUrl": "https://localhost:44322"
      },
      "IdentityService": {
        "RootUrl": "https://localhost:44388"
      },
      "AdministrationService": {
        "RootUrl": "https://localhost:44367"
      },
      "SaasService": {
        "RootUrl": "https://localhost:44381"
      },
      "ProductService": {
        "RootUrl": "https://localhost:44361"
      }
    }
  }

Creating Api Resources

After creating identity resources, api resources are created.

private async Task CreateApiResourcesAsync()
{
    var commonApiUserClaims = new[] { "email", "email_verified", "name", "phone_number", "phone_number_verified", "role" };

    await CreateApiResourceAsync("AccountService", commonApiUserClaims);
    await CreateApiResourceAsync("IdentityService", commonApiUserClaims);
    await CreateApiResourceAsync("AdministrationService", commonApiUserClaims);
    await CreateApiResourceAsync("SaasService", commonApiUserClaims);
    await CreateApiResourceAsync("ProductService", commonApiUserClaims);
}

Creating Api Scopes

Each api resource is defined as a scope.

private async Task CreateApiScopesAsync()
{
    await CreateApiScopeAsync("AccountService");
    await CreateApiScopeAsync("IdentityService");
    await CreateApiScopeAsync("AdministrationService");
    await CreateApiScopeAsync("SaasService");
    await CreateApiScopeAsync("ProductService");
}

Creating Web Gateway Swagger Client

Web Gateway Swagger client is the only swagger client used for all swagger authorization in between microservices and gateways.

private async Task CreateWebGatewaySwaggerClientsAsync()
{
    await CreateSwaggerClientAsync("WebGateway", new[] { "AccountService", "IdentityService", "AdministrationService", "SaasService", "ProductService" });
}

private async Task CreateSwaggerClientAsync(string name, string[] scopes = null)
{
    var commonScopes = new[] { "email", "openid", "profile", "role", "phone", "address" };
    scopes ??= new[] { name };

    // Swagger Client
    var swaggerClientId = $"{name}_Swagger";
    if (!swaggerClientId.IsNullOrWhiteSpace())
    {
        var webGatewaySwaggerRootUrl = _configuration[$"IdentityServer:Clients:{name}:RootUrl"].TrimEnd('/');
        var publicWebGatewayRootUrl = _configuration[$"IdentityServer:Clients:PublicWebGateway:RootUrl"].TrimEnd('/');
        var accountServiceRootUrl = _configuration[$"IdentityServer:Resources:AccountService:RootUrl"].TrimEnd('/');
        var identityServiceRootUrl = _configuration[$"IdentityServer:Resources:IdentityService:RootUrl"].TrimEnd('/');
        var administrationServiceRootUrl = _configuration[$"IdentityServer:Resources:AdministrationService:RootUrl"].TrimEnd('/');
        var saasServiceRootUrl = _configuration[$"IdentityServer:Resources:SaasService:RootUrl"].TrimEnd('/');
        var productServiceRootUrl = _configuration[$"IdentityServer:Resources:ProductService:RootUrl"].TrimEnd('/');

        await CreateClientAsync(
            name: swaggerClientId,
            scopes: commonScopes.Union(scopes),
            grantTypes: new[] { "authorization_code" },
            secret: "1q2w3e*".Sha256(),
            requireClientSecret: false,
            redirectUris: new List<string> {
                $"{webGatewaySwaggerRootUrl}/swagger/oauth2-redirect.html", // WebGateway redirect uri
                $"{publicWebGatewayRootUrl}/swagger/oauth2-redirect.html", // PublicWebGateway redirect uri
                $"{accountServiceRootUrl}/swagger/oauth2-redirect.html", // AccountService redirect uri
                $"{identityServiceRootUrl}/swagger/oauth2-redirect.html", // IdentityService redirect uri
                $"{administrationServiceRootUrl}/swagger/oauth2-redirect.html", // AdministrationService redirect uri
                $"{saasServiceRootUrl}/swagger/oauth2-redirect.html", // SaasService redirect uri
                $"{productServiceRootUrl}/swagger/oauth2-redirect.html", // ProductService redirect uri
            },
            corsOrigins: new[] {
                webGatewaySwaggerRootUrl.RemovePostFix("/"),
                publicWebGatewayRootUrl.RemovePostFix("/"),
                accountServiceRootUrl.RemovePostFix("/"),
                identityServiceRootUrl.RemovePostFix("/"),
                administrationServiceRootUrl.RemovePostFix("/"),
                saasServiceRootUrl.RemovePostFix("/"),
                productServiceRootUrl.RemovePostFix("/")
            }
        );
    }
}

Creating Clients

While public-web and administration service clients are distinct, all the other back-office clients are created by default. Administration service is used to make request to identity service to get user permission data. See administration service for more.

private async Task CreateClientsAsync()
{
    var commonScopes = new[] { "email", "openid", "profile", "role", "phone", "address" };

    //Web Client
    var webClientRootUrl = _configuration["IdentityServer:Clients:MyProjectName_Web:RootUrl"].EnsureEndsWith('/');
    await CreateClientAsync(
        name: "MyProjectName_Web",
        scopes: commonScopes.Union(new[] {
            "AccountService", "IdentityService", "AdministrationService", "SaasService", "ProductService"
        }),
        grantTypes: new[] { "hybrid" },
        secret: "1q2w3e*".Sha256(),
        redirectUris: new List<string> { $"{webClientRootUrl}signin-oidc" },
        postLogoutRedirectUri: $"{webClientRootUrl}signout-callback-oidc",
        frontChannelLogoutUri: $"{webClientRootUrl}Account/FrontChannelLogout",
        corsOrigins: new[] { webClientRootUrl.RemovePostFix("/") }
    );

    //Blazor Client
    var blazorClientRootUrl = _configuration["IdentityServer:Clients:MyProjectName_Blazor:RootUrl"].EnsureEndsWith('/');
    await CreateClientAsync(
        name: "MyProjectName_Blazor",
        scopes: commonScopes.Union(new[] {
            "AccountService", "IdentityService", "AdministrationService", "SaasService", "ProductService"
        }),
        grantTypes: new[] { "authorization_code" },
        secret: "1q2w3e*".Sha256(),
        requireClientSecret: false,
        redirectUris: new List<string> { $"{blazorClientRootUrl}authentication/login-callback" },
        postLogoutRedirectUri: $"{blazorClientRootUrl}authentication/logout-callback",
        corsOrigins: new[] { blazorClientRootUrl.RemovePostFix("/") }
    );

    //Blazor Server Client
    var blazorServerClientRootUrl = _configuration["IdentityServer:Clients:MyProjectName_BlazorServer:RootUrl"].EnsureEndsWith('/');
    await CreateClientAsync(
        name: "MyProjectName_BlazorServer",
        scopes: commonScopes.Union(new[] {
            "AccountService", "IdentityService", "AdministrationService", "SaasService", "ProductService"
        }),
        grantTypes: new[] { "hybrid" },
        secret: "1q2w3e*".Sha256(),
        redirectUris: new List<string> { $"{blazorServerClientRootUrl}signin-oidc" },
        postLogoutRedirectUri: $"{blazorServerClientRootUrl}signout-callback-oidc",
        frontChannelLogoutUri: $"{blazorServerClientRootUrl}Account/FrontChannelLogout",
        corsOrigins: new[] { blazorServerClientRootUrl.RemovePostFix("/") }
    );

    //Public Web Client
    var publicWebClientRootUrl = _configuration["IdentityServer:Clients:MyProjectName_PublicWeb:RootUrl"]
        .EnsureEndsWith('/');
    await CreateClientAsync(
        name: "MyProjectName_PublicWeb",
        scopes: commonScopes.Union(new[] { "AccountService", "AdministrationService", "ProductService" }),
        grantTypes: new[] { "hybrid" },
        secret: "1q2w3e*".Sha256(),
        redirectUris: new List<string> { $"{publicWebClientRootUrl}signin-oidc" },
        postLogoutRedirectUri: $"{publicWebClientRootUrl}signout-callback-oidc",
        frontChannelLogoutUri: $"{publicWebClientRootUrl}Account/FrontChannelLogout",
        corsOrigins: new[] { publicWebClientRootUrl.RemovePostFix("/") }
    );

    //Angular Client
    var angularClientRootUrl = _configuration["IdentityServer:Clients:MyProjectName_Angular:RootUrl"].TrimEnd('/');
    await CreateClientAsync(
        name: "MyProjectName_Angular",
        scopes: commonScopes.Union(new[] {
            "AccountService", "IdentityService", "AdministrationService", "SaasService", "ProductService"
        }),
        grantTypes: new[] { "authorization_code", "LinkLogin", "Impersonation" },
        secret: "1q2w3e*".Sha256(),
        requireClientSecret: false,
        redirectUris: new List<string> { $"{angularClientRootUrl}" },
        postLogoutRedirectUri: $"{angularClientRootUrl}",
        corsOrigins: new[] { angularClientRootUrl }
    );

    //Administration Service Client
    await CreateClientAsync(
        name: "MyProjectName_AdministrationService",
        scopes: commonScopes.Union(new[] { "IdentityService" }),
        grantTypes: new[] { "client_credentials" },
        secret: "1q2w3e*".Sha256(),
        permissions: new[] { IdentityPermissions.Users.Default }
    );
}

If you have created an angular back-office application in your microservice template, you are free to delete blazor server, blazor and web clients.

Auto Migration Data Seeding

IdentityServiceDatabaseMigrationEventHandler is used for seeding language management, identity server and identity data.

private async Task SeedDataAsync(Guid? tenantId, string adminEmail, string adminPassword)
{
    using (CurrentTenant.Change(tenantId))
    {
        if (tenantId == null)
        {
            await _languageManagementDataSeeder.SeedAsync();
            await _identityServerDataSeeder.SeedAsync();
        }

        await _identityDataSeeder.SeedAsync(
            adminEmail,
            adminPassword,
            tenantId
        );
    }
}

Data seeding occurs when database is migrated:

public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
{
    ...
    await SeedDataAsync(
        tenantId: eventData.TenantId,
        adminEmail: IdentityServiceDbProperties.DefaultAdminEmailAddress,
        adminPassword: IdentityServiceDbProperties.DefaultAdminPassword
    );
    ...
}

Or when a tenant is created:

public async Task HandleEventAsync(TenantCreatedEto eventData)
{
    ...
    await SeedDataAsync(
        tenantId: eventData.Id,
        adminEmail: eventData.Properties.GetOrDefault(IdentityDataSeedContributor.AdminEmailPropertyName) ?? IdentityServiceDbProperties.DefaultAdminEmailAddress,
        adminPassword: eventData.Properties.GetOrDefault(IdentityDataSeedContributor.AdminPasswordPropertyName) ?? IdentityServiceDbProperties.DefaultAdminPassword
    );
    ...
}

Or when a tenants connection string is updated:

public async Task HandleEventAsync(TenantConnectionStringUpdatedEto eventData)
{
    ...
    await SeedDataAsync(
        tenantId: eventData.Id,
        adminEmail: IdentityServiceDbProperties.DefaultAdminEmailAddress,
        adminPassword: IdentityServiceDbProperties.DefaultAdminPassword
    );
    ...
}

Changing connection string will cause creation of a new database. This database will be empty if data from old database is not moved to new database. It is left for developer's choice.

DbMigrator Data Seeding

Host data is seeded in MigrateHostAsync method:

private async Task MigrateHostAsync(CancellationToken cancellationToken)
{
    _logger.LogInformation("Migrating Host side...");
    await MigrateAllDatabasesAsync(null, cancellationToken);
    await SeedDataAsync();
}

IdentityServerDataSeedContributor is used for seeding the required IdentityServerDataSeeder and seeding data runs after migrating all the databases:

private async Task SeedDataAsync()
{
    await _dataSeeder.SeedAsync(
        new DataSeedContext(_currentTenant.Id)
            .WithProperty(IdentityDataSeedContributor.AdminEmailPropertyName,
                IdentityServiceDbProperties.DefaultAdminEmailAddress)
            .WithProperty(IdentityDataSeedContributor.AdminPasswordPropertyName,
                IdentityServiceDbProperties.DefaultAdminPassword)
    );
}

Tenant data is seeded after host data is migrated in MigrateTenantsAsync method after tenant database migration is completed:

private async Task MigrateTenantsAsync(CancellationToken cancellationToken)
{
    ...
        using (_currentTenant.Change(tenant.Id))
        {
            ...

            //Seed data
            _logger.LogInformation($"Seeding tenant data: {tenant.Name} ({tenant.Id})");
            await SeedDataAsync();
        }
    ...
}

If you decide to use both DbMigrator and Auto Migration approaches, you need to keep both IdentityServerDataSeeder files located under IdentityService/DbMigrations and DbMigrator projects synchonized.

AdministrationService Data Seeding

AdministrationService uses two different mapped database configurations; AdministrationService and SaasService which are located under appsettings.json file. AdministrationService seeds the initial admin permission data.

Auto Migration Data Seeding

AdministrationServiceDatabaseMigrationEventHandler is used for seeding permissions:

private async Task SeedDataAsync(Guid? tenantId)
{
    using (CurrentTenant.Change(tenantId))
    {
        using (var uow = UnitOfWorkManager.Begin(requiresNew: true, isTransactional: true))
        {
            var multiTenancySide = tenantId == null
                ? MultiTenancySides.Host
                : MultiTenancySides.Tenant;

            var permissionNames = _permissionDefinitionManager
                .GetPermissions()
                .Where(p => p.MultiTenancySide.HasFlag(multiTenancySide))
                .Where(p => !p.Providers.Any() || p.Providers.Contains(RolePermissionValueProvider.ProviderName))
                .Select(p => p.Name)
                .ToArray();

            await _permissionDataSeeder.SeedAsync(
                RolePermissionValueProvider.ProviderName,
                "admin",
                permissionNames,
                tenantId
            );
        
            await uow.CompleteAsync();
        }
    }
}

Data seeding occurs when database is migrated:

public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
{
    ...
    await SeedDataAsync(eventData.TenantId);
	...        
}

Or when a tenant is created:

public async Task HandleEventAsync(TenantCreatedEto eventData)
{
	...
    await SeedDataAsync(eventData.Id);
    ...
}

Or when a tenants connection string is updated:

public async Task HandleEventAsync(TenantConnectionStringUpdatedEto eventData)
{
    ...
    await MigrateDatabaseSchemaAsync(eventData.Id);
    await SeedDataAsync(eventData.Id);
    ...
}

DbMigrator Data Seeding

IDataSeeder is used for seeding the required data.

SaasService Data Seeding

SaasService uses two different mapped database configurations; SaasService and AdministrationService which are located under appsettings.json file. SaasService does not seed any initial data. If an initial saas data is required to be seeded, create a SaasDataSeedContibutor.

SaasService database is only available for the host side.

Auto Migration Data Seeding

SaasServiceDatabaseMigrationEventHandler is used for seeding initial saas data by calling SeedAsync method of newly created SaasDataSeedContibutor after migrating database schema.

public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
{
    ...
    await MigrateDatabaseSchemaAsync(null);
    // Seed data here
    ...
}

SaasServiceDatabaseMigrationEventHandler only handles ApplyDatabaseMigrationsEto by default.

DbMigrator Data Seeding

IDataSeeder is used for seeding the required data and Data seed contributors are automatically discovered by the ABP Framework and executed as a part of the data seed process.

ProductService Data Seeding

ProductService uses three different mapped database configurations; ProductService, AdministrationService and SaasService which are located under appsettings.json file. ProductService does not seed any initial data. If an initial product data is required to be seeded, create a ProductDataSeedContibutor.

Auto Migration Data Seeding

ProductServiceDatabaseMigrationEventHandler is used for seeding initial product data by calling SeedAsync method of newly created ProductDataSeedContibutor after migrating database schema.

public async Task HandleEventAsync(ApplyDatabaseMigrationsEto eventData)
{
    ...
    var schemaMigrated = await MigrateDatabaseSchemaAsync(eventData.TenantId);
    // Seed data here
    ...
}

If the product data needs to be seeded when a tenant is created:

public async Task HandleEventAsync(TenantCreatedEto eventData)
{
    ...
    await MigrateDatabaseSchemaAsync(eventData.Id);
    // Seed data here
    ...
}

Or when a tenants connection string is updated:

public async Task HandleEventAsync(TenantConnectionStringUpdatedEto eventData)
{
    ...
    await MigrateDatabaseSchemaAsync(eventData.Id);
    // Seed data here
    ...
}

DbMigrator Data Seeding

Generic IDataSeeder is used to seed the required data and Data seed contributors are automatically discovered by the ABP Framework and executed as a part of the data seed process.

Contributors

Avatar gterdem Avatar jogoertzen-stantec
Last updated: February 10, 2022 Edit this page on GitHub

Was this page helpful?

Please make a selection.

To help us improve, please share your reason for the negative feedback in the field below.

Please enter a note.

Thank you for your valuable feedback!

Please note that although we cannot respond to feedback, our team will use your comments to improve the experience.

In this document
Community Talks

What’s New with .NET 9 & ABP 9?

21 Nov, 17:00
Online
Watch the Event
Mastering ABP Framework Book
Mastering ABP Framework

This book will help you gain a complete understanding of the framework and modern web application development techniques.

Learn More