ABP Framework 5.0 implemented the strict ordering of outboxes and inboxes for monolithic apps. However, in a microservice or multi-database scenario, distributed events will no longer be linearizability [1] due to the limitations of network latency and infrastructure efficiency. So there will inevitably be physical time disordering events.

Use an illustration from Daniel Wu's article "Messaging Reliability and Ordering" [2] to show you the problem:

If the event being handled has a causality with any other event, there may be problems caused by the two being disordering.

This article will discuss the situations and solutions we may encounter on the event subscriber apps based on the above truth.

Cases

We make the following conventions.

1. We focus on a user score service that subscribes to some distributed events.
2. m1 and m2 are two events that occurred successively.
3. t1 and t2 are when the service receives and handles the events m1 and m2, respectively.
4. t1 < t2 means that t1 is earlier than t2, which is called ordered; t1 > t2 means that t1 is later than t2, which is called disordered.
5. C (configuration) represents the state of the subscriber service. C0 is the initial state, CF is the expected final state, and CW is the wrong final state.

Case 1

• Event m1: event of user A created
• Event m2: event of user B created
• Handler jobs: Create LocalUser entities locally according to m1 and m2, respectively
• Analysis: m1 and m2 are non-causal and ordering insensitive

  • t1 < t2 (ordered):

     

  • t1 > t2 (disordered):

     

We don't need to intervene in it.

Case 2

• Event m1: event of user A created
• Event m2: event of order 1 paid
• Handler jobs: According to m1, create a LocalUser entity locally; according to m2, increase LocalUser.Score
• Analysis: m1 and m2 are causal; m1 and m2 are ordering sensitive, but the "entity not found exception" intercepts the disordering, so handlers are idempotent, which avoids the consistency problem

  • t1 < t2 (ordered):

     

  • t1 > t2 (disordered):

     

We don't need to intervene in it. After m1 is handled, m2 delays retry handling, essentially reaching the order.

Case 3

• Event m1: event of order 1 paid
• Event m2: event of order 1 canceled
• Handler jobs: According to m1, increase LocalUser.Score; according to m2, deduct LocalUser.Score; The score is minimum deducted to 0 and will not be a negative number
• Analysis: m1 and m2 are causal; m1 and m2 are ordering sensitive, and handlers are not idempotent, so there will be a consistency problem

  • t1 < t2 (ordered):

     

  • t1 > t2 (disordered):

     

The user score service creates LocalOrder entities to record the order handling states.

public class LocalOrder : AggregateRoot<Guid>
{
    public bool HasPaidEventHandled { get; set; } // set to true after handling m1
}

If the m2 handler finds OrderCanceledEto.OrderPaidTime != null but LocalOrder.HasPaidEventHandled == false, it throws an exception. After m1 is handled, m2 delays retry handling, essentially reaching the order.

We essentially transformed Case 3 into Case 2, thus achieving idempotency.

After the Change

• t1 < t2 (ordered):

   

• t1 > t2 (disordered):

   

Case 4

• Event m1: event of user A updated (Region changed)
• Event m2: event of order 1 paid
• Handler jobs: According to m1, clear LocalUser.Score if UserEto.Region != LocalUser.Region. According to m2, increase LocalUser.Score
• Analysis: m1 and m2 are causal; m1 and m2 are ordering sensitive, and handlers are not idempotent, so there will be a consistency problem

  • t1 < t2 (ordered):

     

  • t1 > t2 (disordered):

     

We can solve the problem with these changes:

1. Add a new RegionVersion property in the User entity with the default value of 0. It increases by 1 once the user's region is changed.
2. The user score service uses LocalUserRegion.Score to record user scores instead of LocalUser.Score.

   public class LocalUserRegion : AggregateRoot<Guid>
   {
       public Guid UserId { get; set; }
       public string Region { get; set; }
       public int RegionVersion { get; set; }
       public int Score { get; set; }
   }
   

3. When handling m1, if UserEto.RegionVersion is new, create an new LocalUserRegion entity with the initial score of 0, which equals clearing the user's scores once his region is changed.
4. When the user pays, the local service invokes the Identity remote service to query and sets the found UserDto.RegionVersion as OrderPaidEto.UserRegionVersion in the event m2.
5. When handling m2, according to m1, increase the corresponding LocalUserRegion.Score.

We made the handlers idempotent by disentangling the causality of m1 and m2.

After the Change

• t1 < t2 (ordered):

   

• t1 > t2 (disordered):

   

Case 5: ABP Entity Synchronizer

In the DDD practice of the ABP framework, modules use entity synchronizers to redundant data of external entities. A typical case is the BlogUserSynchronizer [3] of the Blogging module. What's unique about this case is that stale events can be skipped handling if not strictly required.

• Event m1: event of User A updated
• Event m2: event of User A updated
• Handler jobs: According to m1 and m2, update the user information in the LocalUser entity.
• Analysis: Once m2 is handled earlier than m1, the stale data will overwrite the latest data, so there will be a consistency problem

  • t1 < t2 (ordered):

     

  • t1 > t2 (disordered):

     

Let's add a new integer property named EntityVersion. Its default value is 0 and increments by one every time the entity changes. When the entity synchronizer gets an EntityUpdatedEto<UserEto> event, skip handling if UserEto.EntityVersion <= LocalUser.EntityVersion is satisfied. That's it, and we solved the problem. I tried implementing the above feature in the ABP framework. See PR #14197 [4].

After the Change

• t1 < t2 (ordered):

   

• t1 > t2 (disordered):

   

Solution Summary

We believe that there are the following ideas for solving event disordering.

1. Try to keep the business logic of the DistributedEventHandler simple to spot potential event disordering problems.
2. In some cases, we can make handlers idempotent by recording the entity's states locally, as Case 3 did above.
3. In some cases, we can disentangle causality by improving the business design, as Case 4 did above.
4. Entity synchronizers can be designed with EntityVersion to avoid synchronizing to stale data.

Conclusion

Even if your app is currently monolithic, you should be concerned about the event disordering. That is a preparation for possible architectural changes in the future. Also, please give up implementing linearizability, as it is impossible in microservices or multi-database scenarios.

In several cases mentioned in this article, it seems that it is not difficult for developers to find the hidden dangers of consistency problems. However, the business is more complex in actual production, and the diverse events will be hard-considered. Even if we find out all possible causalities and deal with them during development, can we ensure that nothing will go wrong when the business changes? The answer is probably no.

There is no silver bullet to the distributed consistency problem. It's always there. Developers can only reduce complexity, dissolve the causality or manually implement idempotency.

References

1. Herlihy, Maurice P.; Wing, Jeannette M. (1987). "Axioms for Concurrent Objects". Proceedings of the 14th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, POPL '87. p. 13
2. Daniel Wu. (2021). "Messaging Reliability and Ordering". https://danielw.cn/messaging-reliability-and-order
3. GitHub abpframework/abp repository. BlogUserSynchronizer.cs. https://github.com/abpframework/abp/blob/1275f2207fc39d850d23472294e456c8504f20d2/modules/blogging/src/Volo.Blogging.Domain/Volo/Blogging/Users/BlogUserSynchronizer.cs
4. GitHub abpframework/abp repository. PR #14197. https://github.com/abpframework/abp/pull/14197

Stepping is a distributed BASE jobs implementation. You can use it as a workflow engine, event outbox/inbox, email/SMS sender, remote invoker, and more.

We have provided documentation for the following languages: English, 简体中文.

What are Job and Step in Stepping?

Job is a distributed transaction unit, and Step is a specific task inside a job.

A job contains one or many steps, and the transaction manager will execute them in order. If step 1 fails, it will be retried until success, and then step 2 starts to execute.

Scenarios for Using Stepping

Want To Execute Steps and Ensure Atomicity

When you start a job, Stepping will eventually complete the steps you require. If the app crashes during the executions, the transaction manager will continue to execute the rest steps after it recovers.

Stepping will complete your steps one by one. If a step fails, it will be tried later until success, which makes the job atomic. Please ensure all your steps can eventually succeed after retrying unless it is a Saga step.

Stepping may already complete the current step when your app crashes during the execution. When your app recovers, Stepping will execute it redundantly. Therefore, all your steps should be idempotent.

Want To Ensure Executing Steps After a DB Transaction Commits

When you start a job with a DB transaction, Stepping will eventually complete the steps you require after the DB transaction commits.

You don't need to worry about the non-atomicity caused by the app crashes after the transaction commits but before the steps' execution. We have handled this case by using the DTM's 2-phase messaging pattern.

Stepping also supports the "multi-tenant with multi-DB" scenario, meaning it works no matter how many different databases there are in your app.

Examples

The transaction manager will eventually complete the added steps:

var job = await distributedJobFactory.CreateJobAsync();

job.AddStep(new RequestBank1TransferOutStep(args)); // step with args
job.AddStep<RequestBank2TransferInStep>(); // step without args

await job.StartAsync();

The Steps document shows how to define a step.

If you want to execute the steps after a DB transaction commits and ensure they will eventually be done:

var db = serviceProvider.GetRequiredService<MyDbContext>(); // example for EF Core
await db.Database.BeginTransactionAsync();

var order = new Order(args);

db.Orders.Add(order);
await db.SaveChangesAsync();

var job = await distributedJobFactory.CreateJobAsync(new EfCoreSteppingDbContext(db));

job.AddStep(new SendOrderCreatedEmailStep(order));
job.AddStep(new SendOrderCreatedSmsStep(order));

await job.StartAsync(); // it will commit the DB transaction

Stepping supports EF Core, ADO.NET(coming soon), and MongoDB.

For details, please see the Usage document.

Installation

See the Installation document.

Supported Transaction Managers

Stepping requires transaction managers. You can choose an implementation you like.

DTM Server

DTM is a mature transaction manager you can use as the TM provider for Stepping. DTM allows you to use many other distributed transaction patterns like Saga, TCC, and XA.

See the DTM document.

Local-TM

Stepping provides a simple built-in TM implementation. The local-TM runs with your app as a local transaction manager. Which app starts a job should be the TM of this job.

See the Local-TM document.

这篇文章分享了使用 DTM 二阶段消息模式解决 issue #10036 的方法。

今天我们要使用 EasyAbp 的 Abp.EventBus.Boxes.Dtm 模块。

DTM 事件箱的介绍

这个模块使用了 DTM 的 二阶段消息 使得 ABP 的事件箱得以支持 多租户多数据库场景。

你需要先阅读 DTM 文档，它将帮助你理解这个模块。

与 ABP 默认事件箱的差异

| | DTM 二阶段消息事件箱 | ABP 5.0+ 默认事件箱 | | :------------------------------: | :------------------: | :----------------------------------------: | | 收发速度 | :heavy_check_mark: | :x: | | 更少的数据传输 | :x: | :heavy_check_mark: | | 保证事件发出
(事务工作单元) | :heavy_check_mark: | :heavy_check_mark: | | 保证事件发出
(非事务工作单元) | :x: | :heavy_check_mark:
(要求消费端解决幂等) | | 避免重复处理(仅限纯DB操作) | :heavy_check_mark: | :heavy_check_mark: | | 支持多租户多数据库 | :heavy_check_mark: | :x: | | 没有增加外部设施 | :x: | :heavy_check_mark: | | 管理面板和报警 | :heavy_check_mark: | :x: |

DTM 发件箱是如何工作的？

假设你正在使用发件箱发布新的事件:

await _distributedEventBus.PublishAsync(eto1, useOutbox: true);
await _distributedEventBus.PublishAsync(eto2, useOutbox: true);  // useOutbox 的默认值即 true

DTM 发件箱会临时存储这些事件。接下来我们看看，在你完成当前工作单元时它会怎么做：

// UnitOfWork.cs 的代码片段
protected override async Task CommitTransactionsAsync()
{
    // 第 1 步：在事务内插入一条记录到 DTM 屏障表，接着发送一条“prepare”请求到 DTM 服务器
    await DtmMessageManager.InsertBarriersAndPrepareAsync(EventBag);

    // 第 2 步: 提交当前 DB 事务
    await base.CommitTransactionsAsync();

    // 第 3 步: 发送一条"submit"请求到 DTM 服务器
    OnCompleted(async () => await DtmMessageManager.SubmitAsync(EventBag));
}

至此，DTM 服务器已经收到了一条“submit”请求。它会调用 app 的 PublishEvents 服务并附上所有事件的数据，后者被调用后会立即发布这些事件到 MQ。

如果你依然对这个模式“如何确保发送”困惑，请查看 DTM 的 二阶段消息文档 以了解更多。

DTM 收件箱是如何工作的？

与 ABP 的默认实现不同，DTM 收件箱从 MQ 收到一个事件后会立即处理（handle）。在所有的 handler 完成他们的工作后，收件箱会沿用当前事务，向 DTM 屏障表插入一条记录。最后它提交了事务并向 MQ 返回 ACK。

所有入箱的事件都拥有一条唯一的 MessageId。拥有相同 MessageId 的事件只会被处理一次，因为我们不能插入 gid (MessageId) 重复的记录到 DTM 屏障表。

正如你注意到的，DTM 服务器没有参与收件箱的工作。🤭

安装和使用

请阅读 https://github.com/EasyAbp/Abp.EventBus.Boxes.Dtm/tree/main#installation.

后记

这样的一个 DTM 事件箱实现并不是完美的解决方案。这里最大的成本是，你需要额外关心 DTM 服务器的可用性。但是如果你遇到了多租户多数据库的场景，它就是现在最佳的选择。

This article shares a new way to solve issue #10036 using the DTM's 2-phase messages pattern.

The module to be used today is EasyAbp's Abp.EventBus.Boxes.Dtm.

Introduction of the DTM Event Boxes Module

This implementation uses DTM's 2-phase messages to support ABP event boxes in the multi-tenant & multi-database scene.

You should see the DTM docs, which helps to understand this module.

Differences From the ABP's Default Event Boxes

| | DTM 2-phase Message Boxes | ABP 5.0+ Default Boxes | |:--------------------------------------------------------------------------:| :-----------------------: | :----------------------------------------------------: | | Speediness | :heavy_check_mark: | :x: | | Less data transfer | :x: | :heavy_check_mark: | | Be guaranteed to publish
(transactional UOW) | :heavy_check_mark: | :heavy_check_mark: | | Be guaranteed to publish
(non-transactional UOW) | :x: | :heavy_check_mark:
(consumers idempotency required) | | Avoid duplicate handling
(with only DB operations) | :heavy_check_mark: | :heavy_check_mark: | | Multi-tenant-database support | :heavy_check_mark: | :x: | | No additional external infrastructure | :x: | :heavy_check_mark: | | Dashboard and Alarm | :heavy_check_mark: | :x: |

How Does the DTM Outbox Work?

You are publishing events using the ABP event outbox:

await _distributedEventBus.PublishAsync(eto1, useOutbox: true);
await _distributedEventBus.PublishAsync(eto2, useOutbox: true);  // The useOutbox is true by default.

The DTM outbox collects them temporarily. Let's see what it will do when you complete the current unit of work:

// Code snippet for UnitOfWork.cs
protected override async Task CommitTransactionsAsync()
{
    // Step 1: inserting a record to the DTM barrier table within the current DB transaction,
    //         and then it sends a "prepare" request to the DTM server.
    await DtmMessageManager.InsertBarriersAndPrepareAsync(EventBag);

    // Step 2: committing the current DB transaction.
    await base.CommitTransactionsAsync();

    // Step 3: sending a "submit" request to the DTM server.
    OnCompleted(async () => await DtmMessageManager.SubmitAsync(EventBag));
}

Now, the DTM server has received a "submit" request. It invokes the app's PublishEvents service with the events' data, and the latter will publish the events to the MQ provider immediately.

If you are still confused about how it is guaranteed to publish, see DTM's 2-phase messages doc for more information.

How Does the DTM Inbox Work?

Unlike ABP's default implementation, the DTM inbox gets an event from MQ and handles it at once. After the handlers finish their work, the inbox inserts a barrier within the current DB transaction. Finally, it commits the transaction and returns ACK to MQ.

All the incoming events have a unique MessageId. Events with the same MessageId only are handled once since we cannot insert a barrier with a duplicate gid (MessageId).

As you may have noticed, the inbox has nothing to do with the DTM Server.🤭

Installation and Usage

Please see https://github.com/EasyAbp/Abp.EventBus.Boxes.Dtm/tree/main#installation.

Postscript

This DTM event-boxes implementation is not a perfect solution. The main cost is that you need to care about the availability of the DTM Server. But if you run into the multi-tenant multi-database scene, it should be the best choice for now.

Introduction

In this article, we will use the AbpHelper GUI to install the EasyAbp.Abp.SettingUi module into your ABP application.

Pre-Requirements

The following tools should be installed on your development machine:

• .NET Core 5.0+
• VS Code or another IDE
• AbpHelper GUI 1.0+

Creating a New Solution

Open the AbpHelper GUI, switch to the "ABP CLI" tab, and create a new app:

Install the SettingUi Modules

Open the new app solution in the AbpHelper GUI and install the modules:

Then we will find that the DependsOns have been added:

Run the App

See https://docs.abp.io/en/abp/latest/Getting-Started-Running-Solution to learn more about how to run an ABP solution.

Run the commands:

cd C:\Temp\TryAbpHelper\src\TryAbpHelper.DbMigrator
dotnet run
cd ..\TryAbpHelper.Web
dotnet run

Log in and try the Setting UI:

Use the default username admin and password 1q2w3E* to log in to the app.

Conclusion

AbpHelper is helpful for us to manage the application's modules.

We can use it to install not only Volosoft's modules but also the community modules.

The module management feature of the AbpHelper depends on EasyAbp's module library, welcome to register your developed modules through PR.

In the previous article, we have completed the basic features of the address book application.

In this article, we will install the PrivateMessaging module to our address book application, so that users can send private messages to each other and get new message notifications on the website.

In the design of the private messaging module, users can send private messages to themselves, but we will override the design to prohibit users from sending private messages to themselves at the end of this article.

What Is a Module in ABP Framework?

Using the ABP framework, you can make some modules in advance, such as external login implementation, private messaging, file upload, blog, forum, and other modules, and package them for later use. When developing a specific application, you can easily install modules into your application, saving a lot of repetitive work.

In addition to making your own modules, you can also install modules maintained by the open-source community on NuGet. Of course, the community is also waiting for your contribution.

Install the Private Messaging Module

The module (EasyAbp.PrivateMessaging) we use is developed and maintained by the EasyAbp organization, and you can even use it for free in commercial applications.

Now we will start to install the module step by step.

Step 1: Install the Module's NuGet Packages

1. Install the NuGet package EasyAbp.PrivateMessaging.Application to the AddressBook.Application project.
2. Install the NuGet package EasyAbp.PrivateMessaging.Application.Contracts to the AddressBook.Application.Contracts project.
3. Install the NuGet package EasyAbp.PrivateMessaging.Domain to the AddressBook.Domain project.
4. Install the NuGet package EasyAbp.PrivateMessaging.Domain.Shared to the AddressBook.Domain.Shared project.
5. Install the NuGet package EasyAbp.PrivateMessaging.EntityFrameworkCore to the AddressBook.EntityFrameworkCore project.
6. Install the NuGet package EasyAbp.PrivateMessaging.HttpApi into the AddressBook.HttpApi project.
7. Install the NuGet package EasyAbp.PrivateMessaging.HttpApi.Client into the AddressBook.HttpApi.Client project.
8. Install the NuGet package EasyAbp.PrivateMessaging.Web into the AddressBook.Web project (you can skip this step if you do not need the UI).

Step 2: Add Module Dependencies and Configurations

1. Add the dependency of the module to the module class of the above projects, for example, we add the attribute [DependsOn(PrivateMessagingApplicationModule)] to the class in the AddressBookApplicationModule.cs of the AddressBook.Application project, and so on.

2. Find the OnModelCreating method in AddressBookMigrationsDbContext.cs of the AddressBook.EntityFrameworkCore.DbMigrations project and add the code builder.ConfigurePrivateMessaging(); to it to configure the EF Core migration of the module.

Step 3: Create EF Core Migration and Update the Database

1. Run the command dotnet ef migrations add Installed_Pm_Module -s ../AddressBook.DbMigrator in the directory of AddressBook.EntityFrameworkCore.DbMigrations project.

2. Run the AddressBook.DbMigrator project, it will automatically update the database.

If you want to learn more about this step, you can read the ABP official documentation.

Step 4: Run the Application

Now run the AddressBook.Web project, we can see that the private messaging module has been installed successfully.

Improvement: Prohibit Users From Sending Private Messages to Themselves

The ABP framework allows us to override the code in the module. Please create a new file MyPrivateMessageAppService.cs in the AddressBook.Application project:

[Dependency(ReplaceServices = true)]
public class MyPrivateMessageAppService : PrivateMessageAppService
{
    // ctor

    public override Task<PrivateMessageDto> CreateAsync(CreateUpdatePrivateMessageDto input)
    {
        if (input.ToUserName == CurrentUser.UserName)
        {
            throw new UserFriendlyException("Don't send messages to yourself");
        }
        
        return base.CreateAsync(input);
    }
}

If a user sends a message to himself, he will see the error message:

Thanks to the extensible modular design of the ABP framework, static files (such as JS and CSS files) and pages can be overridden. Please read the ABP official document for more information: Customizing the Existing Modules

Postscript

If you want to learn more about the private messaging module, please read its documentation.

By the way, the EasyAbp organization has many powerful and useful modules, such as EShop, PaymentService, GiftCardManagement, which can improve your development efficiency and enhance the reliability of applications. If you are interested, please read EasyAbp Guide to learn more.

I look forward to the launch of ABP's official module market (market.abp.io) in the future, providing community modules list, search, and automatic installation features, making it easier for us to install modules.

Next

In the next article, we will upgrade the address book application to a multi-tenant SaaS application with a minor change.

ABP vNext framework (hereinafter called "ABP") is an artwork-level application development framework designed by Volosoft. It is based on Domain-Driven-Design (DDD) and innovatively adopts modular development design. ABP is undoubtedly the most advanced and elegant existence in ASP.NET Core development frameworks.

I believe that with the excellent modular development design and smooth development experience, ABP's position in ASP.NET Core is expected to reach the position of Spring in Java.

Module Development and Application Development

Using the ABP framework, you can make some modules in advance, such as external login implementation, private messaging, file upload, blog, forum, and other modules, and package them for later use. When developing a specific application, you can easily install modules into your application, saving a lot of repetitive work.

In addition to making your own modules, you can also install modules maintained by the open-source community on NuGet. Of course, the community is also waiting for your contribution.

Start to Develop the Address Book Application

Today we will not discuss module development, but start with the simplest application development. I will follow the ABP best practices and show you how to develop an address book application using Abp vNext framework in 5 minutes.

Step 1: Use ABP CLI to Generate the Application Solution

1. Use the command line to install ABP CLI: dotnet tool install -g Volo.Abp.Cli.

2. Use the command line to generate a solution: abp new AddressBook (the solution will be generated in the current directory).

Step 2: Create a "Contact" Entity

For ABP, the contact entity should be an AggregateRoot. For details, please refer to the introduction of Domain-Driven-Design (DDD) in the ABP official document.

1. Create a new directory: aspnet-core/src/AddressBook.Domain/Contacts.

2. Manually create the Contact.cs file in the above directory.

   public class Contact : AggregateRoot<Guid>
   {
       public virtual string Name { get; protected set; }
   
       public virtual string PhoneNumber { get; protected set; }
   
       public virtual string Address { get; protected set; }
   
       public virtual byte? Age { get; protected set; }
   
       public virtual DateTime? Birthday { get; protected set; }
   
       // The constructors will be generated later, you don’t need to add them manually here.
   }
   

3. Run the AddressBook.DbMigrator project, which is to establish the basic structure and data in the database for our application.

Step 3: Generate the Remaining Code

This article uses EasyAbp's open-source AbpHelper GUI to generate code. If you are an ABP commercial user, you can also choose ABP Suite.

1. Download AbpHelper GUI: https://github.com/EasyAbp/AbpHelper.GUI/releases

2. Use the CRUD Code Generator to generate all codes related to the Contact entity.

   

If you are using the AbpHelper GUI for the first time, please install the AbpHelper CLI through the Install or update AbpHelper CLI on the left navigation menu. You can also directly use AbpHelper CLI to complete the above work.

Step 4: Run the Application

1. Run the AddressBook.Web project.

   

2. Log in and explore your application (the default password of the admin user is 1q2w3E*).

   

   You must have noticed that the form has been automatically generated by the abp-dynamic-form tag helper. Moreover, you only need to modify the localization file to customize the words, but I will not demonstrate it here.

3. Contact entity's RESTful API has also been automatically generated, if you need them, visit the route /swagger.

Postscript

Our address book application naturally includes identity management, multi-tenant support, if you plan to learn the ABP framework systematically, please read the official document.

The AbpHelper used in this article is a tool produced by the EasyAbp organization, which can significantly improve your development efficiency and is completely free. In addition, EasyAbp also provides a lot of useful modules, you can read EasyAbp Guide to learn more.

Next

In the next article, I will introduce how to install the PrivateMessaging module to our address book application. This module is also developed and maintained by the EasyAbp organization, and you can even use it for free in commercial applications.