Distributed Event Bus Kafka Integration

This document explains how to configure the Kafka as the distributed event bus provider. See the distributed event bus document to learn how to use the distributed event bus system

Installation

Use the ABP CLI to add Volo.Abp.EventBus.Kafka NuGet package to your project:

  • Install the ABP CLI if you haven't installed before.
  • Open a command line (terminal) in the directory of the .csproj file you want to add the Volo.Abp.EventBus.Kafka package.
  • Run abp add-package Volo.Abp.EventBus.Kafka command.

If you want to do it manually, install the Volo.Abp.EventBus.Kafka NuGet package to your project and add [DependsOn(typeof(AbpEventBusKafkaModule))] to the ABP module class inside your project.

Configuration

You can configure using the standard configuration system, like using the appsettings.json file, or using the options classes.

appsettings.json file configuration

This is the simplest way to configure the Kafka settings. It is also very strong since you can use any other configuration source (like environment variables) that is supported by the AspNet Core.

Example: The minimal configuration to connect to a local kafka server with default configurations

{
  "Kafka": {
    "Connections": {
      "Default": {
        "BootstrapServers": "localhost:9092"
      }
    },
    "EventBus": {
      "GroupId": "MyGroupId",
      "TopicName": "MyTopicName"
    }
  }
}
  • MyGroupId is the name of this application, which is used as the GroupId on the Kakfa.
  • MyTopicName is the topic name.

See the Kafka document to understand these options better.

Connections

If you need to connect to another server than the localhost, you need to configure the connection properties.

Example: Specify the host name (as an IP address)

{
  "Kafka": {
    "Connections": {
      "Default": {
        "BootstrapServers": "123.123.123.123:9092"
      }
    },
    "EventBus": {
      "GroupId": "MyGroupId",
      "TopicName": "MyTopicName"
    }
  }
}

Defining multiple connections is allowed. In this case, you can specify the connection that is used for the event bus.

Example: Declare two connections and use one of them for the event bus

{
  "Kafka": {
    "Connections": {
      "Default": {
        "BootstrapServers": "123.123.123.123:9092"
      },
      "SecondConnection": {
        "BootstrapServers": "321.321.321.321:9092"
      }
    },
    "EventBus": {
      "GroupId": "MyGroupId",
      "TopicName": "MyTopicName",
      "ConnectionName": "SecondConnection"
    }
  }
}

This allows you to use multiple Kafka cluster in your application, but select one of them for the event bus.

You can use any of the ClientConfig properties as the connection properties.

Example: Specify the socket timeout

{
  "Kafka": {
    "Connections": {
      "Default": {
        "BootstrapServers": "123.123.123.123:9092",
        "SocketTimeoutMs": 60000
      }
    }
  }
}

The Options Classes

AbpRabbitMqOptions and AbpRabbitMqEventBusOptions classes can be used to configure the connection strings and event bus options for the RabbitMQ.

You can configure this options inside the ConfigureServices of your module.

Example: Configure the connection

Configure<AbpKafkaOptions>(options =>
{
    options.Connections.Default.BootstrapServers = "123.123.123.123:9092";
    options.Connections.Default.SaslUsername = "user";
    options.Connections.Default.SaslPassword = "pwd";
});

Example: Configure the consumer config

Configure<AbpKafkaOptions>(options =>
{
    options.ConfigureConsumer = config =>
    {
        config.GroupId = "MyGroupId";
        config.EnableAutoCommit = false;
    };
});

Example: Configure the producer config

Configure<AbpKafkaOptions>(options =>
{
    options.ConfigureProducer = config =>
    {
        config.MessageTimeoutMs = 6000;
        config.Acks = Acks.All;
    };
});

Example: Configure the topic specification

Configure<AbpKafkaOptions>(options =>
{
    options.ConfigureTopic = specification =>
    {
        specification.ReplicationFactor = 3;
        specification.NumPartitions = 3;
    };
});

Using these options classes can be combined with the appsettings.json way. Configuring an option property in the code overrides the value in the configuration file.

Contributors


Last updated: October 02, 2020 Edit this page on GitHub

Was this page helpful?

Please make a selection.

To help us improve, please share your reason for the negative feedback in the field below.

Please enter a note.

Thank you for your valuable feedback!

Please note that although we cannot respond to feedback, our team will use your comments to improve the experience.

In this document
Community Talks

Building Modular Monolith Applications Using .NET and ABP Framework

17 Oct, 17:00
Online
Watch the Event
Mastering ABP Framework Book
Mastering ABP Framework

This book will help you gain a complete understanding of the framework and modern web application development techniques.

Learn More