Hangfire Hangs And Not Processing Time To Time. #8770

cangunaydin created one month ago

ABP Framework version: v9
UI Type: Angular
Database System: EF Core ( PostgreSQL)
Tiered (for MVC) or Auth Server Separated (for Angular): yes

Hello my question is about how to implement hangfire. I am using hangfire in my abp project. But i am having some weird behavior. To make the things simpler i can post some basic code for my module and hangfire implementation.

[DependsOn(
    typeof(AbpAspNetCoreMvcUiMultiTenancyModule),
    typeof(AbpAspNetCoreAuthenticationOpenIdConnectModule),
    typeof(PopManagementModule),
    typeof(AbpBlobStoringAzureModule),
    typeof(AbpBackgroundJobsHangfireModule),
    typeof(AbpBackgroundWorkersHangfireModule),
    typeof(AbpAspNetCoreMvcUiLeptonXThemeModule),
    typeof(DoohlinkSharedHostingMicroservicesModule)
)]
public class PopManagementHttpApiHostModule:AbpModule
{
    //etc...

and here is how i configure hangfire.

private void ConfigureHangfire(ServiceConfigurationContext context, IConfiguration configuration)
    {
        context.Services.AddHangfire(config =>
        {
            var options = new PostgreSqlStorageOptions
            {
                DistributedLockTimeout = TimeSpan.FromMinutes(1)
            };

            config.UsePostgreSqlStorage(postgresSqlStorageConfig =>
            {
                postgresSqlStorageConfig.UseNpgsqlConnection(configuration.GetConnectionString("PopManagement"));
            }, options);
        
        });
    }

and here is my project structure (part of my project)

so when i run the project everythings works fine. But the weird part is i don't see any background jobs enqueued in my hangfire dashboard with parameters.

what i mean by that is i have this background job.

public class PopJob : AsyncBackgroundJob<PopJobBto>, ITransientDependency
{
    private readonly IPopRepository _popRepository;
    private readonly IBlobContainer<PopCsvContainer> _popCsvContainer;
    private readonly IGuidGenerator _guidGenerator;

    public PopJob(IPopRepository popRepository, IBlobContainer<PopCsvContainer> popCsvContainer,
        IGuidGenerator guidGenerator)
    {
        _popRepository = popRepository;
        _popCsvContainer = popCsvContainer;
        _guidGenerator = guidGenerator;
    }

    public override async Task ExecuteAsync(PopJobBto args)
    {
        var file = await _popCsvContainer.GetOrNullAsync(args.FileName);
        if (file == null)
        {
            return;
        }

        var csvData = System.Text.Encoding.UTF8.GetString(GetBytesFromStream(file));
        var items= ConvertFromCsv(csvData);
        await _popRepository.InsertBulkAsync(items);
    }

public sealed record PopJobBto(string FileName);

public class PopAppService : PopManagementAppService, IPopAppService
{
    private readonly IBlobContainer<PopCsvContainer> _blobContainer;
    private readonly IClock _clock;
    private readonly IBackgroundJobManager _backgroundJobManager;
    private readonly IDistributedCache<PopCacheItem, PopCacheItemKey> _distributedCache;
    private readonly IPopRepository _popRepository;
    private readonly IScreenRepository _screenRepository;
    private readonly IFileRepository _fileRepository;

    public PopAppService(IBlobContainer<PopCsvContainer> blobContainer, IClock clock,
        IBackgroundJobManager backgroundJobManager, IDistributedCache<PopCacheItem, PopCacheItemKey> distributedCache,
        IPopRepository popRepository, IScreenRepository screenRepository, IFileRepository fileRepository)
    {
        _blobContainer = blobContainer;
        _clock = clock;
        _backgroundJobManager = backgroundJobManager;
        _distributedCache = distributedCache;
        _popRepository = popRepository;
        _screenRepository = screenRepository;
        _fileRepository = fileRepository;
    }
 public async Task CreateAsync(CreatePopDto input)
    {
        //create csv file from the input and send it to azure blob storage
        var csvData = ConvertToCsv(input.Items);
        var csvBytes = GetCsvBytes(csvData);
        // Generate the file name using the first item's DeviceId and the current date and time
        if (!input.Items.Any())
        {
            return;
        }

        var deviceId = input.Items.First().DeviceId;
        var dateTime = _clock.Now.ToString("yyyyMMdd_HHmmss");
        var fileName = $"{deviceId}_{dateTime}.csv";
        await _blobContainer.SaveAsync(fileName, csvBytes);
        await _backgroundJobManager.EnqueueAsync(new PopJobBto(fileName));
    }

so in hangfire i am expecting to see the file name and how it is serialized. But all i see is background workers.

I believe it still uses default fifo implementation cause i can see that it hits background job. But sometimes it hangs and never fire the background job, could be because of the distributed lock i suppose. What am i missing over here. Can you point me to the right direction?

berkansasmaz created one month ago

Support Team .NET Developer

Hello, our relevant team member will respond to you as soon as possible.

liangshiwei created one month ago

Support Team Fullstack Developer

Hi,

so when i run the project everythings works fine. But the weird part is i don't see any background jobs enqueued in my hangfire dashboard with parameters.

Try using the UseAbpHangfireDashboard method and BackgroundJobNameAttribute to define job names.

[BackgroundJobName("MyJobName")]
public class EmailSendingJob
    : AsyncBackgroundJob<EmailSendingArgs>, ITransientDependency
{
    private readonly IEmailSender _emailSender;

    public EmailSendingJob(IEmailSender emailSender)
    {
        _emailSender = emailSender;
    }

    public override async Task ExecuteAsync(EmailSendingArgs args)
    {
        await _emailSender.SendAsync(
            args.EmailAddress,
            args.Subject,
            args.Body
        );
    }
}

liangshiwei created one month ago

Support Team Fullstack Developer

I believe it still uses default fifo implementation cause i can see that it hits background job. But sometimes it hangs and never fire the background job, could be because of the distributed lock i suppose. What am i missing over here. Can you point me to the right direction?

Can you provide the full steps or a simple project to reproduce the problem? I will check it.

cangunaydin created one month ago

Hello @liangshiwei, i figured it out. But i have some questions. The problem was: I had AbpBackgroundJobsEntityFrameworkCoreModule,AbpBackgroundJobsDomainModule and AbpBackgroundJobsDomainSharedModule in my app efcore, domain and domainshared module respectively. I have removed them instead i used

    typeof(AbpBackgroundJobsModule),
    typeof(AbpBackgroundJobsHangfireModule),
    typeof(AbpBackgroundWorkersHangfireModule),

in my httpapi.host project. This fixed the problem however i didn't understand which module is implementing the DefaultBgJob implementation. Can you give me a hint about it probably it is efcoremodule since it should be the infra project. And why AbpBackgroundJobsHangfireModule didn't override it i didn't understand. But now it works in this order.

The second thing i want to ask is about HangfirePeriodicBackgroundWorkerAdapter<BackgroundJobWorker>.DoWorkAsync why is this work triggered every 5 secs? I believe that's the default value. How can that value be changed? Also is this worker are dispatching BackgroundJobs? What is the purpose of this worker can you elaborate? Is it possible to disable it and delete it?

liangshiwei created one month ago

Support Team Fullstack Developer

in my httpapi.host project. This fixed the problem however i didn't understand which module is implementing the DefaultBgJob implementation. Can you give me a hint about it probably it is efcoremodule since it should be the infra project. And why AbpBackgroundJobsHangfireModule didn't override it i didn't understand. But now it works in this order.

They are different implementations. After installing the Hangfire provider, you need to remove other implementations.

The second thing i want to ask is about HangfirePeriodicBackgroundWorkerAdapter<BackgroundJobWorker>.DoWorkAsync why is this work triggered every 5 secs? I believe that's the default value. How can that value be changed? Also is this worker are dispatching BackgroundJobs? What is the purpose of this worker can you elaborate? Is it possible to disable it and delete it?

Please remove AbpBackgroundJobsModule and delete the job in the hangfire dashboard.

cangunaydin created one month ago

Ok. thanks. can you tell me if i have AsyncBackgroundWorkers, Is abp use HangfirePeriodicBackgroundWorkerAdapter to execute them? and which module is implementing BackgroundJobWorker? There are bunch of different nuget packages from Abp.

AbpBackgroundJobsEfCoreModule AbpBackgroundJobsDomainModule AbpBackgroundJobsDomainSharedModule AbpBackgroundJobsModule

only module i am sure to use is AbpBackgroundJobsAbstractions :)