Activities of "stevefisher"

Thanks for this. I've fed this back to the team and it appears we already have a 25 second timeout configured.

 context.Services.AddHttpClient<EzekiaCRM.IClient, EzekiaCRM.Client>(client =>
        {
            var apiKey = configuration.GetValue<string>("EzekiaConfiguration:ApiKey");
            var apiUrl = configuration.GetSection("EzekiaConfiguration:BaseApiUrl").Value;
 
            client.BaseAddress = new Uri(apiUrl);
            client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", apiKey);
            client.Timeout = TimeSpan.FromSeconds(CompanyConsts.ExternalRequestTimeoutInSec);
        }

We don't have the call wrapped in CancellationTokenSource inside the job itself so we could add that but thought it was worth mentioning in case you had any other ideas or that changed anything.

Hi,

Sorry for the somewhat delayed response. The issue has resolved itself now, but it would be good to try and get an idea of what happened and how we can look to prevent the issue in the future. The only things I did during the day on the 28th was to set the IsAbandoned flag on the last jobs that had been retrying before it stopped working and restarted the application. On restart I noticed high CPU usage of the app, but nothing changed. It then seems to have jumped back into life during the early hours on 29th.

In response to your questions:

1. We are using Volo.Abp.BackgroundJobs
2. Single instance in Azure App Service - we had a change deployed on the 15/16th but everything had been functioning after that.
3. No IsJobExecutionEnabled, custom JobExecutionTimeout, DefaultTimeout haven't been customised
4. I’ve included a sample of some of the jobs that were hung prior to me changing the IsAbandoned flag in the linked investigation files (the flag is set in the results I’m sending)
5. Application logs from the point we saw the last mention of Volo.Abp.BackgroundJobs are included in the linked investigation files - after that there is nothing and if I look at the log for 23rd there is nothing at all despite there still being non abandoned jobs in the queue.
6. I’m not sure this one is going to help us now the jobs are processing again. A lock sounds plausible though. We don’t use Redis so everything would be in the SQL database. Is there any locking the application does itself or would it just be a case of SQL locks?

Investigation Files: https://1drv.ms/u/c/f973c8721a5019ff/IQA66z9bj6KkTL0VtwF_LD1yAYMFJUIF6V4roYH347pEPc4?e=NVo0RM Password: DELETED

Hopefully this answers everything but please let me know if I can provide anything else.

Thanks.

We seem to have an issue since last week when our background jobs have stopped processing. Around the 22nd we can see jobs that had previously failed and were having retries performed suddenly stopped retrying. Also new jobs are getting created in the queue but they never get processed.

Our last new deployment of our application was on the 15th, so it feels like that's unrelated to the current issue.

I'm unsure what information would be useful to help diagnose this issue. So please let me know if there's anything I can provide and I'm happy to get that sent over and added to the ticket.