Activities of "maliming"

Hi,

Glad to hear it all went through cleanly. Feel free to open a fresh ticket if anything else pops up.

Thanks.

Hi,

Thanks for confirming — that's actually really useful information, because it tells us the wedge wasn't the EzekiaCRM HTTP call itself. Looking again at the log around the same window, there's a lot more going on that day:

• 14:13 — Azure Blob Storage RetriableStream.ReadAsync cancelled (a profile-picture call)
• 14:57 — IdentityUserManager.GetByIdAsync cancelled
• 19:13 — EF Core SaveChanges cancelled mid-ConsumeResultSetAsync
• 12:57 → 17:35 — repeated String or binary data would be truncated ... AbpSessions.IpAddresses errors

So Azure SQL and Azure Blob were both flaky for a long stretch on the 22nd. Since HttpClient.Timeout only protects the EzekiaCRM call, any other await inside the job (EF Core SaveChangesAsync, a blob read, a SQL connection open, even a downstream IdentityUserManager lookup) has no per-job timeout — BackgroundJobWorker only passes the application's StoppingToken, which never fires. That's almost certainly what kept the worker pinned: one of the other awaits in SendVacancyDataAsync / ApplicantDataSendingJob got stuck on Azure SQL or Blob, not on Ezekia.

What I'd add now

You can do all of this on your side — no ABP changes needed.

1) Wrap the job body in a CancellationTokenSource. Yes, please add it — this is the only catch-all that bounds everything inside the job, not just the HTTP call. ABP's IAsyncBackgroundJob<TArgs>.ExecuteAsync doesn't take a CancellationToken parameter, so you create one yourself:

public class VacancyDataSendingJob : AsyncBackgroundJob<VacancySendingArgs>, ITransientDependency
{
    private readonly ExternalCompanyService _service;

    public override async Task ExecuteAsync(VacancySendingArgs args)
    {
        using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(2));
        await _service.SendVacancyDataAsync(args.vacancyId, cts.Token);
    }
}

Make sure SendVacancyDataAsync actually flows the token down to the EzekiaCRM call and to any SaveChangesAsync / blob read it does. Whatever you pick (we usually suggest 1–2 min) just needs to be longer than a healthy job and shorter than "forever."

2) Set an EF Core CommandTimeout. This protects you against Azure SQL hangs specifically. In your DbContextOptions configuration:

options.UseSqlServer(connectionString, sql =>
{
    sql.CommandTimeout(60);
});

3) Separately, you'll want to fix the AbpSessions.IpAddresses truncation. It's unrelated to the jobs but it was throwing all afternoon on the 22nd and is masking other things in your logs. The column is nvarchar(64) by default and your Cloudflare-fronted requests pile up multiple X-Forwarded-For IPs that exceed it. Either widen the column with a custom EF migration, or override the session-recording behaviour to keep only the leftmost client IP.

With (1) in place you'll never see this exact lockup again — even if Azure SQL or Blob has a bad day, the worst case is the job throws after 2 minutes, gets retried later, and the worker keeps moving.

On our side, we'll look at adding a built-in DefaultJobTimeout option to AbpBackgroundJobOptions so future versions wrap each job in a timeout automatically — but you don't need to wait for that, the snippet above is enough.

Thanks

Hi,

Yes, you should follow the new template and route /api/account-admin/{**catch-all} to the AuthServer cluster, not Identity. Starting from ABP 8.0, the account-admin endpoints (login/auth related settings such as two-factor, recaptcha, external providers, idle timeout, passkey) were moved from IdentityService to AuthServer as the official architecture, and the gateway templates were updated accordingly when we switched from Ocelot to YARP.

So your gateway route should look like this (this is exactly what the latest microservice template ships):

"AccountAdmin": {
  "ClusterId": "AuthServer",
  "Match": {
    "Path": "/api/account-admin/{**catch-all}"
  }
}

On the AuthServer backend side, please double-check two places:

1. MyCompanyName.MyProjectName.AuthServer.csproj references both Account Admin packages (use the same version as the rest of your ABP packages):

<PackageReference Include="Volo.Abp.Account.Pro.Admin.HttpApi" Version="10.x.x" />
<PackageReference Include="Volo.Abp.Account.Pro.Admin.Application" Version="10.x.x" />

2. MyProjectNameAuthServerModule.cs declares the dependency:

[DependsOn(
    // ... existing modules
    typeof(AbpAccountPublicWebOpenIddictModule),
    typeof(AbpAccountPublicApplicationModule),
    typeof(AbpAccountPublicHttpApiModule),
    typeof(AbpAccountAdminApplicationModule),
    typeof(AbpAccountAdminHttpApiModule),
    // ...
)]
public class MyProjectNameAuthServerModule : AbpModule
{
}

If you upgraded with the ABP CLI those usually get applied for you, but for an 8 → 10 upgrade where the AuthServer project may have started life on an older template, it's worth verifying. Once those two pieces are in place, no other backend code change is required — the controllers (AccountSettingsController, route prefix api/account-admin/settings) come from Volo.Abp.Account.Pro.Admin.HttpApi and will be exposed by AuthServer automatically.

After applying the change, you can confirm by hitting AuthServer's swagger directly and looking for the /api/account-admin/... endpoints there.

Thanks

Hi,

Thanks for the logs and CSV — they made the picture pretty clear. Here's what happened:

The default BackgroundJobWorker in ABP processes jobs sequentially in a single foreach, and the whole polling cycle is wrapped in a distributed lock (AbpBackgroundJobWorker). When you don't use Redis, that lock lives in process memory (LocalAbpDistributedLock). The implication: if any single job gets stuck on an await (e.g. an HTTP call that never returns), the worker thread holds the lock forever, every subsequent 5-second polling cycle silently fails to acquire it, and no further jobs are processed and no log lines are produced — exactly what you saw after 12:19:14 on April 22.

In your case, VacancyDataSendingJob / ApplicantDataSendingJob call EzekiaCRM. The default HttpClient.Timeout is 100 seconds, but it only applies to the response headers — once a response starts trickling back slowly (or a TCP socket gets half-open), the call can hang indefinitely. Most likely Ezekia had a hiccup that day, one of those calls never returned, the worker thread parked there, and the lock stayed held. Restarting the app on the 28th cleared the in-memory lock, but the same jobs immediately ran into the same external behavior, which is why CPU went up but the queue still didn't drain. By early on the 29th the upstream connection finally died and the worker resumed.

The other IdentitySessionCleanupBackgroundWorker / TokenCleanupBackgroundWorker entries you can see throughout the log confirm the worker host itself was alive — only the background job worker was wedged.

Fix — give the EzekiaCRM HttpClient an explicit timeout

This is the smallest change that actually prevents the deadlock. Wherever you register the Ezekia client, set a hard Timeout:

context.Services.AddHttpClient<EzekiaCRM.Client>(client =>
{
    client.BaseAddress = new Uri("https://api.ezekiacrm.com/");
    client.Timeout = TimeSpan.FromSeconds(30);
});

Pick a value that's comfortably above your normal Ezekia response time but well under "forever" (30–60s is usually fine). Once that's in place, the worst case is the job throws TaskCanceledException, the executor catches it as a normal failure, increments TryCount, and moves on — the worker is never wedged again.

If you also want belt-and-braces, you can wrap the call in a CancellationTokenSource inside the job itself, but the HttpClient.Timeout alone is enough to prevent the symptom you ran into.

We'll also look into adding a per-job timeout in the default worker on our side so this kind of upstream hang can't silently freeze the whole queue.

Thanks

Hi,

This is the intended behavior, not a bug. The change comes from PR #24775 - Prevent Privilege Escalation: Add Assignment Restrictions for Roles and Permissions, which closes a real privilege escalation hole: a non-admin operator with IdentityPermissions.Users.Update could otherwise assign any role (including admin) to other users, or to themselves.

The new rule in IdentityUserAppService.UpdateRolesAsync is:

• If the current operator is in the admin role (HasAdminRoleAsync() returns true), the input is used as-is.
• Otherwise, the operator can only grant roles they themselves already have. Roles the target user already has but the operator does not have are preserved (so the operator cannot strip a higher-privileged role either).

The "is operator an admin?" check intentionally uses the well-known AbpRoleConsts.AdminRoleName ("admin") because that's the role the framework itself seeds and treats as superuser everywhere (data seeding, permission management, etc.). It's a const, and changing it to a settable property is something we'd rather not do - it would break binary compatibility (a const is inlined at compile time into every consumer), and IdentityDataSeeder does const string adminRoleName = AbpRoleConsts.AdminRoleName;, which only compiles because the right-hand side is a compile-time constant. So we don't recommend renaming the framework's admin role - it's much safer to keep an admin role in your DB even if your business uses different role names elsewhere.

That said, if you really need a different role name to act as the "superuser" for this check, you don't need to override UpdateRolesAsync. We left a smaller, more precise extension point: HasAdminRoleAsync() is protected virtual. Just override that one method in your *.Application project:

using System.Threading.Tasks;
using Microsoft.AspNetCore.Identity;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Options;
using Volo.Abp.Authorization.Permissions;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.Identity;
using Volo.Abp.Identity.Emailing;
using Volo.Abp.Identity.UserInvitations;
using Volo.Abp.Identity.UserSharing;
using Volo.Abp.Threading;

namespace YourProject.Identity;

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IIdentityUserAppService), typeof(IdentityUserAppService))]
public class MyIdentityUserAppService : IdentityUserAppService
{
    public MyIdentityUserAppService(
        IdentityUserManager userManager,
        IIdentityUserRepository userRepository,
        IIdentityRoleRepository roleRepository,
        IOrganizationUnitRepository organizationUnitRepository,
        IIdentityClaimTypeRepository identityClaimTypeRepository,
        IdentityProTwoFactorManager identityProTwoFactorManager,
        IOptions<IdentityOptions> identityOptions,
        IDistributedEventBus distributedEventBus,
        IOptions<AbpIdentityOptions> abpIdentityOptions,
        IPermissionChecker permissionChecker,
        IDistributedCache<IdentityUserDownloadTokenCacheItem, string> downloadTokenCache,
        IDistributedCache<ImportInvalidUsersCacheItem, string> importInvalidUsersCache,
        IdentitySessionManager identitySessionManager,
        IdentityUserTwoFactorChecker identityUserTwoFactorChecker,
        ICancellationTokenProvider cancellationTokenProvider,
        UserSharingManager userSharingManager,
        UserInvitationManager userInvitationManager,
        IIdentityUserInvitationRepository userInvitationRepository,
        IIdentityEmailSender identityEmailSender)
        : base(
            userManager,
            userRepository,
            roleRepository,
            organizationUnitRepository,
            identityClaimTypeRepository,
            identityProTwoFactorManager,
            identityOptions,
            distributedEventBus,
            abpIdentityOptions,
            permissionChecker,
            downloadTokenCache,
            importInvalidUsersCache,
            identitySessionManager,
            identityUserTwoFactorChecker,
            cancellationTokenProvider,
            userSharingManager,
            userInvitationManager,
            userInvitationRepository,
            identityEmailSender)
    {
    }

    protected override Task<bool> HasAdminRoleAsync()
    {
        return Task.FromResult(CurrentUser.IsInRole("YourCustomAdminRoleName"));
    }
}

I verified locally on Volo.Abp.Identity.Pro.Application.Tests against ABP 10.3 - the override is picked up correctly, an operator with the custom admin role claim can assign any role (including ones they don't have themselves), and all the existing role-filtering / unmanageable-role-preservation tests keep passing.

Thanks.

Great 👍

Great!

Hi,

Confirmed — SwitchTenant is registered with the OpenIddict server but its handler is missing from the ABP token-grant dispatcher, that's why you see "not implemented".

Please add the following to your AuthServer module's ConfigureServices as a workaround:

using Volo.Abp.Account.Web.ExtensionGrants;
using Volo.Abp.OpenIddict.ExtensionGrantTypes;

public override void ConfigureServices(ServiceConfigurationContext context)
{
    // ... existing code ...

    Configure<AbpOpenIddictExtensionGrantsOptions>(options =>
    {
        options.Grants[SwitchTenantExtensionGrant.ExtensionGrantName] = new SwitchTenantExtensionGrant();
    });
}

After this, the SwitchTenant flow will dispatch correctly. We'll ship a permanent fix on our side in the next ABP release.

Thanks

Hi,

This is by design, not a bug. Adding a NuGet <PackageReference> is not the same as installing a module on the solution from ABP Suite's perspective — Suite only auto-loads entities from modules that were registered via "Install module on solution".

For custom modules, Suite provides an official extension point: the .suite/module-entity-extension.json file at the root of your module solution. Add the Domain DLL file names of the ABP modules you want to expose as navigation property targets. Example:

{
  "Modules": [
    { "DomainProjectDllFileName": "Volo.Abp.Identity.Pro.Domain.dll" }
  ]
}

Then in the Add Navigation Property dialog, check the Include entities from ABP modules checkbox — IdentityUser (and other entities from the listed DLLs) will appear in the dropdown. Suite will automatically build the solution before scanning, so you don't need to build manually, but the DLL has to physically exist somewhere under <solutionRoot>/src/**/bin/... for Assembly.LoadFrom to pick it up. If your module solution has no host project that copies NuGet dependencies to bin, this method won't find the DLL.

If the DLL approach doesn't work for your project layout, the always-working fallback is to add the navigation property manually — type the target type (Volo.Abp.Identity.IdentityUser), FK property (Guid UserId), display property, etc. directly in the dialog without picking from the dropdown.

Docs:

• https://abp.io/docs/latest/suite/generating-crud-page#extending-with-custom-module-entities
• https://abp.io/docs/latest/suite/generating-crud-page#adding-an-existing-entity-as-a-navigation-property

Thanks

Hi,

Glad the NoSmtpIdentityEmailSender part is working.

The SwitchTenant error is unrelated to UserSharingStrategy = Shared — that switch is correct and required, but the actual blocker is that the Angular OpenIddict application doesn't have the gt:SwitchTenant permission in the database.

You probably added SwitchTenant to ExtensionGrantTypes in the OpenIddict admin UI, but the next DbMigrator run wipes it out. Look at OpenIddictDataSeeder.cs in your DbMigrator project — CreateApplicationAsync does this:

if (!HasSameScopes(client, application))
{
    client.Permissions = JsonSerializer.Serialize(application.Permissions.Select(q => q.ToString()));
    await _applicationManager.UpdateAsync(client.ToModel());
}

HasSameScopes actually compares the full Permissions JSON, so any permission you add manually in the UI (like gt:SwitchTenant) makes the comparison fail and the seeder overwrites the whole permission list with what's defined in code.

Fix: add "SwitchTenant" to the Angular client's grantTypes list in OpenIddictDataSeeder.cs, then re-run DbMigrator:

//Angular Client
await CreateApplicationAsync(
    name: "Angular",
    type: OpenIddictConstants.ClientTypes.Public,
    consentType: OpenIddictConstants.ConsentTypes.Implicit,
    displayName: "Angular Client",
    secret: null,
    grantTypes: new List<string> {
        OpenIddictConstants.GrantTypes.AuthorizationCode,
        OpenIddictConstants.GrantTypes.RefreshToken,
        OpenIddictConstants.GrantTypes.Password,
        "LinkLogin",
        "Impersonation",
        "SwitchTenant"   // <-- add this
    },
    scopes: ...,
    ...
);

After DbMigrator runs, the OpenIddictApplicationPermissions table for the Angular client should contain a row gt:SwitchTenant. Then the Switch tenant action in Angular will succeed.

You can also add new grant types to an application directly from the OpenIddict admin UI (OpenIddict → Applications → edit → ExtensionGrantTypes). It's persisted to the database immediately, but as shown above it will be wiped on the next DbMigrator run unless you keep OpenIddictDataSeeder.cs in sync. So use the UI for quick experiments, and update the seeder for the permanent fix.

Thanks