Activities of "maliming"

Hi,

OpenIddict (and ABP's TokenController.HandleClientCredentialsAsync) only puts the scopes the client explicitly asked for into the access token. The scopes you tick on the application page are an allow-list, not defaults — if the client sends no scope parameter, the issued token has no scope at all, which is why your [Authorize] check fails.

OAuth 2.0 actually allows the authorization server to fall back to a pre-defined default scope (RFC 6749 §3.3 and §4.4.2), but ABP doesn't ship a built-in switch for that. The cleanest way to add it is a small IAbpOpenIddictClaimsPrincipalHandler that, on client_credentials requests with an empty scope, copies the application's oi_scp: permissions into the principal:

using System.Collections.Immutable;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using OpenIddict.Abstractions;
using Volo.Abp.DependencyInjection;
using Volo.Abp.OpenIddict;

public class DefaultClientCredentialsScopeHandler : IAbpOpenIddictClaimsPrincipalHandler, ITransientDependency
{
    public ILogger<DefaultClientCredentialsScopeHandler> Logger { get; set; }
        = NullLogger<DefaultClientCredentialsScopeHandler>.Instance;

    public async Task HandleAsync(AbpOpenIddictClaimsPrincipalHandlerContext context)
    {
        var request = context.OpenIddictRequest;
        if (!string.Equals(request.GrantType, OpenIddictConstants.GrantTypes.ClientCredentials, StringComparison.Ordinal))
        {
            return;
        }

        if (!context.Principal.GetScopes().IsDefaultOrEmpty)
        {
            return; // client requested scopes explicitly, leave it alone
        }

        var clientId = request.ClientId;
        if (string.IsNullOrEmpty(clientId))
        {
            return;
        }

        var applicationManager = context.ScopeServiceProvider.GetRequiredService<IOpenIddictApplicationManager>();
        var scopeManager = context.ScopeServiceProvider.GetRequiredService<IOpenIddictScopeManager>();

        var application = await applicationManager.FindByClientIdAsync(clientId);
        if (application == null)
        {
            return;
        }

        var permissions = await applicationManager.GetPermissionsAsync(application);
        var prefix = OpenIddictConstants.Permissions.Prefixes.Scope;

        var scopes = permissions
            .Where(p => p.StartsWith(prefix, StringComparison.Ordinal))
            .Select(p => p[prefix.Length..])
            .ToImmutableArray();

        if (scopes.IsDefaultOrEmpty)
        {
            return;
        }

        Logger.LogInformation("Injecting configured scopes for {ClientId}: {Scopes}", clientId, string.Join(", ", scopes));

        context.Principal.SetScopes(scopes);

        var resources = new List<string>();
        await foreach (var resource in scopeManager.ListResourcesAsync(scopes))
        {
            resources.Add(resource);
        }
        context.Principal.SetResources(resources);
    }
}

Register it in your module's ConfigureServices:

Configure<AbpOpenIddictClaimsPrincipalOptions>(options =>
{
    options.ClaimsPrincipalHandlers.Add<DefaultClientCredentialsScopeHandler>();
});

One thing to watch out for: register it with Configure, not PreConfigure. PreConfigure runs before the framework's own Configure<AbpOpenIddictClaimsPrincipalOptions> block in AbpOpenIddictAspNetCoreModule, and you'll find the handler gets dropped from the final list. I verified the whole flow locally on rel-10.3 against your entervo_infinite-style client (client_credentials + oi_scp:ConnectorNL permission):

• Request without scope → token comes back with "scope": "ConnectorNL" and "aud": "ConnectorNLResource" (handler kicks in).
• Request with scope=ConnectorNL → identical token (handler is a no-op).

We'll also look into making this a first-class option on the OpenIddict module in a future release, so you don't have to maintain the handler yourself.

Thanks

: )

Hi,

Yes, that's expected — both services do register the AccountSettings controller because both IdentityService.HttpApi and AuthServer declare a dependency on AbpAccountAdminHttpApiModule. This is leftover from the older Ocelot template, where the gateway used to route /api/account-admin/... to IdentityService (port 44388). Starting from ABP 8.0 the gateway route was moved to AuthServer, but the dependency on the IdentityService side was kept in the template, so both services still surface the same endpoints in their swagger.

Functionally it doesn't cause any issue:

• The gateway only routes /api/account-admin/{**catch-all} to the AuthServer cluster, so the controller copy living inside IdentityService is never reached through the gateway.
• You can simply ignore the duplicate appearing in IdentityService swagger — no code change needed on your side.

If you really want to clean it up so IdentityService no longer exposes those endpoints, you can remove the Account Admin references from the IdentityService projects (IdentityService.HttpApi.csproj / IdentityService.Application.csproj and the matching [DependsOn(...)] entries in their module files). But this is purely cosmetic — keeping them as-is matches the current ABP template and is perfectly fine.

Thanks

Hi,

Glad to hear it all went through cleanly. Feel free to open a fresh ticket if anything else pops up.

Thanks.

Hi,

Thanks for confirming — that's actually really useful information, because it tells us the wedge wasn't the EzekiaCRM HTTP call itself. Looking again at the log around the same window, there's a lot more going on that day:

• 14:13 — Azure Blob Storage RetriableStream.ReadAsync cancelled (a profile-picture call)
• 14:57 — IdentityUserManager.GetByIdAsync cancelled
• 19:13 — EF Core SaveChanges cancelled mid-ConsumeResultSetAsync
• 12:57 → 17:35 — repeated String or binary data would be truncated ... AbpSessions.IpAddresses errors

So Azure SQL and Azure Blob were both flaky for a long stretch on the 22nd. Since HttpClient.Timeout only protects the EzekiaCRM call, any other await inside the job (EF Core SaveChangesAsync, a blob read, a SQL connection open, even a downstream IdentityUserManager lookup) has no per-job timeout — BackgroundJobWorker only passes the application's StoppingToken, which never fires. That's almost certainly what kept the worker pinned: one of the other awaits in SendVacancyDataAsync / ApplicantDataSendingJob got stuck on Azure SQL or Blob, not on Ezekia.

What I'd add now

You can do all of this on your side — no ABP changes needed.

1) Wrap the job body in a CancellationTokenSource. Yes, please add it — this is the only catch-all that bounds everything inside the job, not just the HTTP call. ABP's IAsyncBackgroundJob<TArgs>.ExecuteAsync doesn't take a CancellationToken parameter, so you create one yourself:

public class VacancyDataSendingJob : AsyncBackgroundJob<VacancySendingArgs>, ITransientDependency
{
    private readonly ExternalCompanyService _service;

    public override async Task ExecuteAsync(VacancySendingArgs args)
    {
        using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(2));
        await _service.SendVacancyDataAsync(args.vacancyId, cts.Token);
    }
}

Make sure SendVacancyDataAsync actually flows the token down to the EzekiaCRM call and to any SaveChangesAsync / blob read it does. Whatever you pick (we usually suggest 1–2 min) just needs to be longer than a healthy job and shorter than "forever."

2) Set an EF Core CommandTimeout. This protects you against Azure SQL hangs specifically. In your DbContextOptions configuration:

options.UseSqlServer(connectionString, sql =>
{
    sql.CommandTimeout(60);
});

3) Separately, you'll want to fix the AbpSessions.IpAddresses truncation. It's unrelated to the jobs but it was throwing all afternoon on the 22nd and is masking other things in your logs. The column is nvarchar(64) by default and your Cloudflare-fronted requests pile up multiple X-Forwarded-For IPs that exceed it. Either widen the column with a custom EF migration, or override the session-recording behaviour to keep only the leftmost client IP.

With (1) in place you'll never see this exact lockup again — even if Azure SQL or Blob has a bad day, the worst case is the job throws after 2 minutes, gets retried later, and the worker keeps moving.

On our side, we'll look at adding a built-in DefaultJobTimeout option to AbpBackgroundJobOptions so future versions wrap each job in a timeout automatically — but you don't need to wait for that, the snippet above is enough.

Thanks

Hi,

Yes, you should follow the new template and route /api/account-admin/{**catch-all} to the AuthServer cluster, not Identity. Starting from ABP 8.0, the account-admin endpoints (login/auth related settings such as two-factor, recaptcha, external providers, idle timeout, passkey) were moved from IdentityService to AuthServer as the official architecture, and the gateway templates were updated accordingly when we switched from Ocelot to YARP.

So your gateway route should look like this (this is exactly what the latest microservice template ships):

"AccountAdmin": {
  "ClusterId": "AuthServer",
  "Match": {
    "Path": "/api/account-admin/{**catch-all}"
  }
}

On the AuthServer backend side, please double-check two places:

1. MyCompanyName.MyProjectName.AuthServer.csproj references both Account Admin packages (use the same version as the rest of your ABP packages):

<PackageReference Include="Volo.Abp.Account.Pro.Admin.HttpApi" Version="10.x.x" />
<PackageReference Include="Volo.Abp.Account.Pro.Admin.Application" Version="10.x.x" />

2. MyProjectNameAuthServerModule.cs declares the dependency:

[DependsOn(
    // ... existing modules
    typeof(AbpAccountPublicWebOpenIddictModule),
    typeof(AbpAccountPublicApplicationModule),
    typeof(AbpAccountPublicHttpApiModule),
    typeof(AbpAccountAdminApplicationModule),
    typeof(AbpAccountAdminHttpApiModule),
    // ...
)]
public class MyProjectNameAuthServerModule : AbpModule
{
}

If you upgraded with the ABP CLI those usually get applied for you, but for an 8 → 10 upgrade where the AuthServer project may have started life on an older template, it's worth verifying. Once those two pieces are in place, no other backend code change is required — the controllers (AccountSettingsController, route prefix api/account-admin/settings) come from Volo.Abp.Account.Pro.Admin.HttpApi and will be exposed by AuthServer automatically.

After applying the change, you can confirm by hitting AuthServer's swagger directly and looking for the /api/account-admin/... endpoints there.

Thanks

Hi,

Thanks for the logs and CSV — they made the picture pretty clear. Here's what happened:

The default BackgroundJobWorker in ABP processes jobs sequentially in a single foreach, and the whole polling cycle is wrapped in a distributed lock (AbpBackgroundJobWorker). When you don't use Redis, that lock lives in process memory (LocalAbpDistributedLock). The implication: if any single job gets stuck on an await (e.g. an HTTP call that never returns), the worker thread holds the lock forever, every subsequent 5-second polling cycle silently fails to acquire it, and no further jobs are processed and no log lines are produced — exactly what you saw after 12:19:14 on April 22.

In your case, VacancyDataSendingJob / ApplicantDataSendingJob call EzekiaCRM. The default HttpClient.Timeout is 100 seconds, but it only applies to the response headers — once a response starts trickling back slowly (or a TCP socket gets half-open), the call can hang indefinitely. Most likely Ezekia had a hiccup that day, one of those calls never returned, the worker thread parked there, and the lock stayed held. Restarting the app on the 28th cleared the in-memory lock, but the same jobs immediately ran into the same external behavior, which is why CPU went up but the queue still didn't drain. By early on the 29th the upstream connection finally died and the worker resumed.

The other IdentitySessionCleanupBackgroundWorker / TokenCleanupBackgroundWorker entries you can see throughout the log confirm the worker host itself was alive — only the background job worker was wedged.

Fix — give the EzekiaCRM HttpClient an explicit timeout

This is the smallest change that actually prevents the deadlock. Wherever you register the Ezekia client, set a hard Timeout:

context.Services.AddHttpClient<EzekiaCRM.Client>(client =>
{
    client.BaseAddress = new Uri("https://api.ezekiacrm.com/");
    client.Timeout = TimeSpan.FromSeconds(30);
});

Pick a value that's comfortably above your normal Ezekia response time but well under "forever" (30–60s is usually fine). Once that's in place, the worst case is the job throws TaskCanceledException, the executor catches it as a normal failure, increments TryCount, and moves on — the worker is never wedged again.

If you also want belt-and-braces, you can wrap the call in a CancellationTokenSource inside the job itself, but the HttpClient.Timeout alone is enough to prevent the symptom you ran into.

We'll also look into adding a per-job timeout in the default worker on our side so this kind of upstream hang can't silently freeze the whole queue.

Thanks

Hi,

This is the intended behavior, not a bug. The change comes from PR #24775 - Prevent Privilege Escalation: Add Assignment Restrictions for Roles and Permissions, which closes a real privilege escalation hole: a non-admin operator with IdentityPermissions.Users.Update could otherwise assign any role (including admin) to other users, or to themselves.

The new rule in IdentityUserAppService.UpdateRolesAsync is:

• If the current operator is in the admin role (HasAdminRoleAsync() returns true), the input is used as-is.
• Otherwise, the operator can only grant roles they themselves already have. Roles the target user already has but the operator does not have are preserved (so the operator cannot strip a higher-privileged role either).

The "is operator an admin?" check intentionally uses the well-known AbpRoleConsts.AdminRoleName ("admin") because that's the role the framework itself seeds and treats as superuser everywhere (data seeding, permission management, etc.). It's a const, and changing it to a settable property is something we'd rather not do - it would break binary compatibility (a const is inlined at compile time into every consumer), and IdentityDataSeeder does const string adminRoleName = AbpRoleConsts.AdminRoleName;, which only compiles because the right-hand side is a compile-time constant. So we don't recommend renaming the framework's admin role - it's much safer to keep an admin role in your DB even if your business uses different role names elsewhere.

That said, if you really need a different role name to act as the "superuser" for this check, you don't need to override UpdateRolesAsync. We left a smaller, more precise extension point: HasAdminRoleAsync() is protected virtual. Just override that one method in your *.Application project:

using System.Threading.Tasks;
using Microsoft.AspNetCore.Identity;
using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Options;
using Volo.Abp.Authorization.Permissions;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.EventBus.Distributed;
using Volo.Abp.Identity;
using Volo.Abp.Identity.Emailing;
using Volo.Abp.Identity.UserInvitations;
using Volo.Abp.Identity.UserSharing;
using Volo.Abp.Threading;

namespace YourProject.Identity;

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IIdentityUserAppService), typeof(IdentityUserAppService))]
public class MyIdentityUserAppService : IdentityUserAppService
{
    public MyIdentityUserAppService(
        IdentityUserManager userManager,
        IIdentityUserRepository userRepository,
        IIdentityRoleRepository roleRepository,
        IOrganizationUnitRepository organizationUnitRepository,
        IIdentityClaimTypeRepository identityClaimTypeRepository,
        IdentityProTwoFactorManager identityProTwoFactorManager,
        IOptions<IdentityOptions> identityOptions,
        IDistributedEventBus distributedEventBus,
        IOptions<AbpIdentityOptions> abpIdentityOptions,
        IPermissionChecker permissionChecker,
        IDistributedCache<IdentityUserDownloadTokenCacheItem, string> downloadTokenCache,
        IDistributedCache<ImportInvalidUsersCacheItem, string> importInvalidUsersCache,
        IdentitySessionManager identitySessionManager,
        IdentityUserTwoFactorChecker identityUserTwoFactorChecker,
        ICancellationTokenProvider cancellationTokenProvider,
        UserSharingManager userSharingManager,
        UserInvitationManager userInvitationManager,
        IIdentityUserInvitationRepository userInvitationRepository,
        IIdentityEmailSender identityEmailSender)
        : base(
            userManager,
            userRepository,
            roleRepository,
            organizationUnitRepository,
            identityClaimTypeRepository,
            identityProTwoFactorManager,
            identityOptions,
            distributedEventBus,
            abpIdentityOptions,
            permissionChecker,
            downloadTokenCache,
            importInvalidUsersCache,
            identitySessionManager,
            identityUserTwoFactorChecker,
            cancellationTokenProvider,
            userSharingManager,
            userInvitationManager,
            userInvitationRepository,
            identityEmailSender)
    {
    }

    protected override Task<bool> HasAdminRoleAsync()
    {
        return Task.FromResult(CurrentUser.IsInRole("YourCustomAdminRoleName"));
    }
}

I verified locally on Volo.Abp.Identity.Pro.Application.Tests against ABP 10.3 - the override is picked up correctly, an operator with the custom admin role claim can assign any role (including ones they don't have themselves), and all the existing role-filtering / unmanageable-role-preservation tests keep passing.

Thanks.

Great 👍

Great!