Open Closed

Intermittent `GrantedPolicies = 0` for authenticated user #10481


User avatar
0

ABP Support Request: Intermittent GrantedPolicies = 0 for authenticated admin user (ABP 10.0.2)

Hi ABP team,

We are investigating an intermittent production issue where an authenticated user suddenly loses effective permissions (menus/actions disappear) without logging out.

Environment

  • ABP version: 10.0.2
  • App type: ABP commercial-style modular app (Web + HttpApi.Host)
  • Cache stack: IDistributedCache (Redis) + custom in-memory L1 decorator over IDistributedCache
  • Redis: Azure Cache for Redis
  • Hosting: Azure App Service

Context

We already implemented an L1 cache decorator around IDistributedCache to handle the previously observed Redis transient/cancellation path.

That fix helped for Redis outages, but we are now seeing a different failure mode where cache is healthy, user is authenticated, yet GrantedPolicies becomes zero.

What we observed (latest incident)

Incident window (UTC): 2026-02-27 19:33:10Z to 19:35:53Z.

1) User is authenticated and claims look valid

  • Claims diagnostic: User=admin, UserId=e82af67f-ecf7-4559-8140-c9596aa9f91a, TotalClaims=11, Roles=[admin], AllKeyClaimsPresent=true

2) App config reports zero policies for that same authenticated user

Multiple requests in the same window show:

  • AppConfig diagnostic: User=admin has 0 grantedPolicies. Auth IsAuthenticated=true, CurrentUser set=true

Seen on:

  • /
  • /HostDashboard
  • /Abp/ApplicationConfigurationScript
  • /Abp/ApplicationLocalizationScript
  • /Abp/ServiceProxyScript

3) Authorization failures follow immediately

Many permission checks fail for the same request/user after the zero-policy state appears.

4) Redis appears healthy during this exact window

  • Redis status check ... IsConnected=true (periodic checks around the incident)
  • No corresponding Redis outage signature in this incident window

5) No classic application-configuration 500 chain this time

For this window we did not see the older pattern (API 500 + remote call exception/cancellation) that we saw in a separate earlier incident.

Why this is confusing

It looks like a valid, authenticated principal can occasionally resolve to an application configuration with GrantedPolicies = 0, and that value then appears to be reused by cache for a short period.

Questions for ABP team

  1. Are there known scenarios in ABP 10.0.2 where ApplicationConfiguration can be successfully generated with an empty GrantedPolicies dictionary for an authenticated user that otherwise has roles/claims?
  2. Is there any known race condition around dynamic claims/permission value providers that could yield transient empty results without exceptions?
  3. Is ABP expected to cache an empty GrantedPolicies result the same way as a normal one, and is there a recommended guard to avoid caching suspiciously empty results for authenticated users?
  4. Is there a recommended extension point to validate/reject an ApplicationConfigurationDto when IsAuthenticated=true and GrantedPolicies.Count==0?
  5. Are there patches after 10.0.2 related to intermittent empty permission resolution in RemotePermissionChecker / application configuration generation?

Notes

  • We can share anonymized log snippets with trace IDs if needed.
  • We already have diagnostics proving this is not always tied to Redis transport failures.

Thanks in advance. Any guidance on whether this is a known issue or recommended mitigation pattern would be very helpful.


22 Answer(s)
  • User Avatar
    0
    lanpin created

    以下信息可能有用: 不正常时: 在MVC前端访问:https://manage.domain.com/Abp/ApplicationConfigurationScript 时,currentUser包含用户信息。但在redis缓存中,键 c:Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationDto,k:AppName1:ApplicationConfiguration_1d5fa0de27ae42a8a24f5f46ea0793fb_7ad543414c677529e7243a1f648d059c_zh-Hans 中currentUser不包含用户信息。

    The following information may be helpful:

    When incorrect: When accessing https://manage.domain.com/Abp/ApplicationConfigurationScript in the MVC frontend, currentUser contains user information. However, in the Redis cache, currentUser in the key c:Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationDto,k:SmartBus:ApplicationConfiguration_1d5fa0de27ae42a8a24f5f46ea0793fb_7ad543414c677529e7243a1f648d059c_zh-Hans does not contain user information.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    We already implemented an L1 cache decorator around IDistributedCache to handle the previously observed Redis transient/cancellation path.

    If you remove this feature, does the problem still exist? I just want to confirm whether they're related.

    Thanks.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi lanpin

    This might be because your access token has expired or is invalid. We have a CheckTokenExpiration extension method to check it.

    Thanks.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    BTW, you can test with this sultion: https://abp.io/support/questions/10483/v1010-Tiered-Empty-grantedPolicies-in-WebPublic-MVC-when-CmsKit-is-enabled#answer-3a1fbf3e-08f2-5b12-1295-acf09d5eb112

    Thanks.

  • User Avatar
    0

    [maliming] said: hi

    We already implemented an L1 cache decorator around IDistributedCache to handle the previously observed Redis transient/cancellation path.

    If you remove this feature, does the problem still exist? I just want to confirm whether they're related.

    Thanks.

    No, we had to add this to work around the issue though it sounds like it's still an issue (user report now): https://gist.github.com/kfrancis/0064c960832458254dcca9591948a103

    We've deployed that change you mentioned. Can you explain why the additional authenticate call is needed?

  • User Avatar
    0

    No, no change in function. Still an issue.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you try this? https://abp.io/support/questions/10483/v1010-Tiered-Empty-grantedPolicies-in-WebPublic-MVC-when-CmsKit-is-enabled#answer-3a1fc36a-b259-117e-907a-2687f3b22102

    Can you remove the CMS module to see if this is fixed?

    //typeof(CmsKitProPublicWebModule)

    Thanks.

  • User Avatar
    0
    lanpin created

    @maliming 我需要的是用户在登录管理后台之后可以一直保持在线。在添加checkTokenExpiration之后,ManageHost确实可以在出错之前检测,但这并不是我的要求。我想最佳方案应该是Web(MVC)Host可以检查accesstoken是否过期,如果过期则自动使用refreshToken重新获取accessToken并继续完全用户的后续请求,并将新的令牌返回给用户cookies。这整个过程对用户是无感知的。

    What I need is for users to remain online after logging into the admin panel. While adding checkTokenExpiration does allow ManageHost to detect errors before they occur, this isn't what I require. I think the best solution would be for the Web (MVC) Host to check if the access token has expired. If it has, it should automatically obtain a new access token using a refreshToken, continue processing all subsequent user requests, and return the new token to the user's cookies. This entire process should be seamless for the user.


    补充一下: 在未添加CheckTokenExpiration之前,用户登录一段时间后会失去所有权限和菜单,保持不注销并继续等待一段时间,用户权限和菜单会自行恢复。在用户失去权限和菜单时,通过redis-cli flushall强制清空所有缓存,可以立即恢复用户权限和菜单。

    To add to the above: Before adding CheckTokenExpiration, users would lose all permissions and menu access after logging in for a period of time. If they remained logged in and waited for a while, their permissions and menu access would be automatically restored. When a user loses permissions and menu access, a forced clear of all cache using redis-cli flushall can immediately restore their permissions and menu access.

  • User Avatar
    0

    [maliming] said: hi

    Can you try this?

    https://abp.io/support/questions/10483/v1010-Tiered-Empty-grantedPolicies-in-WebPublic-MVC-when-CmsKit-is-enabled#answer-3a1fc36a-b259-117e-907a-2687f3b22102

    Can you remove the CMS module to see if this is fixed?

    //typeof(CmsKitProPublicWebModule)

    Thanks.

    I also did that, no change.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi lanpin

    Can you create a new question?

    Thanks.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi kfrancis

    Another customer says this fixed their problem.

    https://abp.io/support/questions/10483/v1010-Tiered-Empty-grantedPolicies-in-WebPublic-MVC-when-CmsKit-is-enabled#answer-3a1fc36a-b259-117e-907a-2687f3b22102

    Can you try that?

    If it's still not working, can you share your code? I will check and try to reproduce it locally.

    liming.ma@volosoft.com

    Thanks.

  • User Avatar
    0
    lanpin created

    @maliming 我以为我们遇到的是同一个问题,我已经尝试解决并提交了一个PR:https://github.com/abpframework/abp/pull/25011 。 之后我将不再在此主题回复。

    I thought we were having the same problem. I've already tried to solve it and submitted a PR: https://github.com/abpframework/abp/pull/25011 I will no longer reply to this thread.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Thanks @lanpin

  • User Avatar
    0

    @mailming No, still an issue.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi kfrancis

    If it's still not working, can you share your code? I will check and try to reproduce it locally.

    liming.ma@volosoft.com

    Thanks.

  • User Avatar
    0

    So, I'm just packing up the source to send - along with a summary and some recent logs showing the issues.

    We basically have been dealing with the same issue, and all I've been able to do is move the issue so that users weren't getting stuck on login - to - users get in but don't have the correct permissions for some amount of time.

    When we initially encountered major issues related to this, it was in the form of users getting stuck on the login page for about 4 minutes in a redirect loop until the system would finally allow them in. Over time, we've changed it so that users don't get weirdly stuck but now we are seeing more directly that it's all permissions related, in our current case the users login no problem but then randomly they won't have any permissions (grants=0) for some amount of time (shorter than 4 minutes), before they will start coming back gradually.

    I'm sending the source with a folder \volosoft-logs for you, which includes recent related logs (web and auth) and a summary of the issue.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Thanks, I will check it. 👍

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Hi kfrancis,

    I went through your source code and logs. I couldn't build the project locally (missing NuGet feeds/dependencies), but I've pushed some diagnostic changes here: https://github.com/maliming/source/pull/1

    What I changed:

    • Disabled the L1 cache decorator in Web, Web.Public, and HttpApi.Host (since you said it made things worse, let's take it out for now)
    • Added logging in MvcCachedApplicationConfigurationClient and AbpApplicationConfigurationAppService to capture token status, user identity, cache state, etc. when GrantedPolicies=0 happens
    • All diagnostic log entries start with ABP_DIAG: so they're easy to find

    Can you:

    1. Merge these changes and build locally
    2. Check that ABP_DIAG: entries show up in both Web and API output when you log in
    3. Deploy to Azure and wait for the issue to occur
    4. Search logs for ABP_DIAG: and share what you find

    Is https://github.com/kfrancis your GitHub? I can add you to the repo so you can review the PR.

    Thanks

  • User Avatar
    0

    Yes, that's me.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    https://github.com/maliming/source/invitations

  • User Avatar
    0

    What Changed Since Last Update

    We deployed custom diagnostic instrumentation on 2026-03-19 ~16:00 UTC to trace the exact request flow between the Web and API tiers. Three components were added:

    1. DiagnosticApplicationConfigurationClientDecorator (Web tier) — decorates IAbpApplicationConfigurationClientProxy, logs every outbound AppConfig request with user identity, token presence/size, tenant, and culture.
    2. DiagnosticApplicationConfigurationAppService (API tier) — wraps the AppConfig response, logs inbound token status, resolved PolicyCount, and elapsed time.
    3. AccessTokenDiagnosticMiddleware (Web tier) — warns when the access token is within 30 seconds of expiry.

    All diagnostic entries are tagged with ABP_DIAG in the message template for easy filtering.


    Findings (3-hour window, 2026-03-19 16:00–19:45 UTC)

    Web Tier → API Tier: Token IS Being Sent ✅

    66 outbound AppConfig requests logged. Every single one shows:

    • IsAuthenticated = true
    • TokenStatus = Present(1450–1550 chars) — a valid-sized JWT
    • Correct UserId, UserName, and TenantId values

    Example (Web tier, outbound):

    {
      "@t": "2026-03-19T19:45:58.286Z",
      "@mt": "ABP_DIAG: AppConfig fetching from RemoteAPI. User={UserName}, UserId={UserId}, Auth={IsAuthenticated}, TenantId={TenantId}, AccessToken={TokenStatus}, Culture={Culture}",
      "UserName": "admin",
      "UserId": "e82af67f-ecf7-4559-8140-c9596aa9f91a",
      "IsAuthenticated": true,
      "TenantId": null,
      "TokenStatus": "Present(1550chars)",
      "Culture": "en",
      "SourceContext": "CabMD.Web.Diagnostics.DiagnosticApplicationConfigurationClientDecorator",
      "RequestPath": "/HostDashboard"
    }
    

    API Tier: Token Received But Resolved as Anonymous, PolicyCount=0 ❌

    14 API-side entries logged. Every single one shows:

    • TokenStatus = Bearer(825chars) — the Bearer token arrives in the request
    • PolicyCount = 0 — 100% failure rate
    • Logged as Warning: "API AppConfig anonymous request" — despite having a Bearer token

    Example (API tier, inbound):

    {
      "@t": "2026-03-19T16:26:54.796Z",
      "@mt": "ABP_DIAG: API AppConfig anonymous request. Token={TokenStatus}, Policies={PolicyCount}, Elapsed={ElapsedMs}ms, ClientIp={ClientIp}",
      "@l": "Warning",
      "TokenStatus": "Bearer(825chars)",
      "PolicyCount": 0,
      "ElapsedMs": 247.15,
      "ClientIp": "20.220.247.162",
      "SourceContext": "CabMD.Diagnostics.DiagnosticApplicationConfigurationAppService",
      "RequestPath": "/api/abp/application-configuration"
    }
    

    Full API-Side Timeline

    | Timestamp (UTC) | Token | PolicyCount | Elapsed | Source IP | |---|---|---|---|---| | 16:26:54 | Bearer(825chars) | 0 | 247ms | 20.220.247.162 | | 16:36:57 | Bearer(825chars) | 0 | 9,336ms | 10.0.0.253 | | 16:37:16 | Bearer(825chars) | 0 | 236ms | 10.0.0.253 | | 17:09:47 | Bearer(825chars) | 0 | 210ms | 10.0.0.253 | | 17:30:30 | Bearer(825chars) | 0 | 194ms | 10.0.0.253 | | 18:02:33 | Bearer(825chars) | 0 | 197ms | 10.0.0.253 | | 18:11:39 | Bearer(825chars) | 0 | 195ms | 10.0.0.253 | | 18:13:03 | Bearer(825chars) | 0 | 202ms | 10.0.0.253 | | 18:22:25 | Bearer(825chars) | 0 | 209ms | 10.0.0.253 | | 18:36:20 | Bearer(825chars) | 0 | 195ms | 10.0.0.253 | | 18:42:34 | Bearer(825chars) | 0 | 201ms | 10.0.0.253 | | 18:49:31 | Bearer(825chars) | 0 | 197ms | 10.0.0.253 | | 19:22:24 | Bearer(825chars) | 0 | 50ms | 10.0.0.253 | | 19:28:53 | Bearer(825chars) | 0 | 208ms | 10.0.0.253 |

    Notes:

    • 10.0.0.253 = internal VNet traffic (web tier → API tier)
    • 20.220.247.162 = external client
    • The 9,336ms outlier at 16:36 suggests contention in the permission resolution pipeline

    Token Expiry (Minor, Separate Observation)

    5 warnings for user wlahaye showing token within 15–27 seconds of expiry at ~17:41 UTC. This is a natural edge case from rapid page loads near token expiry — not the root cause, since PolicyCount=0 occurs with tokens that have plenty of remaining lifetime.


    Root Cause Isolation

    The diagnostic data narrows the problem to a specific gap:

    Web Tier                          API Tier
    ─────────                         ────────
    User authenticated ✅
    Token attached (1550 chars) ✅
            ──── HTTP ────→
                                      Token received (825 chars) ✅
                                      CurrentUser resolved?       ❌ (anonymous)
                                      PolicyCount resolved?       ❌ (0)
            ←── Response ────
    grantedPolicies = {} ❌
    
    1. Web tier correctly authenticates and attaches a valid access token.
    2. API tier receives the token — it's present in the Authorization header.
    3. API tier does NOT resolve CurrentUser from the token — it treats the request as anonymous.
    4. PolicyCount = 0 on 100% of observed requests — this is not intermittent during this window; it's a consistent failure.

    The problem is inside ABP's Bearer token validation / CurrentUser population pipeline on the API tier, somewhere between receiving the Authorization header and evaluating IPermissionChecker.


    Questions for ABP Team

    1. Is there a known issue where the API tier's AbpApplicationConfigurationAppService.GetAsync() fails to populate ICurrentUser from a valid Bearer token? Our evidence shows the token arrives but the user is treated as anonymous.

    2. Could IdentitySessionDynamicClaimsPrincipalContributor be interfering? We've observed this contributor causing multi-second delays (the "four minute problem"). Could it be failing silently and stripping claims, resulting in an anonymous-looking principal?

    3. What populates CurrentUser on the API tier for Bearer-authenticated requests? Is there a middleware ordering requirement we might be missing, or a DI registration that could be silently failing?

    4. The token shrinks from ~1550 chars (Web tier) to ~825 chars (API tier) — is this expected? The Web tier logs the full access token size; the API tier logs the Authorization header value. Could the token be getting truncated or replaced in transit?

    5. Is there a way to inspect HttpContext.User.Claims inside AbpApplicationConfigurationAppService.GetAsync() to determine whether the Bearer middleware ran but produced an empty principal vs. didn't run at all?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Hey, thanks for the detailed logs — super helpful.

    So we dug into the diagnostic output and your source code, and here's where we landed:

    The Bearer token is definitely reaching the API tier (825 chars every time), but the API is treating every request as anonymous. We checked your HttpApiHostModule and noticed you've disabled dynamic claims:

    options.IsDynamicClaimsEnabled = false;
    options.DynamicContributors.Clear();
    

    So IdentitySessionDynamicClaimsPrincipalContributor isn't even in the picture. The problem is happening earlier — the JwtBearer handler itself is failing to validate the token. When that happens, HttpContext.User stays anonymous, and you get empty GrantedPolicies on every cache-miss request.

    Your API uses AddAbpJwtBearer with Authority from config and Audience = "CabMD". Your AuthServer sets the issuer via serverBuilder.SetIssuer(configuration["AuthServer:Authority"]). ABP calls DisableAccessTokenEncryption() unconditionally — we checked your entire codebase, nothing re-enables it — so the token should be a plain JWT. But 100% of validations are failing, and we can't tell why from the source code alone because both sides read AuthServer:Authority from their own config, and the production values come from Azure env vars (not in appsettings).

    We've pushed a bunch of updates to the PR:

    New diagnostic hooks — we added a DiagnosticJwtBearerPostConfigure that taps directly into the JwtBearer handler events. After deploying, look for these in your API logs:

    • ABP_DIAG: JwtBearer OnAuthenticationFailed — this is the big one. It'll show the exact exception (signature key not found? issuer mismatch? audience mismatch?), plus the JWT's decoded issuer, audience, expiry, the configured Authority, and the raw token itself.
    • ABP_DIAG: JwtBearer OnTokenValidated SUCCESS — if you see this, validation is actually working and we need to look elsewhere.
    • ABP_DIAG: JwtBearer OnMessageReceived — confirms the handler is receiving the token and whether it's actually a JWT.

    IdentityModel verbose logging — we enabled ShowPII = true and EventLevel.Verbose in Program.cs. This writes the full JWT validation pipeline to Logs/identitymodel.txt in the API host's working directory — every step of key lookup, signature check, issuer/audience validation, with nothing masked.

    Log levels bumped to Debug — across HttpApi.Host, Web, and AuthServer, both dev and production configs. Serilog MinimumLevel and Microsoft.AspNetCore.Authentication are both set to Debug now.

    Could you pull the latest, deploy, and then share the following after reproducing the issue?

    • Logs/identitymodel.txt from the API host
    • Complete log files from all three projects — HttpApi.Host, Web, and AuthServer

    The identitymodel.txt alone should tell us exactly what's going wrong with the JWT validation. The full logs from all projects might also reveal something we haven't thought of yet.

Boost Your Development
ABP Live Training
Packages
See Trainings
Mastering ABP Framework Book
The Official Guide
Mastering
ABP Framework
Learn More
Mastering ABP Framework Book
Made with ❤️ on ABP v10.3.0-preview. Updated on March 13, 2026, 12:51
1
ABP Assistant
🔐 You need to be logged in to use the chatbot. Please log in first.