31 Answer(s)

• 

  0

  maliming created 7 months ago

  Support Team Fullstack Developer

  hi

  Can you share all logs.txt and identitymodel.txt to liming.ma@volosoft.com

  Thanks.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 7 months ago

  Support Team Fullstack Developer

  hi

  Your case may be caused by:

  HttpContextIdentityModelRemoteServiceHttpClientAuthenticator.Authenticate()
    → AccessTokenProvider.GetTokenAsync() returns null
    → Falls back to base.Authenticate()
    → IdentityModelAuthenticationService: client credentials grant
    → 825-char token, no 'sub', no 'session_id'
  

  We also reviewed your identitymodel.txt (March 20-23) — 58 out of 95 tokens are client credentials, generated every 30-90 minutes 24/7, including all day Saturday with zero users online. That's your Hangfire workers (CachePingWorker, EDT workers, health checks, etc.) running in the API Host process with no HttpContext — completely legitimate and unaffected by our changes.

  We've pushed updated code to the PR with three layers:

  Layer 1: Token backup + recovery (AccessTokenBackupMiddleware + AccessTokenBackupStore)

  A new middleware saves a copy of the user's access_token to Redis on every authenticated request. When the primary source (OIDC cookie) returns null, both the decorator and the authenticator attempt to recover from this backup. If the backup exists, the request proceeds normally with the recovered token — the user never sees missing menus.

  This is registered in the middleware pipeline:

  app.UseAuthentication();
  app.UseMiddleware<AccessTokenDiagnosticMiddleware>();
  app.UseMiddleware<AccessTokenBackupMiddleware>();  // saves token to Redis
  

  Layer 2: SafeRemoteServiceHttpClientAuthenticator (prevents client credentials fallback)

  Replaces ABP's HttpContextIdentityModelRemoteServiceHttpClientAuthenticator in the Web tier only. When the user's token is unavailable:

  1. Tries to recover from the Redis backup
  2. If backup found → uses it (request succeeds normally)
  3. If backup not found → skips authentication entirely (no client credentials token sent)

  Also detects Polly retry scenarios by checking if the Authorization header is already set when Authenticate() is called — this would indicate the request is being retried by the resilience handler.

  Layer 3: DiagnosticApplicationConfigurationClientDecorator guard (prevents cache poisoning)

  Last line of defense. If both primary token and backup are unavailable, blocks the remote API call and returns an empty AppConfig WITHOUT caching it. The page renders with reduced permissions for that single request only.

  Diagnostic output

  We still need to understand WHY the token goes missing. When the primary source fails, the code dumps:

  • All stored tokens in the OIDC session (access_token, refresh_token, id_token, expires_at)
  • Full AuthenticateAsync() result with Properties.Items
  • Cookie header size (to detect truncation)
  • HttpContext.User identity state
  • Thread ID (to detect async context loss)
  • Whether the request is a Polly retry

  After deploying, watch for these entries in the Web tier logs:

  | Log pattern | What it means | |---|---| | RECOVERED from backup | Token was missing from cookie but recovered from Redis — user experience preserved | | Both primary and backup token sources returned null | First request for a new user where no backup exists yet | | AUTH DUMP — Stored tokens | Shows exactly which tokens are present/missing in the OIDC session | | AUTH DUMP — Cookie header size | If unusually large, cookies may be getting truncated | | Authenticator — called again on a request that ALREADY has Authorization header | Polly retry detected | | Authenticator — BLOCKING client credentials fallback | Fallback prevented |

  Your Hangfire background jobs are unaffected — all changes are in CabMD.Web and CabMD.Web.Public only.

  You can try again and share the new logs.

  Thanks.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  kfrancis@clinicalsupportsystems.com created 7 months ago

  Verified locally. Curious though, is this something that will get improved in abp - or is this due to something we're doing? That part I'm unsure about. For example, if you do make changes to abp and they get released - what should I be looking for in the release notes?

  In case it's helpful, this is what I used to cause the issue locally:

  /// <summary>
  ///     DEV-ONLY decorator that simulates intermittent access token loss by returning null
  ///     on a configurable percentage of requests. Used to verify the three-layer fix for [#10481](https://abp.io/QA/Questions/10481).
  ///     Enable via appsettings:
  ///     "Diagnostics": { "FaultyTokenProvider": { "Enabled": true, "FailurePercent": 30 } }
  ///     Safety: Registration is gated on IsDevelopment() in CabMDWebModule.ConfigureServices.
  /// </summary>
  [Dependency(ReplaceServices = true)]
  [ExposeServices(typeof(IAbpAccessTokenProvider))]
  public class FaultyAccessTokenProviderDecorator : IAbpAccessTokenProvider, ITransientDependency
  {
      // Thread-safe random for failure rate calculation
      private static readonly Random s_random = new();
      private readonly IConfiguration _configuration;
      private readonly IHttpContextAccessor _httpContextAccessor;
      private readonly ILogger<FaultyAccessTokenProviderDecorator> _logger;
  
      public FaultyAccessTokenProviderDecorator(
          IHttpContextAccessor httpContextAccessor,
          IConfiguration configuration,
          ILogger<FaultyAccessTokenProviderDecorator> logger)
      {
          _httpContextAccessor = httpContextAccessor;
          _configuration = configuration;
          _logger = logger;
      }
  
      public async Task<string?> GetTokenAsync()
      {
          // Get the real token (same logic as ABP's HttpContextAbpAccessTokenProvider)
          var httpContext = _httpContextAccessor.HttpContext;
          if (httpContext == null)
          {
              return null;
          }
  
          var realToken = await httpContext.GetTokenAsync("access_token");
  
          // Check if fault injection is enabled (re-read config each time for hot-reload support)
          var enabled = _configuration.GetValue("Diagnostics:FaultyTokenProvider:Enabled", false);
          if (!enabled || string.IsNullOrEmpty(realToken))
          {
              return realToken;
          }
  
          var failurePercent = _configuration.GetValue("Diagnostics:FaultyTokenProvider:FailurePercent", 30);
          failurePercent = Math.Clamp(failurePercent, 0, 100);
  
          var roll = s_random.Next(100);
          if (roll >= failurePercent)
          {
              return realToken;
          }
  
          var userName = httpContext.User.Identity?.Name ?? "unknown";
          var path = httpContext.Request.Path.Value ?? "/";
  
          _logger.LogWarning(
              "ABP_DIAG: SIMULATED token null. " +
              "Real token was present ({TokenLength} chars) but returning null to simulate failure. " +
              "FailurePercent={FailurePercent}, Roll={Roll}, User={UserName}, Path={Path}",
              realToken.Length, failurePercent, roll, userName, path);
  
          return null;
      }
  }
  

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 7 months ago

  Support Team Fullstack Developer

  hi

  To answer your question: this is partly an ABP framework issue. HttpContextIdentityModelRemoteServiceHttpClientAuthenticator should not silently fall back to client credentials when ICurrentUser.IsAuthenticated is true — that fallback produces incorrect results for user-context endpoints regardless of why the token is unavailable. We'll look into improving this in the framework.

  However, we still need to understand why httpContext.GetTokenAsync("access_token") returns null for some authenticated requests in your environment. The backup/recovery mechanism works around the symptom, but we want to find the root cause. The latest code we pushed includes detailed diagnostic output (AUTH DUMP) that fires automatically when the token is missing — it dumps all stored tokens, the full AuthenticateAsync() result, authentication properties, and the cookie header size.

  Could you share the Web tier logs after the deployment has been running for a day or two with real users? Specifically we're looking for:

  • ABP_DIAG: AUTH DUMP entries — these reveal exactly what's present vs missing in the OIDC session when the token loss occurs
  • ABP_DIAG: RECOVERED from backup entries — confirms the backup mechanism is working and that the primary source did return null
  • ABP_DIAG: Authenticator entries at Warning/Error level — shows the state at the HTTP client authenticator layer

  This will tell us whether the issue is cookie-related (truncation, data protection), session-related (ticket store eviction), or something else entirely.

  Thanks.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  kfrancis@clinicalsupportsystems.com created 7 months ago

  Hi maliming,

  We deployed the changes from your PR (SafeRemoteServiceHttpClientAuthenticator, AccessTokenBackupMiddleware, and the updated DiagnosticApplicationConfigurationClientDecorator with cache-poisoning guard) on March 27.

  Results after 3+ days in production (March 27-30):

  • Zero occurrences of ZERO POLICIES or IDENTITY MISMATCH — the GrantedPolicies=0 issue has not recurred
  • Zero occurrences of BLOCKING client credentials — the SafeRemoteServiceHttpClientAuthenticator has not needed to block a fallback
  • Zero occurrences of RECOVERED from backup — the backup recovery has never been triggered
  • Zero occurrences of AUTH DUMP — no authenticated user has experienced token loss
  • Zero occurrences of "access token is NULL" for an authenticated user

  The only access token is NULL entries (~20) are for anonymous/unauthenticated requests (IsAuthenticated: false, UserName: "unknown") on paths like /signalr-hubs/background-job-event/negotiate and /close-popup. These correctly fall through to client credentials as expected.

  Interpretation: Since deploying the three-layer fix, the intermittent token loss has not occurred at all in production. We haven't been able to capture an AUTH DUMP because the condition that triggers it (authenticated user + null token) hasn't happened. This suggests one of two possibilities:

  1. The fix has a preventive effect — perhaps the AccessTokenBackupMiddleware reading the token on every request is subtly changing timing/caching behaviour that was causing the original issue
  2. The issue is truly intermittent, and we simply haven't hit the right conditions yet in 3 days

  We'll continue monitoring. If/when the condition does occur, the AUTH DUMP and backup recovery logging will capture the full diagnostic state you need. Should we leave the diagnostics running indefinitely, or is there a specific timeframe you'd recommend?

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 7 months ago

  Support Team Fullstack Developer

  hi

  To answer your question: it's a bit of both. ABP's silent fallback to client credentials when the user token is missing is a framework-level problem we should fix. But why the token goes null is environment-specific — most likely cookie truncation, Redis ticket eviction, or async context loss in a Polly retry.

  The AUTH DUMP will tell us which one. If it never triggers post-deployment, the cache poisoning fix is doing its job. Keep an eye on the logs for a few more days and let us know if the dump shows up.

  Thanks.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel