Open Closed

Un-responsive Angular Application with a Black screen #7884


User avatar
0
sanobarm@cloudassert.com created
  • ABP Framework version: v8.0.1

  • UI Type: Angular

  • Database System: EF Core (SQL Server)

  • Tiered (for MVC) or Auth Server Separated (for Angular): Auth Server Separated

  • Exception message and full stack trace:Failed to validate the token. Microsoft.IdentityModel.Tokens.SecurityTokenExpiredException: IDX10223: Lifetime validation failed. The token is expired. ValidTo (UTC): '7/31/2024 11:58:08 AM', Current time (UTC): '9/10/2024 12:53:59 PM'. at Microsoft.IdentityModel.Tokens.Validators.ValidateLifetime(Nullable1 notBefore, Nullable1 expires, SecurityToken securityToken, TokenValidationParameters validationParameters) at Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ValidateTokenPayloadAsync(JsonWebToken jsonWebToken, TokenValidationParameters validationParameters, BaseConfiguration configuration) at Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ValidateJWSAsync(JsonWebToken jsonWebToken, TokenValidationParameters validationParameters, BaseConfiguration configuration) 2024-09-10 12:53:59.835 +00:00 [INF] Bearer was not authenticated. Failure message: IDX10223: Lifetime validation failed. The token is expired. ValidTo (UTC): '7/31/2024 11:58:08 AM', Current time (UTC): '9/10/2024 12:53:59 PM'. 2024-09-10 12:53:59.841 +00:00 [INF] Executing endpoint 'Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc)' 2024-09-10 12:53:59.841 +00:00 [INF] Route matched with {area = "abp", action = "Get", controller = "AbpApplicationConfiguration", page = ""}. Executing controller action with signature System.Threading.Tasks.Task`1[Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationDto] GetAsync(Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationRequestOptions) on controller Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController (Volo.Abp.AspNetCore.Mvc).

  • Steps to reproduce the issue: Yesterday all of a sudden there was a complete black screen displayed on the Production Portal. Refreshing the page did not help, however, restarting the API helped and Users were able to login to the application again.

  • Why do we see a back screen during this call - ** Request starting HTTP/1.1 GET https://api.xxxxx.com/api/abp/application-configuration?includeLocalizationResources=false - null null**

  • How can we navigate to Login page when the application does not respond with a black screen. Please refer the screen shot.

  • A lot of users were unable to access the application for almost 2 hours.

As this is a Production issue, I am requesting immediate assistance from you.

Thanks.


9 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share full logs when this happens?

    liming.ma@volosoft.com

    Thanks.

  • User Avatar
    0
    sanobarm@cloudassert.com created

    Please download the Logs from here - sent file through mail.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    I can see from the log that this API is blocking requests. However, the log level is Information and no more info can be obtained. You can change the log level to Debug so that we can get more information in next time.

    And is your Redis work?

    IDX10223: Lifetime validation failed. The token is expired.

    This is a normal log if your access token is expired.


    When this happens:

    1. Is the database in a normal state?
    2. Is the Redis in a normal state?
    3. What about the server's memory and CPU?

    We can only change the log level and try to reproduce the problem.

    There are not many clues at the moment.


    .MinimumLevel.Debug()

    public class Program
    {
        public async static Task<int> Main(string[] args)
        {
            IdentityModelEventSource.ShowPII = true;
    
            IdentityModelEventSource.Logger.LogLevel = EventLevel.Verbose;
            var wilsonTextLogger = new TextWriterEventListener("Logs/identitymodel.txt");
            wilsonTextLogger.EnableEvents(IdentityModelEventSource.Logger, EventLevel.Verbose);
    
            Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Debug()
                .MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
                .Enrich.FromLogContext()
                .WriteTo.Async(c => c.File("Logs/logs.txt"))
                .CreateLogger();
    
  • User Avatar
    0
    sanobarm@cloudassert.com created

    How do you say that the API is blocking requests? From which log can you identify that?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    You can see the requests are taking too much time, and return 499

    The request must be blocked at some step.

    Request finished HTTP/1.1 GET https://api..com/api/app/notifications/get-user-notification - 499 null null 59171973.8186ms
    
    Request finished HTTP/1.1 GET https://api..com/api/abp/application-configuration?includeLocalizationResources=false - 499 null null 59146523.8819ms
    
    Request finished HTTP/1.1 GET https://api..com/api/app/dashboard/recent-newsletters?count=20 - 499 null null 59170152.7414ms
    

    You can change the log level to .MinimumLevel.Debug() so we can get more information next time when this problem occurs again.

    
    public class Program
    {
        public async static Task Main(string[] args)
        {
            IdentityModelEventSource.ShowPII = true;
    
            IdentityModelEventSource.Logger.LogLevel = EventLevel.Verbose;
            var wilsonTextLogger = new TextWriterEventListener("Logs/identitymodel.txt");
            wilsonTextLogger.EnableEvents(IdentityModelEventSource.Logger, EventLevel.Verbose);
    
            Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Debug()
                .MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
                .Enrich.FromLogContext()
                .WriteTo.Async(c => c.File("Logs/logs.txt"))
                .CreateLogger();
    
  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming, As per your instructions added the above lines of code to capture detailed logs. However, after few days of deployment of this code, the application completely crashed and failed to re-start. We looked at logs and we found nothing from the Logs. We thought it was problem with the Azure Platform and raised a ticket. Upon investigating a looking at the VM logs of the machine where the app was hosted the following error was registered.

    <Event><System><Provider Name=".NET Runtime"/><EventID>1026</EventID><Level>1</Level><Task>0</Task><Keywords>Keywords</Keywords><TimeCreated SystemTime="2024-10-16T10:33:03Z"/><EventRecordID>78189937</EventRecordID><Channel>Application</Channel><Computer>WN1MDWK0001E7</Computer><Security/></System><EventData><Data>
       at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
       at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options,
     Int64 preallocationSize, Nullable`1 unixCreateMode)
       at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
       at Microsoft.IdentityModel.Logging.TextWriterEventListener..ctor(String filePath)
       at WDN.HealthySmiles.Program.Main(String[] args) in D:\a\1\s\aspnet-core\src\WDN.HealthySmiles.HttpApi.Host\Program.cs:line 22
       at WDN.HealthySmiles.Program.&lt;Main&gt;(String[] args)&lt;/Data&gt;&lt;/EventData&gt;&lt;/Event&gt;
    

    The application crashed while creating the Logs/identitymodel.txt file. We had to face severe backslash from the Customer as the portal went down and failed to restart, it took two weeks for us to figure out the root cause and roll out a release. Meanwhile to serve our customers and keep application up, we had to scale up the instance count from 1 to 2. Which had doubled our operation cost.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Sorry for that, Maybe the app needs permission to create new txt file

    When you get Logs/logs.txt and Logs/identitymodel.txt please share them with me, and I will check them out.

    Thanks.

  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming, The app has permissions to create the file, the file got created, however all of a sudden it crashed and never restarted. I have shared the logs.txt file. But due to large file size, cannot able to get the identitymodel.txt file https://drive.google.com/file/d/1RK2RKzZ1Dafc0Z9jhiZjCmB2sEGv7sZe/view?usp=sharing

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    I checked the logs and it doesn't have any particular problem.

    IDX10223: Lifetime validation failed. The token is expired. This is a normal log because your access_token has expired. There are also several error logs, which should be the browser/front end canceling the request.

    Your problem is that the application can't respond. This situation is not like a code problem but more like a problem with the server. We can't troubleshoot the problem through the application logs.

    I still have the following questions:

    1. Can you reproduce the problem in some conditions?
    2. Is the database normal when the server doesn't respond?
    3. What about the server's memory and CPU when the server doesn't respond?

    You should be able to view the history of performance indicators such as CPU and memory in Auzre.

    Finally, you can consider changing the running environment for testing.

    Sorry, I don't have much advice.

Made with ❤️ on ABP v9.1.0-preview. Updated on November 11, 2024, 11:11