Un-responsive Angular Application with a Black screen #7884 | Support

sanobarm@cloudassert.com created 2 months ago

ABP Framework version: v8.0.1
UI Type: Angular
Database System: EF Core (SQL Server)
Tiered (for MVC) or Auth Server Separated (for Angular): Auth Server Separated
Exception message and full stack trace:Failed to validate the token. Microsoft.IdentityModel.Tokens.SecurityTokenExpiredException: IDX10223: Lifetime validation failed. The token is expired. ValidTo (UTC): '7/31/2024 11:58:08 AM', Current time (UTC): '9/10/2024 12:53:59 PM'. at Microsoft.IdentityModel.Tokens.Validators.ValidateLifetime(Nullable1 notBefore, Nullable1 expires, SecurityToken securityToken, TokenValidationParameters validationParameters) at Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ValidateTokenPayloadAsync(JsonWebToken jsonWebToken, TokenValidationParameters validationParameters, BaseConfiguration configuration) at Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ValidateJWSAsync(JsonWebToken jsonWebToken, TokenValidationParameters validationParameters, BaseConfiguration configuration) 2024-09-10 12:53:59.835 +00:00 [INF] Bearer was not authenticated. Failure message: IDX10223: Lifetime validation failed. The token is expired. ValidTo (UTC): '7/31/2024 11:58:08 AM', Current time (UTC): '9/10/2024 12:53:59 PM'. 2024-09-10 12:53:59.841 +00:00 [INF] Executing endpoint 'Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController.GetAsync (Volo.Abp.AspNetCore.Mvc)' 2024-09-10 12:53:59.841 +00:00 [INF] Route matched with {area = "abp", action = "Get", controller = "AbpApplicationConfiguration", page = ""}. Executing controller action with signature System.Threading.Tasks.Task`1[Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationDto] GetAsync(Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ApplicationConfigurationRequestOptions) on controller Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.AbpApplicationConfigurationController (Volo.Abp.AspNetCore.Mvc).
Steps to reproduce the issue: Yesterday all of a sudden there was a complete black screen displayed on the Production Portal. Refreshing the page did not help, however, restarting the API helped and Users were able to login to the application again.
Why do we see a back screen during this call - ** Request starting HTTP/1.1 GET https://api.xxxxx.com/api/abp/application-configuration?includeLocalizationResources=false - null null**
How can we navigate to Login page when the application does not respond with a black screen. Please refer the screen shot.
A lot of users were unable to access the application for almost 2 hours.

As this is a Production issue, I am requesting immediate assistance from you.

Thanks.

9 Answer(s)

0

maliming created 2 months ago
Support Team Fullstack Developer

hi

Can you share full logs when this happens?

liming.ma@volosoft.com

Thanks.
0

sanobarm@cloudassert.com created 2 months ago

Please download the Logs from here - sent file through mail.

maliming created 2 months ago

Support Team Fullstack Developer

I can see from the log that this API is blocking requests. However, the log level is Information and no more info can be obtained. You can change the log level to Debug so that we can get more information in next time.

And is your Redis work?

IDX10223: Lifetime validation failed. The token is expired.

This is a normal log if your access token is expired.

When this happens:

Is the database in a normal state?
Is the Redis in a normal state?
What about the server's memory and CPU?

We can only change the log level and try to reproduce the problem.

There are not many clues at the moment.

.MinimumLevel.Debug()

public class Program
{
    public async static Task<int> Main(string[] args)
    {
        IdentityModelEventSource.ShowPII = true;

        IdentityModelEventSource.Logger.LogLevel = EventLevel.Verbose;
        var wilsonTextLogger = new TextWriterEventListener("Logs/identitymodel.txt");
        wilsonTextLogger.EnableEvents(IdentityModelEventSource.Logger, EventLevel.Verbose);

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
            .Enrich.FromLogContext()
            .WriteTo.Async(c => c.File("Logs/logs.txt"))
            .CreateLogger();

0

sanobarm@cloudassert.com created 2 months ago

How do you say that the API is blocking requests? From which log can you identify that?

maliming created 2 months ago

Support Team Fullstack Developer

You can see the requests are taking too much time, and return 499

The request must be blocked at some step.

Request finished HTTP/1.1 GET https://api..com/api/app/notifications/get-user-notification - 499 null null 59171973.8186ms

Request finished HTTP/1.1 GET https://api..com/api/abp/application-configuration?includeLocalizationResources=false - 499 null null 59146523.8819ms

Request finished HTTP/1.1 GET https://api..com/api/app/dashboard/recent-newsletters?count=20 - 499 null null 59170152.7414ms

You can change the log level to .MinimumLevel.Debug() so we can get more information next time when this problem occurs again.


public class Program
{
    public async static Task Main(string[] args)
    {
        IdentityModelEventSource.ShowPII = true;

        IdentityModelEventSource.Logger.LogLevel = EventLevel.Verbose;
        var wilsonTextLogger = new TextWriterEventListener("Logs/identitymodel.txt");
        wilsonTextLogger.EnableEvents(IdentityModelEventSource.Logger, EventLevel.Verbose);

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .MinimumLevel.Override("Microsoft.EntityFrameworkCore", LogEventLevel.Warning)
            .Enrich.FromLogContext()
            .WriteTo.Async(c => c.File("Logs/logs.txt"))
            .CreateLogger();

oncalldev@cloudassert.com created one month ago

Hi maliming, As per your instructions added the above lines of code to capture detailed logs. However, after few days of deployment of this code, the application completely crashed and failed to re-start. We looked at logs and we found nothing from the Logs. We thought it was problem with the Azure Platform and raised a ticket. Upon investigating a looking at the VM logs of the machine where the app was hosted the following error was registered.

<Event><System><Provider Name=".NET Runtime"/><EventID>1026</EventID><Level>1</Level><Task>0</Task><Keywords>Keywords</Keywords><TimeCreated SystemTime="2024-10-16T10:33:03Z"/><EventRecordID>78189937</EventRecordID><Channel>Application</Channel><Computer>WN1MDWK0001E7</Computer><Security/></System><EventData><Data>
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options,
 Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
   at Microsoft.IdentityModel.Logging.TextWriterEventListener..ctor(String filePath)
   at WDN.HealthySmiles.Program.Main(String[] args) in D:\a\1\s\aspnet-core\src\WDN.HealthySmiles.HttpApi.Host\Program.cs:line 22
   at WDN.HealthySmiles.Program.&lt;Main&gt;(String[] args)&lt;/Data&gt;&lt;/EventData&gt;&lt;/Event&gt;

The application crashed while creating the Logs/identitymodel.txt file. We had to face severe backslash from the Customer as the portal went down and failed to restart, it took two weeks for us to figure out the root cause and roll out a release. Meanwhile to serve our customers and keep application up, we had to scale up the instance count from 1 to 2. Which had doubled our operation cost.

0

maliming created one month ago
Support Team Fullstack Developer

hi

Sorry for that, Maybe the app needs permission to create new txt file

When you get Logs/logs.txt and Logs/identitymodel.txt please share them with me, and I will check them out.

Thanks.
0

oncalldev@cloudassert.com created one month ago

Hi maliming, The app has permissions to create the file, the file got created, however all of a sudden it crashed and never restarted. I have shared the logs.txt file. But due to large file size, cannot able to get the identitymodel.txt file https://drive.google.com/file/d/1RK2RKzZ1Dafc0Z9jhiZjCmB2sEGv7sZe/view?usp=sharing
0
maliming created one month ago
Support Team Fullstack Developer
hi

I checked the logs and it doesn't have any particular problem.

IDX10223: Lifetime validation failed. The token is expired. This is a normal log because your access_token has expired. There are also several error logs, which should be the browser/front end canceling the request.

Your problem is that the application can't respond. This situation is not like a code problem but more like a problem with the server. We can't troubleshoot the problem through the application logs.

I still have the following questions:

Can you reproduce the problem in some conditions?

Is the database normal when the server doesn't respond?

What about the server's memory and CPU when the server doesn't respond?

You should be able to view the history of performance indicators such as CPU and memory in Auzre.

Finally, you can consider changing the running environment for testing.

Sorry, I don't have much advice.

Have an answer to this question? Log in and write your answer.