Activities of "s.beckers"

Hi

I'll try that.

Thanks!

While investigating the bundling I saw a lot of logs related to the bundling. Can you check if this is correct, since I see a lot of the same files getting bundled and minified? Or are these from multiple requests? Also, notice the timestamps.

https://cdn.fuzed.app/share/abp/main-http-api-6f5c75958b.log

Thanks!

• ABP Framework version: v8.1.4
• UI Type: Angular
• Database System: EF Core (SQL Server)
• Tiered (for MVC) or Auth Server Separated (for Angular): no
• Exception message and full stack trace:

The following exception occurs when our API is running for a few days in production. Logged out users are not able to login anymore due to this exception which results in a 500 error page instead of the login page.

We're hosting the API as a published release build in a linux docker container. It seems that the BundleManager still watches for file changes? Is this correct behaviour in a release build? Restarting the API container only fixes the issue temporary.

2024-11-05 09:21:35Z [ERR] Connection id ""0HN7TAA59SCFI"", Request id ""0HN7TAA59SCFI:00000003"": An unhandled exception was thrown by the application.
System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.FileProviders.CompositeFileProvider.Watch(String pattern)
   at Volo.Abp.AspNetCore.VirtualFileSystem.WebContentFileProvider.Watch(String filter)
   at Microsoft.Extensions.FileProviders.CompositeFileProvider.Watch(String pattern)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.WatchChanges(BundleCacheItem cacheValue, List`1 files, String bundleRelativePath)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.<>c__DisplayClass17_0.<AddToBundleCache>b__0()
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleCache.GetOrAdd(String bundleName, Func`1 factory)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.AddToBundleCache(String bundleName, IBundler bundler, List`1 bundleFiles)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.GetBundleFilesAsync(BundleConfigurationCollection bundles, String bundleName, IBundler bundler)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.GetStyleBundleFilesAsync(String bundleName)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpTagHelperStyleService.GetBundleFilesAsync(String bundleName)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpTagHelperResourceService.ProcessAsync(ViewContext viewContext, TagHelper tagHelper, TagHelperContext context, TagHelperOutput output, List`1 bundleItems, String bundleName)
   at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpBundleTagHelperService`2.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.<RunAsync>g__Awaited|0_0(Task task, TagHelperExecutionContext executionContext, Int32 i, Int32 count)
   at AspNetCoreGeneratedDocument.Themes_Default_Layouts_Account.<>c__DisplayClass18_0.<<ExecuteAsync>b__0>d.MoveNext() in /src/modules/Fuzed.DefaultTheme/src/Fuzed.AspNetCore.Mvc.UI.Theme.Default/Themes/Default/Layouts/Account.cshtml:line 41
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
   at AspNetCoreGeneratedDocument.Themes_Default_Layouts_Account.ExecuteAsync() in /src/modules/Fuzed.DefaultTheme/src/Fuzed.AspNetCore.Mvc.UI.Theme.Default/Themes/Default/Layouts/Account.cshtml:line 26
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderLayoutAsync(ViewContext context, ViewBufferTextWriter bodyWriter)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|28_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Volo.Abp.AspNetCore.Serilog.AbpSerilogMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Volo.Abp.AspNetCore.Auditing.AbpAuditingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Volo.Abp.AspNetCore.Auditing.AbpAuditingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Volo.Abp.AspNetCore.Security.Claims.AbpDynamicClaimsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Volo.Abp.AspNetCore.Uow.AbpUnitOfWorkMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Volo.Abp.AspNetCore.ExceptionHandling.AbpExceptionHandlingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Volo.Abp.AspNetCore.ExceptionHandling.AbpExceptionHandlingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Volo.Abp.AspNetCore.MultiTenancy.MultiTenancyMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.ApplicationBuilderAbpOpenIddictMiddlewareExtension.<>c__DisplayClass0_0.<<UseAbpOpenIddictValidation>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Volo.Abp.AspNetCore.Security.AbpSecurityHeadersMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Fuzed.FuzedHttpApiHostModule.<>c.<<OnApplicationInitialization>b__1_0>d.MoveNext() in /src/src/Fuzed.HttpApi.Host/FuzedHttpApiHostModule.cs:line 157
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.RequestLocalization.AbpRequestLocalizationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Can you share this project or make a simple project?

I can't share our source code. With the test project I tried to create, I coudn't reproduce the same warnings..

Yes, we still can reproduce the issue in our application without SetMinThreads. When the Redis cache is cleared or expired, if we then request a web page, which triggers a lot of API requests (due loading image thumbnails), then the Redis timeout warnings are logged.

Hi @maliming, I've tried to create a test app, but I couldn't reproduce the same warnings we got from the Redis timeouts. We monitored the thread pool usage:

dotnet-counters monitor --counters System.Runtime -n Fuzed.HttpApi.Host

In the end we ended up updating the min worker threads in our app:

ThreadPool.GetMinThreads(out int minWorkerThreads, out int minCompletionPortThreads);
ThreadPool.SetMinThreads(100, minCompletionPortThreads);

Thanks!

hi @s.beckers

Can you share your test project?

liming.ma@volosoft.com

Thanks.

Hi @maliming, I'll try to create a test project and send it to you.

Is this added to the backlog? Just asking since ServiceBot would like to close this question..

I just found this related question: https://abp.io/support/questions/2880/Redis-performance-issues-in-LanguageManagement-DynamicLocalizationResourceContributor

Please try to remove DynamicLocalizationResourceContributor and try again.

That does the job! Removing the DynamicLocalizationResourceContributor from the language management module gives a big performance boost. Which functionality do we lose by removing this?