It seems I still had to clear my local NuGet cache to resolve the warnings.
When building our backend API we still get warnings for each project. Maybe related to the Cloudflare issue?
MyCompany.DbMigrator.csproj : warning NU1900: Error occurred while getting package vulnerability data: Unable to load the service index for source https://nuget.abp.io/aa3334f2-xxxx-xxxx-xxxx-xxxxxxxxxxxxx/v3/index.json.
MyCompany.Application.csproj : warning NU1900: Error occurred while getting package vulnerability data: Unable to load the service index for source https://nuget.abp.io/aa3334f2-xxxx-xxxx-xxxx-xxxxxxxxxxxxx/v3/index.json.
...
I got the same issue
While investigating the bundling I saw a lot of logs related to the bundling. Can you check if this is correct, since I see a lot of the same files getting bundled and minified? Or are these from multiple requests? Also, notice the timestamps.
https://cdn.fuzed.app/share/abp/main-http-api-6f5c75958b.log
Thanks!
The following exception occurs when our API is running for a few days in production. Logged out users are not able to login anymore due to this exception which results in a 500 error page instead of the login page.
We're hosting the API as a published release build in a linux docker container. It seems that the BundleManager still watches for file changes? Is this correct behaviour in a release build? Restarting the API container only fixes the issue temporary.
2024-11-05 09:21:35Z [ERR] Connection id ""0HN7TAA59SCFI"", Request id ""0HN7TAA59SCFI:00000003"": An unhandled exception was thrown by the application.
System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
at System.IO.FileSystemWatcher.StartRaisingEvents()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
at Microsoft.Extensions.FileProviders.CompositeFileProvider.Watch(String pattern)
at Volo.Abp.AspNetCore.VirtualFileSystem.WebContentFileProvider.Watch(String filter)
at Microsoft.Extensions.FileProviders.CompositeFileProvider.Watch(String pattern)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.WatchChanges(BundleCacheItem cacheValue, List`1 files, String bundleRelativePath)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.<>c__DisplayClass17_0.<AddToBundleCache>b__0()
at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleCache.GetOrAdd(String bundleName, Func`1 factory)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.AddToBundleCache(String bundleName, IBundler bundler, List`1 bundleFiles)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.GetBundleFilesAsync(BundleConfigurationCollection bundles, String bundleName, IBundler bundler)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundleManager.GetStyleBundleFilesAsync(String bundleName)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpTagHelperStyleService.GetBundleFilesAsync(String bundleName)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpTagHelperResourceService.ProcessAsync(ViewContext viewContext, TagHelper tagHelper, TagHelperContext context, TagHelperOutput output, List`1 bundleItems, String bundleName)
at Volo.Abp.AspNetCore.Mvc.UI.Bundling.TagHelpers.AbpBundleTagHelperService`2.ProcessAsync(TagHelperContext context, TagHelperOutput output)
at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.<RunAsync>g__Awaited|0_0(Task task, TagHelperExecutionContext executionContext, Int32 i, Int32 count)
at AspNetCoreGeneratedDocument.Themes_Default_Layouts_Account.<>c__DisplayClass18_0.<<ExecuteAsync>b__0>d.MoveNext() in /src/modules/Fuzed.DefaultTheme/src/Fuzed.AspNetCore.Mvc.UI.Theme.Default/Themes/Default/Layouts/Account.cshtml:line 41
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperExecutionContext.SetOutputContentAsync()
at AspNetCoreGeneratedDocument.Themes_Default_Layouts_Account.ExecuteAsync() in /src/modules/Fuzed.DefaultTheme/src/Fuzed.AspNetCore.Mvc.UI.Theme.Default/Themes/Default/Layouts/Account.cshtml:line 26
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderLayoutAsync(ViewContext context, ViewBufferTextWriter bodyWriter)
at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|28_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Volo.Abp.AspNetCore.Serilog.AbpSerilogMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Volo.Abp.AspNetCore.Auditing.AbpAuditingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Volo.Abp.AspNetCore.Auditing.AbpAuditingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Volo.Abp.AspNetCore.Security.Claims.AbpDynamicClaimsMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Volo.Abp.AspNetCore.Uow.AbpUnitOfWorkMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Volo.Abp.AspNetCore.ExceptionHandling.AbpExceptionHandlingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Volo.Abp.AspNetCore.ExceptionHandling.AbpExceptionHandlingMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Volo.Abp.AspNetCore.MultiTenancy.MultiTenancyMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Builder.ApplicationBuilderAbpOpenIddictMiddlewareExtension.<>c__DisplayClass0_0.<<UseAbpOpenIddictValidation>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Volo.Abp.AspNetCore.Security.AbpSecurityHeadersMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Fuzed.FuzedHttpApiHostModule.<>c.<<OnApplicationInitialization>b__1_0>d.MoveNext() in /src/src/Fuzed.HttpApi.Host/FuzedHttpApiHostModule.cs:line 157
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.RequestLocalization.AbpRequestLocalizationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next)
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.InterfaceMiddlewareBinder.<>c__DisplayClass2_0.<<CreateMiddleware>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Can you share this project or make a simple project?
I can't share our source code. With the test project I tried to create, I coudn't reproduce the same warnings..
Yes, we still can reproduce the issue in our application without SetMinThreads. When the Redis cache is cleared or expired, if we then request a web page, which triggers a lot of API requests (due loading image thumbnails), then the Redis timeout warnings are logged.
Hi @maliming, I've tried to create a test app, but I couldn't reproduce the same warnings we got from the Redis timeouts. We monitored the thread pool usage:
dotnet-counters monitor --counters System.Runtime -n Fuzed.HttpApi.Host
In the end we ended up updating the min worker threads in our app:
ThreadPool.GetMinThreads(out int minWorkerThreads, out int minCompletionPortThreads);
ThreadPool.SetMinThreads(100, minCompletionPortThreads);
Thanks!
hi @s.beckers
Can you share your test project?
liming.ma@volosoft.com
Thanks.
Hi @maliming, I'll try to create a test project and send it to you.