Getting 499 error when tried to load test an API. Sometimes it is returning 499 error status code, sometimes executed successfully with 200 status code. load testing is done using postman and powershell scripts. Added 2 logs of this issue
Why this issue is taking place?
How can we address this issue?
ABP Framework version: v8.0.1
UI Type: Angular
Database System: EF Core (SQL Server)
Tiered (for MVC) or Auth Server Separated (for Angular): yes
Exception message and full stack trace:
Steps to reproduce the issue:
Logs:
* 2024-10-08 06:26:20.246 +00:00 [ERR] An error occurred using a transaction. 2024-10-08 06:26:20.257 +00:00 [WRN] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "Your request is not valid!", "details": "The following errors were detected during validation.\r\n - Failed to read the request form. The client has disconnected\r\n", "data": {}, "validationErrors": [ { "message": "Failed to read the request form. The client has disconnected", "members": [ "" ] } ] }
2024-10-08 06:26:20.257 +00:00 [WRN] ModelState is not valid! See ValidationErrors for details. Volo.Abp.Validation.AbpValidationException: ModelState is not valid! See ValidationErrors for details. at Volo.Abp.AspNetCore.Mvc.Validation.ModelStateValidator.Validate(ModelStateDictionary modelState) at Volo.Abp.AspNetCore.Mvc.Validation.AbpValidationActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-10-08 06:26:20.257 +00:00 [WRN] There are 1 validation errors: Failed to read the request form. The client has disconnected ()
2024-10-08 06:26:20.257 +00:00 [INF] Executing ObjectResult, writing value of type 'Volo.Abp.Http.RemoteServiceErrorResponse'. 2024-10-08 06:26:20.263 +00:00 [INF] Executed action xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi) in 3861068.766ms 2024-10-08 06:26:20.264 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi)' 2024-10-08 06:26:20.295 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-08 06:26:20.295 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-08 06:26:20.313 +00:00 [INF] Request finished HTTP/1.1 POST https://......./api/app/xxx-events - 499 null application/json; charset=utf-8 3858506.3645ms
* 2024-10-08 05:00:40.062 +00:00 [WRN] Savepoints are disabled because Multiple Active Result Sets (MARS) is enabled. If 'SaveChanges' fails, then the transaction cannot be automatically rolled back to a known clean state. Instead, the transaction should be rolled back by the application before retrying 'SaveChanges'. See https://go.microsoft.com/fwlink/?linkid=2149338 for more information and examples. To identify the code which triggers this warning, call 'ConfigureWarnings(w => w.Throw(SqlServerEventId.SavepointsDisabledBecauseOfMARS))'. 2024-10-08 05:00:40.063 +00:00 [ERR] An error occurred using a transaction. 2024-10-08 05:00:40.133 +00:00 [ERR] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "An internal error occurred during your request!", "details": null, "data": {}, "validationErrors": null }
2024-10-08 05:00:40.133 +00:00 [ERR] A task was canceled.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable
1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList
1 entriesToSave, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func
4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken)
at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.
10 Answer(s)
-
0
hi
Can you share the full code of your controller, including the DTO class?
Thanks
-
0
Hi maliming,
I have shared the controller's full code and the respective DTO which used in creation API(CreateAsync method in controller is used for creating)
https://drive.google.com/file/d/1ht7AuADgwew5gbqZNQyuin4DnfSJybWe/view?usp=sharing https://drive.google.com/file/d/1JxNj2bM7eEK5wNzP1f6NHCM506-GIWNH/view?usp=sharing
Thanks
-
0
hi
Can you try the following solution:
- Remove
public IFormFile[]? Files { get; set; }
and test again. - Use
IRemoteStreamContent
to replace theIFromFile
and test again. https://abp.io/docs/latest/framework/architecture/domain-driven-design/application-services#working-with-streams
Thanks
- Remove
-
0
Hi maliming,
- Replaced the IformFile with IRemoteStreamContent and load testing is in progress, will update here once load testing is over
- How do you think this will resolve the issue?
Thanks
-
0
hi
- ok, I am waiting for your good news.
- This is just to rule out possible causes.
-
0
Hi maliming, Again got the same error. Kindly help this resolved
please find the logs below: Logs1:
- 2024-10-10 06:01:03.830 +00:00 [INF] Request finished HTTP/1.1 GET https://....../health-status - 200 null application/json 12.3292ms 2024-10-10 06:01:07.942 +00:00 [WRN] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "Your request is not valid!", "details": "The following errors were detected during validation.\r\n - Failed to read the request form. The client has disconnected\r\n", "data": {}, "validationErrors": [ { "message": "Failed to read the request form. The client has disconnected", "members": [ "" ] } ] }
2024-10-10 06:01:07.942 +00:00 [WRN] ModelState is not valid! See ValidationErrors for details. Volo.Abp.Validation.AbpValidationException: ModelState is not valid! See ValidationErrors for details. at Volo.Abp.AspNetCore.Mvc.Validation.ModelStateValidator.Validate(ModelStateDictionary modelState) at Volo.Abp.AspNetCore.Mvc.Validation.AbpValidationActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-10-10 06:01:07.948 +00:00 [WRN] There are 1 validation errors: Failed to read the request form. The client has disconnected ()
2024-10-10 06:01:07.949 +00:00 [INF] Executing ObjectResult, writing value of type 'Volo.Abp.Http.RemoteServiceErrorResponse'. 2024-10-10 06:01:07.949 +00:00 [INF] Executed action xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi) in 254023.2564ms 2024-10-10 06:01:07.949 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi)' 2024-10-10 06:01:07.968 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-10 06:01:07.970 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-10 06:01:08.015 +00:00 [INF] Request finished HTTP/1.1 POST https://....../api/app/xxx-events - 499 null application/json; charset=utf-8 254100.7634ms
Logs2: 2. 2024-10-10 04:59:32.772 +00:00 [WRN] Savepoints are disabled because Multiple Active Result Sets (MARS) is enabled. If 'SaveChanges' fails, then the transaction cannot be automatically rolled back to a known clean state. Instead, the transaction should be rolled back by the application before retrying 'SaveChanges'. See https://go.microsoft.com/fwlink/?linkid=2149338 for more information and examples. To identify the code which triggers this warning, call 'ConfigureWarnings(w => w.Throw(SqlServerEventId.SavepointsDisabledBecauseOfMARS))'. 2024-10-10 04:59:32.773 +00:00 [ERR] An error occurred using a transaction. 2024-10-10 04:59:32.776 +00:00 [ERR] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "An internal error occurred during your request!", "details": null, "data": {}, "validationErrors": null }
2024-10-10 04:59:32.776 +00:00 [ERR] A task was canceled. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable
1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable
1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList
1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func
4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Thanks
-
0
hi
The client has disconnected
A task was canceled.
The errors look like the browser request was canceled during the file upload process or the http request,
-
0
Hi maliming, The API randomly shuts down and re-starts automatically in production. Because of this there is considerable downtime and our customers are severely affected. We have reported this issue several times and there had not been proper fix, response or support. I would like to seriously escalated to this and would request an immediate response from you. Please refer the logs and provide us a solution in an hour or two max. There are too many 499 errors in the API. Please let us know more on this.
https://drive.google.com/file/d/downloaded
-
0
hi
The log shows that an error occurred during the process of
ReadFormAsync/ReadAsync
. It has nothing to do with the abp code. It is related to IIS. You can try switching tooutofprocess
mode and try again.Change the
hostingModel="outofprocess"
in web.config file.https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/web-config?view=aspnetcore-8.0#configuration-of-aspnet-core-module-with-webconfig
https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/out-of-process-hosting?view=aspnetcore-8.0
2024-10-16 05:32:43.154 +00:00 [ERR] Unexpected exception in "IISHttpContext.ReadBody". System.ObjectDisposedException: Cannot access a disposed object. Object name: 'MemoryPool'. at System.Buffers.MemoryPoolThrowHelper.ThrowObjectDisposedException(ExceptionArgument argument) at System.IO.Pipelines.Pipe.RentMemory(BufferSegment segment, Int32 sizeHint) at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint) at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody() 2024-10-16 05:32:43.156 +00:00 [ERR] Unexpected exception in "IISHttpContext.ReadBody". System.ObjectDisposedException: Cannot access a disposed object. Object name: 'MemoryPool'. at System.Buffers.MemoryPoolThrowHelper.ThrowObjectDisposedException(ExceptionArgument argument) at System.IO.Pipelines.Pipe.RentMemory(BufferSegment segment, Int32 sizeHint) at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint) at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody() at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken) at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken) at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context) at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList`1 factories) at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.TryCreateAsync(ActionContext actionContext, IList`1 factories) at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-10-16 05:32:43.168 +00:00 [INF] Executed action xxx.xxx.Controllers.xxxEvents.xxxEventController.CreateAsync (xxx.xxx.HttpApi) in 155926.4606ms 2024-10-16 05:32:43.168 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.xxxEvents.xxxEventController.CreateAsync (xxx.xxx.HttpApi)'
-
0
If abp code has problems your app will not restart.