Open Closed

Getting 499 error for an API #8056


User avatar
0
oncalldev@cloudassert.com created

Getting 499 error when tried to load test an API. Sometimes it is returning 499 error status code, sometimes executed successfully with 200 status code. load testing is done using postman and powershell scripts. Added 2 logs of this issue

  • Why this issue is taking place?

  • How can we address this issue?

  • ABP Framework version: v8.0.1

  • UI Type: Angular

  • Database System: EF Core (SQL Server)

  • Tiered (for MVC) or Auth Server Separated (for Angular): yes

  • Exception message and full stack trace:

  • Steps to reproduce the issue:

Logs:

* 2024-10-08 06:26:20.246 +00:00 [ERR] An error occurred using a transaction. 2024-10-08 06:26:20.257 +00:00 [WRN] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "Your request is not valid!", "details": "The following errors were detected during validation.\r\n - Failed to read the request form. The client has disconnected\r\n", "data": {}, "validationErrors": [ { "message": "Failed to read the request form. The client has disconnected", "members": [ "" ] } ] }

2024-10-08 06:26:20.257 +00:00 [WRN] ModelState is not valid! See ValidationErrors for details. Volo.Abp.Validation.AbpValidationException: ModelState is not valid! See ValidationErrors for details. at Volo.Abp.AspNetCore.Mvc.Validation.ModelStateValidator.Validate(ModelStateDictionary modelState) at Volo.Abp.AspNetCore.Mvc.Validation.AbpValidationActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-10-08 06:26:20.257 +00:00 [WRN] There are 1 validation errors: Failed to read the request form. The client has disconnected ()

2024-10-08 06:26:20.257 +00:00 [INF] Executing ObjectResult, writing value of type 'Volo.Abp.Http.RemoteServiceErrorResponse'. 2024-10-08 06:26:20.263 +00:00 [INF] Executed action xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi) in 3861068.766ms 2024-10-08 06:26:20.264 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi)' 2024-10-08 06:26:20.295 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-08 06:26:20.295 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-08 06:26:20.313 +00:00 [INF] Request finished HTTP/1.1 POST https://......./api/app/xxx-events - 499 null application/json; charset=utf-8 3858506.3645ms

* 2024-10-08 05:00:40.062 +00:00 [WRN] Savepoints are disabled because Multiple Active Result Sets (MARS) is enabled. If 'SaveChanges' fails, then the transaction cannot be automatically rolled back to a known clean state. Instead, the transaction should be rolled back by the application before retrying 'SaveChanges'. See https://go.microsoft.com/fwlink/?linkid=2149338 for more information and examples. To identify the code which triggers this warning, call 'ConfigureWarnings(w => w.Throw(SqlServerEventId.SavepointsDisabledBecauseOfMARS))'. 2024-10-08 05:00:40.063 +00:00 [ERR] An error occurred using a transaction. 2024-10-08 05:00:40.133 +00:00 [ERR] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "An internal error occurred during your request!", "details": null, "data": {}, "validationErrors": null }

2024-10-08 05:00:40.133 +00:00 [ERR] A task was canceled. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.


10 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share the full code of your controller, including the DTO class?

    Thanks

  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming,

    I have shared the controller's full code and the respective DTO which used in creation API(CreateAsync method in controller is used for creating)

    https://drive.google.com/file/d/1ht7AuADgwew5gbqZNQyuin4DnfSJybWe/view?usp=sharing https://drive.google.com/file/d/1JxNj2bM7eEK5wNzP1f6NHCM506-GIWNH/view?usp=sharing

    Thanks

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you try the following solution:

    1. Remove public IFormFile[]? Files { get; set; } and test again.
    2. Use IRemoteStreamContent to replace the IFromFile and test again. https://abp.io/docs/latest/framework/architecture/domain-driven-design/application-services#working-with-streams

    Thanks

  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming,

    1. Replaced the IformFile with IRemoteStreamContent and load testing is in progress, will update here once load testing is over
    2. How do you think this will resolve the issue?

    Thanks

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    1. ok, I am waiting for your good news.
    2. This is just to rule out possible causes.
  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming, Again got the same error. Kindly help this resolved

    please find the logs below: Logs1:

    1. 2024-10-10 06:01:03.830 +00:00 [INF] Request finished HTTP/1.1 GET https://....../health-status - 200 null application/json 12.3292ms 2024-10-10 06:01:07.942 +00:00 [WRN] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "Your request is not valid!", "details": "The following errors were detected during validation.\r\n - Failed to read the request form. The client has disconnected\r\n", "data": {}, "validationErrors": [ { "message": "Failed to read the request form. The client has disconnected", "members": [ "" ] } ] }

    2024-10-10 06:01:07.942 +00:00 [WRN] ModelState is not valid! See ValidationErrors for details. Volo.Abp.Validation.AbpValidationException: ModelState is not valid! See ValidationErrors for details. at Volo.Abp.AspNetCore.Mvc.Validation.ModelStateValidator.Validate(ModelStateDictionary modelState) at Volo.Abp.AspNetCore.Mvc.Validation.AbpValidationActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-10-10 06:01:07.948 +00:00 [WRN] There are 1 validation errors: Failed to read the request form. The client has disconnected ()

    2024-10-10 06:01:07.949 +00:00 [INF] Executing ObjectResult, writing value of type 'Volo.Abp.Http.RemoteServiceErrorResponse'. 2024-10-10 06:01:07.949 +00:00 [INF] Executed action xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi) in 254023.2564ms 2024-10-10 06:01:07.949 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.Events.EventController.CreateAsync (xxx.xxx.HttpApi)' 2024-10-10 06:01:07.968 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-10 06:01:07.970 +00:00 [DBG] Added 0 entity changes to the current audit log 2024-10-10 06:01:08.015 +00:00 [INF] Request finished HTTP/1.1 POST https://....../api/app/xxx-events - 499 null application/json; charset=utf-8 254100.7634ms

    Logs2: 2. 2024-10-10 04:59:32.772 +00:00 [WRN] Savepoints are disabled because Multiple Active Result Sets (MARS) is enabled. If 'SaveChanges' fails, then the transaction cannot be automatically rolled back to a known clean state. Instead, the transaction should be rolled back by the application before retrying 'SaveChanges'. See https://go.microsoft.com/fwlink/?linkid=2149338 for more information and examples. To identify the code which triggers this warning, call 'ConfigureWarnings(w => w.Throw(SqlServerEventId.SavepointsDisabledBecauseOfMARS))'. 2024-10-10 04:59:32.773 +00:00 [ERR] An error occurred using a transaction. 2024-10-10 04:59:32.776 +00:00 [ERR] ---------- RemoteServiceErrorInfo ---------- { "code": null, "message": "An internal error occurred during your request!", "details": null, "data": {}, "validationErrors": null }

    2024-10-10 04:59:32.776 +00:00 [ERR] A task was canceled. System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager) at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.

    Thanks

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The client has disconnected

    A task was canceled.

    The errors look like the browser request was canceled during the file upload process or the http request,

  • User Avatar
    0
    oncalldev@cloudassert.com created

    Hi maliming, The API randomly shuts down and re-starts automatically in production. Because of this there is considerable downtime and our customers are severely affected. We have reported this issue several times and there had not been proper fix, response or support. I would like to seriously escalated to this and would request an immediate response from you. Please refer the logs and provide us a solution in an hour or two max. There are too many 499 errors in the API. Please let us know more on this.

    https://drive.google.com/file/d/downloaded

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The log shows that an error occurred during the process of ReadFormAsync/ReadAsync. It has nothing to do with the abp code. It is related to IIS. You can try switching to outofprocess mode and try again.

    Change the hostingModel="outofprocess" in web.config file.

    https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/web-config?view=aspnetcore-8.0#configuration-of-aspnet-core-module-with-webconfig

    https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/out-of-process-hosting?view=aspnetcore-8.0

    2024-10-16 05:32:43.154 +00:00 [ERR] Unexpected exception in "IISHttpContext.ReadBody".
    System.ObjectDisposedException: Cannot access a disposed object.
    Object name: 'MemoryPool'.
       at System.Buffers.MemoryPoolThrowHelper.ThrowObjectDisposedException(ExceptionArgument argument)
       at System.IO.Pipelines.Pipe.RentMemory(BufferSegment segment, Int32 sizeHint)
       at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint)
       at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
    2024-10-16 05:32:43.156 +00:00 [ERR] Unexpected exception in "IISHttpContext.ReadBody".
    System.ObjectDisposedException: Cannot access a disposed object.
    Object name: 'MemoryPool'.
       at System.Buffers.MemoryPoolThrowHelper.ThrowObjectDisposedException(ExceptionArgument argument)
       at System.IO.Pipelines.Pipe.RentMemory(BufferSegment segment, Int32 sizeHint)
       at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint)
       at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
       at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
       at System.IO.Pipelines.Pipe.GetReadAsyncResult()
       at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
       at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
       at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
       at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList`1 factories)
       at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.TryCreateAsync(ActionContext actionContext, IList`1 factories)
       at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
    --- End of stack trace from previous location ---
       at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
    2024-10-16 05:32:43.168 +00:00 [INF] Executed action xxx.xxx.Controllers.xxxEvents.xxxEventController.CreateAsync (xxx.xxx.HttpApi) in 155926.4606ms
    2024-10-16 05:32:43.168 +00:00 [INF] Executed endpoint 'xxx.xxx.Controllers.xxxEvents.xxxEventController.CreateAsync (xxx.xxx.HttpApi)'
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    If abp code has problems your app will not restart.

Made with ❤️ on ABP v9.1.0-preview. Updated on October 22, 2024, 09:35