Open Closed

Intermittent AbpRemoteCallException Responses when making Synchronous Communication between Microservices #4824


User avatar
0
jeflux created
  • ABP Framework version: v7.0.1
  • UI type: Angular
  • DB provider: EF Core
  • Tiered (MVC) or Identity Server Separated (Angular): yes
  • Exception message and stack trace:
  at Volo.Abp.Http.Client.ClientProxying.ClientProxyBase`1.ThrowExceptionForResponseAsync(HttpResponseMessage response)
   at Volo.Abp.Http.Client.ClientProxying.ClientProxyBase`1.RequestAsync(ClientProxyRequestContext requestContext)
   at Volo.Abp.Http.Client.ClientProxying.ClientProxyBase`1.RequestAsync[T](ClientProxyRequestContext requestContext)
   at Volo.Abp.Http.Client.ClientProxying.ClientProxyBase`1.RequestAsync[T](String methodName, ClientProxyRequestTypeValue arguments)
   at Volo.Abp.Identity.ClientProxies.IdentityUserClientProxy.CreateAsync(IdentityUserCreateDto input)
   at Tryl.InvitationService.Invitations.InvitationsAppService.CreateWithUserAsync(InvitationCreateDto input) in /src/src/InvitationService.Application/Invitations/InvitationsAppServiceExt.cs:line 89
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.GlobalFeatures.GlobalFeatureInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Authorization.AuthorizationInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Validation.ValidationInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Auditing.AuditingInterceptor.ProceedByLoggingAsync(IAbpMethodInvocation invocation, AbpAuditingOptions options, IAuditingHelper auditingHelper, IAuditLogScope auditLogScope)
   at Volo.Abp.Auditing.AuditingInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at Castle.DynamicProxy.AsyncInterceptorBase.ProceedAsynchronous[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo)
   at Volo.Abp.Castle.DynamicProxy.CastleAbpMethodInvocationAdapterWithReturnValue`1.ProceedAsync()
   at Volo.Abp.Uow.UnitOfWorkInterceptor.InterceptAsync(IAbpMethodInvocation invocation)
   at Volo.Abp.Castle.DynamicProxy.CastleAsyncAbpInterceptorAdapter`1.InterceptAsync[TResult](IInvocation invocation, IInvocationProceedInfo proceedInfo, Func`3 proceed)
   at lambda_method2604(Closure, Object)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

Steps to reproduce the issue: We cannot reliably re-produce the issue as it is intermittent. But what we are currently seeing:

There are three services currently involved: Service A, B, and C

During one process, Service A will call Service B to create a user (not the Identity Service). If required, Service B will then call Service C (which is the Identity Service) to create the user record.

We frequently get Service A return a 404 with this response:

---------- RemoteServiceErrorInfo ----------
{
  "code": "NotFound",
  "message": "Not Found",
  "details": null,
  "data": null,
  "validationErrors": null
}

However, in some instances, Service A successfully reaches Service B, but Service B has the same type of 404 error reaching Service C.

It is never predictable. Sometimes I'll get 20 successful Service A requests in a row, then several fail. Sometimes they seem to almost alternate between pass/fail. It took us a while to realize that sometimes it was the call between B & C failing and not A & B.

Our assumption is that we have an incorrect configuration either in our ABP application or maybe in our hosting environment. What we're looking for are recommendations on how we could identify what the heck is happening. (In the above stacktrace, this is logging from Service B failing to reach Service C).

The "Not found" response, does that indicate it found the correct API and just not the correct resource? Or could it not reach the target service at all? Any suggestions are appreciated


3 Answer(s)
  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Can you share a project that can reproduce the problem? thanks, shiwei.liang@volosoft.com

  • User Avatar
    0
    jeflux created

    Can you share a project that can reproduce the problem? thanks, shiwei.liang@volosoft.com

    We can't share the project, but it also can't be reproduced locally. I probably should have mentioned that this is only happening in a hosted environment (Kubernetes cluster on Azure).

    We followed: https://docs.abp.io/en/commercial/latest/startup-templates/microservice/synchronous-interservice-communication

    I'm really just looking for information about what exactly is happening to get this exception? Is the endpoint simply not found? The not found response always comes back immediately, it's never a timeout situation.

  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    I'm really just looking for information about what exactly is happening to get this exception? Is the endpoint simply not found? The not found response always comes back immediately, it's never a timeout situation.

    We never had the problem so I don't know why. Can you share more logs? I will try to help you.

Made with ❤️ on ABP v9.1.0-preview. Updated on December 10, 2024, 06:38