Open Closed

Concurrency exception when I use PaymentModule with own PaymentRequestGateway #8608

User avatar
0
ageiter created
  • ABP Framework version: v8.2.2
  • UI Type: Blazor Server
  • Database System: EF Core (SQL Server)
  • Tiered (for MVC) or Auth Server Separated (for Angular): no

We use the PaymentModule for payment processing via Stripe. Because we want to store various additional information (e.g. about the invoice) in the payment request, we have derived from the StripePaymentGateway and overwritten the CompleteAsync & HandleWebhookAsync methods, for example.

Now we always have concurrency exceptions because the PaymentModule in the PaymentRequestAppService probably also updates the PaymentRequest at the same time.

We have secured our methods with a semaphore so that the processing of our webhook events is synchronized and they cannot access the PaymentRequest at the same time. I have the code from your StripePaymentRequestGateway and we have overwritten all the corresponding methods there, so no relevant code is executed there.

But what happens in your PaymentRequestAppService is still a black box for us.

  1. could you provide me with the code? Preferably from the whole PaymentModule, if that is not possible, then at least from the PaymentRequestAppService.

  2. how should the problem be solved from your point of view? If you send events to the distributed event bus in the app service, then I won't be able to synchronize this with my own event processing, will I?

Thanks, Adrian

Go to accepted answer
38 Answer(s)
  • User Avatar
    0
    ageiter created

    Here is an excerpt from the log file, I have tried to make as many log messages as possible. But unfortunately I can't see where exactly the exceptions are happening, as this seems to come from your module.

    Due to the character limitation in two parts...

    Part 1:

    2025-01-08 09:49:16.002 +00:00 [INF] PalmaPaymentRequestAppService.CreateAsync, ThreadId='38'
2025-01-08 09:49:16.103 +00:00 [INF] PalmaPaymentRequestAppService.UpdateWithExtraPropertiesAsync: PaymentRequest '9e476206-1f24-2f92-334b-3a17577e5568' updated with extra properties: TenantId='70938969-ef4a-ba91-a4ca-3a16f1add23d', TerminationId='01dfb6ef-fb21-6f9d-8b85-3a16f1af3030', ThreadId='21'
2025-01-08 09:49:16.614 +00:00 [INF] PalmaPaymentRequestAppService.StartAsync, ThreadId='21'
2025-01-08 09:49:17.940 +00:00 [INF] PalmaStripePaymentGateway.StartAsync: Start Stripe request.   PaymentRequest='9e476206-1f24-2f92-334b-3a17577e5568'
2025-01-08 09:49:42.494 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync, paymentGateway='stripe', eventType='customer.subscription.created', ThreadId='39'
2025-01-08 09:49:42.505 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'customer.subscription.created' with id 'evt_1QevtRHPiPVaq9xpEy1e75VL'
2025-01-08 09:49:42.511 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionCreatedAsync: Begin handle Stripe event 'customer.subscription.created' with id 'evt_1QevtRHPiPVaq9xpEy1e75VL'.   Important payload data: SubscriptionId='sub_1QevtLHPiPVaq9xpu4IS2vSf', PaymentRequestId='9e476206-1f24-2f92-334b-3a17577e5568', ThreadId='39'
2025-01-08 09:49:42.653 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync, paymentGateway='stripe', eventType='invoice.finalized', ThreadId='39'
2025-01-08 09:49:42.755 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'invoice.finalized' with id 'evt_1QevtRHPiPVaq9xpOSrJK4Kk'
2025-01-08 09:49:42.777 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync, paymentGateway='stripe', eventType='checkout.session.completed', ThreadId='38'
2025-01-08 09:49:42.787 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionCreatedAsync: End handle Stripe event 'customer.subscription.created'.   Payment request updated with ExternalSubscriptionId='sub_1QevtLHPiPVaq9xpu4IS2vSf', ThreadId='39'
2025-01-08 09:49:42.804 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'checkout.session.completed' with id 'evt_1QevtRHPiPVaq9xp7aJCb7vy'
2025-01-08 09:49:42.819 +00:00 [WRN] PalmaStripePaymentGateway.HandleCheckoutSessionCompletedAsync: Unhandled Stripe event 'checkout.session.completed' with id 'evt_1QevtRHPiPVaq9xp7aJCb7vy'
2025-01-08 09:49:42.814 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: Begin handle Stripe event 'invoice.finalized' with id 'evt_1QevtRHPiPVaq9xpOSrJK4Kk'.   Important payload data: SubscriptionId='sub_1QevtLHPiPVaq9xpu4IS2vSf', PaymentRequestId='9e476206-1f24-2f92-334b-3a17577e5568', InvoiceId='in_1QevtL***', CustomerName='Buyer Paypal', CustomerEmail='buyer.paypal@***', AmountDue='12000', AmountPaid='0', AmountRemaining=='12000', InvoiceUrl='https://invoice.stripe.com/i/a***', ThreadId='39'
2025-01-08 09:49:42.883 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '9e476206-1f24-2f92-334b-3a17577e5568': ExternalCustomerName='Buyer Paypal', ExternalCustomerEmailAddress='buyer.paypal@***', AmountDue='12000', AmountPaid='0', AmountRemaining='12000', InvoiceUrl='https://invoice.stripe.com/i/a***', ThreadId='39'
2025-01-08 09:49:42.894 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: End handle Stripe event 'invoice.finalized'.   Payment request updated with invoice data, ThreadId='39'
2025-01-08 09:49:42.922 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync, paymentGateway='stripe', eventType='invoice.payment_succeeded', ThreadId='38'
2025-01-08 09:49:42.963 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'invoice.payment_succeeded' with id 'evt_1QevtSHPiPVaq9xpu6c9BDJ2'
2025-01-08 09:49:42.974 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: Begin handle Stripe event 'invoice.payment_succeeded' with id 'evt_1QevtSHPiPVaq9xpu6c9BDJ2'.   Important payload data: SubscriptionId='sub_1QevtLHPiPVaq9xpu4IS2vSf', PaymentRequestId='9e476206-1f24-2f92-334b-3a17577e5568', InvoiceId='in_1QevtLHPiPVaq9xpj9VPg1w0', CustomerName='Buyer Paypal', CustomerEmail='buyer.paypal@***', AmountDue='12000', AmountPaid='12000', AmountRemaining=='0', InvoiceUrl='https://invoice.stripe.com/i/a***', ThreadId='38'
2025-01-08 09:49:43.023 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync, paymentGateway='stripe', eventType='customer.subscription.updated', ThreadId='39'
2025-01-08 09:49:43.027 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'customer.subscription.updated' with id 'evt_1QevtSHPiPVaq9xpIZFViU7N'
2025-01-08 09:49:43.219 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '9e476206-1f24-2f92-334b-3a17577e5568': ExternalCustomerName='Buyer Paypal', ExternalCustomerEmailAddress='buyer.paypal@***', AmountDue='12000', AmountPaid='12000', AmountRemaining='0', InvoiceUrl='https://invoice.stripe.com/i/a***', ThreadId='39'
2025-01-08 09:49:43.221 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: End handle Stripe event 'invoice.payment_succeeded'.   Payment request updated with invoice data, ThreadId='39'
  • User Avatar
    0
    ageiter created

    Part 2:

    2025-01-08 09:49:43.229 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionUpdatedAsync: Begin handle Stripe event 'customer.subscription.updated' with id 'evt_1QevtSHPiPVaq9xpIZFViU7N'.   Important payload data: SubscriptionId='sub_1QevtLHPiPVaq9xpu4IS2vSf', PaymentRequestId='9e476206-1f24-2f92-334b-3a17577e5568', PlanId='price_1PuqpAHPiPVaq9xpirMIDMuj', PeriodEndDate='08.01.2026 09:49:35', ThreadId='39'
2025-01-08 09:49:43.240 +00:00 [WRN] There is an entry which is not saved due to concurrency exception:
PaymentRequest {Id: 9e476206-1f24-2f92-334b-3a17577e5568} Modified

2025-01-08 09:49:43.241 +00:00 [WRN] There is an entry which is not saved due to concurrency exception:
PaymentRequest {Id: 9e476206-1f24-2f92-334b-3a17577e5568} Modified

2025-01-08 09:49:43.251 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '9e476206-1f24-2f92-334b-3a17577e5568': PeriodEndDateUtc='08.01.2026 09:49:35', ThreadId='38'
2025-01-08 09:49:43.261 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionUpdatedAsync: End handle Stripe event 'customer.subscription.updated'.   Payment request updated with PeriodEndDateUtc='08.01.2026 09:49:35', ThreadId='38'
2025-01-08 09:49:43.262 +00:00 [ERR] ---------- RemoteServiceErrorInfo ----------
{
  "code": null,
  "message": "The data you have submitted has already changed by another user/client. Please discard the changes you've done and try from the beginning.",
  "details": null,
  "data": null,
  "validationErrors": null
}

2025-01-08 09:49:43.287 +00:00 [ERR] ---------- RemoteServiceErrorInfo ----------
{
  "code": null,
  "message": "The data you have submitted has already changed by another user/client. Please discard the changes you've done and try from the beginning.",
  "details": null,
  "data": null,
  "validationErrors": null
}

2025-01-08 09:49:43.270 +00:00 [ERR] The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
Volo.Abp.Data.AbpDbConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
 ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithRowsAffectedOnlyAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
2025-01-08 09:49:43.291 +00:00 [ERR] The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
Volo.Abp.Data.AbpDbConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
 ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithRowsAffectedOnlyAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
2025-01-08 09:49:44.065 +00:00 [INF] PalmaPaymentRequestAppService.CompleteAsync, paymentGateway='stripe', ThreadId='35'
2025-01-08 09:49:44.263 +00:00 [INF] PalmaStripePaymentGateway.CompleteAsync: Begin complete Stripe request.   PaymentRequest='9e476206-1f24-2f92-334b-3a17577e5568'
2025-01-08 09:49:44.271 +00:00 [INF] PalmaStripePaymentGateway.CompleteAsync: End complete Stripe request.   PaymentRequest='9e476206-1f24-2f92-334b-3a17577e5568' updated with currency & state, ThreadId='35'
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Send an email to liming.ma@volosoft.com

    I will share the source code of PaymentRequestAppService, then you can troubleshot the problem.

    Thanks.

  • User Avatar
    0
    ageiter created

    Hi, thanks for sending me the following files:

    • PaymentRequestAppService.cs
    • PaymentCompletedHandler.cs
    • SubscriptionCanceledHandler.cs
    • SubscriptionCreatedHandler.cs
    • SubscriptionUpdatedHandler.cs

    Unfortunately I could not find the cause in these files...

    Where else is an update made on the PaymentRequest?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share full logs.txt?

    See https://abp.io/support/questions/8622/How-to-enable-Debug-logs-for-troubleshoot-problems

  • User Avatar
    0
    ageiter created

    I wanted to activate EnableSensitiveDataLogging according to your instructions.

    But then I get the following error:

    InvalidOperationException: No database provider has been configured for this DbContext. A provider can be configured by overriding the 'DbContext.OnConfiguring' method or by using 'AddDbContext' on the application service provider. If 'AddDbContext' is used, then also ensure that your DbContext type accepts a DbContextOptions<TContext> object in its constructor and passes it to the base constructor for DbContext.

    I have solved the log configuration via appsettings.json. I think the following should be enough, right?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    No need for EnableSensitiveDataLogging

    You can share current logs.

    Thanks.

  • User Avatar
    0
    ageiter created

    I wanted to overwrite the configuration in Azure with the environment variables. This worked for the default MinimumLevel. For the EntityFramework, however, I don't think this works... I have no debug log messages in the file.

    But I will send you the log file that I am currently receiving by e-mail.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    You can set the log level in code.

  • User Avatar
    0
    ageiter created

    Yes, I know, but then a complete deployment is necessary and that takes about 30 minutes... ;-) Do you need the EF messages?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Yes.

  • User Avatar
    0
    ageiter created

    I have sent you the log file by email. Also my implementation of StripePaymentGateway and PaymentRequestAppService.

    Some background information: We need to save additional data from Stripe (invoice, CustomerId for link to the Stripe customer portal, ...). Since the WebHooks are processed in parallel and access the same PaymentRequest object, there are concurrency exceptions. I tried to secure this with a semaphore, but at the same time some of your code is still being executed somewhere. I have tried to switch off all distributed events so that I have it completely under my own control, but I have not yet succeeded.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The exception lines, There are concurrent requests.

    UPDATE [PayPaymentRequests] SET [ConcurrencyStamp] = @p0, [ExtraProperties] = @p1

    The ConcurrencyStamp of PayPaymentRequests has been updated.

  • User Avatar
    0
    ageiter created

    I don't know what you mean exactly. This call alone does not lead to an exception. And I would like to know why this object is updated somewhere else outside of my code.

    I have protected the HandleWebhookAsync method in my PalmaPaymentRequestAppService with a semaphore. This way, the various webhook events should not interfere with each other:

    In addition, the error only occurs later when HandleInvoiceAsync has already been successfully processed.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Please try to set autoSave to turewhile updating the payment request.

    await PaymentRequestRepository.UpdateAsync(paymentRequest, autoSave: true);

    Then share the logs again.

    Thanks.

    1
  • User Avatar
    0
    ageiter created

    Unfortunately the same problem, even if I have now set autoSave: true for all UpdateAsync methods. I have sent you the log file.

  • User Avatar
    0
    ageiter created

    Sorry, but the log file is not complete... Loglevel was wrong again. I'll send another one later.

  • User Avatar
    0
    ageiter created

    I haven't mentioned anything else here, but I hope you have seen the mail with the logs (Log3.txt).

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Please try to add lock for Task<PaymentRequestWithDetailsDto> CompleteAsync(string paymentGateway, Dictionary<string, string> parameters) of PaymentRequestAppService

    Override it and use _webhookSemaphore

     await _webhookSemaphore.WaitAsync();

CompleteAsync

_webhookSemaphore.Release();

    Thanks

  • User Avatar
    0
    ageiter created

    I have now implemented this as you said. But unfortunately that doesn't help either. I have been working on this problem for several days now and with all my observations and tests I can say that the exception does not always occur at the same time. It always tends to occur in the second half of the workflow. And it is definitely triggered from outside my code. Unfortunately, you can't see exactly from where in the exception's StackTrace.

    That's why I suspected that you might still be sending distributed events somewhere, which then access the PaymentRequest. I overwrote everything I knew and I didn't find anything in the classes / handlers you sent me by email. Is there anywhere else in your codebase from the payment module (or in relation to the subscription) that still handles corresponding events?

  • User Avatar
    0
    ageiter created

    Here is an excerpt from the log (I have not included the debug messages and other unnecessary entries so that you can see the process a little better):

    2025-01-14 07:20:54.728 +00:00 [INF] PalmaPaymentRequestAppService.CreateAsync, Begin create payment request.   ThreadId='28'
2025-01-14 07:20:55.284 +00:00 [INF] PalmaPaymentRequestAppService.CreateAsync, End create payment request.   ThreadId='28'
2025-01-14 07:20:55.771 +00:00 [INF] PalmaPaymentRequestAppService.UpdateWithExtraPropertiesAsync: PaymentRequest '26c70db7-96d2-8ddd-164e-3a1775dcaad2' updated with extra properties: TenantId='70938969-ef4a-ba91-a4ca-3a16f1add23d', TerminationId='01dfb6ef-fb21-6f9d-8b85-3a16f1af3030'.   ThreadId='39'
2025-01-14 07:20:56.747 +00:00 [INF] PalmaPaymentRequestAppService.StartAsync, Begin start payment request for payment gateway 'stripe'.   ThreadId='39'
2025-01-14 07:20:58.002 +00:00 [INF] PalmaStripePaymentGateway.StartAsync: Start Stripe request.   PaymentRequest='26c70db7-96d2-8ddd-164e-3a1775dcaad2'
2025-01-14 07:20:58.014 +00:00 [INF] PalmaPaymentRequestAppService.StartAsync, End start payment request for payment gateway 'stripe'.   ThreadId='39'
2025-01-14 07:21:20.862 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'customer.subscription.created' for payment gateway 'stripe'.   ThreadId='35'
2025-01-14 07:21:20.865 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'customer.subscription.updated' for payment gateway 'stripe'.   ThreadId='39'
2025-01-14 07:21:20.862 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'invoice.payment_succeeded' for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:21.066 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'invoice.finalized' for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:21.170 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'customer.subscription.created' with id 'evt_1Qh4R9HPiPVaq9xpthx9Gf80'
2025-01-14 07:21:21.335 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionCreatedAsync: Begin handle Stripe event 'customer.subscription.created' with id 'evt_1Qh4R9HPiPVaq9xpthx9Gf80'.   Important payload data: CustomerId='cus_RaEuASr7DTwiPR', SubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', PaymentRequestId='26c70db7-96d2-8ddd-164e-3a1775dcaad2', ThreadId='35'
2025-01-14 07:21:21.648 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'checkout.session.completed' for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:21.659 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionCreatedAsync: End handle Stripe event 'customer.subscription.created'.   Payment request updated with ExternalSubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', ThreadId='28'
2025-01-14 07:21:21.667 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: End handle webhook event 'customer.subscription.created' for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:21.680 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'customer.subscription.updated' with id 'evt_1Qh4R9HPiPVaq9xpvZLiwRNx'
2025-01-14 07:21:21.738 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionUpdatedAsync: Begin handle Stripe event 'customer.subscription.updated' with id 'evt_1Qh4R9HPiPVaq9xpvZLiwRNx'.   Important payload data: SubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', PaymentRequestId='26c70db7-96d2-8ddd-164e-3a1775dcaad2', PlanId='price_1PuqpAHPiPVaq9xpirMIDMuj', PeriodEndDate='14.01.2026 07:21:17', ThreadId='35'
2025-01-14 07:21:21.931 +00:00 [INF] PalmaStripePaymentGateway.ChangeEditionEndDateForTenantAsync: Change edition end date for tenant '70938969-ef4a-ba91-a4ca-3a16f1add23d' from '14.01.2026 07:21:17' -> '14.01.2026 07:21:17'
2025-01-14 07:21:21.968 +00:00 [INF] PalmaStripePaymentGateway.UpdateTenantExtraPropertiesAsync: Add extra properties to Tenant '70938969-ef4a-ba91-a4ca-3a16f1add23d': ExternalSubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', ExternalPlanId='price_1PuqpAHPiPVaq9xpirMIDMuj', ExternalCustomerId='cus_RaEuASr7DTwiPR', ThreadId='28'
2025-01-14 07:21:21.986 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '26c70db7-96d2-8ddd-164e-3a1775dcaad2': PeriodEndDateUtc='14.01.2026 07:21:17', ThreadId='28'
2025-01-14 07:21:22.038 +00:00 [INF] PalmaStripePaymentGateway.HandleCustomerSubscriptionUpdatedAsync: End handle Stripe event 'customer.subscription.updated'.   Payment request updated with PeriodEndDateUtc='14.01.2026 07:21:17', ThreadId='28'
2025-01-14 07:21:22.046 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: End handle webhook event 'customer.subscription.updated' for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:22.319 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'invoice.payment_succeeded' with id 'evt_1Qh4RAHPiPVaq9xpeIZBYPAJ'
2025-01-14 07:21:22.379 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: Begin handle Stripe event 'invoice.payment_succeeded' with id 'evt_1Qh4RAHPiPVaq9xpeIZBYPAJ'.   Important payload data: SubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', PaymentRequestId='26c70db7-96d2-8ddd-164e-3a1775dcaad2', InvoiceId='in_1Qh4R7HPiPVaq9xpwZhHWp3t', CustomerId='cus_RaEuASr7DTwiPR', ThreadId='30'
2025-01-14 07:21:22.396 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '26c70db7-96d2-8ddd-164e-3a1775dcaad2': ExternalCustomerId='cus_RaEuASr7DTwiPR', AmountDue='12000', AmountPaid='12000', AmountRemaining='0', ThreadId='30'
2025-01-14 07:21:22.417 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: End handle Stripe event 'invoice.payment_succeeded'.   Payment request updated with invoice data, ThreadId='39'
2025-01-14 07:21:22.419 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: End handle webhook event 'invoice.payment_succeeded' for payment gateway 'stripe'.   ThreadId='39'
2025-01-14 07:21:22.426 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'invoice.finalized' with id 'evt_1Qh4RAHPiPVaq9xpmghDqy22'
2025-01-14 07:21:22.427 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: Begin handle Stripe event 'invoice.finalized' with id 'evt_1Qh4RAHPiPVaq9xpmghDqy22'.   Important payload data: SubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', PaymentRequestId='26c70db7-96d2-8ddd-164e-3a1775dcaad2', InvoiceId='in_1Qh4R7HPiPVaq9xpwZhHWp3t', CustomerId='cus_RaEuASr7DTwiPR', ThreadId='30'
2025-01-14 07:21:22.448 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '26c70db7-96d2-8ddd-164e-3a1775dcaad2': ExternalCustomerId='cus_RaEuASr7DTwiPR', AmountDue='12000', AmountPaid='0', AmountRemaining='12000', ThreadId='30'
2025-01-14 07:21:22.495 +00:00 [WRN] There is an entry which is not saved due to concurrency exception:
PaymentRequest {Id: 26c70db7-96d2-8ddd-164e-3a1775dcaad2} Modified

2025-01-14 07:21:22.496 +00:00 [ERR] PalmaPaymentRequestAppService.HandleWebhookAsync: Exception when handling webhook event 'invoice.finalized' for payment gateway 'stripe'.   ExceptionMessage='The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.'
2025-01-14 07:21:22.504 +00:00 [WRN] There is an entry which is not saved due to concurrency exception:
PaymentRequest {Id: 26c70db7-96d2-8ddd-164e-3a1775dcaad2} Modified

2025-01-14 07:21:22.522 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'checkout.session.completed' with id 'evt_1Qh4R9HPiPVaq9xpsGIoml0k'
2025-01-14 07:21:22.525 +00:00 [WRN] PalmaStripePaymentGateway.HandleWebhookAsync: Unhandled Stripe event 'checkout.session.completed' with id 'evt_1Qh4R9HPiPVaq9xpsGIoml0k'
2025-01-14 07:21:22.535 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: End handle webhook event 'checkout.session.completed' for payment gateway 'stripe'.   ThreadId='30'
2025-01-14 07:21:22.574 +00:00 [ERR] ---------- RemoteServiceErrorInfo ----------
{
  "code": null,
  "message": "The data you have submitted has already changed by another user/client. Please discard the changes you've done and try from the beginning.",
  "details": null,
  "data": null,
  "validationErrors": null
}

2025-01-14 07:21:22.581 +00:00 [ERR] The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
Volo.Abp.Data.AbpDbConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
 ---> Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded. See https://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyExceptionAsync(RelationalDataReader reader, Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithRowsAffectedOnlyAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Update.Internal.SqlServerModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Volo.Abp.EntityFrameworkCore.AbpDbContext`1.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Volo.Abp.Uow.UnitOfWork.SaveChangesAsync(CancellationToken cancellationToken)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.SaveChangesAsync(ActionExecutingContext context, IUnitOfWorkManager unitOfWorkManager)
   at Volo.Abp.AspNetCore.Mvc.Uow.AbpUowActionFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
2025-01-14 07:21:24.500 +00:00 [INF] PalmaPaymentRequestAppService.CompleteAsync, Begin complete payment request for payment gateway 'stripe'.   ThreadId='30'
2025-01-14 07:21:24.713 +00:00 [INF] PalmaStripePaymentGateway.CompleteAsync: Begin complete Stripe request.   PaymentRequest='26c70db7-96d2-8ddd-164e-3a1775dcaad2'
2025-01-14 07:21:24.741 +00:00 [INF] PalmaStripePaymentGateway.CompleteAsync: End complete Stripe request.   PaymentRequest='26c70db7-96d2-8ddd-164e-3a1775dcaad2' updated with currency & state, ThreadId='28'
2025-01-14 07:21:24.761 +00:00 [INF] PalmaPaymentRequestAppService.CompleteAsync, End complete payment request for payment gateway 'stripe'.   ThreadId='28'
2025-01-14 07:21:39.400 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: Begin handle webhook event 'invoice.finalized' for payment gateway 'stripe'.   ThreadId='35'
2025-01-14 07:21:39.419 +00:00 [INF] PalmaStripePaymentGateway.HandleWebhookAsync: Receive Stripe event 'invoice.finalized' with id 'evt_1Qh4RAHPiPVaq9xpmghDqy22'
2025-01-14 07:21:39.421 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: Begin handle Stripe event 'invoice.finalized' with id 'evt_1Qh4RAHPiPVaq9xpmghDqy22'.   Important payload data: SubscriptionId='sub_1Qh4R7HPiPVaq9xpdJEx46hQ', PaymentRequestId='26c70db7-96d2-8ddd-164e-3a1775dcaad2', InvoiceId='in_1Qh4R7HPiPVaq9xpwZhHWp3t', CustomerId='cus_RaEuASr7DTwiPR', AmountDue='12000', AmountPaid='0', AmountRemaining=='12000', ThreadId='35'
2025-01-14 07:21:39.440 +00:00 [INF] PalmaStripePaymentGateway.UpdatePaymentRequestWithExtraPropertiesAsync: Add extra properties to PaymentRequest '26c70db7-96d2-8ddd-164e-3a1775dcaad2': ExternalCustomerId='cus_RaEuASr7DTwiPR', AmountDue='12000', AmountPaid='0', AmountRemaining='12000', ThreadId='39'
2025-01-14 07:21:39.460 +00:00 [INF] PalmaStripePaymentGateway.HandleInvoiceAsync: End handle Stripe event 'invoice.finalized'.   Payment request updated with invoice data, ThreadId='39'
2025-01-14 07:21:39.462 +00:00 [INF] PalmaPaymentRequestAppService.HandleWebhookAsync: End handle webhook event 'invoice.finalized' for payment gateway 'stripe'.   ThreadId='39'
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Please search where PaymentRequestRepository.UpdateAsync(paymentRequest) is called. Concurrency exception occurs here.

    Please use try catch and set autoSave: true. You will get the exception stack.

    try
{
    var paymentRequest = await PaymentRequestRepository.GetAsync(paymentRequestId);
    
    // update paymentRequest
    
    await PaymentRequestRepository.UpdateAsync(paymentRequest, autoSave: true);
}
catch(e)
{
    Logger.LogError("PaymentRequestRepository.UpdateAsync", e);
}

    Thanks.

  • User Avatar
    0
    ageiter created

    That's exactly what I want from you. Please search for “PaymentRequestRepository.UpdateAsync” in your code. The error is triggered from outside my code (as I wrote above). And in my case I have protected all places with locks and made corresponding log entries. But unfortunately I have no access to your code from PaymentModule & Co.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Three places.

    Please change your code as below, then share the logs again.

    Thanks

  • User Avatar
    0
    ageiter created

    Thank you for searching for the relevant places. Unfortunately, none of this was new to me...

    I have now made the try-catch block you requested around all PaymentRequestRepository.UpdateAsync calls. But I don't think that will do any help, as I already had it in the PalmaPaymentRequestAppService.

    Does anything else happen between the PaymentRequestAppService and the StripePaymentGateway, other than the requests being passed through?

    I will send you the new log file by e-mail in about 30 minutes...

Assignee
User avatar maliming
Labels
None yet
Made with ❤️ on ABP v9.2.0-preview. Updated on January 16, 2025, 11:47