Database exceptions while running in Azure, how to handle without EnableRetryOnFailure #7848 | ABP Support | ABP.IO | ABP Support

okains created 12 days ago

ABP Framework version: v8.2.3
UI Type: Blazor Server
Database System: EF Core (SQL Server)
Tiered (for MVC) or Auth Server Separated (for Angular): yes
Exception message and full stack trace:
Steps to reproduce the issue:

Hi,

I have been deploying to Azure Deployment Slots, and I am getting transient db errors in the logs on the Auth project that are causing me to often get 400 Errors returned. The log errors are initially in this format:

2024-09-06 15:20:30.990 +00:00 [ERR] An error occurred using the connection to database 'ESv2-testing' on server 'tcp:hathor-hk.database.windows.net,1433'.

Resulting in this:

Following the suggestion of ChatGPT, I added EnableRetryOnFailure:

Configure<AbpDbContextOptions>(options => { /* The main point to change your DBMS. * See also ESv2DbContextFactory for EF Core tooling. */ options.UseSqlServer(sqlOptions => sqlOptions.EnableRetryOnFailure(5, TimeSpan.FromSeconds(10), null)); });

But it seems this may not be supported in ABP.

I am now getting the following in my logs :

2024-09-06 13:08:13.988 +00:00 [ERR] An exception occurred while iterating over the results of a query for context type 'Volo.Abp.TextTemplateManagement.EntityFrameworkCore.TextTemplateManagementDbContext'. System.InvalidOperationException: The configured execution strategy 'SqlServerRetryingExecutionStrategy' does not support user-initiated transactions. Use the execution strategy returned by 'DbContext.Database.CreateExecutionStrategy()' to execute all the operations in the transaction as a retriable unit. at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.OnFirstExecution()

Is there another approach to handling these errors?

This particular exception that is logged: [ERR] An exception occurred while iterating over the results of a query for context type 'Volo.Abp.TextTemplateManagement.EntityFrameworkCore.TextTemplateManagementDbContext'.

Is this maybe the root cause here? This doesn't happen all of the time though, any idea on what is going on here?

Thanks,

Karim Ainsworth

15 Answer(s)

0

maliming created 10 days ago
Support Team Fullstack Developer

hi

ABP doesn't support the connection-resiliency(EnableRetryOnFailure) feature.
0

okains created 9 days ago

Hi,

Yeah I understand that, question is how to handle this without being able to use EnableRetryOnFailure. Is there a best practice with ABP for this?

Is there any plan on implementing this in a future version?

Thanks,

Karim
0

maliming created 9 days ago
Support Team Fullstack Developer

hi

You can only write code to catch the exception and retry.

EnableRetryOnFailure is not compatible with abp design.
0

okains created 9 days ago

OK but this is happening during the login process, seems to be on the db connection that pulls the OpenIddictApplications record for the BlazorWebAppTiered client.

So I would need to pull the source code for login then and try and handle it in there? Any info on what method I should focus on , or any other ideas on how to best do this?
0

maliming created 9 days ago
Support Team Fullstack Developer

hi

OK but this is happening during the login process, seems to be on the db connection that pulls the OpenIddictApplications record for the BlazorWebAppTiered client.

Can you share full logs?

liming.ma@volosoft.com
0

okains created 8 days ago

Hi,

I have shared the full debug logs of the site load and login attempt. Here is a screenshot of the db error that I believe is causing these 400 errors:

Thanks,

Karim

maliming created 8 days ago

Support Team Fullstack Developer

HTTP 400 is about the Openiddict.

[INF] Client validation failed because 'https://esv2-test.azurewebsites.net/signin-oidc' was not a valid redirect_uri for ESv2_BlazorWebAppTiered.
[INF] The authorization request was rejected because the redirect_uri was invalid: 'https://esv2-test.azurewebsites.net/signin-oidc'.

0

okains created 8 days ago

That client validation failed INF is only present when the DB connection ERR happens. The configuration is fine, it works fine occasionally and I have double and tripled checked Environment Variables and the redirectURLs in the OpenIddictApplications table. Everything is at it should be.

This is why I had said that I think that DB error is happening when trying to read the OpenIddictApplications table, so no data coming back therefore the redirect_uri mismatch. Just a guess but I have been testing this for days and that seems to be the 1 thing that makes a bit of sense. That is why I wanted to retry that db call in the first place.

This is happening on an AppService in Azure with 3 deployment slots, Test / Staging / Production . It seems that only 1 deployment slot will work at a time, the other 2 then throw the ' error using the connection to the database ' error and then redirect uri mismatch. And there are no config changes at all being made.
0

maliming created 8 days ago
Support Team Fullstack Developer

hi

Maybe the index error causes the system to cache the wrong data.

You can try removing the await OpenIdApplicationRepository.GetListAsync(); from the index page and testing again.
0

okains created 6 days ago

Hi,

OK I commented out the // Application = await OpenIdApplicationRepository.GetListAsync(); line,

I am still getting the : [ERR] An error occurred using the connection to database 'ESv2-testing' on server 'tcp:hathor-hk.database.windows.net,1433'. error intermittently.

Any idea of what I can try next on this? This is a pretty critical error, we can deploy our app reliably at this point. Any help would be much appreciated.

Thanks,

Karim
0

maliming created 6 days ago
Support Team Fullstack Developer

Can you share a project so I can reproduce the problem locally?
0

okains created 2 days ago

Hi,

You have access to this repo in GitHub, it is the same repo as before. The problem is though that this doesn't happen locally, it is only showing up when we deploy to Azure. And not all the time.

I just need to figure out a fix or workaround for this in some way so that we can deploy properly. If you have any ideas or can look at the code and see if there is anything obviously wrong with the Auth project that would be great.

Thanks,

Karim
0

maliming created 2 days ago
Support Team Fullstack Developer

Can the problem be reproduced by connecting to the remote database locally?
0

okains created 15 hours ago

Hi,

OK let me try that. It actually worked for about a day, no 400 errors however there was another Auth error on the web app that I created a separate ticket for. But now back to the 400 error on the login page. I think all of these issues are down to this db connection problem we are having.

I will try locally and let you know.

Thanks,

Karim
0

okains created 15 hours ago

OK I made a copy of the ESv2-testing db on Azure, called it ESv2-support. Updated the OpenIddictApplications table and added my localhost urls where needed. Set the db connection locally on Auth / API projects to the ESv2-support db on Azure.

Was able to get to the login page, logged in, then got an auth error but there were no [ERR] entries in the Auth log, so no issues with the connection it seems, though only ran it one time for now.

Can check back on this in a few hours. If any further ideas please let me know.

Thanks for your help,

Karim

Have an answer to this question? Log in and write your answer.