Activities of "jkrause"

Hi liangshiwei,

Sometimes rubber ducking works in mysterious ways.

We found out on Azure that we are using Deployment slots, so what happened is that after a successful deployment "Slot B" is receiving 0% of traffic and "Slot A" is getting 100% of traffic. However, B is still running, meaning that the BackgroundWorker is also still running.

This seems like a very plausible explanation of how certain data was being 'hijacked' by a ghost process. We don't know why the messages are immediately flagged as abandoned by that instance, we cannot access its logging, but at the same time we kinda don't care. So we now deleted that deployment slot and will monitor the table again, but we're confident this should solve our mystery!

Thanks for your patience and time to troubleshoot with us.

Hi,

Only BackgroundJobWorker is operating the AbpBackgroundJobs table https://github.com/abpframework/abp/blob/rel-8.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L42-L112

It will try to retry to execute the Job If an exception occurs. When the max retry count is reached, it will abandon the job without logging the exception https://github.com/abpframework/abp/blob/rel-8.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L82

I understand how the 'normal' process works, but what is happening is abnormal behavior. The reason I am mentioning that the exception is not being logged is because we specifically mentioned that we are using our own worker that has explicit logging which is not showing up. If the Abandoning were being done by our worker, it would have produced logging, it is not.

There is no other process running on the web server (app service) other than our ABP application (w3wp.exe).

The combination of RetryCount = 1 and IsAbandoned = 1 also indicates that the normal behavior is not being triggered, because otherwise it would have retried the job, but it does not. "It" immediately concludes that the job is unprocessable, which is indicated by the fact that those 100 jobs never reached the worker.

What modification can we make to ABP plumbing to see which process/code is sending the update statement?

We already added the MachineName to JobInfo to ensure that the production app service is the one adding the messages to the queue. We'd love to add more information to ExtraProperties, but while the Entity has this property, the Dto does not, so it won't be persisted after mapping.

We now have 7 unprocessed jobs in the table, setting them to IsAbandoned = 0, the logs show that our worker is immediately processing them and the table is empty. So 'something' is preventing from those messages ever reaching our worker.

@liangshiwei thank you for your initial analysis. Is there any underlying explanation why this particular project is affected by it? We started a complete new project using ABP Suite 5.2 and even without the DisplayAttribute this project is fast and responsive.

@liangshiwei, what mail address can I forward the details to? Thanks.

@albert

You misread my comment completely. I am showing you the before and after from using RC and then Stable. It changes the make-up of the csproj files by adding modules directly as NuGet packages instead of project references.

Adding a module to a freshly generated website using ABP Suite v5.0.0 (Stable) yields the following error:

Error occurred while adding the module "Acme.Abp.Crm" to the solution "AcmeProject"...

Could not find a class derived from AbpModule in the project '<path-to-project>/<project-name>.csproj'.

This while there is a fully functional project and the AcmeAbpModule.cs file is located in the root of that location and is inheriting the AbpModule class, as it is a completely unmodified new project created by ABP Suite.

In addition, despite the error, the module is created in the .\modules\Acme.Abo.Crm location, and the csproj file is modified to load it. However, upon inspecting the file, it assumes that the project can be referenced as a NuGet package with version 5.0.0 as the assumed version?

In ABP Suite 5.0.0-rc.2 <ProjectReference Include="..\..\modules\Acme.Abp.Crm\src\Acme.Abp.Crm.Web\Acme.Abp.Crm.Web.csproj" />

In ABP Suite 5.0.0 (Stable) <PackageReference Include="Acme.Abp.Crm.Web" Version="5.0.0" />

This breaks all the generated code in the project and logically it will not compile anymore. What am I missing here? Maybe it's time for a reboot..

Hi @albert,

After updating Suite to RC2 I get this error:

[12:00:50 ERR] Connection id "0HMDR0CIU98V0", Request id "0HMDR0CIU98V0:00000002": An unhandled exception was thrown by the application. Volo.Abp.AbpException: Could not find the bundle file '/libs/abp/core/abp.css' for the bundle 'Lepton.Global'!

Seems the wwwroot folder is missing in the %USERPROFILE%\.dotnet\tools\.store\volo.abp.suite\5.0.0-rc.2\volo.abp.suite\5.0.0-rc.2\tools\net6.0\any folder? When I created that folder and copied the files from the %USERPROFILE%\.dotnet\tools\.store\volo.abp.suite\5.0.0-rc.2\volo.abp.suite\5.0.0-rc.2\staticwebassets\ to that location, it started up and was usable again.

So either the staticwebassets are not being found, or the wwwroot is missing in the publish? (assumptions)

Thanks.

maliming,

I messed up and totally forgot to edit the NuGet.config to remove a certain detail from the feed.

Apologies for this oversight, would it be possible/required to invalidate that key and obtain a new one?

Sorry.

I have a simple project, that is roughly the same size in ZIP as our actual project, but whatever.

You can find the source here: Deleted

The application is setup to use LocalDb 2019 (came with VS2022 installer). After giving the default admin user permissions to see the module, click on the Companies menu item and you will get the error. Saving a new Company gives the same error.

Getting just this simple "clean" application to compile and start was frustration, as the generated files did not compile and some ConsoleTest project refused to build, so I removed that from the solution.

(The ctor generated for Company resulted in an illegal order of parameters in combination with nullable types, so I had to fix that manually)

I will try to reproduce this in a "barren" project, but it might very well not happen, which is exactly why I am asking troubleshooting for our actual project. There are no code samples available anywhere about creating a custom module that needs to use the User and Organization.

I'll report back if I can extract all the effort I did, not happy with this answer. Our actual project is not that much "bigger" than the starting template with 1 custom module containing half a dozen entities.