Activities of "enisn"

Hi,

Which solution template are you currently using the ABP Suite on?

There can be a race condition while acquiring the lock, hard to determine.

Can you reproduce it with newly created solution? Can you share us a minimal reproduction steps or the minimal application that reproduces the problem so we can check if there is a problem in framework level

It seems the team solved the problem, it'll be inlcuded in the next release

The microservices template in multi-tenant applications using outbox/inbox as it comes by default works properly only if the host and all tenants share the same database. Is that correct?

Yes, it's how it works for now. We'll consider a development on it soon. We put it ti our roadmap but it's tough topic to find a generic solution. You may customize services according to your own requirements for now.

On another note, I think it would be good to consider a simple way to configure the microservices template to support at least one database per tenant, since the SaaS module that implements this template includes that functionality, which is inconsistent with the microservices template.

Saas module is designed much earlier than this feature & micro-service template. Saas module doesn't know about other modules and features. It works with Layerd & Non-Layered templates very well, it also works with even MicroService only if you disable/remove inbox/outbox feature. But in that case there'll be no guarantee all the events will be delivered and executed correctly and applying the same transaction with DB operations and events.

Hi Enis,

How can I sure about if AbpAuthorizationService works on your gRPC request?

You can override it in your application and check if it's triggered to make sure it's executed. It implements Microsoft's default Microsoft.AspNetCore.Authorization.IAuthorizationService and replaces it in the dependency injection. If it's not triggered you'll need to configure it properly by following Microsoft's documentation: https://learn.microsoft.com/en-us/aspnet/core/grpc/authn-and-authz?view=aspnetcore-9.0

using System;
using System.Collections.Generic;
using System.Security.Claims;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Authorization;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Volo.Abp.Authorization;
using Volo.Abp.DependencyInjection;
using Volo.Abp.Security.Claims;

namespace YourComapnyName.YourProjetName.Web;

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IAuthorizationService))]
public class CustomAbpAuthorizationService : AbpAuthorizationService
{
    public CustomAbpAuthorizationService(IAuthorizationPolicyProvider policyProvider, IAuthorizationHandlerProvider handlers, ILogger<DefaultAuthorizationService> logger, IAuthorizationHandlerContextFactory contextFactory, IAuthorizationEvaluator evaluator, IOptions<AuthorizationOptions> options, ICurrentPrincipalAccessor currentPrincipalAccessor, IServiceProvider serviceProvider) : base(policyProvider, handlers, logger, contextFactory, evaluator, options, currentPrincipalAccessor, serviceProvider)
    {
    }

    public override Task<AuthorizationResult> AuthorizeAsync(ClaimsPrincipal user, object? resource, IEnumerable<IAuthorizationRequirement> requirements)
    {
        // debugger here
        Console.WriteLine("AuthorizeAsync is called");
        return base.AuthorizeAsync(user, resource, requirements);
    }

    public override Task<AuthorizationResult> AuthorizeAsync(ClaimsPrincipal user, object? resource, string policyName)
    {
        // debugger here
        Console.WriteLine("AuthorizeAsync is called");
        return base.AuthorizeAsync(user, resource, policyName);
    }
}

It's hard to understand the problem right no with this information. More detailed logs or debugging might be required. You're encountering a complex issue involving authorization failures, database connection problems, and potential caching issues within your application. Let's break down the problem and outline a systematic approach to debugging and resolving it.

Understanding the Problem

1. Authorization Failures:

• The logs clearly show authorization failures:
  • PermissionRequirement: Project.Project.Edit
  • PermissionRequirement: Project.Project.ScheduleEdit
• This indicates that the application is attempting to check permissions, even when requesting application configuration.
• The fact that this happens for api/abp/application-configuration which is normally available to all users is very strange.

2.Database Connection Issues:

• The logs report database connection errors:
  • An error occurred using the connection to database 'sqldb-myapp' on server 'sql-myserver.database.windows.net'.
  • System.Threading.Tasks.TaskCanceledException: A task was canceled.
• These errors suggest that the application is trying to access the database, but the connection is being canceled or failing. This is happening during the feature management store call.
• This is happening during the feature management store call which is called to check feature permissions.

3.Caching Issues:

• System.OperationCanceledException: The operation was canceled.
• at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAsync(String key, CancellationToken token)
• This indicates that a cached request to redis was cancelled. This is likely linked to the database connection issues.

4. Recurring Requests:

• The requests occur at 5-minute intervals, pointing to a potential background task or scheduled operation.

Debugging Steps

1. Identify the Source of the Recurring Requests

• Scheduled Tasks: Check your application for any scheduled tasks or background jobs that might be making these requests. Look for BackgroundJobManager, IBackgroundJob, or similar implementations.
• External Monitoring: Verify that no external monitoring tools or health checks are configured to make these requests.
• Client-Side Code: although unlikely, check any client side javascript code for any calls to the application configuration endpoint. Middleware: Check for custom middleware that might be making these calls.

2. Investigate Authorization Issues:

• Permission Configuration: Review your permission configuration (PermissionDefinitionProvider) to ensure that Project.Project.Edit and Project.Project.ScheduleEdit are correctly defined.
• Feature Checks: The stack trace indicates that FeatureChecker is involved. Investigate your feature definitions and how they are used in conjunction with permissions.
• Authentication: Ensure that your authentication setup is correct. If the application is incorrectly trying to authenticate, it might be triggering permission checks.
• Override AbpApplicationConfigurationAppService: Override the GetAuthConfigAsync method within your application's AbpApplicationConfigurationAppService to log the current user's claims and permissions. This can help you understand why authorization is failing.

3. Diagnose Database Connection Problems:

• Connection String: Double-check your database connection string for accuracy.
• Database Server Health: Monitor the health of your SQL Server instance, including CPU, memory, and disk usage. Connection Limits: Verify that you are not exceeding connection limits on your database server.
• Transient Fault Handling: Implement transient fault handling (e.g., using Polly) to retry database operations after temporary connection failures.
• EntityFramework Core Logging: Enable detailed EF Core logging to get more information about the database queries and connection attempts.

4. Analyze Caching Behavior:

• Redis Health: Check the health of your Redis server.
• Redis Configuration: Verify your Redis configuration, including connection timeouts and other settings.
• Caching Keys: Log the caching keys being used to identify any patterns or issues.
• Disable Caching Temporarily: Temporarily disable Redis caching to see if the database connection issues persist. This will help isolate the problem.

5. Environment Differences:

• Configuration: Compare the configuration files and environment variables between your local, test, and production environments. Pay close attention to database connection strings, caching settings, and feature flags.
• Infrastructure: Investigate any differences in infrastructure between your environments, such as network latency, load balancers, or firewalls.

Code Example (Overriding GetAuthConfigAsync)

using System.Threading.Tasks;
using Microsoft.AspNetCore.Authorization;
using Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations;
using Volo.Abp.DependencyInjection;
using Microsoft.AspNetCore.Http;
using System.Linq;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Volo.Abp.Localization;
using Volo.Abp.MultiTenancy;
using Volo.Abp.Authorization;
using Volo.Abp.Authorization.Permissions;
using Volo.Abp.Users;
using Volo.Abp.Settings;
using Volo.Abp.Features;
using Volo.Abp.Timing;
using Volo.Abp.AspNetCore.Mvc.ApplicationConfigurations.ObjectExtending;

namespace YourProject.Controllers
{
    [Dependency(ReplaceServices = true)]
    [ExposeServices(typeof(IAbpApplicationConfigurationAppService), typeof(AbpApplicationConfigurationAppService))]
    public class CustomAbpApplicationConfigurationAppService : AbpApplicationConfigurationAppService
    {
        private readonly ILogger<CustomAbpApplicationConfigurationAppService> _logger;
        private readonly IHttpContextAccessor _httpContextAccessor;

        public CustomAbpApplicationConfigurationAppService(IOptions<AbpLocalizationOptions> localizationOptions,
            IOptions<AbpMultiTenancyOptions> multiTenancyOptions,
            IServiceProvider serviceProvider,
            IAbpAuthorizationPolicyProvider abpAuthorizationPolicyProvider,
            IPermissionDefinitionManager permissionDefinitionManager,
            DefaultAuthorizationPolicyProvider defaultAuthorizationPolicyProvider,
            IPermissionChecker permissionChecker,
            IAuthorizationService authorizationService,
            ICurrentUser currentUser,
            ISettingProvider settingProvider,
            ISettingDefinitionManager settingDefinitionManager,
            IFeatureDefinitionManager featureDefinitionManager,
            ILanguageProvider languageProvider,
            ITimezoneProvider timezoneProvider,
            IOptions<AbpClockOptions> abpClockOptions,
            ICachedObjectExtensionsDtoService cachedObjectExtensionsDtoService,
            IOptions<AbpApplicationConfigurationOptions> options,
            ILogger<CustomAbpApplicationConfigurationAppService> logger,
            IHttpContextAccessor httpContextAccessor) : base(localizationOptions, multiTenancyOptions, serviceProvider, abpAuthorizationPolicyProvider, permissionDefinitionManager, defaultAuthorizationPolicyProvider, permissionChecker, authorizationService, currentUser, settingProvider, settingDefinitionManager, featureDefinitionManager, languageProvider, timezoneProvider, abpClockOptions, cachedObjectExtensionsDtoService, options)
        {
            _logger = logger;
            _httpContextAccessor = httpContextAccessor;
        }

        protected override async Task<ApplicationAuthConfigurationDto> GetAuthConfigAsync()
        {
            _logger.LogInformation("Custom GetAuthConfigAsync called.");

            var httpContext = _httpContextAccessor.HttpContext;
            if (httpContext?.User?.Claims != null)
            {
                _logger.LogInformation("User Claims:");
                foreach (var claim in httpContext.User.Claims)
                {
                    _logger.LogInformation($"Claim Type: {claim.Type}, Value: {claim.Value}");
                }
            }
            if(httpContext != null){
                _logger.LogInformation("User authenticated: " + httpContext.User.Identity.IsAuthenticated);
            }

            var authConfig = await base.GetAuthConfigAsync();
            return authConfig;
        }
    }
}

Due to the inherent nature of the Inbox/Outbox pattern, all events are processed by a background job. When you separate your tenant's databases, that background job will not check the tenant's segregated database. To address this issue, there are two options available:

Extend/Customize the background worker to ensure it checks both databases and handles the events accordingly.

Operate two separate application instances, each working with a different database.

Hence, Point 1 (Defining a single database per tenant) is feasible with monolithic applications that do not utilize the Inbox/Outbox pattern in conjunction with BackgroundJobs.RabbitMQ.

However, Point 2 (Defining a separate database per tenant and microservice) is not achievable without intervention as outlined above. Whenever the Inbox/Outbox pattern with transaction is employed, it will not function properly with disparate databases.

See the following implementation for understanding better:

• https://github.com/abpframework/abp/blob/336882dac24c88438f7d09fe667f423b189cda31/framework/src/Volo.Abp.EventBus/Volo/Abp/EventBus/Distributed/InboxProcessManager.cs
• https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.EventBus/Volo/Abp/EventBus/Distributed/OutboxSenderManager.cs

Also implementations of IInboxProcessor and IOutboxSender

• https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.EventBus/Volo/Abp/EventBus/Distributed/InboxProcessor.cs
• https://github.com/abpframework/abp/blob/dev/framework/src/Volo.Abp.EventBus/Volo/Abp/EventBus/Distributed/OutboxSender.cs

Can you make sure if AbpAuthorizationService works on your gRPC request?

Hi,

ABP doesn't provide an authentication pipeline for gRPC by default, if you make the regular ASP.NET Core authentication mechanism, ABP policies can work together. In your case it seems ASP.NET Core can't execute its own properties.

Since gRPC requests does not have header like HTTP Requests by default, you may add and resolve them manually.

Hi,

I'll ask the LeptonX team and inform you about his issue soon