Open Closed

About the kubernetes multipod support(Again) #5064


User avatar
0
cagriceylan created

I open this support question again because of closing the others which is => https://support.abp.io/QA/Questions/4990#answer-3a0aefd0-52c0-7e1b-f4d5-5269a1d63ef7

We tried this suggestion but it did not provide a solution to our problem. When we tried the suggestion, application wrote the following information to redis**. But when we run the application on multipod, we encountered a situation where we could not get a response different pods again. When we receive a token from a pod, the other pod does not accept this token and returns 401(unauthorized) as a response.

How can we solve this problem? I would appreciate if you could share a different suggestion or sample code.

<key id="fd8f3160-6b8f-47ab--3fb052d8f24a" version="1"> <creationDate>2023-05-15T07:53:02.8500225Z</creationDate> <activationDate>2023-05-15T07:53:02.6656766Z</activationDate> <expirationDate>2023-08-13T07:53:02.6656766Z</expirationDate> <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb979382960"> <descriptor> <encryption algorithm="AES_256_CBC" /> <validation algorithm="HMACSHA256" /> <masterKey p4:requiresEncryption="true" xmlns:p4="http://schemas.asp.net/2015/03/dataProtection"> <!-- Warning: the key below is in an unencrypted form. --> <value>VQMZlolN7EQpyoPqtxfw7vcr7Pza0Cbv1EXBSvFEsvN5QT*********SEAzlI+sEcGrlTx45Y5M/7iFP4phg==</value> </masterKey> </descriptor> </descriptor> </key>

Check the docs before asking a question: https://docs.abp.io/en/commercial/latest/ Check the samples, to see the basic tasks: https://docs.abp.io/en/commercial/latest/samples/index The exact solution to your question may have been answered before, please use the search on the homepage.

If you're creating a bug/problem report, please include followings:

  • ABP Framework version: v7.0.0
  • UI type: MVC
  • DB provider: EF Core
  • Tiered (MVC) or Identity Server Separated (Angular): Tiered MVC
  • Exception message and stack trace:
  • Steps to reproduce the issue:"

7 Answer(s)
  • User Avatar
    0
    alper created
    Support Team Director

    do you have a single sign on endpoint? like https://account.abp.io/

  • User Avatar
    0
    cagriceylan created

    Yes, we have. We use corporate sso endpoint https://sso.setur.com.tr but our problem is api side. We get token for example https://localhost/connect/token pod1, when loadbalancer forwards the token we get from here(pod1(kubernetes)) to a different pod(pod2(kubernetes)), we get 401(unauthorized) in api services

    For example We get token pod1 (https://localhost/connect/token) we can use this token https://localhost/api/app/configs bearer pod1_token => we get success When loadbalancer forward another pods; for example; pod2 => https://localhost/api/app/configs bearer pod1_token => we get 401 unauthorized

  • User Avatar
    0
    gterdem created
    Senior .NET Developer

    Can you explain your deployment scenario in more details? You have a tiered application containing AuthServer, MVC app and API.Host.

    • Is it a local staging environment k8s cluster or production environment?
    • Are you running all of them on the same k8s network?
    • Where is the load-balancer located? Is there other load balancers between MVC app and APIs? Is there specific load balancer configurations for it?

    Related to https://support.abp.io/QA/Questions/4990#answer-3a0aefd0-52c0-7e1b-f4d5-5269a1d63ef7 did you configure all of the applications (AuthServer, MVC app and the API.Host) with data-protection?:

    var redis = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]);
    dataProtectionBuilder.PersistKeysToStackExchangeRedis(redis, "MyApp-Protection-Keys");
    

    Can you diagnose the error from getting the access_token to sent API as bearer token?

    It may be related to HTTPS headers not being forwarded.

    If that's the case, you can try adding forwarded headers configuration to your AuthServer module ConfigureServices:

    context.Services.Configure<ForwardedHeadersOptions>(options =>
    {
        options.ForwardedHeaders = ForwardedHeaders.XForwardedFor |
                                   ForwardedHeaders.XForwardedProto |
                                   ForwardedHeaders.XForwardedHost;
        options.KnownNetworks.Clear();
        options.KnownProxies.Clear();
    });
    

    and add:

    public override void OnApplicationInitialization(ApplicationInitializationContext context)
    {
        var app = context.GetApplicationBuilder();
        var env = context.GetEnvironment();
        app.UseForwardedHeaders(); // Before any HTTPS redirection midware
        ...
        
    }
    
  • User Avatar
    0
    cagriceylan created

    We have no separate environment for AuthServer. We use build in openiddict to web application with api.

    Like this:

    There are 4 environment such as; playground, testing, staging and production and whole environments run k8s on the same network each.

    There are only one specific configuration load balancer is sticky session.

    There is a web application with api which runs k8s and we implement this solution on it.

    We tried seperated application(authserver, web, apihost) and there are many other problems with communication eachothers and we changed the architecture this.

  • User Avatar
    0
    gterdem created
    Senior .NET Developer

    If you are using non-tiered mvc application, which is not using jwt for authentication; you can use session just fine. Sticky session is related to your load balancer configuration and ABP does not add any configuration related to it. You need to add that configuration manually like any .net application behind a load balancer. The problem seems to be related to your sticky session configuration.

    However if you use tiered application, it will be using jwt and you'll have to use redis and data protection for consistency between replicated pods.

    I don't think I can replicate your environment. You can share detailed error messages for better diagnose to see if we can help further.

  • User Avatar
    0
    alper created
    Support Team Director

    This issue has been resolved. maybe @cagriceylan can explain how we solved it

  • User Avatar
    0
    cagriceylan created

    Firstly thanks for your online support Mr. Alper & Mr. Halil

    As Mr. @alper said, our problem has been solved, and we have been continuing our tests for a while, and we did not see a problem.

    The steps we take are as follows;

    1- Add the Microsoft.AspNetCore.DataProtection.StackExchangeRedis package to Web project. 2- Redis is integrated into the web project to following steps.

    • Add this two following methods yourprojectnameWebModule class ConfigurePersistKeysToStackExchangeRedis ve ConfigureRedis
     private void ConfigurePersistKeysToStackExchangeRedis(ServiceConfigurationContext context, IConfiguration configuration)
        {
            ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(configuration["Redis:ConnectionString"]);
            context.Services
                .AddDataProtection()
                .SetApplicationName("loyalty-web-ui")
                .PersistKeysToStackExchangeRedis(redis, "Loyalty.Web.UI-Protection-Keys");
        }
    
    private void ConfigureRedis(ServiceConfigurationContext context, IConfiguration configuration)
        {
            ConnectionMultiplexer multiplexer = ConnectionMultiplexer.Connect(configuration["Redis:ConnectionString"]);
            context.Services.AddSingleton<IConnectionMultiplexer>(multiplexer);
        }
    
    • Also add this code blocks to ConfigureServices method
        ConnectionMultiplexer multiplexer = ConnectionMultiplexer.Connect(configuration["Redis:ConnectionString"]);
    
    context.Services.AddSingleton\<IConnectionMultiplexer>(multiplexer);
    

    3- Run the following command in the web project directory for install Volo.Abp.Caching.StackExchangeRedis

    • C:\Users\username\source\repos\yourprojectname\src\Web
        abp add-package Volo.Abp.Caching.StackExchangeRedis
    
    • After this process, the [DependsOn(typeof(AbpCachingStackExchangeRedisModule))] attribute should be added yourprojectnameWebModule automatically.

    4-

    • Add the GetSigningCertificate method to yourprojectnameWebModule
    private X509Certificate2 GetSigningCertificate(IWebHostEnvironment hostingEnv, IConfiguration configuration)
        {
            var fileName = "authserver.pfx";
            var passPhrase = "32165487-6547-6544-1236-123654789654"; 
            var file = Path.Combine(hostingEnv.ContentRootPath, fileName);
            if (!System.IO.File.Exists(file)) 
            {
                throw new FileNotFoundException($"Signing Certificate couldn't found: {file}"); 
            }
            return new X509Certificate2(file, passPhrase);
        }
    
    • Add this code blocks to PreConfigureServices method in yourprojectnameWebModule
            var hostingEnvironment = context.Services.GetHostingEnvironment();
            var configuration = context.Services.GetConfiguration();
    
            if (!hostingEnvironment.IsDevelopment())
            {
                PreConfigure<AbpOpenIddictAspNetCoreOptions>(options =>
                {
                    options.AddDevelopmentEncryptionAndSigningCertificate = false;
                });
    
                PreConfigure<OpenIddictServerBuilder>(builder =>
                {
                    builder.AddSigningCertificate(GetSigningCertificate(hostingEnvironment, configuration));
                    builder.AddEncryptionCertificate(GetSigningCertificate(hostingEnvironment, configuration));
                    builder.SetIssuer(new Uri(configuration["AuthServer:Authority"]));
                });
            }
    
    • Run the following command in the web project directory
        * dotnet dev-certs https -v -ep ./authserver.pfx -p 32165487-6547-6544-1236-123654789654
    
Made with ❤️ on ABP v9.1.0-preview. Updated on December 05, 2024, 12:19