Deployment microservices solution in K8S public cloud #8704

carl.hallqvist created 2 months ago

ABP Framework version: v9.X
UI Type: Blazor WASM
Database System: EF Core SQL Server
Tiered (for MVC) or Auth Server Separated (for Angular): yes/no
Exception message and full stack trace:
Steps to reproduce the issue:

In the ABP 7 we successfully made deployments in public cloud (DigitalOcean) using the generated Helm charts. We modified the values.yaml file and configured the domain names, with TLS certificates using Certmanager and Lets Encrypt.

Now, the Helm charts looking totally different and seems to be used for local deployment?

Can you please give short instructions how to deploy on public cloud?

Thanks

berkansasmaz created 2 months ago

Support Team .NET Developer

Hello, first of all, sorry for the late reply, we somehow missed this.

Along with the microservice template, it provides K8s configurations ready for deployment to a local or remote K8s cluster. By default, this configuration uses the Stagging .NET environment. However, you can override this and other values, as you have done before.

I have shared the documents that I think will make it easier for you to do this with you below:

https://abp.io/docs/latest/solution-templates/microservice/helm-charts-and-kubernetes
https://abp.io/docs/latest/studio/kubernetes#abp-studio-working-with-kubernetes

Basically, between the last version of ABP and version 7.*.* the suffix “local” was added to the filenames because all environment variables are configured for local. You need to change them for production. In addition, the feature of building images with a different tag each time when building images has been added so that deployment can be done from scratch each time.

carl.hallqvist created one month ago

Thanks,

Thanks! I almost got it working with minimal changes. But I got this error at the blazer WASM load after using TLS:

berkansasmaz created one month ago

Support Team .NET Developer

Can you confirm whether the site has a valid certificate?

We have a document written about certification in the production environment, if you have not read it, I recommend you to take a look there. See more: https://abp.io/docs/latest/deployment/ssl

You can also find more information about deployment here.

carl.hallqvist created one month ago

Hi,

Yes I can confirm that a valid certificate is present. Attaching screenshot. Thanks.

berkansasmaz created one month ago

Support Team .NET Developer

Can you send the log record of Blazor Server? Also, if your application is tiered, can you send the log record of AuthServer and HttpApiHost?

carl.hallqvist created one month ago

Hi, here it comes:

Blazor Backend (K8S)

2025-02-08T13:00:38.046+01:00[myproject-blazor]      Request starting HTTP/1.1 GET http://staging.mysecretdomain.com/appsettings.json - - -
2025-02-08T13:00:38.046+01:00[myproject-blazor]info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
2025-02-08T13:00:38.046+01:00[myproject-blazor]      Executing endpoint 'Microsoft.AspNetCore.Routing.RouteEndpoint'
2025-02-08T13:00:38.046+01:00[myproject-blazor]info: Microsoft.AspNetCore.StaticAssets.StaticAssetsInvoker[6]
2025-02-08T13:00:38.046+01:00[myproject-blazor]      The file appsettings.json was not modified
2025-02-08T13:00:38.046+01:00[myproject-blazor]info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
2025-02-08T13:00:38.046+01:00[myproject-blazor]      Executed endpoint 'Microsoft.AspNetCore.Routing.RouteEndpoint'
2025-02-08T13:00:38.047+01:00[myproject-blazor]info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-02-08T13:00:38.047+01:00[myproject-blazor]      Request finished HTTP/1.1 GET http://staging.mysecretdomain.com/appsettings.json - 304 - application/json 0.6110ms

AuthServer (K8S)

2025-02-08T13:01:51.535+01:00[myproject-authserver][12:01:51 INF] Request finished HTTP/1.1 GET http://myproject-authserver:80/metrics - 200 null application/openmetrics-text; version=1.0.0; charset=utf-8 14.5306ms
2025-02-08T13:02:06.520+01:00[myproject-authserver][12:02:06 INF] Request starting HTTP/1.1 GET http://myproject-authserver:80/metrics - null null
2025-02-08T13:02:06.525+01:00[myproject-authserver][12:02:06 INF] Executing endpoint 'Prometheus metrics'
2025-02-08T13:02:06.536+01:00[myproject-authserver][12:02:06 INF] Executed endpoint 'Prometheus metrics'
2025-02-08T13:02:06.536+01:00[myproject-authserver][12:02:06 INF] Request finished HTTP/1.1 GET http://myproject-authserver:80/metrics - 200 null application/openmetrics-text; version=1.0.0; charset=utf-8 15.9733ms

Safari developer log

[Error] Inläsningsfel av källkarta (x5)
[Error] Failed to load resource: the server responded with a status of 404 () (bootstrap.min.css.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (bootstrap.bundle.min.js.map;, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (bootstrap.min.css.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (dotnet.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (dotnet.runtime.js.map, line 0)
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
      Loaded ABP modules:
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
      - MyProject.Blazor.Client.MyProjectBlazorClientModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
        - Volo.Abp.Autofac.WebAssembly.AbpAutofacWebAssemblyModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.Autofac.AbpAutofacModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
            - Volo.Abp.Castle.AbpCastleCoreModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.AspNetCore.Components.WebAssembly.AbpAspNetCoreComponentsWebAssemblyModule
......
......
        - MyProject.AdministrationService.MyProjectAdministrationServiceContractsModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
        - Volo.Abp.Identity.Pro.Blazor.Server.WebAssembly.AbpIdentityProBlazorWebAssemblyModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.Identity.Pro.Blazor.AbpIdentityProBlazorModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.SettingManagement.Blazor.WebAssembly.AbpSettingManagementBlazorWebAssemblyModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
            - Volo.Abp.SettingManagement.AbpSettingManagementHttpApiClientModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.Identity.AbpIdentityHttpApiClientModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
        - Volo.Abp.Account.Pro.Admin.Blazor.WebAssembly.AbpAccountAdminBlazorWebAssemblyModule
[Info] info: Volo.Abp.AbpApplicationBase[0] (dotnet.runtime.o8gq1i8bk6.js, line 3)
          - Volo.Abp.Account.Pro.Admin.Blazor.AbpAccountAdminBlazorModule

berkansasmaz created one month ago

Support Team .NET Developer

I would like to examine it in more detail. If it is appropriate, you can send your sample application to support@abp.io with a text like below:

Please forward my mail to Berkan Sasmaz regarding ticket #8704.

berkansasmaz created one month ago

Support Team .NET Developer

I saw the mail you sent, but when I clicked on the link of your application, I could not get a meaningful result when it sent a request to localhost, but there is still something I would like you to try.

Can you update the Program class of your MyApp.Blazor project as in the picture below?

Then update your ingress file as in the picture below:

If these do not work, if you can send your project to the drive and send the link from the mail, I think we can get much faster results if it is also suitable for you.

carl.hallqvist created one month ago

Thanks Berkan,

I was adding suggested patch above with same results unfortunately. However, I was sending you a Onedrive link, did you manage to download it? As you say, it will simplify troubleshooting.

Best regards, Carl

berkansasmaz created one month ago

Support Team .NET Developer

Hello there,

I think there's been some confusion. I can't find the Onedrive link, the only e-mail you sent me seems to be the one in the picture below. Any chance you can send it again?

berkansasmaz created one month ago

Support Team .NET Developer

I have received your e-mail now and I will review it as soon as possible and get back to you.

berkansasmaz created one month ago

Support Team .NET Developer

Hello again,

I intuitively had a few ideas after reviewing your code. Since I can't be completely sure, we will proceed with a little trial and error, sorry for that, but I think we will succeed as a result.

I see that you put the client application behind Nginx as below. Actually there is no need for that.

When I create a Blazor Wasm project from scratch, appsettings.json is mapped as follows:

Obviously this seemed like it could be a problem.

Also, could you try my suggestions regarding from-ingress and the custom middleware I mentioned in my previous message, not in Blazor but in the AuthServer application instead? I somehow suspect that the following code is not working.

  if (!configuration.GetValue<bool>("AuthServer:RequireHttpsMetadata"))
        {
            Configure<OpenIddictServerAspNetCoreOptions>(options =>
            {
                options.DisableTransportSecurityRequirement = true;
            });
            
            Configure<ForwardedHeadersOptions>(options =>
            {
                options.ForwardedHeaders = ForwardedHeaders.XForwardedProto; // this line
            });
        }

        context.Services.ForwardIdentityAuthenticationForBearer(OpenIddictValidationAspNetCoreDefaults.AuthenticationScheme);

carl.hallqvist created one month ago

Thanks for the advice, I have implemented these but unfortunately without success. I have also verified that the correct Docker image is being used in the Kubernetes deployment. In the Blazor container, I have opened a pod shell and verified the app/wwwroot/appsettings.json, and its contents are correct as follows:

root@myproject-blazor-6d98cf46bf-lrdqn:/app/wwwroot# cat appsettings.json
{
  "App": {
    "SelfUrl": "https://staging.myproject.io"
  },
  "AuthServer": {
    "Authority": "https://auth.staging.myproject.io",
    "ClientId": "Blazor",
    "ResponseType": "code"
  },
  "RemoteServices": {
    "Default": {
      "BaseUrl": "https://gateway-web.staging.myproject.io"
    }
  },
  "AbpCli": {
    "Bundle": {
      "Mode": "BundleAndMinify", /* Options: None, Bundle, BundleAndMinify */
      "Name": "global"
    }
  }
}

I have also executed the following:

root@myproject-blazor-6d98cf46bf-lrdqn:/app# grep "localhost" -r
wwwroot/_content/Volo.Abp.AspNetCore.Components.Web.LeptonXTheme/side-menu/libs/bootstrap-icons/package.json:    "test:linkinator": "linkinator _site --recurse --silent --skip \"^(?!http://localhost)\"",
wwwroot/_content/Volo.Abp.AspNetCore.Components.Web.LeptonXTheme/side-menu/libs/bootstrap-icons/README.md:Then open `http://localhost:4000` in your browser.
root@myproject-blazor-6d98cf46bf-lrdqn:/app#

So far, I have not found anything indicating a misconfiguration. Where is localhost:44366 defined? Feels like it is hardcoded in some dll or so? Do you have any further ideas?

Best regards, Carl

berkansasmaz created one month ago

Support Team .NET Developer

Hi,

Thank you for the information. I understand your problem.

If you use UseStaticFiles middleware instead of MapStaticAssets you will see that there is no problem. The problem is that the MapStaticAssets middleware does not support modifying static files after build/publishing.

You can use UseStaticFiles middleware instead of MapStaticAssets if you want to solve the problem. Or you can follow the steps below:

Create AbpStaticFileProvider:

public class AbpStaticFileProvider : IFileProvider
{
    private readonly Matcher _matcher;
    private readonly IFileProvider _fileProvider;

    /// <param name="fileProvider">The file provider to be used to get the files.</param>
    /// <param name="fileNamePatterns">The file name patterns to include when serving static files (e.g., "appsettings*.json").
    /// Supports glob patterns. See <see href="https://learn.microsoft.com/en-us/dotnet/core/extensions/file-globbing">Glob patterns documentation</see>.
    /// </param>
    public AbpStaticFileProvider(IReadOnlyList<string> fileNamePatterns, IFileProvider fileProvider)
    {
        _fileProvider = fileProvider;
        _matcher = new Matcher(StringComparison.OrdinalIgnoreCase);
        _matcher.AddIncludePatterns(fileNamePatterns);
    }

    public IDirectoryContents GetDirectoryContents(string subpath)
    {
        return new NotFoundDirectoryContents();
    }

    public IFileInfo GetFileInfo(string subpath)
    {
        return _matcher.Match(Path.GetFileName(subpath)).HasMatches ?
            _fileProvider.GetFileInfo(subpath) :
            new NotFoundFileInfo(subpath);
    }

    public IChangeToken Watch(string filter)
    {
        return NullChangeToken.Singleton;
    }
}

You can then invoke the UseStaticAssets middleware before the MapStaticAssets middleware as follows:

app.UseStaticFiles(new StaticFileOptions
{
    ~~~~FileProvider = new AbpStaticFileProvider(["appsettings*.json"], env.WebRootFileProvider)
});

carl.hallqvist created one month ago

Thanks Berkan,

Looking good! Its just the last thing that is not working:

[Error] Failed to load resource: the server responded with a status of 404 () (bootstrap.min.css.map, line 0) https://staging.myproject.io/bootstrap.min.css.map

[Error] Failed to load resource: the server responded with a status of 404 () (dotnet.js.map, line 0) https://staging.myproject.io/_framework/dotnet.js.map

[Error] Failed to load resource: the server responded with a status of 404 () (dotnet.runtime.js.map, line 0) https://staging.myproject.io/_framework/dotnet.runtime.js.map

[Error] Failed to load resource: the server responded with a status of 404 () (bootstrap.bundle.min.js.map;, line 0) https://staging.myproject.io/bootstrap.bundle.min.js.map;