404 error because of inconsistent bundle hash keys across multiple instances on running MVC web service with round robin setup #7209 | Support

tony.chen@sjrb.ca created 6 months ago

ABP Framework version: v8.1.2
UI Type: MVC
Database System: EF Core (PostgreSQL)
Tiered (for MVC): yes
Exception message and full stack trace: 404 on loading css and js
Steps to reproduce the issue: put round robin on routing and deploy 2+ mvc web service instances

The issue is lazy bundling on script and css. Each instance has it's own unique hash key. (ex: /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053). When we have multiple instances with round robin setup in loading balance, users get 404 response on loading script and css source files.

Is there any way to pre-generate the source file during build stage? so application doesn't do bundling on the fly and avoid inconsistent hash keys across all instances?

15 Answer(s)

0

maliming created 6 months ago
Support Team Fullstack Developer

hi

The 8923ECD9CC3A022B71E966D19950185C is the md5 of scripts content.

So, it should be the same for all instances.
0

tony.chen@sjrb.ca created 6 months ago

Hello Ming, This is the problem I am trying to solving. When first time opens the page after every new deployment, I can't get stylesheet (404). I always need to refresh the page.

First Time:

Second Time after refreshed page
0

maliming created 6 months ago
Support Team Fullstack Developer

hi

Can you share the logs of these two requests?

I can see some other bundle files are correctly loaded, but only this CSS gets a 404.
0
tony.chen@sjrb.ca created 6 months ago
This is only happening on loading balanced MVC app. If I am only running one instance it's working fine. Since the bundle is generated on the fly when opens index page, this is what happened (my theory):

Request index page from server 1

Loading bundle files from server 2 and server 3 (round-robin on loading balancer)

Server 2 and Server 3 return 404 not found on .js and .css bundles (because these two servers' index page never get called in first place)

Can ABP add an attribute such as preGenerating=true into <abp-style-bundle>,<abp-style>, <abp-script>, <abp-script-bundle> tags?

====================================================================================================== Logs: (I don't see any log successful log when bundle loaded ) Timestamp: 2024-05-19T22:50:49.894-06:00, Host: autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:49.257-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:48.529-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:48.284-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:46.814-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:46.167-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

2024-05-19T22:50:44.641-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

TimeStamp: 2024-05-19T22:50:44.365-06:00, Host: autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}
0

maliming created 6 months ago
Support Team Fullstack Developer

Thank you for the information. I will check it out in depth.

maliming created 6 months ago

Support Team Fullstack Developer

Try to add a CachedDynamicFileProvider class to get files from Redis cache.

CachedDynamicFileProvider:

using System;
using System.Collections.Generic;
using Microsoft.Extensions.FileProviders;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.VirtualFileSystem;

namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;

[Dependency(ReplaceServices = true)]
public class CachedDynamicFileProvider : DynamicFileProvider
{
    protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }

    public CachedDynamicFileProvider(IDistributedCache<InMemoryFileInfoCacheItem> cache)
    {
        Cache = cache;
    }

    public override IFileInfo GetFileInfo(string? subpath)
    {
        if (subpath == null)
        {
            return new NotFoundFileInfo(subpath!);
        }

        var file = DynamicFiles.GetOrDefault(NormalizePath(subpath));
        if (file == null && (subpath.Contains(".js", StringComparison.OrdinalIgnoreCase) || subpath.Contains(".css", StringComparison.OrdinalIgnoreCase)))
        {
            var cacheItem = Cache.Get(NormalizePath(subpath));
            if (cacheItem == null)
            {
                return new NotFoundFileInfo(subpath);
            }

            var inMemoryFile = new InMemoryFileInfo(NormalizePath(subpath), cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(NormalizePath(subpath), inMemoryFile, (key, value) => inMemoryFile);
            return inMemoryFile;
        }

        return file ?? new NotFoundFileInfo(subpath);
    }

    public override void AddOrUpdate(IFileInfo fileInfo)
    {
        var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
        Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
        DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
        ReportChange(filePath!);
    }

    public override bool Delete(string filePath)
    {
        Cache.Remove(filePath);
        if (!DynamicFiles.TryRemove(filePath, out _))
        {
            return false;
        }

        ReportChange(filePath);
        return true;
    }
}

InMemoryFileInfoCacheItem

using System;
using Volo.Abp.MultiTenancy;

namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;

[Serializable]
[IgnoreMultiTenancy]
public class InMemoryFileInfoCacheItem
{
    public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
    {
        DynamicPath = dynamicPath;
        Name = name;
        FileContent = fileContent;
    }

    public string DynamicPath { get; set; }

    public string Name { get; set; }

    public byte[] FileContent { get; set; }
}

0

maliming created 6 months ago
Support Team Fullstack Developer

https://github.com/abpframework/abp/pull/19870
0

tony.chen@sjrb.ca created 6 months ago

Thanks Ming! Do you know when this fix will be released? Looks like I need wait base class get changed, I can't overwrite AddOrUpdate and Delete functions.

maliming created 6 months ago

Support Team Fullstack Developer

You can create a new class and copy the code from DynamicFileProvider.cs and CachedBundleDynamicFileProvider.cs

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IDynamicFileProvider))]
public class YourDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency

tony.chen@sjrb.ca created 6 months ago

Hello Ming, I tried this way, but I don't see any cache created in redis.

I put a break point in this function but never called.

public void AddOrUpdate(IFileInfo fileInfo)
{
var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
ReportChange(filePath!);
}

However, this function get called constantly during the loading time

 public override IFileInfo GetFileInfo(string? subpath)
 {
....
}

I even did this to replace everything

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(CachedDynamicFileProvider), typeof(DynamicFileProvider), typeof(IDynamicFileProvider))]
public class CachedDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
{
...
}

maliming created 6 months ago

Support Team Fullstack Developer

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;
using Volo.Abp.AspNetCore.Mvc.UI.Bundling;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.MultiTenancy;
using Volo.Abp.VirtualFileSystem;

namespace MyCompanyName.MyProjectName.Web;

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IDynamicFileProvider))]
public class CachedBundleDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
{
 protected override IDictionary<string, IFileInfo> Files => DynamicFiles;

    protected ConcurrentDictionary<string, IFileInfo> DynamicFiles { get; }
    protected ConcurrentDictionary<string, ChangeTokenInfo> FilePathTokenLookup { get; }
    protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }
    protected IOptions<AbpBundlingOptions> BundlingOptions { get; }

    public CachedBundleDynamicFileProvider(
        IDistributedCache<InMemoryFileInfoCacheItem> cache,
        IOptions<AbpBundlingOptions> bundlingOptions)
    {
        Cache = cache;
        BundlingOptions = bundlingOptions;

        FilePathTokenLookup = new ConcurrentDictionary<string, ChangeTokenInfo>(StringComparer.OrdinalIgnoreCase); ;
        DynamicFiles = new ConcurrentDictionary<string, IFileInfo>();
    }

    public override IFileInfo GetFileInfo(string? subpath)
    {
        var fileInfo = base.GetFileInfo(subpath);

        if (!subpath.IsNullOrWhiteSpace() && fileInfo is NotFoundFileInfo &&
            subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
        {
            var filePath = NormalizePath(subpath);
            var cacheItem = Cache.Get(filePath);
            if (cacheItem == null)
            {
                return fileInfo;
            }

            fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) => fileInfo);
        }

        return fileInfo;
    }

    public void AddOrUpdate(IFileInfo fileInfo)
    {
        var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
        Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
        DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
        ReportChange(filePath!);
    }

    public bool Delete(string filePath)
    {
        Cache.Remove(filePath);
        if (!DynamicFiles.TryRemove(filePath, out _))
        {
            return false;
        }

        ReportChange(filePath);
        return true;
    }

    public override IChangeToken Watch(string filter)
    {
        return GetOrAddChangeToken(filter);
    }

    private IChangeToken GetOrAddChangeToken(string filePath)
    {
        if (!FilePathTokenLookup.TryGetValue(filePath, out var tokenInfo))
        {
            var cancellationTokenSource = new CancellationTokenSource();
            var cancellationChangeToken = new CancellationChangeToken(cancellationTokenSource.Token);
            tokenInfo = new ChangeTokenInfo(cancellationTokenSource, cancellationChangeToken);
            tokenInfo = FilePathTokenLookup.GetOrAdd(filePath, tokenInfo);
        }

        return tokenInfo.ChangeToken;
    }

    private void ReportChange(string filePath)
    {
        if (FilePathTokenLookup.TryRemove(filePath, out var tokenInfo))
        {
            tokenInfo.TokenSource.Cancel();
        }
    }

    protected struct ChangeTokenInfo
    {
        public ChangeTokenInfo(
            CancellationTokenSource tokenSource,
            CancellationChangeToken changeToken)
        {
            TokenSource = tokenSource;
            ChangeToken = changeToken;
        }

        public CancellationTokenSource TokenSource { get; }

        public CancellationChangeToken ChangeToken { get; }
    }
}



[Serializable]
[IgnoreMultiTenancy]
public class InMemoryFileInfoCacheItem
{
    public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
    {
        DynamicPath = dynamicPath;
        Name = name;
        FileContent = fileContent;
    }

    public string DynamicPath { get; set; }

    public string Name { get; set; }

    public byte[] FileContent { get; set; }
}

tony.chen@sjrb.ca created 5 months ago

Still no lucky... I copied all your code.

This if statement always returns false on this logic: subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase) <-- I tired in both debug and release builds.

        if (!subpath.IsNullOrWhiteSpace() &amp;&amp; fileInfo is NotFoundFileInfo &amp;&amp;
            subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
        {
            var filePath = NormalizePath(subpath);
            var cacheItem = Cache.Get(filePath);
            if (cacheItem == null)
            {
                return fileInfo;
            }

            fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) =&gt; fileInfo);
        }

0

maliming created 5 months ago
Support Team Fullstack Developer

hi

The BundlingOptions.Value.BundleFolderName should be __bundles

And the __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js contains the __bundles

You can set a breakpoint and try to navigate to /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053 in the browser.
0

tony.chen@sjrb.ca created 5 months ago

Thanks! working now.
0

maliming created 6 months ago
Support Team Fullstack Developer

Great!

Have an answer to this question? Log in and write your answer.