Open Closed

404 error because of inconsistent bundle hash keys across multiple instances on running MVC web service with round robin setup #7209

User avatar
0
tony.chen@sjrb.ca created
  • ABP Framework version: v8.1.2
  • UI Type: MVC
  • Database System: EF Core (PostgreSQL)
  • Tiered (for MVC): yes
  • Exception message and full stack trace: 404 on loading css and js
  • Steps to reproduce the issue: put round robin on routing and deploy 2+ mvc web service instances

The issue is lazy bundling on script and css. Each instance has it's own unique hash key. (ex: /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053). When we have multiple instances with round robin setup in loading balance, users get 404 response on loading script and css source files.

Is there any way to pre-generate the source file during build stage? so application doesn't do bundling on the fly and avoid inconsistent hash keys across all instances?

Go to accepted answer
15 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The 8923ECD9CC3A022B71E966D19950185C is the md5 of scripts content.

    So, it should be the same for all instances.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Hello Ming, This is the problem I am trying to solving. When first time opens the page after every new deployment, I can't get stylesheet (404). I always need to refresh the page.

    First Time:

    Second Time after refreshed page

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share the logs of these two requests?

    I can see some other bundle files are correctly loaded, but only this CSS gets a 404.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    This is only happening on loading balanced MVC app. If I am only running one instance it's working fine. Since the bundle is generated on the fly when opens index page, this is what happened (my theory):

    1. Request index page from server 1
    2. Loading bundle files from server 2 and server 3 (round-robin on loading balancer)
    3. Server 2 and Server 3 return 404 not found on .js and .css bundles (because these two servers' index page never get called in first place)

    Can ABP add an attribute such as preGenerating=true into <abp-style-bundle>,<abp-style>, <abp-script>, <abp-script-bundle> tags?

    ====================================================================================================== Logs: (I don't see any log successful log when bundle loaded ) Timestamp: 2024-05-19T22:50:49.894-06:00, Host: autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:49.257-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:48.529-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:48.284-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:46.814-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:46.167-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:44.641-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    TimeStamp: 2024-05-19T22:50:44.365-06:00, Host: autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Thank you for the information. I will check it out in depth.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Try to add a CachedDynamicFileProvider class to get files from Redis cache.

    CachedDynamicFileProvider:

    using System;
using System.Collections.Generic;
using Microsoft.Extensions.FileProviders;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.VirtualFileSystem;

namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;

[Dependency(ReplaceServices = true)]
public class CachedDynamicFileProvider : DynamicFileProvider
{
    protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }

    public CachedDynamicFileProvider(IDistributedCache<InMemoryFileInfoCacheItem> cache)
    {
        Cache = cache;
    }

    public override IFileInfo GetFileInfo(string? subpath)
    {
        if (subpath == null)
        {
            return new NotFoundFileInfo(subpath!);
        }

        var file = DynamicFiles.GetOrDefault(NormalizePath(subpath));
        if (file == null && (subpath.Contains(".js", StringComparison.OrdinalIgnoreCase) || subpath.Contains(".css", StringComparison.OrdinalIgnoreCase)))
        {
            var cacheItem = Cache.Get(NormalizePath(subpath));
            if (cacheItem == null)
            {
                return new NotFoundFileInfo(subpath);
            }

            var inMemoryFile = new InMemoryFileInfo(NormalizePath(subpath), cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(NormalizePath(subpath), inMemoryFile, (key, value) => inMemoryFile);
            return inMemoryFile;
        }

        return file ?? new NotFoundFileInfo(subpath);
    }

    public override void AddOrUpdate(IFileInfo fileInfo)
    {
        var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
        Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
        DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
        ReportChange(filePath!);
    }

    public override bool Delete(string filePath)
    {
        Cache.Remove(filePath);
        if (!DynamicFiles.TryRemove(filePath, out _))
        {
            return false;
        }

        ReportChange(filePath);
        return true;
    }
}

    InMemoryFileInfoCacheItem

    using System;
using Volo.Abp.MultiTenancy;

namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;

[Serializable]
[IgnoreMultiTenancy]
public class InMemoryFileInfoCacheItem
{
    public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
    {
        DynamicPath = dynamicPath;
        Name = name;
        FileContent = fileContent;
    }

    public string DynamicPath { get; set; }

    public string Name { get; set; }

    public byte[] FileContent { get; set; }
}
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    https://github.com/abpframework/abp/pull/19870

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Thanks Ming! Do you know when this fix will be released? Looks like I need wait base class get changed, I can't overwrite AddOrUpdate and Delete functions.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    You can create a new class and copy the code from DynamicFileProvider.cs and CachedBundleDynamicFileProvider.cs

    [Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IDynamicFileProvider))]
public class YourDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
  • User Avatar
    0
    tony.chen@sjrb.ca created

    Hello Ming, I tried this way, but I don't see any cache created in redis.

    I put a break point in this function but never called.

    public void AddOrUpdate(IFileInfo fileInfo)
{
var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
ReportChange(filePath!);
}

    However, this function get called constantly during the loading time

     public override IFileInfo GetFileInfo(string? subpath)
 {
....
}

    I even did this to replace everything

    [Dependency(ReplaceServices = true)]
[ExposeServices(typeof(CachedDynamicFileProvider), typeof(DynamicFileProvider), typeof(IDynamicFileProvider))]
public class CachedDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
{
...
}
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Threading;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Options;
using Microsoft.Extensions.Primitives;
using Volo.Abp.AspNetCore.Mvc.UI.Bundling;
using Volo.Abp.Caching;
using Volo.Abp.DependencyInjection;
using Volo.Abp.MultiTenancy;
using Volo.Abp.VirtualFileSystem;

namespace MyCompanyName.MyProjectName.Web;

[Dependency(ReplaceServices = true)]
[ExposeServices(typeof(IDynamicFileProvider))]
public class CachedBundleDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
{
 protected override IDictionary<string, IFileInfo> Files => DynamicFiles;

    protected ConcurrentDictionary<string, IFileInfo> DynamicFiles { get; }
    protected ConcurrentDictionary<string, ChangeTokenInfo> FilePathTokenLookup { get; }
    protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }
    protected IOptions<AbpBundlingOptions> BundlingOptions { get; }

    public CachedBundleDynamicFileProvider(
        IDistributedCache<InMemoryFileInfoCacheItem> cache,
        IOptions<AbpBundlingOptions> bundlingOptions)
    {
        Cache = cache;
        BundlingOptions = bundlingOptions;

        FilePathTokenLookup = new ConcurrentDictionary<string, ChangeTokenInfo>(StringComparer.OrdinalIgnoreCase); ;
        DynamicFiles = new ConcurrentDictionary<string, IFileInfo>();
    }

    public override IFileInfo GetFileInfo(string? subpath)
    {
        var fileInfo = base.GetFileInfo(subpath);

        if (!subpath.IsNullOrWhiteSpace() && fileInfo is NotFoundFileInfo &&
            subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
        {
            var filePath = NormalizePath(subpath);
            var cacheItem = Cache.Get(filePath);
            if (cacheItem == null)
            {
                return fileInfo;
            }

            fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) => fileInfo);
        }

        return fileInfo;
    }

    public void AddOrUpdate(IFileInfo fileInfo)
    {
        var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
        Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
        DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
        ReportChange(filePath!);
    }

    public bool Delete(string filePath)
    {
        Cache.Remove(filePath);
        if (!DynamicFiles.TryRemove(filePath, out _))
        {
            return false;
        }

        ReportChange(filePath);
        return true;
    }

    public override IChangeToken Watch(string filter)
    {
        return GetOrAddChangeToken(filter);
    }

    private IChangeToken GetOrAddChangeToken(string filePath)
    {
        if (!FilePathTokenLookup.TryGetValue(filePath, out var tokenInfo))
        {
            var cancellationTokenSource = new CancellationTokenSource();
            var cancellationChangeToken = new CancellationChangeToken(cancellationTokenSource.Token);
            tokenInfo = new ChangeTokenInfo(cancellationTokenSource, cancellationChangeToken);
            tokenInfo = FilePathTokenLookup.GetOrAdd(filePath, tokenInfo);
        }

        return tokenInfo.ChangeToken;
    }

    private void ReportChange(string filePath)
    {
        if (FilePathTokenLookup.TryRemove(filePath, out var tokenInfo))
        {
            tokenInfo.TokenSource.Cancel();
        }
    }

    protected struct ChangeTokenInfo
    {
        public ChangeTokenInfo(
            CancellationTokenSource tokenSource,
            CancellationChangeToken changeToken)
        {
            TokenSource = tokenSource;
            ChangeToken = changeToken;
        }

        public CancellationTokenSource TokenSource { get; }

        public CancellationChangeToken ChangeToken { get; }
    }
}



[Serializable]
[IgnoreMultiTenancy]
public class InMemoryFileInfoCacheItem
{
    public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
    {
        DynamicPath = dynamicPath;
        Name = name;
        FileContent = fileContent;
    }

    public string DynamicPath { get; set; }

    public string Name { get; set; }

    public byte[] FileContent { get; set; }
}
  • User Avatar
    0
    tony.chen@sjrb.ca created

    Still no lucky... I copied all your code.

    This if statement always returns false on this logic: subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase) <-- I tired in both debug and release builds.

            if (!subpath.IsNullOrWhiteSpace() &amp;&amp; fileInfo is NotFoundFileInfo &amp;&amp;
            subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
        {
            var filePath = NormalizePath(subpath);
            var cacheItem = Cache.Get(filePath);
            if (cacheItem == null)
            {
                return fileInfo;
            }

            fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
            DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) =&gt; fileInfo);
        }
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The BundlingOptions.Value.BundleFolderName should be __bundles

    And the __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js contains the __bundles

    You can set a breakpoint and try to navigate to /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053 in the browser.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Thanks! working now.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Great!

Assignee
User avatar maliming
Labels
None yet
Boost Your Development
ABP Live Training
Packages
See Trainings
Mastering ABP Framework Book
The Official Guide
Mastering
ABP Framework
Learn More
Mastering ABP Framework Book
Made with ❤️ on ABP v10.1.0-preview. Updated on October 02, 2025, 08:00