Open Closed

404 error because of inconsistent bundle hash keys across multiple instances on running MVC web service with round robin setup #7209


User avatar
0
tony.chen@sjrb.ca created
  • ABP Framework version: v8.1.2
  • UI Type: MVC
  • Database System: EF Core (PostgreSQL)
  • Tiered (for MVC): yes
  • Exception message and full stack trace: 404 on loading css and js
  • Steps to reproduce the issue: put round robin on routing and deploy 2+ mvc web service instances

The issue is lazy bundling on script and css. Each instance has it's own unique hash key. (ex: /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053). When we have multiple instances with round robin setup in loading balance, users get 404 response on loading script and css source files.

Is there any way to pre-generate the source file during build stage? so application doesn't do bundling on the fly and avoid inconsistent hash keys across all instances?


15 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The 8923ECD9CC3A022B71E966D19950185C is the md5 of scripts content.

    So, it should be the same for all instances.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Hello Ming, This is the problem I am trying to solving. When first time opens the page after every new deployment, I can't get stylesheet (404). I always need to refresh the page.

    First Time:

    Second Time after refreshed page

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you share the logs of these two requests?

    I can see some other bundle files are correctly loaded, but only this CSS gets a 404.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    This is only happening on loading balanced MVC app. If I am only running one instance it's working fine. Since the bundle is generated on the fly when opens index page, this is what happened (my theory):

    1. Request index page from server 1
    2. Loading bundle files from server 2 and server 3 (round-robin on loading balancer)
    3. Server 2 and Server 3 return 404 not found on .js and .css bundles (because these two servers' index page never get called in first place)

    Can ABP add an attribute such as preGenerating=true into <abp-style-bundle>,<abp-style>, <abp-script>, <abp-script-bundle> tags?

    ====================================================================================================== Logs: (I don't see any log successful log when bundle loaded ) Timestamp: 2024-05-19T22:50:49.894-06:00, Host: autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:49.257-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:48.529-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:48.284-06:00, autonomy-a-5b4bf6f7c4-lbwgt {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"44fdd624fdf2355b5f5f1f18d42888f8","SpanId":"1d8c7bc0ccdc3f74","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"9778beb7-a34b-4979-9df3-6cd9f714fc73","ActionName":"Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared.Controllers.ErrorController.Index (Volo.Abp.AspNetCore.Mvc.UI.Theme.Shared)","RequestId":"0HN3OGTV9H30I:00000003","RequestPath":"/Error","ConnectionId":"0HN3OGTV9H30I","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:46.814-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (945874 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:46.167-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js (38 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    2024-05-19T22:50:44.641-06:00, autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","RenderedMessage":"Bundled __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (385593 bytes)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

    TimeStamp: 2024-05-19T22:50:44.365-06:00, Host: autonomy-a-5b4bf6f7c4-5l7tn {"Level":"Information","MessageTemplate":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","RenderedMessage":"Bundling __bundles/LeptonX.Global.35000FF442CE30AB15B2C70BCDC5040F.css (18 files)","TraceId":"09320413213231c9ec29b5d73f80aa6a","SpanId":"b0fa115f47cfb3ff","Properties":{"SourceContext":"Volo.Abp.AspNetCore.Mvc.UI.Bundling.BundlerBase","ActionId":"fb40100f-1632-4e09-b8bc-081f3dbc636e","ActionName":"/Index","RequestId":"0HN3OGSGL8JGK:00000001","RequestPath":"/","ConnectionId":"0HN3OGSGL8JGK","UserId":"3a1285a4-f1e8-d0e5-308e-b99b48a3a164"}}

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Thank you for the information. I will check it out in depth.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Try to add a CachedDynamicFileProvider class to get files from Redis cache.

    CachedDynamicFileProvider:

    using System;
    using System.Collections.Generic;
    using Microsoft.Extensions.FileProviders;
    using Volo.Abp.Caching;
    using Volo.Abp.DependencyInjection;
    using Volo.Abp.VirtualFileSystem;
    
    namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;
    
    [Dependency(ReplaceServices = true)]
    public class CachedDynamicFileProvider : DynamicFileProvider
    {
        protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }
    
        public CachedDynamicFileProvider(IDistributedCache<InMemoryFileInfoCacheItem> cache)
        {
            Cache = cache;
        }
    
        public override IFileInfo GetFileInfo(string? subpath)
        {
            if (subpath == null)
            {
                return new NotFoundFileInfo(subpath!);
            }
    
            var file = DynamicFiles.GetOrDefault(NormalizePath(subpath));
            if (file == null && (subpath.Contains(".js", StringComparison.OrdinalIgnoreCase) || subpath.Contains(".css", StringComparison.OrdinalIgnoreCase)))
            {
                var cacheItem = Cache.Get(NormalizePath(subpath));
                if (cacheItem == null)
                {
                    return new NotFoundFileInfo(subpath);
                }
    
                var inMemoryFile = new InMemoryFileInfo(NormalizePath(subpath), cacheItem.FileContent, cacheItem.Name);
                DynamicFiles.AddOrUpdate(NormalizePath(subpath), inMemoryFile, (key, value) => inMemoryFile);
                return inMemoryFile;
            }
    
            return file ?? new NotFoundFileInfo(subpath);
        }
    
        public override void AddOrUpdate(IFileInfo fileInfo)
        {
            var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
            Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
            DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
            ReportChange(filePath!);
        }
    
        public override bool Delete(string filePath)
        {
            Cache.Remove(filePath);
            if (!DynamicFiles.TryRemove(filePath, out _))
            {
                return false;
            }
    
            ReportChange(filePath);
            return true;
        }
    }
    
    

    InMemoryFileInfoCacheItem

    using System;
    using Volo.Abp.MultiTenancy;
    
    namespace Volo.Abp.AspNetCore.Mvc.UI.Bundling;
    
    [Serializable]
    [IgnoreMultiTenancy]
    public class InMemoryFileInfoCacheItem
    {
        public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
        {
            DynamicPath = dynamicPath;
            Name = name;
            FileContent = fileContent;
        }
    
        public string DynamicPath { get; set; }
    
        public string Name { get; set; }
    
        public byte[] FileContent { get; set; }
    }
    
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    https://github.com/abpframework/abp/pull/19870

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Thanks Ming! Do you know when this fix will be released? Looks like I need wait base class get changed, I can't overwrite AddOrUpdate and Delete functions.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    You can create a new class and copy the code from DynamicFileProvider.cs and CachedBundleDynamicFileProvider.cs

    [Dependency(ReplaceServices = true)]
    [ExposeServices(typeof(IDynamicFileProvider))]
    public class YourDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
    
    
  • User Avatar
    0
    tony.chen@sjrb.ca created

    Hello Ming, I tried this way, but I don't see any cache created in redis.

    I put a break point in this function but never called.

    public void AddOrUpdate(IFileInfo fileInfo)
    {
    var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
    Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
    DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
    ReportChange(filePath!);
    }
    

    However, this function get called constantly during the loading time

     public override IFileInfo GetFileInfo(string? subpath)
     {
    ....
    }
    

    I even did this to replace everything

    [Dependency(ReplaceServices = true)]
    [ExposeServices(typeof(CachedDynamicFileProvider), typeof(DynamicFileProvider), typeof(IDynamicFileProvider))]
    public class CachedDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
    {
    ...
    }
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    using System;
    using System.Collections.Concurrent;
    using System.Collections.Generic;
    using System.Threading;
    using Microsoft.Extensions.FileProviders;
    using Microsoft.Extensions.Options;
    using Microsoft.Extensions.Primitives;
    using Volo.Abp.AspNetCore.Mvc.UI.Bundling;
    using Volo.Abp.Caching;
    using Volo.Abp.DependencyInjection;
    using Volo.Abp.MultiTenancy;
    using Volo.Abp.VirtualFileSystem;
    
    namespace MyCompanyName.MyProjectName.Web;
    
    [Dependency(ReplaceServices = true)]
    [ExposeServices(typeof(IDynamicFileProvider))]
    public class CachedBundleDynamicFileProvider : DictionaryBasedFileProvider, IDynamicFileProvider, ISingletonDependency
    {
     protected override IDictionary<string, IFileInfo> Files => DynamicFiles;
    
        protected ConcurrentDictionary<string, IFileInfo> DynamicFiles { get; }
        protected ConcurrentDictionary<string, ChangeTokenInfo> FilePathTokenLookup { get; }
        protected IDistributedCache<InMemoryFileInfoCacheItem> Cache { get; }
        protected IOptions<AbpBundlingOptions> BundlingOptions { get; }
    
        public CachedBundleDynamicFileProvider(
            IDistributedCache<InMemoryFileInfoCacheItem> cache,
            IOptions<AbpBundlingOptions> bundlingOptions)
        {
            Cache = cache;
            BundlingOptions = bundlingOptions;
    
            FilePathTokenLookup = new ConcurrentDictionary<string, ChangeTokenInfo>(StringComparer.OrdinalIgnoreCase); ;
            DynamicFiles = new ConcurrentDictionary<string, IFileInfo>();
        }
    
        public override IFileInfo GetFileInfo(string? subpath)
        {
            var fileInfo = base.GetFileInfo(subpath);
    
            if (!subpath.IsNullOrWhiteSpace() && fileInfo is NotFoundFileInfo &&
                subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
            {
                var filePath = NormalizePath(subpath);
                var cacheItem = Cache.Get(filePath);
                if (cacheItem == null)
                {
                    return fileInfo;
                }
    
                fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
                DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) => fileInfo);
            }
    
            return fileInfo;
        }
    
        public void AddOrUpdate(IFileInfo fileInfo)
        {
            var filePath = fileInfo.GetVirtualOrPhysicalPathOrNull();
            Cache.GetOrAdd(filePath!, () => new InMemoryFileInfoCacheItem(filePath!, fileInfo.ReadBytes(), fileInfo.Name));
            DynamicFiles.AddOrUpdate(filePath!, fileInfo, (key, value) => fileInfo);
            ReportChange(filePath!);
        }
    
        public bool Delete(string filePath)
        {
            Cache.Remove(filePath);
            if (!DynamicFiles.TryRemove(filePath, out _))
            {
                return false;
            }
    
            ReportChange(filePath);
            return true;
        }
    
        public override IChangeToken Watch(string filter)
        {
            return GetOrAddChangeToken(filter);
        }
    
        private IChangeToken GetOrAddChangeToken(string filePath)
        {
            if (!FilePathTokenLookup.TryGetValue(filePath, out var tokenInfo))
            {
                var cancellationTokenSource = new CancellationTokenSource();
                var cancellationChangeToken = new CancellationChangeToken(cancellationTokenSource.Token);
                tokenInfo = new ChangeTokenInfo(cancellationTokenSource, cancellationChangeToken);
                tokenInfo = FilePathTokenLookup.GetOrAdd(filePath, tokenInfo);
            }
    
            return tokenInfo.ChangeToken;
        }
    
        private void ReportChange(string filePath)
        {
            if (FilePathTokenLookup.TryRemove(filePath, out var tokenInfo))
            {
                tokenInfo.TokenSource.Cancel();
            }
        }
    
        protected struct ChangeTokenInfo
        {
            public ChangeTokenInfo(
                CancellationTokenSource tokenSource,
                CancellationChangeToken changeToken)
            {
                TokenSource = tokenSource;
                ChangeToken = changeToken;
            }
    
            public CancellationTokenSource TokenSource { get; }
    
            public CancellationChangeToken ChangeToken { get; }
        }
    }
    
    
    
    [Serializable]
    [IgnoreMultiTenancy]
    public class InMemoryFileInfoCacheItem
    {
        public InMemoryFileInfoCacheItem(string dynamicPath, byte[] fileContent, string name)
        {
            DynamicPath = dynamicPath;
            Name = name;
            FileContent = fileContent;
        }
    
        public string DynamicPath { get; set; }
    
        public string Name { get; set; }
    
        public byte[] FileContent { get; set; }
    }
    
    
  • User Avatar
    0
    tony.chen@sjrb.ca created

    Still no lucky... I copied all your code.

    This if statement always returns false on this logic: subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase) <-- I tired in both debug and release builds.

            if (!subpath.IsNullOrWhiteSpace() &amp;&amp; fileInfo is NotFoundFileInfo &amp;&amp;
                subpath.Contains(BundlingOptions.Value.BundleFolderName, StringComparison.OrdinalIgnoreCase))
            {
                var filePath = NormalizePath(subpath);
                var cacheItem = Cache.Get(filePath);
                if (cacheItem == null)
                {
                    return fileInfo;
                }
    
                fileInfo = new InMemoryFileInfo(filePath, cacheItem.FileContent, cacheItem.Name);
                DynamicFiles.AddOrUpdate(filePath, fileInfo, (key, value) =&gt; fileInfo);
            }
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    The BundlingOptions.Value.BundleFolderName should be __bundles

    And the __bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js contains the __bundles

    You can set a breakpoint and try to navigate to /__bundles/LeptonX.Global.8923ECD9CC3A022B71E966D19950185C.js?_v=638515137047135053 in the browser.

  • User Avatar
    0
    tony.chen@sjrb.ca created

    Thanks! working now.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Great!

Made with ❤️ on ABP v9.1.0-preview. Updated on December 10, 2024, 06:38