Open Closed

Redis Distributed Lock exception #4249


User avatar
0
bozkan created

If you're creating a bug/problem report, please include followings:

  • ABP Framework version: v5.2.2
  • UI type: Angular
  • DB provider: EF Core
  • Tiered (MVC) or Identity Server Separated (Angular): yes
  • Exception message and stack trace:
[10:11:56 ERR] ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [&lt;&gt;f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
   at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable`1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in /_/src/StackExchange.Redis/LuaScript.cs:line 117
   at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1245
   at Medallion.Threading.Internal.Helpers.InternalSafeCreateTask[TState,TTask,TResult](Func`2 taskFactory, TState state) in /_/DistributedLock.Core/Internal/Helpers.cs:line 48
--- End of stack trace from previous location ---
   at Medallion.Threading.Redis.RedLock.RedLockAcquire.TryAcquireAsync() in /_/DistributedLock.Redis/RedLock/RedLockAcquire.cs:line 94
   at Medallion.Threading.Redis.RedisDistributedLock.TryAcquireAsync(CancellationToken cancellationToken) in /_/DistributedLock.Redis/RedisDistributedLock.cs:line 73
   at Medallion.Threading.Internal.BusyWaitHelper.WaitAsync[TState,TResult](TState state, Func`3 tryGetValue, TimeoutValue timeout, TimeoutValue minSleepTime, TimeoutValue maxSleepTime, CancellationToken cancellationToken) in /_/DistributedLock.Core/Internal/BusyWaitHelper.cs:line 29
   at Medallion.Threading.Internal.Helpers.Convert[TDerived,TBase](ValueTask`1 task, ValueTaskConversion _) in /_/DistributedLock.Core/Internal/Helpers.cs:line 24
   at Volo.Abp.DistributedLocking.MedallionAbpDistributedLock.TryAcquireAsync(String name, TimeSpan timeout, CancellationToken cancellationToken)
   at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext)
   at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()

* **Steps to reproduce the issue**:"

We use redis as distributed lock provider configured like this:

context.Services.AddSingleton<IDistributedLockProvider>(sp =>
            {
                var connection = ConnectionMultiplexer
                    .Connect(configuration["Redis:Configuration"]);
                return new RedisDistributedSynchronizationProvider(connection.GetDatabase());
            });

When we run the host project, several distributed lock exceptions occur and log to the console and we couldn't find the reason.


18 Answer(s)
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    If the problem cannot be reproduced every time, you can observe it for a while.

  • User Avatar
    0
    bozkan created

    hi

    If the problem cannot be reproduced every time, you can observe it for a while.

    Hi maliming,

    Problem is being reproduced every time.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    ok, I will check it.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    I can't reproduce the problem, can you try the following code?

    context.Services.AddSingleton<IDistributedLockProvider>(sp =>
    {
        var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]);
        return new RedisDistributedSynchronizationProvider(connection.GetDatabase());
    });
    
    
    app.Use(async (httpContext, next) =>
    {
        await using (var handler = await httpContext.RequestServices.GetRequiredService<IAbpDistributedLock>().TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10)))
        {
            httpContext.Response.StatusCode = 200;
            return;
        }
        await next(httpContext);
    });
    
  • User Avatar
    0
    bozkan created

    I can't reproduce the problem, can you try the following code?

    context.Services.AddSingleton<IDistributedLockProvider>(sp => 
    { 
        var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]); 
        return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); 
    }); 
     
    
    app.Use(async (httpContext, next) => 
    { 
        await using (var handler = await httpContext.RequestServices.GetRequiredService<IAbpDistributedLock>().TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10))) 
        { 
            httpContext.Response.StatusCode = 200; 
            return; 
        } 
        await next(httpContext); 
    }); 
    

    The second code block produces error like below:

    When I change it like this, the problem is not solved:

    app.Use(async (httpContext, next) => { await using (var handler = await httpContext.RequestServices.GetRequiredService() .TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10))) { httpContext.Response.StatusCode = 200; await next(httpContext); } });

    Btw, we do not use IAbpDistributedLock but the IDistributedLockProvider interface for distributed lock.

    public class MyBackgroundJob : AsyncBackgroundJob<MyRequest>, ITransientDependency
    {
        private readonly IMyDomainService _myDomainService;
        private readonly IDistributedLockProvider _distributedLockProvider;
    
        public MyBackgroundJob(IMyDomainService myDomainService,
            IDistributedLockProvider distributedLockProvider)
        {
            _myDomainService = myDomainService;
            _distributedLockProvider = distributedLockProvider;
        }
    
        public override async Task ExecuteAsync(MyRequest args)
        {
            var distributedLock = _distributedLockProvider.CreateLock(MyConsts.MyBackgroundJobName);
    
            await using (var handle = await distributedLock.TryAcquireAsync())
            {
                if (handle != null)
                {
                    await _myDomainService.MyMethod(args);
                }
            }
        }
    }
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Please try to use IAbpDistributedLock.

  • User Avatar
    0
    bozkan created

    Documentation suggests using IDistributedLockProvider instead of IAbpDistributedLock (https://docs.abp.io/en/abp/latest/Distributed-Locking)

    Btw, is IAbpDistributedLock really distributed or in-process? Because we are in a real distributed environment and need a real distributed lock.

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Documentation suggests using IDistributedLockProvider instead of IAbpDistributedLock

    I don't think so.

    https://docs.abp.io/en/abp/latest/Distributed-Locking#using-the-iabpdistributedlock-service

    Btw, is IAbpDistributedLock really distributed or in-process? Because we are in a real distributed environment and need a real distributed lock.

    It will be really distributed when you use the AbpDistributedLockingModule and context.Services.AddSingleton<IDistributedLockProvider>

  • User Avatar
    0
    bozkan created

    Ok then, how should I change this configuration to use IAbpDistributedLock

    context.Services.AddSingleton<IDistributedLockProvider>(sp => { var connection = ConnectionMultiplexer .Connect(configuration["Redis:Configuration"]); return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); });

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    See https://github.com/abpframework/abp/blob/rel-7.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L42

    public class MyBackgroundJob : AsyncBackgroundJob, ITransientDependency
    {
        private readonly IMyDomainService _myDomainService;
        private readonly IAbpDistributedLock _distributedLock;
    
        public MyBackgroundJob(IMyDomainService myDomainService,
            IAbpDistributedLock distributedLock)
        {
            _myDomainService = myDomainService;
            _distributedLock = distributedLock;
        }
    
        public override async Task ExecuteAsync(MyRequest args)
        {
            await using (var handle = await _distributedLock.TryAcquireAsync("DistributedLockName"))
            {
                if (handle != null)
                {
                    await _myDomainService.MyMethod(args);
                }
            }
        }
    }
    
    
  • User Avatar
    0
    bozkan created

    See https://github.com/abpframework/abp/blob/rel-7.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L42

    public class MyBackgroundJob : AsyncBackgroundJob, ITransientDependency 
    { 
        private readonly IMyDomainService _myDomainService; 
        private readonly IAbpDistributedLock _distributedLock; 
     
        public MyBackgroundJob(IMyDomainService myDomainService, 
            IAbpDistributedLock distributedLock) 
        { 
            _myDomainService = myDomainService; 
            _distributedLock = distributedLock; 
        } 
     
        public override async Task ExecuteAsync(MyRequest args) 
        { 
            await using (var handle = await _distributedLock.TryAcquireAsync("DistributedLockName")) 
            { 
                if (handle != null) 
                { 
                    await _myDomainService.MyMethod(args); 
                } 
            } 
        } 
    } 
     
    

    Thanks but this is the usage part, but I wonder about the configuration required in ConfigureServices, which was like this before:

    context.Services.AddSingleton<IDistributedLockProvider>(sp => { var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]); return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); });

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    https://github.com/abpframework/abp/blob/1b67ad4c9adadf4b4dbfcacf075905be1ae59548/templates/app/aspnet-core/src/MyCompanyName.MyProjectName.AuthServer/MyProjectNameAuthServerModule.cs#L140-L145

  • User Avatar
    0
    bozkan created

    I tried but it didn't solve the problem. And from the stack trace, I see that the error is not caused by my distributed lock usage but by the ABP's internal polling for background jobs. Additionally, the error is occurring on the console every time JobPollPeriod interval passes.

    ---> System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
       at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable`1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in /_/src/StackExchange.Redis/LuaScript.cs:line 117
       at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1245
       at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in /_/DistributedLock.Redis/RedLock/RedLockRelease.cs:line 69
       --- End of inner exception stack trace ---
       at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in /_/DistributedLock.Redis/RedLock/RedLockRelease.cs:line 78
       at Medallion.Threading.Redis.RedLock.RedLockHandle.DisposeAsync() in /_/DistributedLock.Redis/RedLock/RedLockHandle.cs:line 52
       at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext)
       **at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()**
    
  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    Can you test this?

    https://support.abp.io/QA/Questions/4249#answer-33f8b9ca-b52b-7ceb-d9e9-3a086a8d65c0

  • User Avatar
    0
    bozkan created

    hi

    Can you test this?

    https://support.abp.io/QA/Questions/4249#answer-33f8b9ca-b52b-7ceb-d9e9-3a086a8d65c0

    I did, but it didn't solve the issue:

    [13:19:59 ERR] One or more errors occurred. (ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')) System.AggregateException: One or more errors occurred. (ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')) ---> System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]') at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in //src/StackExchange.Redis/LuaScript.cs:line 117 at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 1245 at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in //DistributedLock.Redis/RedLock/RedLockRelease.cs:line 69 --- End of inner exception stack trace --- at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in //DistributedLock.Redis/RedLock/RedLockRelease.cs:line 78 at Medallion.Threading.Redis.RedLock.RedLockHandle.DisposeAsync() in /_/DistributedLock.Redis/RedLock/RedLockHandle.cs:line 52 at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext) at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    hi

    That is to say, there is no error in using IAbpDistributedLock manually, but there is an error in the background worker.

    it's weird

  • User Avatar
    0
    bozkan created

    hi

    That is to say, there is no error in using IAbpDistributedLock manually, but there is an error in the background worker.

    it's weird

    Yes, it looks like that. Do you have any idea what the reason might be?

  • User Avatar
    0
    maliming created
    Support Team Fullstack Developer

    Can you share a project to reproduce the problem?
    liming.ma@volosoft.com

Made with ❤️ on ABP v9.1.0-preview. Updated on November 01, 2024, 05:35