Redis Distributed Lock exception #4249 | Support

bozkan created 2 years ago

If you're creating a bug/problem report, please include followings:

ABP Framework version: v5.2.2
UI type: Angular
DB provider: EF Core
Tiered (MVC) or Identity Server Separated (Angular): yes
Exception message and stack trace:

[10:11:56 ERR] ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [&lt;&gt;f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
   at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable`1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in /_/src/StackExchange.Redis/LuaScript.cs:line 117
   at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1245
   at Medallion.Threading.Internal.Helpers.InternalSafeCreateTask[TState,TTask,TResult](Func`2 taskFactory, TState state) in /_/DistributedLock.Core/Internal/Helpers.cs:line 48
--- End of stack trace from previous location ---
   at Medallion.Threading.Redis.RedLock.RedLockAcquire.TryAcquireAsync() in /_/DistributedLock.Redis/RedLock/RedLockAcquire.cs:line 94
   at Medallion.Threading.Redis.RedisDistributedLock.TryAcquireAsync(CancellationToken cancellationToken) in /_/DistributedLock.Redis/RedisDistributedLock.cs:line 73
   at Medallion.Threading.Internal.BusyWaitHelper.WaitAsync[TState,TResult](TState state, Func`3 tryGetValue, TimeoutValue timeout, TimeoutValue minSleepTime, TimeoutValue maxSleepTime, CancellationToken cancellationToken) in /_/DistributedLock.Core/Internal/BusyWaitHelper.cs:line 29
   at Medallion.Threading.Internal.Helpers.Convert[TDerived,TBase](ValueTask`1 task, ValueTaskConversion _) in /_/DistributedLock.Core/Internal/Helpers.cs:line 24
   at Volo.Abp.DistributedLocking.MedallionAbpDistributedLock.TryAcquireAsync(String name, TimeSpan timeout, CancellationToken cancellationToken)
   at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext)
   at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()

* **Steps to reproduce the issue**:"

We use redis as distributed lock provider configured like this:

context.Services.AddSingleton<IDistributedLockProvider>(sp =>
            {
                var connection = ConnectionMultiplexer
                    .Connect(configuration["Redis:Configuration"]);
                return new RedisDistributedSynchronizationProvider(connection.GetDatabase());
            });

When we run the host project, several distributed lock exceptions occur and log to the console and we couldn't find the reason.

18 Answer(s)

0

maliming created 2 years ago
Support Team Fullstack Developer

hi

If the problem cannot be reproduced every time, you can observe it for a while.
0

bozkan created 2 years ago

hi

If the problem cannot be reproduced every time, you can observe it for a while.

Hi maliming,

Problem is being reproduced every time.
0

maliming created 2 years ago
Support Team Fullstack Developer

ok, I will check it.

maliming created 2 years ago

Support Team Fullstack Developer

I can't reproduce the problem, can you try the following code?

context.Services.AddSingleton<IDistributedLockProvider>(sp =>
{
    var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]);
    return new RedisDistributedSynchronizationProvider(connection.GetDatabase());
});

app.Use(async (httpContext, next) =>
{
    await using (var handler = await httpContext.RequestServices.GetRequiredService<IAbpDistributedLock>().TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10)))
    {
        httpContext.Response.StatusCode = 200;
        return;
    }
    await next(httpContext);
});

bozkan created 2 years ago

I can't reproduce the problem, can you try the following code?

context.Services.AddSingleton<IDistributedLockProvider>(sp => 
{ 
    var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]); 
    return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); 
});

app.Use(async (httpContext, next) => 
{ 
    await using (var handler = await httpContext.RequestServices.GetRequiredService<IAbpDistributedLock>().TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10))) 
    { 
        httpContext.Response.StatusCode = 200; 
        return; 
    } 
    await next(httpContext); 
});

The second code block produces error like below:

When I change it like this, the problem is not solved:

app.Use(async (httpContext, next) => { await using (var handler = await httpContext.RequestServices.GetRequiredService() .TryAcquireAsync("AbpBackgroundJobWorker", TimeSpan.FromSeconds(10))) { httpContext.Response.StatusCode = 200; await next(httpContext); } });

Btw, we do not use IAbpDistributedLock but the IDistributedLockProvider interface for distributed lock.

public class MyBackgroundJob : AsyncBackgroundJob<MyRequest>, ITransientDependency
{
    private readonly IMyDomainService _myDomainService;
    private readonly IDistributedLockProvider _distributedLockProvider;

    public MyBackgroundJob(IMyDomainService myDomainService,
        IDistributedLockProvider distributedLockProvider)
    {
        _myDomainService = myDomainService;
        _distributedLockProvider = distributedLockProvider;
    }

    public override async Task ExecuteAsync(MyRequest args)
    {
        var distributedLock = _distributedLockProvider.CreateLock(MyConsts.MyBackgroundJobName);

        await using (var handle = await distributedLock.TryAcquireAsync())
        {
            if (handle != null)
            {
                await _myDomainService.MyMethod(args);
            }
        }
    }
}

0

maliming created 2 years ago
Support Team Fullstack Developer

hi

Please try to use IAbpDistributedLock.
0

bozkan created 2 years ago

Documentation suggests using IDistributedLockProvider instead of IAbpDistributedLock (https://docs.abp.io/en/abp/latest/Distributed-Locking)

Btw, is IAbpDistributedLock really distributed or in-process? Because we are in a real distributed environment and need a real distributed lock.
0

maliming created 2 years ago
Support Team Fullstack Developer

Documentation suggests using IDistributedLockProvider instead of IAbpDistributedLock

I don't think so.

https://docs.abp.io/en/abp/latest/Distributed-Locking#using-the-iabpdistributedlock-service

Btw, is IAbpDistributedLock really distributed or in-process? Because we are in a real distributed environment and need a real distributed lock.

It will be really distributed when you use the AbpDistributedLockingModule and context.Services.AddSingleton<IDistributedLockProvider>
0

bozkan created 2 years ago

Ok then, how should I change this configuration to use IAbpDistributedLock

context.Services.AddSingleton<IDistributedLockProvider>(sp => { var connection = ConnectionMultiplexer .Connect(configuration["Redis:Configuration"]); return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); });

maliming created 2 years ago

Support Team Fullstack Developer

See https://github.com/abpframework/abp/blob/rel-7.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L42

public class MyBackgroundJob : AsyncBackgroundJob, ITransientDependency
{
    private readonly IMyDomainService _myDomainService;
    private readonly IAbpDistributedLock _distributedLock;

    public MyBackgroundJob(IMyDomainService myDomainService,
        IAbpDistributedLock distributedLock)
    {
        _myDomainService = myDomainService;
        _distributedLock = distributedLock;
    }

    public override async Task ExecuteAsync(MyRequest args)
    {
        await using (var handle = await _distributedLock.TryAcquireAsync("DistributedLockName"))
        {
            if (handle != null)
            {
                await _myDomainService.MyMethod(args);
            }
        }
    }
}

bozkan created 2 years ago

See https://github.com/abpframework/abp/blob/rel-7.0/framework/src/Volo.Abp.BackgroundJobs/Volo/Abp/BackgroundJobs/BackgroundJobWorker.cs#L42

public class MyBackgroundJob : AsyncBackgroundJob, ITransientDependency 
{ 
    private readonly IMyDomainService _myDomainService; 
    private readonly IAbpDistributedLock _distributedLock; 
 
    public MyBackgroundJob(IMyDomainService myDomainService, 
        IAbpDistributedLock distributedLock) 
    { 
        _myDomainService = myDomainService; 
        _distributedLock = distributedLock; 
    } 
 
    public override async Task ExecuteAsync(MyRequest args) 
    { 
        await using (var handle = await _distributedLock.TryAcquireAsync("DistributedLockName")) 
        { 
            if (handle != null) 
            { 
                await _myDomainService.MyMethod(args); 
            } 
        } 
    } 
}

Thanks but this is the usage part, but I wonder about the configuration required in ConfigureServices, which was like this before:

context.Services.AddSingleton<IDistributedLockProvider>(sp => { var connection = ConnectionMultiplexer.Connect(configuration["Redis:Configuration"]); return new RedisDistributedSynchronizationProvider(connection.GetDatabase()); });

0

maliming created 2 years ago
Support Team Fullstack Developer

https://github.com/abpframework/abp/blob/1b67ad4c9adadf4b4dbfcacf075905be1ae59548/templates/app/aspnet-core/src/MyCompanyName.MyProjectName.AuthServer/MyProjectNameAuthServerModule.cs#L140-L145

bozkan created 2 years ago

I tried but it didn't solve the problem. And from the stack trace, I see that the error is not caused by my distributed lock usage but by the ABP's internal polling for background jobs. Additionally, the error is occurring on the console every time JobPollPeriod interval passes.

---> System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType0`2[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')
   at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable`1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in /_/src/StackExchange.Redis/LuaScript.cs:line 117
   at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1245
   at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in /_/DistributedLock.Redis/RedLock/RedLockRelease.cs:line 69
   --- End of inner exception stack trace ---
   at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in /_/DistributedLock.Redis/RedLock/RedLockRelease.cs:line 78
   at Medallion.Threading.Redis.RedLock.RedLockHandle.DisposeAsync() in /_/DistributedLock.Redis/RedLock/RedLockHandle.cs:line 52
   at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext)
   **at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()**

0

maliming created 2 years ago
Support Team Fullstack Developer

hi

Can you test this?

https://support.abp.io/QA/Questions/4249#answer-33f8b9ca-b52b-7ceb-d9e9-3a086a8d65c0
0

bozkan created 2 years ago

hi

Can you test this?

https://support.abp.io/QA/Questions/4249#answer-33f8b9ca-b52b-7ceb-d9e9-3a086a8d65c0

I did, but it didn't solve the issue:

[13:19:59 ERR] One or more errors occurred. (ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')) System.AggregateException: One or more errors occurred. (ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]')) ---> System.ArgumentException: ps (Parameter 'Expected [lock] to be a field or gettable property on [<>f__AnonymousType02[[StackExchange.Redis.RedisKey, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46],[StackExchange.Redis.RedisValue, StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46]]]') at StackExchange.Redis.LuaScript.ExtractParameters(Object ps, Nullable1 keyPrefix, RedisKey[]& keys, RedisValue[]& args) in //src/StackExchange.Redis/LuaScript.cs:line 117 at StackExchange.Redis.RedisDatabase.ScriptEvaluateAsync(LuaScript script, Object parameters, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 1245 at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in //DistributedLock.Redis/RedLock/RedLockRelease.cs:line 69 --- End of inner exception stack trace --- at Medallion.Threading.Redis.RedLock.RedLockRelease.ReleaseAsync() in //DistributedLock.Redis/RedLock/RedLockRelease.cs:line 78 at Medallion.Threading.Redis.RedLock.RedLockHandle.DisposeAsync() in /_/DistributedLock.Redis/RedLock/RedLockHandle.cs:line 52 at Volo.Abp.BackgroundJobs.BackgroundJobWorker.DoWorkAsync(PeriodicBackgroundWorkerContext workerContext) at Volo.Abp.BackgroundWorkers.AsyncPeriodicBackgroundWorkerBase.DoWorkAsync()
0

maliming created 2 years ago
Support Team Fullstack Developer

hi

That is to say, there is no error in using IAbpDistributedLock manually, but there is an error in the background worker.

it's weird
0

bozkan created 2 years ago

hi

That is to say, there is no error in using IAbpDistributedLock manually, but there is an error in the background worker.

it's weird

Yes, it looks like that. Do you have any idea what the reason might be?
0

maliming created 2 years ago
Support Team Fullstack Developer

Can you share a project to reproduce the problem?
liming.ma@volosoft.com

Have an answer to this question? Log in and write your answer.