Open Closed

redis commands are slow #5653


User avatar
0
portx-dev created

Hi,

We found some redis commands are very slow(1~53s), and fails in test environment. We use amazon elasticache(redis) for our test environment.

slow command examples are below.

HMGET t:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,c:Volo.Abp.LanguageManagement.Texts,k:PortX:PortX_ja HMGET t:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,c:Volo.Abp.LanguageManagement.Texts,k:PortX:AbpValidation_en HMGET c:Volo.Saas.Tenants.Tenant,k:PortX:i:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,n:null EXPIRE t:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,c:Volo.Abp.LanguageManagement.Texts,k:PortX:AbpValidation_en EXPIRE t:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,c:Volo.Abp.LanguageManagement.Texts,k:PortX:AbpValidation_ja EXPIRE c:Volo.Saas.Tenants.Tenant,k:PortX:i:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,n:null HMGET t:3a0c1d64-4d96-0ab6-9c31-0a332a2b8a8e,c:Volo.Abp.PermissionManagement.PermissionGrant,k:PortX:pn:R,pk:admin,n:PortX.RegularProjectRequestHistories.Create

We had a lot of time to investigate the reason with Datadog APM and redis insight, but we've not found the exact reason.

[investigation result]

  • the most portion of the commands are fast(12.81ms by 99 percentile). few commands are very slow(1~53s).
  • server resource is enough (cpu/memory utilization is under 40%)
  • redis resource is enough (cpu/memory utilization is under 40%)
  • values returned by slow commands are not so large.
  • we don't customize framework code which execute redis commands.
  • we found the value of the key t:*:Volo.Abp.LanguageManagement.Texts,k:PortX:PortX_ja is almost empty: { "dictionary": {} }

With these slow commands, we couldn't meet our customers needs.

We doubt

  • Amazon Elasticache(redis) have bad compatibility with ABP framework?
  • We use Amazon Elasticache(redis) with default parameters. There is room for customization?
  • Should we customize some configuration in ABP framework?
  • We mis config some setting?

ABP Framework version: Commercial 7.2 UI type: Angular DB provider: EF Core


4 Answer(s)
  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    Actually, ABP uses Microsoft's distributed Redis cache package

    Can this problem be reproduced with the local Redis server? Could you provide the full steps to reproduce? I will check it.

    I have no suggestion if it cannot be reproduced

  • User Avatar
    0
    portx-dev created

    I would like to stop using redis to isolate the problem. How can I stop using redis?

  • User Avatar
    0
    portx-dev created

    By the way, we use "Amazon ElastiCache for Redis". When using AWS, is there a recommended redis service?

  • User Avatar
    0
    liangshiwei created
    Support Team Fullstack Developer

    Hi,

    Yes, you can use docker to deploy Redis in the instance.

    Actually, Redis is not necessary but the distributed cache.

    There are many options:

    • Memory distributed cache: Highest performance, but not suitable for distributed or microservice applications
    • Redis distributed cache: Good performance
    • SqlServer distributed cache: General performance
    • NCache distributed cache: Highest performance.

    You can check the document: https://learn.microsoft.com/en-us/aspnet/core/performance/caching/distributed?view=aspnetcore-7.0#establish-distributed-caching-services

Made with ❤️ on ABP v9.1.0-preview. Updated on December 05, 2024, 12:19