We have a very serious issue in our production environment where the above mentioned queries have started to take a lot of time. Somethimes they work fast but quite often they take several minutes. Other features of our application work smoothly so overall resource constrictions have been ruled out. Now we need help from you on how to dissect this problem since this is pure Abp code.
5 Answer(s)
-
0
Hi, since you are using a tiered application, we suggest to use Redis as the distributed cache. If you are not using it now, or it has some transient problems, getting localization values may take a while. But, normally after first fetching the localization entries, it should work faster.
So, can you confirm that your Redis server is always up, and there is no transient errors or downtimes on your Redis server?
-
0
Yes, we are using Redis already. My initial thought was that it cannot be the Redis, since we see no overload indicators in Azure Redis monitoring. However we see some intermittent timeout exceptions and in those I can see, if I understand correctly, some pretty high number of waiting calls (async-ops count), which could indicate some Redis overload.
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5859ms elapsed, timeout is 5000ms), command=EXPIRE, next: EXPIRE c:Volo.Abp.MultiTenancy.TenantConfiguration,k:SCM_PROD:i:dba2a1ba-5a58-f4f6-ec6a-3a03f2c223b2,n:null, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: DequeueResult, ws: Idle, in: 0, last-in: 0, cur-in: 63, sync-ops: 19115, **async-ops: 1215359**, serverEndpoint: redis-paas-PROD-002.redis.cache.windows.net:6380, conn-sec: 27571.89, aoc: 0, mc: 1/1/0, mgr: 9 of 10 available, clientName: AW0SDWK000002(SE.Redis-v2.7.27.49176), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=30,Free=32737,Min=2,Max=32767), POOL: (Threads=30,QueuedItems=4,CompletedItems=3836531,Timers=54), v: 2.7.27.49176 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) -
0
[LW] said: Yes, we are using Redis already. My initial thought was that it cannot be the Redis, since we see no overload indicators in Azure Redis monitoring. However we see some intermittent timeout exceptions and in those I can see, if I understand correctly, some pretty high number of waiting calls (async-ops count), which could indicate some Redis overload.
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5859ms elapsed, timeout is 5000ms), command=EXPIRE, next: EXPIRE c:Volo.Abp.MultiTenancy.TenantConfiguration,k:SCM_PROD:i:dba2a1ba-5a58-f4f6-ec6a-3a03f2c223b2,n:null, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: DequeueResult, ws: Idle, in: 0, last-in: 0, cur-in: 63, sync-ops: 19115, **async-ops: 1215359**, serverEndpoint: redis-paas-PROD-002.redis.cache.windows.net:6380, conn-sec: 27571.89, aoc: 0, mc: 1/1/0, mgr: 9 of 10 available, clientName: AW0SDWK000002(SE.Redis-v2.7.27.49176), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=30,Free=32737,Min=2,Max=32767), POOL: (Threads=30,QueuedItems=4,CompletedItems=3836531,Timers=54), v: 2.7.27.49176 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)This might be the reason. If the cache response doesn't arrive within a specific time, the data will be retrieved from the database instead.
-
0
That actually was my initial assumption, that it tries to receive the data from the database and the database query is actually the slowing factor here, but looking at our azure monitoring, there is no dependency to database so the execution does not (at least every time) go through the database.
I guess this has to be related to Redis somehow, but not in the way you described, if I understood correctly
-
0
Hi, while you are checking the Redis Server and fixing the overload problem, do you need anything from our side?
