Activities of "cangunaydin"

• ABP Framework version: v7.4.0
• UI Type: Angular
• Database System: EF Core (PostgreSQL)
• Tiered (for MVC) or Auth Server Separated (for Angular): yes

Hello I am trying to find out bottlenecks inside my app with stress testing. I am preparing the server to handle more than 5000 device requests. It is going to be 1 get / post request in every 5 secs for each device. Each device is sending a soap xml format when they do post and when they do get, they are getting a response as xml from the server. To serve xml to device, I use Volo.Abp.TextTemplating.Scriban

I prepare my controllers to connect to database less frequently so i do not create bottleneck in database. To do that i use Microsoft Orleans. Holding the data in memory then saving it to database in every 5 mins since it is not a critical data that comes from the device. However for some critical data sometimes I need to persist the data, this is happening at first request from each device. I use k6 for stress test. I prepared a test for 3000 devices. I also make a setup for pgbouncer in front of postgresql for connection pooling. Things are okay until 2000 devices after this number I am getting this exception.

2023-10-16 22:03:31.077 +02:00 [ERR] The operation was canceled. System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int32 millisecondsTimeout, CancellationToken cancellationToken) at Volo.Abp.Threading.SemaphoreSlimExtensions.LockAsync(SemaphoreSlim semaphoreSlim, CancellationToken cancellationToken) at Volo.Abp.Caching.DistributedCache2.GetOrAddAsync(TCacheKey key, Func1 factory, Func1 optionsFactory, Nullable1 hideErrors, Boolean considerUow, CancellationToken token) at Volo.Abp.TextTemplateManagement.TextTemplates.DatabaseTemplateContentContributor.GetOrNullAsync(TemplateContentContributorContext context) at Volo.Abp.TextTemplating.TemplateContentProvider.GetContentOrNullAsync(ITemplateContentContributor[] contributors, TemplateContentContributorContext context) at Volo.Abp.TextTemplating.TemplateContentProvider.GetContentOrNullAsync(TemplateDefinition templateDefinition, String cultureName, Boolean tryDefaults, Boolean useCurrentCultureIfCultureNameIsNull) at Volo.Abp.TextTemplating.TemplateRenderingEngineBase.GetContentOrNullAsync(TemplateDefinition templateDefinition) at Volo.Abp.TextTemplating.Scriban.ScribanTemplateRenderingEngine.RenderSingleTemplateAsync(TemplateDefinition templateDefinition, Dictionary2 globalContext, Object model) at Volo.Abp.TextTemplating.Scriban.ScribanTemplateRenderingEngine.RenderInternalAsync(String templateName, Dictionary2 globalContext, Object model) at Volo.Abp.TextTemplating.Scriban.ScribanTemplateRenderingEngine.RenderAsync(String templateName, Object model, String cultureName, Dictionary2 globalContext) at Volo.Abp.TextTemplating.AbpTemplateRenderer.RenderAsync(String templateName, Object model, String cultureName, Dictionary2 globalContext) at Doohlink.MagicInfo.Envelopes.Renderers.EnvelopeRenderingService1.RenderAsync(EnvelopeHeader header, TModel body) in C:\Development\Projects\Examples\Doohlink\aspnet-core\modules\Doohlink.MagicInfo\src\Doohlink.MagicInfo.Domain\Envelopes\Renderers\EnvelopeRenderingService.cs:line 30 at Doohlink.MagicInfo.Handlers.CommandHandler.HandleAsync(Envelope1 envelope) in C:\Development\Projects\Examples\Doohlink\aspnet-core\modules\Doohlink.MagicInfo\src\Doohlink.MagicInfo.Application\Handlers\CommandHandler.cs:line 90 at Doohlink.MagicInfo.Handlers.EnvelopeHandler.HandlePostAsync(String body) in C:\Development\Projects\Examples\Doohlink\aspnet-core\modules\Doohlink.MagicInfo\src\Doohlink.MagicInfo.Application\Handlers\EnvelopeHandler.cs:line 62

and after the stress test finished, the application can not connect to redis anymore. i need to flush redis. Also you can see that inside the logs.

2023-10-16 22:42:47.163 +02:00 [WRN] The message timed out in the backlog attempting to send because no connection became available, command=HMGET, timeout: 5000, outbound: 0KiB, inbound: 0KiB, inst: 0, qu: 89, qs: 0, aw: True, bw: CheckingForTimeoutComplete, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 833, async-ops: 49078, serverEndpoint: localhost:6379, conn-sec: 1484.52, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: DESKTOP-NHAEDKT(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=47,Free=32720,Min=6,Max=32767), POOL: (Threads=52,QueuedItems=441,CompletedItems=2340351,Timers=7635), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) StackExchange.Redis.RedisTimeoutException: The message timed out in the backlog attempting to send because no connection became available, command=HMGET, timeout: 5000, outbound: 0KiB, inbound: 0KiB, inst: 0, qu: 89, qs: 0, aw: True, bw: CheckingForTimeoutComplete, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 833, async-ops: 49078, serverEndpoint: localhost:6379, conn-sec: 1484.52, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: DESKTOP-NHAEDKT(SE.Redis-v2.6.122.38350), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=47,Free=32720,Min=6,Max=32767), POOL: (Threads=52,QueuedItems=441,CompletedItems=2340351,Timers=7635), v: 2.6.122.38350 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2099 at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 405 at Microsoft.Extensions.Caching.StackExchangeRedis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) at Microsoft.Extensions.Caching.StackExchangeRedis.RedisCache.GetAndRefresh(String key, Boolean getData) at Volo.Abp.Caching.DistributedCache2.Get(TCacheKey key, Nullable`1 hideErrors, Boolean considerUow) 2023-10-16 22:42:47.163 +02:00 [WRN] ---------- Exception Data ---------- Redis-Message = HMGET c:Volo.Abp.LanguageManagement.Texts,k:Adzup:AbpExceptionHandling_en Redis-Timeout = 5000 Redis-Write-State = Idle Redis-Read-State = ReadAsync Redis-OutboundDeltaKB = 0KiB Redis-InboundDeltaKB = 0KiB Redis-OpsSinceLastHeartbeat = 0 Redis-Queue-Awaiting-Write = 89 Redis-Queue-Awaiting-Response = 0 Redis-Active-Writer = True Redis-Backlog-Writer = CheckingForTimeoutComplete Redis-Inbound-Bytes = 0 Redis-Inbound-Pipe-Bytes = 0 Redis-Outbound-Pipe-Bytes = 0 Redis-Last-Result-Bytes = 0 Redis-Inbound-Buffer-Bytes = 0 Redis-Sync-Ops = 833 Redis-Async-Ops = 49078 Redis-Server-Endpoint = localhost:6379 Redis-Server-Connected-Seconds = 1484.52 Redis-Abort-On-Connect = 1 Redis-Multiplexer-Connects = 1/1/0 Redis-Manager = 10 of 10 available Redis-Client-Name = DESKTOP-NHAEDKT(SE.Redis-v2.6.122.38350) Redis-ThreadPool-IO-Completion = (Busy=0,Free=1000,Min=1,Max=1000) Redis-ThreadPool-Workers = (Busy=47,Free=32720,Min=6,Max=32767) Redis-ThreadPool-Items = (Threads=52,QueuedItems=441,CompletedItems=2340351,Timers=7635) Redis-Busy-Workers = 47 Redis-Version = 2.6.122.38350 redis-command = HMGET c:Volo.Abp.LanguageManagement.Texts,k:Adzup:AbpExceptionHandling_en request-sent-status = WaitingInBacklog redis-server = localhost:6379

So couple of questions, - I understand from the exception that it is trying to get the template text from DatabaseTemplateContentContributor, at Volo.Abp.TextTemplateManagement.TextTemplates.DatabaseTemplateContentContributor.GetOrNullAsync(TemplateContentContributorContext context) I assume this is coming from Text Template Management Module (https://docs.abp.io/en/commercial/latest/modules/text-template-management#text-template-management-module) and since this is the last contributor, it is the first to try.

Since my text templates doesn't use any localization is it possible to use VirtualFileTemplateContentContributor instead? Of course I can add another ITemplateContentContributor. I just wonder if there is any other way to avoid trying the last contributor and only use VirtualFileTemplateContentContributor, since I don't want the code to go to redis cache or database at all. All I need is to get the template from virtual file system and replace it with the model.

- Also I wonder what is happening in this setup, Is the bottleneck happening from the redis cache or from the database connection? At first thought, I assume it is sth going on with Redis Cache, Cause if the templates are cached, it won't go to database after some time for the same template. But strange thing over here is if i increase the connection pool size in database, I can increase the concurrent users that the server handles. With 150 max connections and max pool size of 150 i can handle 2000 users, if i increase the pool size to 250 then the server can handle 3000 users with the same test. So it seems database connection pool has a role over here, but I am not expecting for the code to create a connection with database. Why it is happening? Can it be a bug when it tries to get the template from the cache? so it goes to database instead?

By the way I disabled auditlogging all over the application with this configuration. any help would be appreciated.

Configure<AbpAuditingOptions>(options =>
        {
            options.IsEnabled = false; //Disables the auditing system
        });

Hello, I have sent it to you.

And I still think that since it is an anonymous call, it shouldn't depend on "Current Tenant Id" while you are downloading the file. Cause you already have a token id to download the file.

Hello, Can you try to impersonate from the admin side for the tenant and try to download? I think that's the problem.

I have sent the email, i have also added docker compose file for postgres and redis. you can check it out if you want.

• ABP Framework version: v7.4.0
• UI Type: No Ui
• Database System: EF Core (SQL Server, Oracle, MySQL, PostgreSQL, etc..)
• Tiered (for MVC) or Auth Server Separated (for Angular): no

Hello, I think razor text templating have a memory leak. And i suppose it is sth. with caching. To produce the problem here are the steps, I can also send a sample app for this.

1. create a new app with abp cli. abp new Acme.BookStore -u none -csf
2. then add Volo.Abp.TextTemplating.Razor module with cli. I added this inside application module but you can add it wherever you want. abp add-package Volo.Abp.TextTemplating.Razor
3. Then create a model for the template I have added a model that takes a list. here it is.

public class Command
{
   
    public bool ReportIndicate { get; set; }

    public string CommandId { get; set; }
    public string MoCmd { get; set; }

    public MoSequence MoSequence { get; set; }


    public Command()
    {
    }

    public Command(bool reportIndicate, string commandId, string moCmd, MoSequence moSequence)
    {
        Check.NotNullOrWhiteSpace(commandId, nameof(commandId));
        Check.NotNullOrWhiteSpace(moCmd, nameof(moCmd));
        Check.NotNull(moSequence, nameof(moSequence));

        ReportIndicate = reportIndicate;
        CommandId = commandId;
        MoCmd = moCmd;
        MoSequence = moSequence;
    }


}
public class MoSequence
{
    public List<Mo> MoList { get; set; }


    public MoSequence()
    {
        MoList = new List<Mo>();
    }

    public void AddMo(string moPath, string moValue)
    {
        MoList.Add(new Mo(moPath, moValue));
    }

    public void AddMo(string moPath)
    {
        MoList.Add(new Mo(moPath));
    }
}

public class Mo
{
    public string MoPath { get; set; }

    public string MoValue { get; set; }

    public Mo()
    {

    }
    public Mo(string moPath)
    {
        MoPath = moPath;
    }

    public Mo(string moPath, string moValue)
    {
        MoPath = moPath;
        MoValue = moValue;
    }
}

4. Now create a template for the command model. here you can find it.Add it as an embedded resource

@inherits Volo.Abp.TextTemplating.Razor.RazorTemplatePageBase<Acme.BookStore.Models.Command>
<srm:COMMAND>
    <srm:REPORT_INDICATE>@(Model.ReportIndicate ? "true" : "false")</srm:REPORT_INDICATE>
    <srm:COMMAND_ID>@Model.CommandId</srm:COMMAND_ID>
    <srm:MO_CMD>@Model.MoCmd</srm:MO_CMD>
    <srm:MO_SEQUENCE>
        @foreach (var item in Model.MoSequence.MoList)
        {
            <srm:MO>
                <srm:MO_PATH>@item.MoPath</srm:MO_PATH>
                @if (item.MoValue != null)
                {
                    <srm:MO_VALUE>@item.MoValue</srm:MO_VALUE>
                }
                else
                {
                    <srm:MO_VALUE />
                }
            </srm:MO>
        }
    </srm:MO_SEQUENCE>
</srm:COMMAND>

5. Create a TemplateDefinitionProvider.

public class DemoTemplateDefinitionProvider : TemplateDefinitionProvider
{
    public override void Define(ITemplateDefinitionContext context)
    {
        context.Add(
            new TemplateDefinition("Demo") //template name: "Demo"
                .WithRazorEngine()
                .WithVirtualFilePath(
                    "/Template/Demo.cshtml", //template content path
                    isInlineLocalized: true
                )
        );
    }
}

6. Add Configuration to your module

public override void ConfigureServices(ServiceConfigurationContext context)
    {
        Configure<AbpAutoMapperOptions>(options =>
        {
            options.AddMaps<BookStoreApplicationModule>();
        });
        Configure<AbpRazorTemplateCSharpCompilerOptions>(options =>
        {
            options.References.Add(MetadataReference.CreateFromFile(typeof(BookStoreApplicationModule).Assembly.Location));
        });
        Configure<AbpVirtualFileSystemOptions>(options =>
        {
            options.FileSets.AddEmbedded<BookStoreApplicationModule>("Acme.BookStore");
        });
    }

7) Create an appservice that will return the output.

public class TestAppService : BookStoreAppService, ITestAppService
{
    private readonly ITemplateRenderer _templateRenderer;

    public TestAppService(ITemplateRenderer templateRenderer)
    {
        _templateRenderer = templateRenderer;
    }

    public async Task<string> GetOutput()
    {
        var moSequence = new MoSequence();
        moSequence.AddMo(".MO.MONITOR_OPERATION.BOOTSTRAP.DEV_CODE", "105");
        moSequence.AddMo(".MO.MONITOR_OPERATION.BOOTSTRAP.DEV_TYPE", "SPLAYER" + Random.Shared.Next().ToString());
        moSequence.AddMo(".MO.MONITOR_OPERATION.BOOTSTRAP.DEV_MDNM", "QB13R");
        var command = new Command(false,
            "31470ade5bc2fd42-60c7af5-4725-8703-33dd729f63cea7aa5e4fbc0e",
            ".MO.MONITOR_OPERATION.BOOTSTRAP",
            moSequence);
        var result = await _templateRenderer.RenderAsync(
           "Demo", //the template name
          command
       );
        return result;

    }
}

8. Now you should be able to hit the endpoint. and it should return sth similar like this.

<srm:COMMAND>
    <srm:REPORT_INDICATE>false</srm:REPORT_INDICATE>
    <srm:COMMAND_ID>31470ade5bc2fd42-60c7af5-4725-8703-33dd729f63cea7aa5e4fbc0e</srm:COMMAND_ID>
    <srm:MO_CMD>.MO.MONITOR_OPERATION.BOOTSTRAP</srm:MO_CMD>
    <srm:MO_SEQUENCE>
            <srm:MO>
                <srm:MO_PATH>.MO.MONITOR_OPERATION.BOOTSTRAP.DEV_CODE</srm:MO_PATH>
                    <srm:MO_VALUE>105</srm:MO_VALUE>
            </srm:MO>
            <srm:MO>
                <srm:MO_PATH>.MO.MONITOR_OPERATION.BOOTSTRAP.DEV_TYPE</srm:MO_PATH>
                    <srm:MO_VALUE>SPLAYER737483572</srm:MO_VALUE>
            </srm:MO>
            <srm:MO>
                <srm:MO_PATH>.MO.MONITOR_OPERATION.BOOTSTRAP.DEV_MDNM</srm:MO_PATH>
                    <srm:MO_VALUE>QB13R</srm:MO_VALUE>
            </srm:MO>
    </srm:MO_SEQUENCE>
</srm:COMMAND>

9. Now you need to call this endpoint with some frequency so you can see what is going on in memory. I have used dotMemory and k6 for it. here is my k6 .js file

import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
    insecureSkipTLSVerify: true,
    noConnectionReuse: false,
    scenarios: {
        per_vu_scenario: {
            executor: "per-vu-iterations",
            vus: 5,
            iterations: 30,
            startTime: "0s",
            maxDuration: '2m',
        },
    },
};

export default function () {
    // Here, we set the endpoint to test.
    const response = http.get('https://localhost:44395/api/app/test/output');

    // An assertion
    check(response, {
        'is status 200': (x) => x.status === 200
    });

    sleep(3);
}

10. if you run this js file from command line tool.

k6 run load.js

and check the memory spike in dotmemory you will see sth similar like this.

I analyze this with dotnet-dump and dotmemory. It has lots of free memory in unmanaged memory heap. mostly strings. I didn't check the source code. It could be sth wrong from caching so it caches the same thing in every request, but not sure, didn't deep dive into it.

If i switch to Volo.Abp.TextTemplating.Scriban, this doesn't happen and you can see the memory steady, no spikes. I switch to scriban and change all my templates with it. Hope i could clarify the problem if you need a sample app i can send it if you give an email address.

Hello, If you want, I can send you a sample app if you give me an email address ok here are the steps.

1. Create the new project from abp cli abp new Doohlink -t app-pro -u angular -dbms PostgreSQL --separate-auth-server -m maui -csf

2. Add Volo.FileManagement module (run the command inside aspnet-core folder) abp add-module Volo.FileManagement

3. Arrange Postgres and Redis. Change appsettings.json according to that. (I use docker containers for that.)

4. Run Dbmigrator.

5. Run the Application (AuthServer and HttpApi.Host)

6. do yarn install in angular app.

7. Configure the angular app. Add Config Module and Feature Module.

8. run angular app with yarn start

9. now you should see file management menu item.

10. upload an image.

11. Then try to download that image.

12. you will see an authentication error.

i hope this is enough information, as i say if you can not reproduce i can send you the sample app.

• ABP Framework version: v7.4.0
• UI Type: Angular
• Database System: EF Core (SQL Server, Oracle, MySQL, PostgreSQL, etc..)

Hello, I recently upgraded my abp version from 7.3.2 to 7.4.0, I don't know if this is related with upgrade, maybe this wasn't working fine before.

In angular part of the gdpr module, when i look at the source code, it checks the cookie consent, if it is true it doesn't show the accept cookie bar as I understand.

here is the code for checking the cookie, here cookie key is '.AspNet.Consent'

 protected checkCookieConsent() {
    const hasCookieConsent =
      decodeURIComponent(this.document.cookie)
        .split(' ')
        .indexOf(this.cookieKey + '=true') > -1;

    if (hasCookieConsent) {
      this.showCookieConsent = false;
    }
  }

When i look at my cookies on chrome,I can see that it is true.

when i debug this, i can see that array from the code decodeURIComponent(this.document.cookie.split(' ') has an item like

here you can see indexOf shouldn't be .indexOf(this.cookieKey + '=true') but it should be .indexOf(this.cookieKey + '=true;') to make it work (';' semicolon at the end). I use chrome by the way. Maybe it works with other browsers, i didn't test it with others. Is there any trick that I can do to fix it fast?

• ABP Framework version: v7.4.0
• UI Type: Angular
• Database System: EF Core (SQL Server, Oracle, MySQL, PostgreSQL, etc..)
• Tiered (for MVC) or Auth Server Separated (for Angular): yes

Hello I recently upgraded to Abp v7.4.0, I am using file management module with abp, I customize it according to my needs and it was working fine. Now i have a problem with file download. I think there is a bug related with it. Since i customize it i couldn't be sure but I didn't override download part before, so it is likely from the new version.

Anyway here is the problem. For download to work first angular is doing a backend call to get a token. Then doing get request by using javascript window.open here is the file management module angular code.

  downloadFile(file: FileInfo) {
    return this.fileDescriptorService.getDownloadToken(file.id).pipe(
      tap((res) => {
        window.open(
          `${this.apiUrl}/api/file-management/file-descriptor/download/${file.id}?token=${res.token}`,
          '_self'
        );
      })
    );
  }

"this.fileDescriptorService.getDownloadToken" call is an authenticated but file-descriptor/download call is anonymous call.

On the backend side, when IDistributedCache is used it sets the token for the current tenant. so it normalizes the cache key. here is the code for it.

 public virtual async Task<DownloadTokenResultDto> GetDownloadTokenAsync(Guid id)
 {
     var token = Guid.NewGuid().ToString();

     await DownloadTokenCache.SetAsync(
         token,
         new FileDownloadTokenCacheItem { FileDescriptorId = id },
         new DistributedCacheEntryOptions
         {
             AbsoluteExpirationRelativeToNow = TimeSpan.FromSeconds(60)
         });

     return new DownloadTokenResultDto
     {
         Token = token
     };
 }

here the current tenant id is set. So the problem is when you do the second call to /api/file-management/file-descriptor/download/ endpoint with window.open since it is anonymous and token is not set it doesn't get the current tenant id. here is the code for it.

 [AllowAnonymous]
 public virtual async Task<IRemoteStreamContent> DownloadAsync(Guid id, string token)
 {
     var downloadToken = await DownloadTokenCache.GetAsync(token);
     if (downloadToken == null || downloadToken.FileDescriptorId != id)
     {
         throw new AbpAuthorizationException("Invalid download token: " + token);
     }

     FileDescriptor fileDescriptor;
     using (DataFilter.Disable<IMultiTenant>())
     {
         fileDescriptor = await FileDescriptorRepository.GetAsync(id);
     }
     var stream = await BlobContainer.GetAsync(id.ToString());
     return new RemoteStreamContent(stream, fileDescriptor?.Name);
 }

here you get authorization exception since cache key is not normalized over here. even if i override and change the current tenant id to null, it gets the token but this time BlobContainer can not find the file since this file belongs to tenant. What i came up with is to send tenantId from user interface as a query string and override FileDescriptorController like this.

also i override the angular part and inject the new service as a download service. sth like

@Injectable()
export class CreativeDownloadService {
  apiName = 'FileManagement';

  get apiUrl() {
    return this.environment.getApiUrl(this.apiName);
  }

  constructor(
    private restService: RestService,
    private fileDescriptorService: FileDescriptorService,
    private environment: EnvironmentService,
    private configStateService: ConfigStateService
  ) {}

  downloadFile(file: FileInfo) {
    const currentUser = this.configStateService.getOne("currentUser"); 
    return this.fileDescriptorService.getDownloadToken(file.id).pipe(
      tap((res) => {
        window.open(
          `${this.apiUrl}/api/file-management/file-descriptor/download/${file.id}?token=${res.token}&__tenant=${currentUser?.tenantId}`,
          '_self'
        );
      })
    );
  }
}

and then provide it.

I don't know how this was working before. I wonder if new version changed sth, by the way i use redis cache. Also sth I didn't understand is according to docs

giving query string __tenant should set CurrentTenant but it doesn't do so. Is this related with [AllowAnonymous] attribute? Thanks for the help and waiting for your reply.

Thank you for the help.