AppConnections (bound sockets) climbing with zero HTTP requests #10597

4 Answer(s)

• 

  0

  AI-Bot created 5 months ago

  Support Team ABP Agent

  • ABP itself does not have a built-in “polling loop” that opens outbound SQL connections during idle periods (i.e., with zero HTTP requests) unless you’ve enabled/installed something that runs in the background (e.g., Hangfire server, ABP Background Workers, health checks, distributed cache backplane, etc.).
  • In a typical MVC (non-tiered) ABP v10 app, the main sources of “connections while idle” are:
    1. Your job scheduler/worker (Hangfire server), and/or
    2. Any ABP Background Worker you (or a module) registered, and/or
    3. ASP.NET Core / App Service platform behaviors (health probes, always-on pings, etc.—not ABP).

  ---

  • What ABP can do in the background (only if you enabled it):

    • Background Workers / Background Jobs infrastructure exists in ABP, but it doesn’t do anything by itself unless workers/jobs are registered and running in your process. ABP’s background workers are designed to run periodically/in a loop while the app is up.
    • If you integrated ABP with Hangfire for background processing, then your Hangfire server is exactly the kind of component that will keep opening SQL connections independent of HTTP traffic (polling, fetching jobs, heartbeats, etc.).

  • ABP features you mentioned are not timer-based connection creators by themselves:

    • Dynamic claims (IsDynamicClaimsEnabled = true): claims are typically evaluated during authentication / request pipeline usage, not via a standalone timer that continuously opens DB connections during total app idleness.
    • SecurityStampValidator: this is part of ASP.NET Core Identity cookie validation; it runs when principals are validated (request-driven), not as a periodic background timer with zero requests.
    • MultiTenantConnectionStringResolver / tenant resolution: resolves connection strings when a DbContext/UoW is created (request/job-driven). No “tenant polling” loop is expected.

  ---

  • For your 3 questions (directly):

    1. Does ABP have internal background processes that open outbound connections independently of HTTP requests?
       Only if you are running background workers/jobs (including Hangfire integration) in that same process. ABP provides the infrastructure, but it doesn’t continuously “refresh cache / flush audit logs / renew locks” on a timer by default in an otherwise idle app.
       If you are running Hangfire server, that alone is sufficient to explain SQL connections during “idle HTTP”.

    2. With dynamic claims + multi-tenancy, do SecurityStampValidator or MultiTenantConnectionStringResolver create connections on a timer?
       No—those are request/job execution driven, not timer-driven background loops. They may open DB connections when invoked, but they’re not expected to steadily increase socket count with truly zero activity.

    3. Similar socket accumulation with ABP + Hangfire.SqlServer on Azure App Service?
       I don’t have a verified ABP-specific source that ties ABP to socket leaks here. However, Hangfire.SqlServer on App Service is a common suspect for persistent SQL connectivity and frequent polling. If sockets are accumulating (not just stable), that points more to a leak/misconfiguration or platform/network exhaustion than to ABP framework internals.

  ---

  • Practical guidance (ABP-focused):
    1. Make sure only one Hangfire Server is running per instance (double-registration can multiply SQL polling/threads). In ABP, ensure you didn’t accidentally start Hangfire server twice (e.g., both AddHangfireServer() and ABP Hangfire integration starting it).
    2. If you don’t need Hangfire processing in the web app, move Hangfire Server to a separate Worker service (common ABP pattern) so your MVC app doesn’t host background polling.
    3. Inventory ABP Background Workers: if you (or a module) added any AsyncPeriodicBackgroundWorkerBase workers, they will run continuously and can trigger DB usage.

  ---

  • References
    • https://abp.io/docs/latest/framework/infrastructure/background-workers
    • https://abp.io/docs/latest/framework/infrastructure/background-jobs
    • https://abp.io/docs/latest/framework/infrastructure/
    • https://abp.io/docs/latest/framework/infrastructure/distributed-locking
    • https://abp.io/docs/latest/framework/infrastructure/audit-logging

  ---

  This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 5 months ago

  Support Team Fullstack Developer

  Hi,

  I checked our Hangfire module source code and also went through the Hangfire.SqlServer source code itself. There's no connection leak on either side — all connections are properly released with try-finally blocks, and the connection pool is handled by ADO.NET as expected.

  ABP also doesn't have any hidden background polling that would create connections on its own. Things like SecurityStampValidator and MultiTenantConnectionStringResolver only run during HTTP requests, not on timers.

  So the socket growth you're seeing is most likely not coming from ABP or Hangfire itself. Here are a few things worth checking:

  1. Check your own code for connection leaks

  Look for any place where you manually create SqlConnection, HttpClient, or other disposable objects without wrapping them in using. This is the most common cause of socket growth. For example:

  // Bad - connection never disposed
  var conn = new SqlConnection(connString);
  conn.Open();
  conn.Execute("...");
  
  // Good
  using var conn = new SqlConnection(connString);
  conn.Open();
  conn.Execute("...");
  

  Also check if you're creating HttpClient instances per request instead of reusing them — each new HttpClient opens a new socket.

  2. Azure SNAT port exhaustion

  This is a very common issue on Azure App Service. When your app makes many short-lived outbound connections (to SQL, Redis, blob storage, etc.), Azure's SNAT ports can get exhausted, which causes exactly the symptoms you described. You can check this in Azure Portal → your App Service → Diagnose and Solve Problems → search for "SNAT".

  More info: https://learn.microsoft.com/en-us/azure/app-service/troubleshoot-intermittent-outbound-connection-errors

  3. Monitor what's actually using the sockets

  You can use dotnet-counters to see what's happening:

  dotnet-counters monitor --process-id <PID> --counters "Microsoft.Data.SqlClient.EventSource,System.Net.Sockets,System.Net.Http"
  

  This will show you SQL connection pool stats, socket counts, and HTTP connection counts separately, so you can tell which component is actually leaking.

  4. Reduce Hangfire connection usage

  Even though Hangfire isn't leaking, you can reduce its connection throughput:

  services.AddHangfire(config =>
      config.UseSqlServerStorage(connectionString, new SqlServerStorageOptions
      {
          QueuePollInterval = TimeSpan.FromSeconds(30)
      }));
  
  services.AddHangfireServer(options =>
  {
      options.WorkerCount = 5;
  });
  

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  LiSong created 5 months ago

  I found there are 4 ABP-owned workers: TokenCleanupBackgroundWorker IdentitySessionCleanupBackgroundWorker ExcelFileCleanupWorker ExpiredAuditLogDeleterWorker

  I want to change all of them run nightly to make the system more stable.

  1. 2 of them are running hourly now, is it ok to make them run daily?
  2. how can I update their settings, I opened the hangfire admin page, but it seems it's readonly, correct?

  thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 5 months ago

  Support Team Fullstack Developer

  Hi,

  • TokenCleanupBackgroundWorker — runs every 1 hour (cleans up expired OpenIddict tokens older than 14 days)
  • IdentitySessionCleanupBackgroundWorker — runs every 1 hour (cleans up inactive sessions older than 30 days)
  • ExcelFileCleanupWorker — already runs daily at 23:00
  • ExpiredAuditLogDeleterWorker — already runs daily at 23:00

  So only the first 2 are running hourly. Changing them to daily is totally fine — they're cleaning up data that's 14-30 days old, so a few hours delay doesn't matter at all.

  You're right that the Hangfire dashboard is read-only for recurring jobs. You need to configure them in your code:

  Configure<TokenCleanupOptions>(options =>
  {
      options.CleanupPeriod = (int)TimeSpan.FromDays(1).TotalMilliseconds;
  });
  
  Configure<IdentitySessionCleanupOptions>(options =>
  {
      options.CleanupPeriod = (int)TimeSpan.FromDays(1).TotalMilliseconds;
  });
  

  However, I should mention that these 4 workers are very unlikely to be the cause of your socket growth issue. Even at hourly frequency, they only create a few database connections per run. That's nothing compared to the ~120 sockets/minute growth you're seeing.

  The socket issue is more likely caused by something else in your application — maybe manually created SqlConnection or HttpClient instances that aren't being disposed, or Azure SNAT port exhaustion. I'd recommend using dotnet-counters to monitor what's actually consuming the sockets:

  dotnet-counters monitor --process-id <PID> --counters "Microsoft.Data.SqlClient.EventSource,System.Net.Sockets,System.Net.Http"
  

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel