Issue creating users in production #10332

2 Answer(s)

• 

  0

  AI-Bot created 10 days ago

  Support Team ABP Agent

  1. Solution

  The error is a pure SQL Server execution timeout:

  Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

  The command that times out is the standard ABP Identity user‑creation batch:

  UPDATE [AbpRoles] ...
  INSERT INTO [AbpUserRoles] ...
  UPDATE [AbpUsers] ...
  

  So EF/ABP are working as expected; SQL Server is taking >30 seconds to complete this batch in production (but not in dev). That means:

  • Either the AbpRoles, AbpUsers, or AbpUserRoles rows are blocked/locked by some other transaction, or
  • The database server is under heavy load / IO issues, or
  • There is a very long‑running transaction (e.g. migration, maintenance, or custom code) holding locks.

  You can resolve and mitigate this with the following steps.

  ---

  A. Immediate checks on the SQL Server

  Run these queries directly on the production database:

  1. Check blocking sessions

  -- See who is blocking whom
  SELECT
      r.session_id,
      r.status,
      r.command,
      r.blocking_session_id,
      r.wait_type,
      r.wait_time,
      r.wait_resource,
      t.text AS sql_text
  FROM sys.dm_exec_requests r
  OUTER APPLY sys.dm_exec_sql_text(r.sql_handle) t
  WHERE r.session_id > 50;  -- Ignore system sessions
  

  Look for:

  • blocking_session_id <> 0
  • Long wait_time
  • wait_type like LCK_M_* (lock waits) on AbpRoles, AbpUsers, or AbpUserRoles.

  2. Check open transactions

  SELECT
      s.session_id,
      s.login_name,
      s.host_name,
      r.status,
      r.command,
      r.blocking_session_id,
      t.transaction_id,
      t.transaction_type,
      t.transaction_state,
      at.name as transaction_name
  FROM sys.dm_tran_active_transactions t
  JOIN sys.dm_tran_session_transactions st ON t.transaction_id = st.transaction_id
  JOIN sys.dm_exec_sessions s ON st.session_id = s.session_id
  LEFT JOIN sys.dm_exec_requests r ON r.session_id = s.session_id
  LEFT JOIN sys.dm_tran_active_snapshot_database_transactions at 
         ON t.transaction_id = at.transaction_id;
  

  If you see:

  • Long‑lived transactions holding locks on Abp* tables, kill them (carefully) or stop the service that opened them.

  3. Ensure all migrations are applied

  From the ABP DbMigrator (or equivalent migration service for Identity):

  • Run migrations against the production Identity database.
  • Confirm AbpUsers, AbpRoles, AbpUserRoles schemas match dev.

  A partially run migration can leave locks or long operations.

  ---

  B. ABP / EF Core mitigation (increase command timeout)

  For an immediate workaround (to stop production user‑creation failures while you investigate), increase the EF Core command timeout for the IdentityService DbContext.

  In your IdentityService EF Core configuration module (e.g. IdentityServiceEntityFrameworkCoreModule or the module where you configure AbpDbContextOptions), add:

  using Microsoft.EntityFrameworkCore;
  using Volo.Abp.EntityFrameworkCore;
  
  public override void ConfigureServices(ServiceConfigurationContext context)
  {
      Configure<AbpDbContextOptions>(options =>
      {
          options.Configure<IdentityServiceDbContext>(c =>
          {
              c.UseSqlServer(
                  c.ExistingConnection ?? 
                  c.ConnectionString,
                  sqlOptions =>
                  {
                      // Increase command timeout (seconds)
                      sqlOptions.CommandTimeout(120);
                  });
          });
      });
  }
  

  Notes:

  • IdentityServiceDbContext is the DbContext from your Identity microservice (CloverleafCMS.IdentityService.Data.IdentityServiceDbContext in your stack trace).
  • This does not fix blocking, but prevents transient slow operations from throwing at 30 seconds.

  ---

  C. Check for custom code holding locks

  If you have any custom logic that:

  • Runs inside long unit of work scopes, or
  • Iterates over many users/roles and updates them in one large transaction, or
  • Uses manual database transactions,

  make sure those operations:

  1. Use short‑lived units of work.
  2. Are not running concurrently with normal user‑creation.
  3. Use background jobs at low traffic times if they touch Identity tables heavily.

  ABP Unit of Work best practices:

  • Prefer short transactions per operation (per request / per app service method).
  • Avoid long loops that update many AbpUsers/AbpRoles in a single UoW.

  ---

  D. Sanity checks in the Identity microservice

  1. Confirm there is no custom trigger or external process on AbpUsers, AbpRoles, AbpUserRoles that could be slow or deadlocking.
  2. Confirm connection string for IdentityService in production really points to the expected DB (not some shared, heavily used DB).
  3. Check SQL Server error log and performance metrics (CPU, IO, deadlocks) during a failing user‑creation attempt.

  ---

  If you do the SQL blocking checks while reproducing the problem (trigger a user creation and immediately inspect sys.dm_exec_requests / sys.dm_tran_locks), you will see exactly which session / query is blocking the AbpRoles / AbpUsers updates. Removing or fixing that blocker is the real solution; the increased timeout just buys time to keep production functional.

  ---

  This response was generated by AI. If it didn’t solve your problem, feel free to post a new answer — our support team will jump in to help you soon.

  Save Cancel
• 

  0

  rogercprops created 10 days ago

  We can close this for now. I ran the queries above and saw that the authserver had suspended transactions. I restarted the pod and the problem went away.

  Save Cancel

  1