Re-indexing all AI workspace datasources bug #10789

4 Answer(s)

• 

  0

  maliming created 2 months ago

  Support Team Fullstack Developer

  Hi,

  ReIndex All enqueues one background job per document, so all of them are expected to run one after another. To find out why the ones after the first stay on Processing, we need to see what the second document's job does.

  Could you enable Debug logs for these namespaces and run ReIndex All once more:

  "Logging": {
    "LogLevel": {
      "Volo.AIManagement": "Debug",
      "Volo.Abp.BackgroundJobs": "Debug"
    }
  }
  

  Here's how to enable Debug logs if you need it: https://abp.io/support/questions/8622/How-to-enable-Debug-logs-for-troubleshoot-problems

  Then send the full log starting from when you click ReIndex All. The key part is what happens right after the first Successfully indexed document ... line — for the second document, does the log:

  • never show Starting document indexing ... (the job didn't start),
  • show Starting ... but then no Successfully indexed and no error (it's stuck on a step), or
  • show a Failed to index document ...?

  Two quick questions so we look in the right place:

  1. Which background job provider are you using — the default one or Hangfire?
  2. Which embedder provider is configured for the workspace (OpenAI, Azure OpenAI, or a local/self-hosted one)?

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  LW created 2 months ago

  Even with debug logging, I don't see much more information, should there be? WHERE t.SCHED_NAME = @schedulerName AND t.TRIGGER_NAME = @triggerName AND t.TRIGGER_GROUP = @triggerGroup 2026-07-20 08:42:36.866 +03:00 [DBG] Prepared SQL: UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = @newState WHERE SCHED_NAME = @schedulerName AND TRIGGER_NAME = @triggerName AND TRIGGER_GROUP = @triggerGroup AND TRIGGER_STATE = @oldState AND NEXT_FIRE_TIME = @nextFireTime 2026-07-20 08:42:36.867 +03:00 [DBG] Prepared SQL: INSERT INTO QRTZ_FIRED_TRIGGERS (SCHED_NAME, ENTRY_ID, TRIGGER_NAME, TRIGGER_GROUP, INSTANCE_NAME, FIRED_TIME, SCHED_TIME, STATE, JOB_NAME, JOB_GROUP, IS_NONCONCURRENT, REQUESTS_RECOVERY, PRIORITY) VALUES(@schedulerName, @triggerEntryId, @triggerName, @triggerGroup, @triggerInstanceName, @triggerFireTime, @triggerScheduledTime, @triggerState, @triggerJobName, @triggerJobGroup, @triggerJobStateful, @triggerJobRequestsRecovery, @triggerPriority) 2026-07-20 08:42:36.869 +03:00 [DBG] Batch acquisition of 1 triggers 2026-07-20 08:42:37.129 +03:00 [INF] Starting document indexing for DataSource "513627ef-a624-fa94-cd0c-3a226d65b3fc" in Workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:38.191 +03:00 [INF] Deleted 8 vector embeddings for data source "513627ef-a624-fa94-cd0c-3a226d65b3fc" in workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:38.193 +03:00 [INF] Deleted 8 old embeddings for DataSource "513627ef-a624-fa94-cd0c-3a226d65b3fc" 2026-07-20 08:42:38.210 +03:00 [INF] Deleted 8 old document chunks for DataSource "513627ef-a624-fa94-cd0c-3a226d65b3fc" 2026-07-20 08:42:38.258 +03:00 [INF] Processing document with content type: text/markdown. Available extractors: 3 2026-07-20 08:42:38.258 +03:00 [INF] Extractor MarkdownExtractor can handle text/markdown: true 2026-07-20 08:42:38.258 +03:00 [INF] Extractor PdfExtractor can handle text/markdown: false 2026-07-20 08:42:38.259 +03:00 [INF] Extractor PlainTextExtractor can handle text/markdown: false 2026-07-20 08:42:38.293 +03:00 [DBG] Split text into 27 paragraphs 2026-07-20 08:42:38.293 +03:00 [INF] Created 8 chunks from text (total length: 4197 chars) 2026-07-20 08:42:38.382 +03:00 [INF] Processed document "513627ef-a624-fa94-cd0c-3a226d65b3fc" into 8 chunks 2026-07-20 08:42:39.531 +03:00 [INF] Generated embeddings for batch 1 (0-7 of 8 chunks) 2026-07-20 08:42:39.765 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_0 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:39.824 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_1 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:39.893 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_2 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:39.962 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_3 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:40.025 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_4 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:40.087 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_5 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:40.145 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_6 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:40.207 +03:00 [DBG] Stored vector embedding 513627ef-a624-fa94-cd0c-3a226d65b3fc_7 for workspace "3c8fa649-0f85-1ea8-c9bc-3a226cb0398f" 2026-07-20 08:42:40.235 +03:00 [INF] Successfully indexed document "513627ef-a624-fa94-cd0c-3a226d65b3fc": 8 chunks created and embedded 2026-07-20 08:42:40.236 +03:00 [DBG] Trigger instruction : DeleteTrigger 2026-07-20 08:42:40.239 +03:00 [DBG] Lock 'TRIGGER_ACCESS' is desired by: "ab953fcd-0ff5-4d5c-bc7b-6de4ac86dbf2" 2026-07-20 08:42:40.239 +03:00 [DBG] Prepared SQL: SELECT * FROM QRTZ_LOCKS WITH (UPDLOCK,ROWLOCK) WHERE SCHED_NAME = @schedulerName AND LOCK_NAME = @lockName 2026-07-20 08:42:40.240 +03:00 [DBG] Lock 'TRIGGER_ACCESS' is being obtained: "ab953fcd-0ff5-4d5c-bc7b-6de4ac86dbf2" 2026-07-20 08:42:40.240 +03:00 [DBG] Lock 'TRIGGER_ACCESS' given to: "ab953fcd-0ff5-4d5c-bc7b-6de4ac86dbf2" 2026-07-20 08:42:40.240 +03:00 [DBG] Prepared SQL: SELECT TRIGGER_STATE, NEXT_FIRE_TIME, JOB_NAME, JOB_GROUP FROM QRTZ_TRIGGERS WHERE SCHED_NAME = @schedulerName AND TRIGGER_NAME = @triggerName AND TRIGGER_GROUP = @triggerGroup 2026-07-20 08:42:40.242 +03:00 [DBG] Prepared SQL: DELETE FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = @schedulerName AND TRIGGER_NAME = @triggerName AND TRIGGER_GROUP = @triggerGroup 2026-07-20 08:42:40.243 +03:00 [DBG] Prepared SQL: DELETE FROM QRTZ_TRIGGERS WHERE SCHED_NAME = @schedulerName AND TRIGGER_NAME = @triggerName AND TRIGGER_GROUP = @triggerGroup 2026-07-20 08:42:40.246 +03:00 [DBG] Prepared SQL: SELECT COUNT(TRIGGER_NAME) FROM QRTZ_TRIGGERS WHERE SCHED_NAME = @schedulerName AND JOB_NAME = @jobName AND JOB_GROUP = @jobGroup 2026-07-20 08:42:40.247 +03:00 [DBG] Prepared SQL: DELETE FROM QRTZ_JOB_DETAILS WHERE SCHED_NAME = @schedulerName AND JOB_NAME = @jobName AND JOB_GROUP = @jobGroup 2026-07-20 08:42:40.247 +03:00 [DBG] Deleting job: DEFAULT.Volo.AIManagement.BackgroundJobs.IndexDocumentJobArgs

  It just indexes the first one and then the job is done. We use Quartz. I have done this with Azure OpenAI and local Ollama embedders with the same result.

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 2 months ago

  Support Team Fullstack Developer

  Hi,

  The DeleteTrigger and Deleting job lines are just Quartz's normal cleanup after a one-shot job finishes, so they don't mean the other documents' jobs were deleted.

  The real signal is the JobKey:

  DEFAULT.Volo.AIManagement.BackgroundJobs.IndexDocumentJobArgs
  

  That key is the job arguments type name. ABP's stock QuartzBackgroundJobManager never sets a job identity, so Quartz gives every enqueue a fresh GUID key and each document gets its own job. Your log uses the args type name as a fixed key instead, and with one shared key only a single IndexDocumentJob can live in the scheduler at a time, so ReIndex All (which enqueues one job per document) only runs the first.

  That fixed key isn't ABP's default, so something in your app is setting it. To pin down exactly where and give you the precise fix, could you share either:

  1. The class that implements or replaces IBackgroundJobManager, plus your AddQuartz / background-job registration code, or
  2. A small project that reproduces it (a fresh app with your Quartz configuration and a couple of documents is enough). A private GitHub repo with https://github.com/maliming invited, or a zip to liming.ma@volosoft.com, both work.

  Once I can see where the key is set, the fix is to let each enqueue get its own job (a unique key per enqueue), with a per-document lock or dedup check if the fixed key was added to avoid indexing the same document twice.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel
• 

  0

  maliming created 2 months ago

  Support Team Fullstack Developer

  Hi,

  Thanks, that confirms it. In SCMQuartzBackgroundJobManager the Quartz JobKey is built from BackgroundJobNameAttribute.GetName<TArgs>() (which falls back to the args type full name when it has no [BackgroundJobName]), plus _{TenantId} for IMultiTenant args and _{UniqueKey} for IUniqueBackgroundJob args. That works for your own jobs, but the module's IndexDocumentJobArgs implements neither interface, it's just a class with a DataSourceId.

  So every reindex job ends up with the same key, Volo.AIManagement.BackgroundJobs.IndexDocumentJobArgs. ReIndex All enqueues one job per document, but while the first job and its trigger still exist in Quartz, your CheckExists branch hits "a job already exists and is not in error, let's not schedule a new job" and returns without scheduling. Nothing throws, so ReIndexAllAsync finishes normally and resets every document (which the UI then shows as Processing), but only the first one actually has a job.

  The fix is to give jobs that don't opt into your identity scheme a unique key per enqueue, like stock ABP does. Where you add it matters, it has to be the fallback of the IUniqueBackgroundJob check, not the tenant else, otherwise a non-tenant IUniqueBackgroundJob would get a GUID and its UniqueKey and lose its stable identity:

  if (args is IUniqueBackgroundJob uniqueBackgroundJob && !string.IsNullOrEmpty(uniqueBackgroundJob.UniqueKey))
  {
      jobName = $"{jobName}_{uniqueBackgroundJob.UniqueKey}";
  }
  else
  {
      jobName = $"{jobName}_{Guid.NewGuid():N}";
  }
  

  This makes anything that isn't an IUniqueBackgroundJob run on its own, like ABP's default manager. If some of your own jobs relied on the CheckExists dedup to avoid running twice, mark those with IUniqueBackgroundJob (with a stable UniqueKey) so they keep that behaviour. These jobs aren't durable, so the GUID keys don't accumulate, Quartz removes each one once it finishes and its trigger is gone.

  A few practical notes:

  • Documents already stuck in Processing won't recover on their own, run ReIndex All again after deploying.
  • With the fix, every document's job is enqueued to start right away, so how many run in parallel comes down to the Quartz worker threads available across your cluster. Worth a quick check against your embedding provider's rate limits for workspaces with a lot of documents.
  • In 10.5 the indexing job doesn't take a per-document lock, so avoid re-triggering ReIndex on a document that's still indexing, two overlapping runs on the same data source can delete each other's chunks/vectors. That's existing 10.5 behaviour, not something this change introduces.

  Unrelated, but I noticed it in the config you sent: quartz.scheduler.instanceId is set to a fixed value. If you run more than one node against this clustered store, each node needs a distinct instanceId (usually AUTO), a shared one interferes with Quartz's cluster and failover handling.

  Thanks

  Markdown supported.

  Copy, paste, or drag & drop images and files (max 100 MB per file, 100 MB total per post)

  Save Cancel