Lucidworks Platform and Search Service Disruption

Incident Report for Lucidworks Platform

Postmortem

Summary

On August 7, 2026, Lucidworks was alerted to several application credentials being invalidated across Connected Search, Lucidworks AI, and Lucidworks Platform, and declared a Sev1 incident. Lucidworks found that a regular cleanup job had begun erroneously designating active integrations for deletion starting on August 5, 2026, resulting in credentials that were actively in use being incorrectly invalidated and recreated. Lucidworks later determined that this same issue also invalidated embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking. We disabled the cleanup job on August 7 to stop further disruptions, rotated credentials for confirmed impacted customers, and deployed a permanent fix on August 12. No further customer-facing errors have been detected since the fix was deployed.

Root Cause

A recent code change to the backend service that manages customer integrations with a third-party authentication and authorization provider altered how that service checked for outdated integrations across a customer's full set of applications. As a result, the service could only see a partial view of each customer's active integrations rather than the complete list. Due to this, it incorrectly treated valid, active integrations as no longer in use and recreated them, generating new backend credentials in the process. Recreating an integration this way had a secondary effect: it also invalidated the embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking, in addition to the backend Platform credentials. This behavior occurred in irregular, clustered bursts rather than on a steady, predictable schedule, which made the resulting customer impact appear intermittent and difficult to correlate to a single cause.

Lucidworks determined the root cause by correlating the timing of the incorrect credential and embed token changes with a recent code change to the affected service, and confirmed the diagnosis by tracing how that service determined which integrations were still active. Lucidworks confirmed the fix by deploying a corrected version of the service and validating that credential and embed token regeneration had stopped.

Lucidworks Actions

Lucidworks has taken and will take the following actions as a result of this incident:

  • Disabled the automated integration-reconciliation process to immediately stop further credential and embed token disruptions
  • Deployed a corrected version of the affected service and re-enabled the reconciliation process after validating the fix
  • Rotated third-party authentication and authorization credentials for confirmed impacted customer environments
  • We will audit and add consistent customer, application, and request tracking to logs across affected services to reduce future diagnosis time
  • We will evaluate an underlying third-party library defect identified during this investigation and determine whether an upgrade can remove the need for its current workaround
  • We will review how integration recreation is handled to reduce the blast radius of future defects, so that a single error cannot invalidate embed tokens for unrelated functionality
  • We will review our proactive customer outreach process to ensure affected customers are notified consistently, regardless of whether a customer contacts Support directly themselves

Recommended Client Actions

Lucidworks previously recommended that affected customers verify that any embedded Agent Studio widgets, Analytics signal collection Beacons, or usage metrics tracking are functioning normally following this incident, and contact Lucidworks Support if authentication errors persist. Lucidworks communicated this recommendation directly to affected customers prior to publication of this report.

Posted Aug 18, 2026 - 00:24 UTC

Resolved

After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials.

Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments.  Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance.

Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10.

Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.
Posted Aug 08, 2026 - 02:43 UTC

Monitoring

We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5.  The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced.

We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers.  We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue.

A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment.  We will post another update here as soon as that has been deployed.
Posted Aug 07, 2026 - 23:14 UTC

Investigating

A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.
Posted Aug 07, 2026 - 22:54 UTC
This incident affected: Lucidworks Platform, Lucidworks AI, and Connected Search.