Resolved -
After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials.
Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments. Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance.
Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10.
Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.
Aug 8, 02:43 UTC
Monitoring -
We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5. The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced.
We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers. We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue.
A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment. We will post another update here as soon as that has been deployed.
Aug 7, 23:14 UTC
Investigating -
A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.
Aug 7, 22:54 UTC
