SaaS Platform Service Routing Disruption

Incident Report for Lucidworks Platform

Postmortem

Summary

On April 24, 2026, at 16:05 UTC, the Lucidworks SaaS Platform experienced an issue that made the entire platform temporarily unavailable and all requests returned IO errors. Lucidworks Engineering was made aware of the issue and reverted the change on April 24, 2026, at 16:08 UTC. The reversion took a few minutes to propagate through the system and successful responses were restored by April 24, 2026, at 16:13 UTC.

Root Cause

The incident was caused by a configuration change to the SaaS Platform gateway components that was deployed to production at 16:05 UTC on April 24, 2026. This change introduced an error in the configuration template that caused it to render syntactically invalid output in the production environment. When the deployment system processed this invalid configuration, it interpreted the malformed output as an indication that the gateway components (routing and load-balancing infrastructure) were no longer needed and removed these components from the platform. This caused all incoming requests to fail with IO errors as there were no gateway instances available to handle traffic. Lucidworks engineers were actively monitoring this deployment as it rolled out, and immediately noticed the problem that the deployment introduced. The configuration change was quickly reverted at 16:08 UTC. The deployment infrastructure then correctly reinstated the gateway components, which became fully operational by 16:13 UTC and restored normal platform operation.

Lucidworks Actions

Lucidworks has taken the following actions as a result of this incident:

  • Added pre-deployment validation to verify that configuration templates render syntactically valid output before deployment, and that changes of this nature are deployed to a development environment before deploying the changes to production
  • Modified the deployment workflow to require additional verification steps before pull requests can be merged

Recommended Client Actions

Lucidworks recommends that clients subscribe to Lucidworks status updates to receive real-time notifications about Lucidworks SaaS Platform incidents. To enable this feature, click Subscribe to Updates at status.lucidworks.com.

Posted May 18, 2026 - 23:02 UTC

Resolved

The Lucidworks SaaS Platform experienced an issue that caused a total disruption of service routing, rendering the platform inaccessible. Users were unable to log in or manage any platform configurations during this period. End users were unable to access search results or interact with any deployed services. The incident was caused by an unintended deployment of a configuration change at 4:06 UTC that should not have gone to production. We resolved the issue by performing a full rollback of the faulty change, which was completed at 4:13 UTC. A postmortem report will be posted with the full incident details.
Posted Apr 24, 2026 - 16:06 UTC