Incident Report for Springboard Health

A configuration change in our ArgoCD continuous deployment platform caused nondeterministic behavior when applied to our Kubernetes ingress controllers.  Though this change was successfully applied in our development environment, when the same configuration was deployed to the production Springboard UI, it put the cluster into an inaccessible state.  We manually intervened to roll back the change as quickly as possible, but found that ArgoCD was still intermittently mis-applying the configuration in this environment.  Further troubleshooting efforts identified a stale cache as the culprit; once we forced a hard reset of ArgoCD’s cache, the correct configuration was applied, and the configuration UI returned to a fully functional state.

No Connected Search indexing or query functionality was affected during this incident; the extent of the impact was limited to the web-based configuration interface, and no data loss occurred.

Posted Oct 05, 2023 - 20:14 UTC

We have confirmed that reverting our earlier change has corrected the root cause of this incident, and is once again fully functional.
Posted Sep 29, 2023 - 18:24 UTC
The interface is once again available and functional. No indexing or search query outage was experienced during this incident. We're continuing to monitor to ensure there are no lingering issues.
Posted Sep 29, 2023 - 17:55 UTC
We have identified the source of the issue and have reverted the recent change that caused it. The fix is actively rolling out. Some users will see recovery and be able to log in to already, while for others it may not show full recovery until the rollout has fully completed.
Posted Sep 29, 2023 - 17:29 UTC
The configuration UI at is currently unavailable and returning errors when users attempt to access it. Lucidworks engineers are actively investigating the problem.

This incident does NOT affect any indexing or search functionality; it is only preventing configuration changes to existing Connected Search applications.

We will post updates as soon as we have more details to share.
Posted Sep 29, 2023 - 17:14 UTC
This incident affected: Lucidworks Platform (User Logins & Configuration UI).