Beacon configs failing to load
Incident Report for Lucidworks Platform
Postmortem

Summary

Beginning at approximately 10:17am PDT and lasting until service restoration at approximately 11:52am, the Lucidworks Signals Beacon on client sites was not functional due to an error in retrieving its configuration from the Lucidworks Platform. During this time, some signals from new site visits were not collected.

Root Cause

A database change that was intended to only be applied in our development environment for testing purposes was inadvertently applied to the production database, which resulted in the creation of new, empty tables for storing Beacon configurations and authorization information. For the duration of this incident, when the Beacon initialized and attempted to retrieve its configuration from the Lucidworks Platform at page load time, the call failed with an HTTP 401 Unauthorized response, and the Beacon would silently fail to load. This behavior, while not site-impacting in any way, does result in the inability to send user behavior signals to Lucidworks Analytics, and so for the duration of this incident some analytics data for affected clients will be incomplete.

The database change was rolled back, which allowed the Platform to once again successfully return Beacon configurations, which had persisted in the previously-used table throughout the duration of the incident.  No configuration data was lost, and Signals Beacon functionality has been fully restored.

Lucidworks Actions

We will update our Beacon and Signals APIs to return HTTP 500 errors when the configuration cannot be loaded, in order to generate a higher-severity alert for our teams so that we can more urgently react to similar issues in the future.

We will also enhance the Beacon to cache the most recently retrieved configuration in local browser storage so that there is a fallback in addition to a potential backoff strategy to minimize throwing errors.

Recommended Client Actions

There are no recommended client actions as a result of this event. Affected customers with additional questions or concerns may contact us via Lucidworks Support.

Posted Jul 24, 2024 - 19:20 PDT

Resolved
The fix has been verified and we've confirmed that full functionality has been restored. No Signals Beacon configurations were lost during this incident.
Posted Jul 24, 2024 - 11:57 PDT
Monitoring
We identified the source of the error as being a database configuration update that was inadvertently applied to our production site, rather than in our development environment as intended. We have rolled back that change, and the Beacon is able to successfully load its configuration again. We're continuing to monitor to ensure there are no ongoing effects of this incident.
Posted Jul 24, 2024 - 11:40 PDT
Investigating
The Lucidworks Signals Beacon is currently unable to load its configuration due to a server-side error on the Lucidworks Platform. As a result, while existing site visits will continue to send signals to Lucidworks Analytics, new page loads will not be able to initialize the Beacon and some signals will not be collected. We are working to restore full functionality as soon as possible, and will provide timely updates as this incident is addressed.
Posted Jul 24, 2024 - 10:30 PDT
This incident affected: Lucidworks Analytics (Beacon & Signals Ingestion).