Beginning at approximately 10:17am PDT and lasting until service restoration at approximately 11:52am, the Lucidworks Signals Beacon on client sites was not functional due to an error in retrieving its configuration from the Lucidworks Platform. During this time, some signals from new site visits were not collected.
A database change that was intended to only be applied in our development environment for testing purposes was inadvertently applied to the production database, which resulted in the creation of new, empty tables for storing Beacon configurations and authorization information. For the duration of this incident, when the Beacon initialized and attempted to retrieve its configuration from the Lucidworks Platform at page load time, the call failed with an HTTP 401 Unauthorized response, and the Beacon would silently fail to load. This behavior, while not site-impacting in any way, does result in the inability to send user behavior signals to Lucidworks Analytics, and so for the duration of this incident some analytics data for affected clients will be incomplete.
The database change was rolled back, which allowed the Platform to once again successfully return Beacon configurations, which had persisted in the previously-used table throughout the duration of the incident. No configuration data was lost, and Signals Beacon functionality has been fully restored.
We will update our Beacon and Signals APIs to return HTTP 500 errors when the configuration cannot be loaded, in order to generate a higher-severity alert for our teams so that we can more urgently react to similar issues in the future.
We will also enhance the Beacon to cache the most recently retrieved configuration in local browser storage so that there is a fallback in addition to a potential backoff strategy to minimize throwing errors.
There are no recommended client actions as a result of this event. Affected customers with additional questions or concerns may contact us via Lucidworks Support.