A configuration change in our ArgoCD continuous deployment platform caused nondeterministic behavior when applied to our Kubernetes ingress controllers.  Though this change was successfully applied in our development environment, when the same configuration was deployed to the production Springboard UI, it put the cluster into an inaccessible state.  We manually intervened to roll back the change as quickly as possible, but found that ArgoCD was still intermittently mis-applying the configuration in this environment.  Further troubleshooting efforts identified a stale cache as the culprit; once we forced a hard reset of ArgoCD’s cache, the correct configuration was applied, and the configuration UI returned to a fully functional state.

No Connected Search indexing or query functionality was affected during this incident; the extent of the impact was limited to the web-based configuration interface, and no data loss occurred.