[Major] Issue with Uploads, Downloads, Logins, and Public API
[Major] Issue with Uploads, Downloads, Logins, and Public API
May 17, 11:00am
May 17, 11:00am
We recently addressed issues affecting uploads, downloads, logins, and public API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future. On May 17, 2022 between 11:05 AM PDT and 11:07 AM PDT, some users may have experienced difficulties while working in Box. During this time, uploads, downloads, logins, and public API experienced some failures or errors. **Analysis** The issue occurred when the leader node of an internal clustered system failed a soft restart. During a maintenance window, an internal infrastructure system that uses a leader-standby clustering model failed to elect a new leader. As a result, some users experienced issues with uploads, downloads, logins, and public API. Additionally, during the impacted timeframe, internal client services of the system did not rely on cached data as a fall-back, which exacerbated the issue. We were able to resolve this issue by issuing a hard restart of the affected node. We are continuing to investigate to determine the root cause of the issue and are developing additional corrective actions to prevent similar issues from occurring in the future. **Corrective Actions** The following corrective actions have been completed or are planned: * Performing a hard restart of the affected node to return it to a normal state. * Identifying and addressing the underlying root cause of the cluster leader-election issue. * Improving the clustering configuration to permit making a cluster passive before maintenance. * Conducting further analysis to determine why internal services did not utilize cached internal data during the impacted timeframe. We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. Sincerely, The Box Team