We had an issue today where apps were unable to sync new changes from our backend servers for around 90 minutes

. During this period, some apps displayed sync errors, or continuously tried to sync, without getting the latest changes.

The issue was caused by an update in the sync processing code on our backends. The update optimizes some aspects of sync reprocessing, which should ultimately result in improved sync performance. Unfortunately, a bug in this update caused a small number of apps to immediate trigger sync reprocessing.

Even though it was a small percentage of apps performing reprocessing, it was enough to saturate the sync processing queue in our ZA backend. This caused sync processing for other apps to be delayed, which prevented users from those apps from syncing the latest changes.

We resolved the issue by scaling up the sync processing capacity, and cancelling reprocessing for apps where it was not required.

To avoid this issue from occurring in the future, we are making changes to the sync processing system to ensure that reprocessing is never prioritized over processing the latest changes. This would ensure that even when a large number of apps are busy reprocessing sync rules, the latest changes would continue to be synced in all apps.