Towards the end of last week, the success team received a few reports of undelivered tasks and agendas. The engineering team investigated and found that the part of our system that queues up background tasks—like emails and texts—occasionally backed up to the extent that a task timed out after a number of retries. There are several queues in the system that share a finite set of resources, and things like QuickBooks syncs and search indexing sometimes generate a large number of background tasks.
The problem went away on its own with no obvious cause, and troubleshooting became more challenging without the ability to see the system under load. Engineers focused on other pressing support and maintenance issues. Unfortunately, the condition degraded over the long weekend until it affected many accounts on Monday, May 29th. This was probably exacerbated by higher-than-normal, post-Memorial Day weekend activity. For example: everyone arriving at the office on Tuesday and running a big QuickBooks sync at the same time.
We’re addressing this specific issue, and guarding against future occurrences, by doing the following:
We are sorry this happened and are grateful to you for bearing with us.