Delivery issues with some email notifications
Incident Report for Kickserv
Postmortem

Towards the end of last week, the success team received a few reports of undelivered tasks and agendas. The engineering team investigated and found that the part of our system that queues up background tasks—like emails and texts—occasionally backed up to the extent that a task timed out after a number of retries. There are several queues in the system that share a finite set of resources, and things like QuickBooks syncs and search indexing sometimes generate a large number of background tasks.

The problem went away on its own with no obvious cause, and troubleshooting became more challenging without the ability to see the system under load. Engineers focused on other pressing support and maintenance issues. Unfortunately, the condition degraded over the long weekend until it affected many accounts on Monday, May 29th. This was probably exacerbated by higher-than-normal, post-Memorial Day weekend activity. For example: everyone arriving at the office on Tuesday and running a big QuickBooks sync at the same time.

We’re addressing this specific issue, and guarding against future occurrences, by doing the following:

  • Adding more monitoring for task notifications and other essential background tasks in the system. In the event that the notification queue starts to enter a retry situation, we will alert engineering and reprioritize the notification queue higher so that tasks do not get dropped.
  • Prioritizing updates to infrastructure software. The part of our system that runs background tasks is starting to get out of date, and newer versions offer improved performance and stability.

We are sorry this happened and are grateful to you for bearing with us.

Posted May 31, 2023 - 10:19 CDT

Resolved
This incident has been resolved.
Posted May 31, 2023 - 09:36 CDT
Update
We are continuing to monitor for any further issues.
Posted May 30, 2023 - 15:58 CDT
Monitoring
We have identified this as an issue with our email sending queue. We updated some settings on the queue and emails are sending normally. Please let us know if you experience any further delivery issues. Thanks!
Posted May 30, 2023 - 15:52 CDT
Investigating
We've received reports that email notifications are not being delivered in some cases. We're looking into it now.
Posted May 30, 2023 - 09:20 CDT
This incident affected: Email Notifications.