Errors when adding job charges
Incident Report for Kickserv
Postmortem

On Tuesday, July 27, our engineers deployed an update to permissions on our items API. This particular API call is is normally used to list products and services, and should only be accessible to users who have a role of Owner or Admin. However, after deployment, we received reports of problems adding job charges. After investigating, we found that the same API call used in the item list view was also being used to search for an item while adding new job charges. This meant that Staff and Tech users were no longer able to add a job charge without running into an error.

As usual, the remedy for this sort of bug involves two steps: 1) stop the immediate symptoms; and 2) address the underlying architectural issue. We took care of step one by putting permissions back the way they were, and job charges are now functioning properly again. For step two, we’ll need to make the API work the way it always should’ve worked: use a different API call when searching for an item to add as a job charge. This is a very different use case than showing the main list of items: one is an administrative function, and the other is used in the normal job workflow. They should be treated differently.

We’ll also be reviewing our internal system documentation to come up with a more reliable way of tracking what API calls are used in what places. The Kickserv codebase is large and constantly changing, and some parts of it are close to ten years old—which is a lot of lines of code for a developer to have to keep track of! We’re always looking for ways to improve quality of life for our engineers as well as our customers. We regret that this happened, and thanks for bearing with us.

Posted Jul 28, 2021 - 06:59 PDT

Resolved
Staff and technician users are once again able to add job charges in the web and mobile apps. Thanks for your patience.
Posted Jul 28, 2021 - 06:37 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 28, 2021 - 06:27 PDT
Update
This issue currently affects only users in the staff or tech roles. Admins and owners are unaffected. The fix is minor and is in progress.
Posted Jul 27, 2021 - 09:19 PDT
Identified
We are currently seeing errors attempting to add charges to a job. This is related to a minor permissions change we just added to our API. The fix is in progress and shouldn't take long. Thanks for bearing with us.
Posted Jul 27, 2021 - 09:17 PDT
This incident affected: Web Application (kickservapp.com), iOS App, and Android App.