Alerting Disruption - Auvik API: Alert History Returning 400 Bad Request Error

Duration of incident

Discovered: Date Jan, 20, 2022 - 16:04 UTC
Resolved: Date Jan 20,2022 - 19:452 UTC

Cause

An update was deployed to Auvik’s alerting API code that resulted in a broken dependency with other APIs.

Effect

Auvik was unable to process API alerts for the duration of the incident.

Action taken

01/20/2022

16:04 UTC - A customer opens a ticket of a “400 bad request error” to Auvik support concerning alert APIs. Several other alerts follow closely thereafter.
16:17 UTC- The offending change is identified by the engineering team.
16:19 UTC - A fix is deployed to the staged environment to test the results.
18:45 UTC - The fix has been validated in the staged environment and approved. The team proceeded to implement to the production environment.
19::52 UTC - The fix deployed to the production environments has been validated.

Future consideration(s)

Separate components of the API service were involved with the update. There was improper understanding of the dependencies between these components and the internal backward compatibility of the service.

• Auvik will institute an internal documented list of services and component dependencies
• Auvik will develop a more robust versioning process to prevent improvements causing incompatibility