Duplicate FortiNet firewalls created on the EU2 cluster.

Incident Report for Auvik Networks Inc.

Postmortem

Service Degraded - FortiGate firewalls duplicated in the EU2 cluster, causing a flood of alerts.

Root Cause Analysis

Duration of the incident

Discovered: Dec 11, 2025 14:00 – UTC
Resolved: Dec 17, 2025 06:45 – UTC

Customer impact

Clients with FortiGate firewalls in the EU2 region experienced:

  • A significant increase in “device offline” alerts due to status flapping between duplicate and original device entries.
  • Duplicate firewalls appear in inventory views, leading to confusion in device management.
  • Inaccurate device statistics and monitoring gaps.
  • In some cases, customers attempted to delete duplicates, which resulted in the loss of historical data on the original device.

No other device types or regions were affected.

Cause

A system change intended to improve how high-availability firewall configurations were processed introduced unexpected behavior. Under certain conditions, the platform was unable to detect when multiple records referenced the same firewall device consistently. As a result, some firewalls were incorrectly detected as new devices, causing duplicate entries and inconsistent operational status reporting.

Effect

The issue resulted in elevated alert volumes across impacted tenants and caused instability in device-status reporting for the affected firewalls. Customers experienced increased operational overhead as they reviewed unexpected alerts and duplicate device entries, and engineering teams were required to intervene to restore accurate device associations and remove the duplicated records.fFuture consideration(s)

  • Strengthen validation of device-supplied information before it affects inventory or monitoring.
  • Implement alerting for sudden spikes in offline/online transitions to detect similar issues earlier.
  • Improve observability of device lifecycle behavior to identify anomalies proactively.
  • Use feature-flagged or staged rollouts for changes that affect device processing logic.
  • Incorporate production-like data into pre-deployment testing to better anticipate unexpected device behaviors.
Posted Dec 19, 2025 - 11:19 EST

Resolved

The incident has been fully resolved, and all services are operating normally.

Customers should no longer experience any related issues. If you continue to experience problems, please don't hesitate to contact Auvik Support.

We will provide a Root Cause Analysis (RCA) once it is available.
Posted Dec 11, 2025 - 18:46 EST

Update

We are continuing to work on a fix for this issue.
Posted Dec 11, 2025 - 17:48 EST

Update

Our team has begun isolating the devices requiring correction.
Auvik has placed the EU2 cluster in maintenance mode to continue working on the issue and prevent further false alerts.

Impact:
All alerting for the EU2 cluster has been placed in maintenance mode.
The following services are not affected: All other clusters.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Posted Dec 11, 2025 - 16:54 EST

Update

Our team continues to investigate the suspected cause of the duplicate Fortinet firewalls and is taking steps to remediate the issue.
Auvik will place the EU2 cluster in maintenance mode to continue working on the issue and prevent further false alerts.

Impact:
All alerting for the EU2 cluster has been placed in maintenance mode.
The following services are not affected: All other clusters.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Posted Dec 11, 2025 - 15:37 EST

Identified

Our team continues to investigate the suspected cause of the duplicate Fortinet firewalls and is taking steps to remediate the issue.

Impact:
Customers may continue to experience alerts from duplicate Fortinet firewalls.
The following services are not affected: All other clusters and vendors.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Posted Dec 11, 2025 - 15:00 EST

Monitoring

We have applied changes to address the issue. Services are beginning to revert to normal, and we are monitoring closely for stability.

Impact:
Extraneous Fortinet firewalls will continue to be flushed from affected tenants; however, if you continue to encounter problems, please report them to Auvik Support.

Next Steps:
A final update will be posted once we confirm the resolution.
Posted Dec 11, 2025 - 12:01 EST

Update

Our team has begun deleting duplicate Fortinet firewalls.

Impact:
Customers may continue to receive excessive resolved alerts while duplicates are cleared from the system.
The following services are not affected: Any other devices or clusters.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are implementing mitigation measures and will provide progress updates.
Posted Dec 11, 2025 - 10:48 EST

Identified

Our team has identified a suspected cause of the Fortinet firewall duplication and is taking steps to remediate the issue.

Impact:
Customers may continue to receive some excessive resolved alerts while duplicates are cleared from the system.
The following services are not affected: Any other devices or clusters.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are implementing mitigation measures and will provide progress updates.
Posted Dec 11, 2025 - 09:43 EST

Investigating

We are currently investigating reports of duplicate devices being created, which could cause alert flooding.

Impact:
Customers may experience a high false alert definitions
The following services are not affected by all other clusters and monitoring.

Next Steps:
Our team is working to identify contributing factors. Updates will follow as more information becomes available.
Posted Dec 11, 2025 - 09:30 EST
This incident affected: Network Mgmt (eu2.my.auvik.com).
← Current Status Powered by Atlassian Statuspage