Postmortem -
Read details
Dec 19, 11:19 EST
Resolved -
The incident has been fully resolved, and all services are operating normally.
Customers should no longer experience any related issues. If you continue to experience problems, please don't hesitate to contact Auvik Support.
We will provide a Root Cause Analysis (RCA) once it is available.
Dec 11, 18:46 EST
Update -
We are continuing to work on a fix for this issue.
Dec 11, 17:48 EST
Update -
Our team has begun isolating the devices requiring correction.
Auvik has placed the EU2 cluster in maintenance mode to continue working on the issue and prevent further false alerts.
Impact:
All alerting for the EU2 cluster has been placed in maintenance mode.
The following services are not affected: All other clusters.
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Dec 11, 16:54 EST
Update -
Our team continues to investigate the suspected cause of the duplicate Fortinet firewalls and is taking steps to remediate the issue.
Auvik will place the EU2 cluster in maintenance mode to continue working on the issue and prevent further false alerts.
Impact:
All alerting for the EU2 cluster has been placed in maintenance mode.
The following services are not affected: All other clusters.
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Dec 11, 15:37 EST
Identified -
Our team continues to investigate the suspected cause of the duplicate Fortinet firewalls and is taking steps to remediate the issue.
Impact:
Customers may continue to experience alerts from duplicate Fortinet firewalls.
The following services are not affected: All other clusters and vendors.
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are applying new mitigation measures and will provide updates on progress.
Dec 11, 15:00 EST
Monitoring -
We have applied changes to address the issue. Services are beginning to revert to normal, and we are monitoring closely for stability.
Impact:
Extraneous Fortinet firewalls will continue to be flushed from affected tenants; however, if you continue to encounter problems, please report them to Auvik Support.
Next Steps:
A final update will be posted once we confirm the resolution.
Dec 11, 12:01 EST
Update -
Our team has begun deleting duplicate Fortinet firewalls.
Impact:
Customers may continue to receive excessive resolved alerts while duplicates are cleared from the system.
The following services are not affected: Any other devices or clusters.
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are implementing mitigation measures and will provide progress updates.
Dec 11, 10:48 EST
Identified -
Our team has identified a suspected cause of the Fortinet firewall duplication and is taking steps to remediate the issue.
Impact:
Customers may continue to receive some excessive resolved alerts while duplicates are cleared from the system.
The following services are not affected: Any other devices or clusters.
Please report any related issues to Auvik Support so we can track and assist further.
Next Steps:
We are implementing mitigation measures and will provide progress updates.
Dec 11, 09:43 EST
Investigating -
We are currently investigating reports of duplicate devices being created, which could cause alert flooding.
Impact:
Customers may experience a high false alert definitions
The following services are not affected by all other clusters and monitoring.
Next Steps:
Our team is working to identify contributing factors. Updates will follow as more information becomes available.
Dec 11, 09:30 EST
