Resolved -
The solution for the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud has been successfully deployed and is working. The source of the disruption has been resolved, and services have been fully restored.
A Root Cause Analysis (RCA) will follow after a full review has been completed.
Oct 2, 10:20 EDT
Update -
We are continuing to monitor for any further issues.
Oct 1, 10:12 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We will be installing an out-of-band collector update Saturday, October 1, 2022, at 8 AM ET. This will force a restart of the collector service(s). We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 30, 10:28 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We have rolled out a new fix to the CA1 cluster and are monitoring the results. We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 30, 10:00 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We will roll out a revised fix to the CA1 cluster at 8:30 AM EDT on September 30, 2022. This will cause collectors to reboot as they do during the bi-weekly maintenance. We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 29, 15:30 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We have rolled out a prepared fix to the CA1 cluster and are monitoring the results. We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 29, 10:09 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We will roll out a prepared fix to the CA1 cluster at 7 AM EDT on September 30, 2022. This will cause collectors to reboot as they do during the bi-weekly maintenance. We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 28, 15:28 EDT
Update -
We’ve identified the source of the service disruption with some of the collectors not reconnecting to the cloud in a timely manner or periodically losing connection and having a delayed reconnection to the cloud. This is a code-related issue that we continue to work through to resolve with a permanent resolution. We appreciate your patience as we continue to work toward a resolution. We will continue to post updates to this page.
Sep 28, 07:09 EDT
Update -
We’ve identified the source of the service disruption with some collectors not reconnecting to the cloud in a timely manner We are continuing to test a fix to maintain a consistent collector connection and continue to monitor the situation. We’ll keep you posted on a resolution.
Sep 27, 09:05 EDT
Update -
We’ve identified the source of the service disruption with some collectors not reconnecting to the cloud in a timely manner We are currently testing a fix to maintain a consistent collector connection and continue to monitor the situation. We’ll keep you posted on a resolution.
Sep 26, 14:00 EDT
Update -
We’ve identified the source of the service disruption with some collectors not reconnecting to the cloud in a timely manor and continue to monitor the situation. We are continuing to develop a fix to maintain a consistent collector connection. We’ll keep you posted on a resolution.
Sep 26, 08:50 EDT
Monitoring -
We’ve identified the source of the service disruption the collectors. The situation seems to have stabilized. We will continue to monitor the situation for the remainder of the weekend and provide the next update on Monday September 26th.
Sep 24, 15:30 EDT
Identified -
We've identified the source of the service disruption with the collectors. After our standard maintenance a subset of collectors did not immediately come back online. We are working as quickly as possible to resolve this issue.
Sep 24, 14:50 EDT
Update -
We are continuing to investigate the disruption to the collectors. We will continue to provide updates as they become available.
Sep 24, 13:33 EDT
Update -
As the final few collectors come back online we are continuing our investigation to ensure stability for our customers. We will continue to provide updates as they become available.
Sep 24, 12:35 EDT
Update -
We are continuing to investigate the disruption to the collectors. We will continue to provide updates as they become available.
Sep 24, 11:24 EDT
Update -
Nearly all collectors have reconnected to the cloud, but a few are still not reconnecting. We continue to investigate and will provide further updates as they become available.
Sep 24, 10:29 EDT
Investigating -
Some customers' collectors are not reconnecting to the cloud, resulting in network monitoring data not being updated in the cloud. We will continue to provide updates as they become available.
Sep 24, 09:38 EDT
