Frequency Foundry - Notice history

Resolved Engineers have resolved this issue as of 2:11 PM EST Posted 4 minutes ago. Jan 11, 2022 - 14:16 EST

Resolved - Voice Services functionality has been restored. All users are able to use Voice Services without issue. Currently, all support channels are operating as normal. Our Network Administrators identified the root cause as a DDoS attack that impacted Voice Services functionality across multiple data centers. We have now mitigated the issue. All services are now working as expected. Due to a high volume of requested call backs around this issue, a return call may be delayed or not completed. For any further questions, please call or chat.

From Click Dimensions,

After investigation our teams have determined that some emails sent during the outage period have failed to send. The Amazon outage was first reported by Amazon at 9:37 AM PST on 12/7/2021 and was reported as resolved at 4:35 PM PST. During this time, some Email Sends sent by accounts in our US Data Center failed to send despite showing as "Sent" within D365/CRM.

We have put together guidance on how to identify and re-process Email Sends that were unable to send due to the Amazon Web Services outage that impacted our US Data Center on 12/7/2021. It can be found here: https://support.clickdimensions.com/hc/en-us/articles/4415953657613

We recommend customers review the linked guidance and determine if emails were impacted, and if the impacted email sends should be sent now.

AWS has listed the issue as resolved. We are monitoring for any potential lingering effects to our infrastructure.

This incident has been resolved

From Click Dimensions:

Our team has implemented a fix and we are monitoring the results. At this time we are seeing newly created email sends complete the send successfully. However, email sends that are in error recovery are still in the process of recovering/sending. These emails should send successfully once they recover from error recovery.

The issue has been identified and a fix is being implemented.

Please note all issues related to the major network outages as result of severe weather in BC are now resolved.

Our engineering team worked tirelessly to restore services using our alternate routes in collaboration with other major telecom providers.

UPDATE: 14:25 ET Dear Valued Customer(s),

We have made significant progress recovering the majority of services that have been affected by the severe weather events in British Columbia. At this point, we have most of our routes back up and running on permanent or temporary facilities, while we await the remaining routes to be permanently restored by our telco partners from fiber cuts resulting from the landslides.

For next few days, you may experience intermittent call completion or voice quality, as we continue to fine tune our network, and bring it back to full capacity and redundancy.

Due to severity of impact from weather events, and lack of alternate routes available, we currently have four CLEC regions that have not been fully restored. We are working with our partners to fully restore all services as quickly and safely as possible. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

• Penticton • Cranbrook

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team

UPDATE: 10:15AM ET Dear Valued Customer(s),

We have made significant progress recovering the majority of services that have been affected by the severe weather events in British Columbia. At this point, we have most of our routes back up and running on permanent or temporary facilities, while we await the remaining routes to be permanently restored by our telco partners from fiber cuts resulting from the landslides.

For next few days, you may experience intermittent call completion or voice quality, as we continue to fine tune our network, and bring it back to full capacity and redundancy.

Due to severity of impact from weather events, and lack of alternate routes available, we currently have four CLEC regions that have not been fully restored. We are working with our partners to fully restore all services as quickly and safely as possible. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

• Kamloops • Penticton • Vernon • Cranbrook

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team

UPDATE: 10:15AM ET Dear Valued Customer(s),

We have made significant progress recovering the majority of services that have been affected by the severe weather events in British Columbia. At this point, we have most of our routes back up and running on permanent or temporary facilities, while we await the remaining routes to be permanently restored by our telco partners from fiber cuts resulting from the landslides.

For next few days, you may experience intermittent call completion or voice quality, as we continue to fine tune our network, and bring it back to full capacity and redundancy.

Due to severity of impact from weather events, and lack of alternate routes available, we currently have four CLEC regions that have not been fully restored. We are working with our partners to fully restore all services as quickly and safely as possible. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

• Kamloops • Penticton • Vernon • Cranbrook

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team

UPDATE: 15:45 PM ET

Dear Valued Customer(s),

We continue to face major network outages as result of severe weather in BC. Our Engineering teams have been working throughout the night to provide alternate options as this network event has impacted multiple service providers in the region.

At this point, certain services have been temporarily restored using alternate paths. During this time you may experience intermittent call completion or voice quality.

Here are the regions that are mostly impacted: Kamloops Penticton Vernon Cranbrook Vancouver - Reduced capacity Abbottsford Victoria Nanaimo

We are working with our partners to fully restore all services as quickly and safely as possible. Emergency crews are attempting to repair fiber cuts to multiple circuits affected by landslides in the region. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team

UPDATE: 12:55PM ET

Dear Valued Customer(s),

We continue to face major network outages as result of severe weather in BC. Our Engineering teams have been working throughout the night to provide alternate options as this network event has impacted multiple service providers in the region.

At this point, certain services have been temporarily restored using alternate paths. During this time you may experience intermittent call completion or voice quality.

We are working with our partners to fully restore all services as quickly and safely as possible. Emergency crews are attempting to repair fiber cuts to multiple circuits affected by landslides in the region. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team

UPDATE: 10:35AM ET

Dear Valued Customer(s),

We continue to face major network outages as result of severe weather in BC. Our Engineering teams have been working throughout the night to provide alternate options as this network event has impacted multiple service providers in the region.

At this point, certain services have been temporarily restored using alternate paths. During this time you may experience intermittent call completion or voice quality.

We are working with our partners to fully restore all services as quickly and safely as possible. Emergency crews are attempting to repair fiber cuts to multiple circuits affected by landslides in the region. Estimated Time of Recovery (ETR) is not confirmed yet but rest assured that all hands are on deck to resolve the issue as soon as possible.

We apologize for any inconvenience and thank you for your understanding.

Sincerely, Business Technical Services Team (ThinkTel)

UPDATE: 18:15 ET

We are working with our partners on providing alternatives options as this is a wide-spread carrier network outage, impacted multiple providers. We are working with our partners to restore services as quickly and safely as possible. Estimated Time of Recovery (ETR) is not confirmed yet but be rest assured that all hands are on deck to resolve the issue as soon as possible.

We apologize for any inconvenience and thank you for your understanding.

Dear Valued Customer(s),

Be advised that we are experiencing a major outage on multiple legs of our network caused by severe weather events in Western Canada that is affecting multiple Telco’s across Canada.

Our Engineering team is working quickly in collaboration with our partner carriers to re-route the impacted traffic/services.

We expect service restoral to be back up and running within the next 45min. Next update: 16:45PM ET.

We apologize for any inconvenience and thank you for your understanding.

Sincerely,

Business Technical Services Team

This incident has been resolved. Posted 3 hours ago. Nov 10, 2021 - 13:31 UTC

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented. Posted 3 hours ago. Nov 09, 2021 - 17:58 UTC