Some of our voices are down
Resolved·Full outage

This incident has been resolved

Wed, Nov 16, 2022, 04:38 PM
(2 years ago)
·
Affected components
AudioStack
Updates

Resolved

This incident has been resolved

Wed, Nov 16, 2022, 04:38 PM

Identified

We received this from Azure.

Impact Statement: Starting at 10:32 UTC on 16 Nov 2022, you have been identified as a customer using Speech Service in West Europe who may experience 5xx errors when trying to use Text-to-speech (TTS). Both Neural voices and Custom Neural voices are affected.

Current Status: Our investigation points to a networking issue as the potential root cause. This caused a subset of nodes to become unhealthy, resulting in the 500 level errors mentioned above. We have performed recovery actions to restore the affected nodes functionality and at this stage customers may start experiencing signs of recovery. We will continue monitoring error rates closely to ensure mitigation, while working on identifying the full root cause. The next update will be provided in 2 hours, or as events warrant.

Wed, Nov 16, 2022, 04:22 PM(15 minutes earlier)

Identified

We received word from our third party provider.

Impact Statement: Starting at 10:32 UTC on 16 Nov 2022, you have been identified as a customer using Speech Service in West Europe who may experience 5xx errors when trying to use Text-to-speech (TTS). Both Neural voices and Custom Neural voices are affected.

Current Status: We continue our investigation of the underlying root cause and exploring potential mitigation options. Customers using Custom Neural Voices might be seeing signs of recovery. The next update will be provided in 2 hours, or as events warrant.

We only use Neural voices

Wed, Nov 16, 2022, 02:25 PM(1 hour earlier)

Identified

We got the following update from Azure

Impact Statement: Starting at 10:32 UTC on 16 Nov 2022 you have been identified as a customer using Text-to-speech (TTS) in West Europe who may experience 5xx errors when trying to use the service. Both Neural voices and Custom Neural voices are affected.

Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.

Wed, Nov 16, 2022, 01:44 PM(41 minutes earlier)

Identified

We are continuing to wait for a response from our third party API provider. We apologise for any inconvenience. Our other voices are working fine.

Wed, Nov 16, 2022, 01:28 PM(15 minutes earlier)

Identified

We've identified some of the issues and are speaking with Microsoft to resolve this. This affects other companies than just us btw according to twitter.

Wed, Nov 16, 2022, 12:30 PM(57 minutes earlier)

Investigating

We are continuing to investigate

Wed, Nov 16, 2022, 11:54 AM(35 minutes earlier)

Investigating

It appears that all of our azure voices are down. This affects various voices. Other voices are operational

It also affects our voice cloning processes

Wed, Nov 16, 2022, 11:07 AM(47 minutes earlier)