EU Network Outage — Post-Mortem

Incident Date: April 7–8, 2026 Duration: Apr 07, 2026 23:30 UTC – Apr 08, 2026 04:13 UTC (4h 43min) Affected Service: LoRaWAN Network (Azure – Europe Central)

Summary

On April 7, 2026 at 23:30 UTC, devices in our EU region began going offline. The trigger was a carrier-side outage at Transatel, our cellular SIM provider for EU gateways. While Transatel worked to restore service, we proactively activated the Azure IoT Hub redundancy feature on the affected hub to limit further blast radius.

That activation surfaced a previously unseen code regression in Azure's failover orchestration: a bulk delete operation intended to clear stale records in the destination region instead targeted the source region's device registration collection. Device metadata was deleted before replication completed, dropping live device connections.

The combination of the cellular trigger and the Azure regression extended the outage to 4 hours 43 minutes. Of the Azure customers affected by the same regression, our hub was the only one that did not auto-recover and required manual intervention from the Azure product team. Full service was restored at 04:13 UTC on April 8.

We take full ownership of the customer impact, including the portion caused by our decision to activate redundancy. This post-mortem describes what happened in detail and the concrete changes we are making.

Impact

• Service affected: LoRaWAN Network — Azure Europe Central
• Region: EU
• Customer-visible symptoms: Devices offline; sensor telemetry not reaching the platform; reconnections delayed during recovery by DNS propagation
• Time to mitigation: ~98 minutes (23:30 UTC → 01:08 UTC)
• Time to full resolution: 4h 43min total — the tail after mitigation was driven by DNS propagation as devices reconnected
• Data loss: Telemetry generated during the outage window was not delivered upstream. Historical data and account/device configuration were not affected.
• Cellular service: Restored same morning via a Transatel fleet-wide detach.

Timeline

All times UTC. Entries published on our public status page during the incident are marked (status page).

Root Cause

This incident was a two-stage cascading failure: a primary trigger that was correctly diagnosed, followed by a mitigation action that exposed a latent bug in our underlying cloud platform.

Primary trigger — Transatel cellular outage (EU)

A carrier-side incident at Transatel, our cellular SIM provider for EU, prevented affected gateways from relaying sensor traffic upstream. The myDevices network server, application layer, and alerting remained operational throughout. Transatel resolved their incident via a fleet-wide detach.

Cascading failure — Azure IoT Hub failover regression

To limit further blast radius during the Transatel incident, we activated Azure IoT Hub redundancy on the affected hub. This triggered a code regression in Azure's failover orchestration: a bulk delete operation intended to clean up stale device records in the destination region's storage instead targeted the source region's device registration collection. Device metadata was deleted before replication completed, dropping live device connections.

Important context on this regression:

• The redundancy feature was not new to us, and not untested. We had previously activated it in other regions under heavier load without issue. The failure mode that occurred here had not been surfaced by that prior usage — it appears to have been a region- or timing-specific manifestation of the regression.
• Multiple Azure customers hit the same bug. All affected hubs except ours auto-recovered. Ours required manual escalation to the Azure product team.
• Continuous backups existed and were used for recovery, but the 24-hour retention window complicated the process.

Why this took longer to resolve than the original cellular incident alone

Two factors drove duration past what a carrier outage on its own would have caused:

1. The mitigation became the larger incident. Activating redundancy was intended to reduce risk. The latent Azure regression instead turned it into the dominant source of customer impact.
2. Our hub did not auto-recover. When the regression hit, peer hubs at other Azure customers recovered automatically. Ours did not, and closing that gap required manual escalation to the Azure product team rather than an automated platform-side fix.

The recovery tail beyond mitigation (DNS propagation as devices reconnected) accounts for the remaining time between the 01:08 failover and the 04:13 all-clear.

What we are changing

These are concrete commitments, not aspirational lessons.

Reducing the chance this happens again

1. Mitigation gate during active incidents. We are formalizing the rule that no failover or redundancy feature will be activated mid-incident without (a) a pre-tested runbook for the specific scenario, and (b) an explicit cost/benefit check against the option of waiting. Proactive mitigations are themselves changes and will be treated with the same caution as a production deploy.
2. Per-region, per-configuration validation. Validation under load in some regions is not a guarantee of behavior in all regions. We are adding periodic synthetic activations of failover features in production-equivalent environments for the specific regions and configurations our customers depend on.
3. Backup retention. A 24-hour continuous-backup window is too tight for the state criticality of an industrial IoT platform. We are working with Azure to extend retention for our deployments, with the commitment captured in writing.
4. Closing the auto-recovery gap. We are working with Azure to understand why our hub did not auto-recover when peer hubs did, and will close that gap — whether the root cause is configuration on our side or detection on theirs. We will not rely on the cloud platform noticing when our service is degraded.

Reducing impact if it happens again

1. Independent monitoring of device-metadata counts. Today we monitor connection state. We are adding monitors on device-registration count anomalies so a bulk-delete event would page us directly, rather than being detected only via downstream symptoms.
2. Carrier-side observability. Transatel-side detach events should surface in our ops dashboards. We are closing this gap so future carrier-side incidents do not require an escalation to the carrier just to diagnose.
3. Status page coverage of upstream layers. Our public status page covered the Azure layer of this incident but did not separately document the cellular trigger. We are reviewing how we communicate multi-layered incidents publicly so the customer narrative matches the full picture.

What we want our customers to know

A few direct acknowledgments:

• The decision to activate redundancy was made in good faith to reduce risk to our customers. It produced the opposite outcome. The underlying trigger was a vendor regression, but the decision to activate was ours, and we own it.
• Both Transatel and Azure responded constructively during the incident and we have strong working relationships with both. That said, we will not be relying on either vendor to detect when our service is degraded — that detection has to be ours.
• The most important takeaway for us is also the simplest: mitigations are changes. Any action taken during an incident, even one intended to reduce risk, has its own failure modes and needs the same caution as a production deploy. That posture is now reflected in our incident response process.

If you have questions about how this incident affected your devices or your data, please contact your account team.