Artie - Notice history

From 5:56 to 6:16 PM PDT on September 29, streaming was degraded for pipelines in our us-east-1 data plane. Right after a planned migration of our compute nodes, an engineer applied an unrelated infrastructure change built on an outdated configuration which undid the migration and replaced those nodes in bulk, causing new nodes to come up that were not attached to public IPs and sending data processing alerts for affected pipelines.

We identified the issue internally at 5:58 PM, and most pipelines recovered by 6:16 PM as the replacement nodes came online with our static egress IPs. We restarted the few remaining pipelines by hand and confirmed all pipelines were healthy by 6:45 PM. Data was delayed, not lost, and no action is required.

We apologize for the disruption and have implemented the following changes to ensure it doesn't happen again:
- additional guardrails on the infrastructure deploy process to catch rogue changes that aren't based on the latest configuration or introduce unintended drift
- process improvements to tighten infrastructure reviews across the team
- exploring foundational infrastructure improvements to reduce pipeline time to recovery when underlying nodes are replaced

The incident has been resolved and streaming is operational -- we will follow up with an incident breakdown shortly.