Artie Status - Incident history https://artie.instatus.com Artie Mon, 27 Jul 2026 23:00:00 +0000 Infrastructure upgrade for Artie Dashboard Type: Maintenance Duration: 2 hours Affected Components: Artie Dashboard Jul 22, 22:56:39 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Jul 27, 23:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 28, 01:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>22:56:39</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> <p><small>Jul <var data-var='date'> 27</var>, <var data-var='time'>23:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 28</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Mon, 27 Jul 2026 23:00:00 +0000 https://artie.instatus.com/maintenance/cmrwom8w604i80klll561fkrv https://artie.instatus.com/maintenance/cmrwom8w604i80klll561fkrv Spark cluster upgrade in us-east-1 Type: Maintenance Duration: 1 hour Affected Components: AWS US-East-1 Data Plane Jun 26, 19:38:47 GMT+0 - Identified - This upgrade only involves pipelines with Iceberg destinations. No impact is expected. Jun 29, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully. Jun 29, 21:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>19:38:47</var> GMT+0</small><br><strong>Identified</strong> - This upgrade only involves pipelines with Iceberg destinations. No impact is expected..</p> <p><small>Jun <var data-var='date'> 29</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Jun <var data-var='date'> 29</var>, <var data-var='time'>21:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Mon, 29 Jun 2026 21:00:00 +0000 https://artie.instatus.com/maintenance/cmqvc3ndj05xf2poapcgh0mtg https://artie.instatus.com/maintenance/cmqvc3ndj05xf2poapcgh0mtg Scheduled database maintenance Type: Maintenance Affected Components: Artie Dashboard Jun 26, 00:27:12 GMT+0 - Completed - We're performing planned maintenance to improve the reliability of the Artie Dashboard. During this window the dashboard may be briefly unavailable or slower to load. Your data pipelines are not affected and will continue running normally. No data is impacted. Type: Maintenance

Affected Components:

<p><small>Jun <var data-var='date'> 26</var>, <var data-var='time'>00:27:12</var> GMT+0</small><br><strong>Completed</strong> - We're performing planned maintenance to improve the reliability of the Artie Dashboard. During this window the dashboard may be briefly unavailable or slower to load. Your data pipelines are not affected and will continue running normally. No data is impacted..</p> ]]> Fri, 26 Jun 2026 00:27:12 +0000 https://artie.instatus.com/maintenance/cmqu6yoro018p2jqpct37b7nu https://artie.instatus.com/maintenance/cmqu6yoro018p2jqpct37b7nu Networking upgrade for Artie Dashboard Type: Maintenance Duration: 30 minutes Affected Components: Artie Dashboard Jun 22, 18:19:51 GMT+0 - Identified - Jun 25, 00:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 25, 00:30:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 30 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 22</var>, <var data-var='time'>18:19:51</var> GMT+0</small><br><strong>Identified</strong> - .</p> <p><small>Jun <var data-var='date'> 25</var>, <var data-var='time'>00:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 25</var>, <var data-var='time'>00:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Thu, 25 Jun 2026 00:00:00 +0000 https://artie.instatus.com/maintenance/cmqpjiq5j00is2kpijbyhooc7 https://artie.instatus.com/maintenance/cmqpjiq5j00is2kpijbyhooc7 Database upgrade Type: Maintenance Duration: 10 hours Affected Components: Artie Dashboard May 19, 01:14:00 GMT+0 - Completed - Database upgrade is complete. May 19, 01:04:00 GMT+0 - Identified - We are running scheduled maintenance to upgrade our database. Type: Maintenance

Duration: 10 hours

Affected Components:

<p><small>May <var data-var='date'> 19</var>, <var data-var='time'>01:14:00</var> GMT+0</small><br><strong>Completed</strong> - Database upgrade is complete..</p> <p><small>May <var data-var='date'> 19</var>, <var data-var='time'>01:04:00</var> GMT+0</small><br><strong>Identified</strong> - We are running scheduled maintenance to upgrade our database..</p> ]]> Tue, 19 May 2026 01:04:00 +0000 https://artie.instatus.com/maintenance/cmpbxuc7s000zqull0ehw6g83 https://artie.instatus.com/maintenance/cmpbxuc7s000zqull0ehw6g83 Dashboard API timing out Type: Incident Duration: 1 hour Affected Components: Artie Dashboard Apr 7, 18:00:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 7, 19:00:27 GMT+0 - Resolved - This incident has been resolved. We will prepare a post-mortem for this incident shortly. Type: Incident

Duration: 1 hour

Affected Components:

<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>19:00:27</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We will prepare a post-mortem for this incident shortly..</p> ]]> Tue, 7 Apr 2026 18:00:00 +0000 https://artie.instatus.com/incident/cmnoy1ndj054az9eh3mdnab88 https://artie.instatus.com/incident/cmnoy1ndj054az9eh3mdnab88 NGINX Upgrade Type: Maintenance Duration: 1 hour Affected Components: Artie Dashboard Mar 25, 23:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 26, 00:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 25, 23:00:00 GMT+0 - Identified - We are planning to do some maintenance upgrades to our Artie dashboard. Our dashboard and dashboard API will be temporarily degraded during this period. Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>23:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 26</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning to do some maintenance upgrades to our Artie dashboard. Our dashboard and dashboard API will be temporarily degraded during this period..</p> ]]> Wed, 25 Mar 2026 23:00:00 +0000 https://artie.instatus.com/maintenance/cmn2frzdp0urifbud6dqtlw6c https://artie.instatus.com/maintenance/cmn2frzdp0urifbud6dqtlw6c Dashboard API is down Type: Incident Duration: 51 minutes Affected Components: Artie Dashboard Feb 20, 02:10:00 GMT+0 - Resolved - Root cause: our NGINX controller became unresponsive, which caused the Dashboard API to be unavailable. As a result, customers were unable to log into the UI or make pipeline changes. Service has now been fully restored and systems are operating normally. We’ll continue to monitor closely to ensure stability. Please reach out to our team if you experience any lingering issues. Feb 20, 01:22:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 51 minutes

Affected Components:

<p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>02:10:00</var> GMT+0</small><br><strong>Resolved</strong> - Root cause: our NGINX controller became unresponsive, which caused the Dashboard API to be unavailable. As a result, customers were unable to log into the UI or make pipeline changes. Service has now been fully restored and systems are operating normally. We’ll continue to monitor closely to ensure stability. Please reach out to our team if you experience any lingering issues..</p> <p><small>Feb <var data-var='date'> 20</var>, <var data-var='time'>01:22:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Fri, 20 Feb 2026 01:22:00 +0000 https://artie.instatus.com/incident/cmlu8y5u20a5cwndff1ziqgpg https://artie.instatus.com/incident/cmlu8y5u20a5cwndff1ziqgpg Networking issues in use1 Type: Incident Duration: 7 hours and 54 minutes Affected Components: Artie Dashboard, AWS US-East-1 Data Plane Jan 31, 00:30:00 GMT+0 - Identified - A config change has affected networking for pipelines in the us-east-1 data plane. We are continuing to work on a fix for this incident. Jan 31, 03:26:57 GMT+0 - Monitoring - There's a CNI error. We are seeing some clients recover and are still working towards a full recovery. Jan 31, 04:39:17 GMT+0 - Monitoring - We’ve identified intermittent DNS issues on a specific node and are actively working to resolve them. We’re working to restore full reliability. Jan 31, 05:14:24 GMT+0 - Monitoring - We are now working with AWS to diagnose the partial DNS failure. Jan 31, 07:42:31 GMT+0 - Monitoring - We are continuing to see intermittent DNS errors. The issue has been escalated with AWS, and an AWS engineer is joining the investigation shortly. We’ll share another update as we learn more. Jan 31, 08:24:08 GMT+0 - Resolved - We’ve identified the root cause of the DNS instability: an out-of-order VPC CNI upgrade led to an undocumented version mismatch, which caused conflicts in our DNS routing rules. The issue has now been resolved, and we are closely monitoring all pipelines to ensure continued stability. Feb 10, 01:34:05 GMT+0 - Postmortem - ### Incident Overview * **Date:** January 30, 2025 * **Affected AWS region:** us-east-1 * **Duration:** partial outage 6.5 hours, total outage 1 hour ### Summary On January 30, a routine infrastructure upgrade to our Kubernetes networking layer caused a service disruption in one of our primary cloud regions. What began as a partial outage escalated into a full outage affecting data pipelines for customers in the us-east-1 region. We sincerely apologize for the impact this had on your operations. Service was fully restored the same evening after our team identified and resolved the underlying issue. It was a version mismatch between critical networking components in our cluster. ### Impact We understand how important reliability is to your business, and we are deeply sorry for the disruption. During the outage: * **Service was unavailable** in the us-east-1 region for 7.5 hours. * A small number of affected customers experienced **data replication interruptions**, which our team worked to remediate promptly. We recognize that any downtime is unacceptable, and we take full responsibility for this incident. ### What happened Our engineering team was performing a planned upgrade to a networking add-on (vpc-cni) to increase capacity in our Kubernetes infrastructure. During the upgrade: 1. The change was applied successfully in a non-production environment but encountered compatibility issues in the production cluster. 2. A rollback was not possible because the previous component version was no longer supported by the underlying platform. 3. The issue was compounded by version mismatches across several interdependent networking components, which made diagnosis more complex than expected. Our team identified the root cause as an outdated networking component (kube-proxy) that was incompatible with the rest of the upgraded stack. Once the fix was applied, service was restored. ### What we're sorry about We want to be transparent; this incident was avoidable. We moved forward with a change that, in hindsight, required more thorough validation and coordination. Specifically: * The production environment differed from the test environment in ways we did not fully account for. * We did not have sufficient safeguards in place to catch the version incompatibility before it caused an outage. * Our incident response process was slower than it should have been, and we did not escalate to our cloud provider's engineering team quickly enough. We are sorry for the disruption and for falling short of the reliability you expect from us. ### Our plan going forward We are actively carrying out a comprehensive set of improvements to prevent incidents like this from happening again: **Stronger change controls** * We have strengthened our change management process so that all planned infrastructure migrations require formal approval and a minimum of two engineers working together to apply changes. * Non-reversible changes are restricted to business hours (Monday through Thursday) to ensure full team availability and vendor support coverage. **Improved testing and validation** * We are auditing and standardizing component versions across all of our clusters to eliminate hidden drift between environments. * Production and non-production clusters are being aligned to ensure test results are representative. **Faster incident response** * We are formalizing a war room procedure with clearly defined roles: a primary engineer making changes, a secondary approving them, an incident commander coordinating, and a communications lead keeping customers informed. * We are establishing clear severity definitions and escalation timelines so that vendor support is engaged earlier when needed. **Infrastructure resilience** * We are working toward running multiple clusters per region so that workloads can be migrated if a single cluster is compromised. * We are building the capability to move customer pipelines between clusters with minimal disruption. * We are proactively engaging our cloud provider to review and approve infrastructure upgrade plans before they are executed. --- We value your trust and are committed to earning it back through these concrete actions. If you have any questions or concerns about this incident or our remediation plan, please do not hesitate to reach out to your account contact or our support team. Type: Incident

Duration: 7 hours and 54 minutes

Affected Components: ,

<p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>00:30:00</var> GMT+0</small><br><strong>Identified</strong> - A config change has affected networking for pipelines in the us-east-1 data plane. We are continuing to work on a fix for this incident..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>03:26:57</var> GMT+0</small><br><strong>Monitoring</strong> - There's a CNI error. We are seeing some clients recover and are still working towards a full recovery..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>04:39:17</var> GMT+0</small><br><strong>Monitoring</strong> - We’ve identified intermittent DNS issues on a specific node and are actively working to resolve them. We’re working to restore full reliability..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>05:14:24</var> GMT+0</small><br><strong>Monitoring</strong> - We are now working with AWS to diagnose the partial DNS failure..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>07:42:31</var> GMT+0</small><br><strong>Monitoring</strong> - We are continuing to see intermittent DNS errors. The issue has been escalated with AWS, and an AWS engineer is joining the investigation shortly. We’ll share another update as we learn more..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>08:24:08</var> GMT+0</small><br><strong>Resolved</strong> - We’ve identified the root cause of the DNS instability: an out-of-order VPC CNI upgrade led to an undocumented version mismatch, which caused conflicts in our DNS routing rules. The issue has now been resolved, and we are closely monitoring all pipelines to ensure continued stability..</p> <p><small>Feb <var data-var='date'> 10</var>, <var data-var='time'>01:34:05</var> GMT+0</small><br><strong>Postmortem</strong> - ### Incident Overview * **Date:** January 30, 2025 * **Affected AWS region:** us-east-1 * **Duration:** partial outage 6.5 hours, total outage 1 hour ### Summary On January 30, a routine infrastructure upgrade to our Kubernetes networking layer caused a service disruption in one of our primary cloud regions. What began as a partial outage escalated into a full outage affecting data pipelines for customers in the us-east-1 region. We sincerely apologize for the impact this had on your operations. Service was fully restored the same evening after our team identified and resolved the underlying issue. It was a version mismatch between critical networking components in our cluster. ### Impact We understand how important reliability is to your business, and we are deeply sorry for the disruption. During the outage: * **Service was unavailable** in the us-east-1 region for 7.5 hours. * A small number of affected customers experienced **data replication interruptions**, which our team worked to remediate promptly. We recognize that any downtime is unacceptable, and we take full responsibility for this incident. ### What happened Our engineering team was performing a planned upgrade to a networking add-on (vpc-cni) to increase capacity in our Kubernetes infrastructure. During the upgrade: 1. The change was applied successfully in a non-production environment but encountered compatibility issues in the production cluster. 2. A rollback was not possible because the previous component version was no longer supported by the underlying platform. 3. The issue was compounded by version mismatches across several interdependent networking components, which made diagnosis more complex than expected. Our team identified the root cause as an outdated networking component (kube-proxy) that was incompatible with the rest of the upgraded stack. Once the fix was applied, service was restored. ### What we're sorry about We want to be transparent; this incident was avoidable. We moved forward with a change that, in hindsight, required more thorough validation and coordination. Specifically: * The production environment differed from the test environment in ways we did not fully account for. * We did not have sufficient safeguards in place to catch the version incompatibility before it caused an outage. * Our incident response process was slower than it should have been, and we did not escalate to our cloud provider's engineering team quickly enough. We are sorry for the disruption and for falling short of the reliability you expect from us. ### Our plan going forward We are actively carrying out a comprehensive set of improvements to prevent incidents like this from happening again: **Stronger change controls** * We have strengthened our change management process so that all planned infrastructure migrations require formal approval and a minimum of two engineers working together to apply changes. * Non-reversible changes are restricted to business hours (Monday through Thursday) to ensure full team availability and vendor support coverage. **Improved testing and validation** * We are auditing and standardizing component versions across all of our clusters to eliminate hidden drift between environments. * Production and non-production clusters are being aligned to ensure test results are representative. **Faster incident response** * We are formalizing a war room procedure with clearly defined roles: a primary engineer making changes, a secondary approving them, an incident commander coordinating, and a communications lead keeping customers informed. * We are establishing clear severity definitions and escalation timelines so that vendor support is engaged earlier when needed. **Infrastructure resilience** * We are working toward running multiple clusters per region so that workloads can be migrated if a single cluster is compromised. * We are building the capability to move customer pipelines between clusters with minimal disruption. * We are proactively engaging our cloud provider to review and approve infrastructure upgrade plans before they are executed. --- We value your trust and are committed to earning it back through these concrete actions. If you have any questions or concerns about this incident or our remediation plan, please do not hesitate to reach out to your account contact or our support team..</p> ]]> Sat, 31 Jan 2026 00:30:00 +0000 https://artie.instatus.com/incident/cml1qnb9z01f9jyy58zozjs8q https://artie.instatus.com/incident/cml1qnb9z01f9jyy58zozjs8q