Slow and Queueing Test Runs

Incident Report for Testery

Resolved

This incident has been resolved.
Posted Jul 02, 2025 - 00:34 UTC

Monitoring

The emergency database upgrade is completed and systems are now operational; however, accounts may still experience longer than usual wait times as the available runners in their accounts work through any queued test runs. We will continue to monitor.
Posted Jul 01, 2025 - 22:12 UTC

Identified

Due to significantly higher than usual volumes of test runs, some of our databases require an emergency upgrade. We are working on it.
Posted Jul 01, 2025 - 19:38 UTC

Investigating

We are currently investigating a production incident that is resulting in test runs slowing down and queuing up. We are working now to identify the root cause and will provide further updates as we work to resolve the issue.
Posted Jul 01, 2025 - 18:02 UTC
This incident affected: API (example) and Test Runners (Linux).