Incident report for 2026-10-27

docs/releases/status.md

@@ -16,6 +16,69 @@ Severity:
 
 The severity is used to decide how much we invest in preventative measures, detection, mitigation plans, and rehearsals.
 
+## 2026 October 28th: login errors (Severity: SIGNIFICANT):
+
+### Timeline (GMT/UTC)
+
+18:46 First automated alert to developers
+19:09 First user email to support address
+21:30 Investigation found an expired token for Microsoft users, which needed refreshing
+21:48 Problem solved
+
+### Analysis
+
+Users already logged in were not affected. Users logging in were affected. Logs show that 24 users were affected (other users relied on existing logins). There were ~85 users on the platform during the episode. 
+
+The cause of the problem stems from switching to a new auth system 12 months ago, and not planning to refresh the token that was due to expire. The speed of resonse was due to being out of office hours and two developers on leave.
+
+### Actions
+
+Calendar events have been set to refresh tokens in advance of their expiry. An additional developer is also being recruited.
+
+N=0.24, effect = 4, duration = 3. Severity = 2.88 (SIGNIFICANT)
+
+## 2026 Oct 27th: Partial Outages (Severity: SIGNIFICANT):
+
+### Timeline (GMT/UTC)
+
+09:12 Singular task hit 100% CPU Utilisation and users experienced instability.
+
+09:28 Unhealthy task was ended, and a new task started automatically.
+
+09:39 CPU Utilisation returned to normal 
+
+We also detected significant periods of loading and slow responses in the evenings on 30/10/26, 31/10/26, 2/11/26. In those periods, the automated response was more rapid and the effects were less severe.
+
+### Analysis
+
+1. For the first detected issue, approximately 156 users were live on the application, which requires a moderate scale up of servers ('tasks'). This scale up should be automated based on the load of the tasks.
+
+2. Automated task scaling relies on sensing that a task is overloaded. In this case, despite the ~100% CPU utilisation, the task was not recognised by the automated systems as reaching capacity. Therefore, rather than the expected ~2 minutes to expand capacity, it took 16 minutes for an automated response. This is the reason for an extended period of instability.
+
+3. The number of live tasks, in addition to scaling with usage, is scheduled to have a higher minimum during busy periods. The first incident happened in the hour before the scheduled 'busy' preiods.
+
+4. For the other incidents (other than the first one), memory limits were the cause for loading and slow responses. 
+
+
+### Actions
+
+All of the following have been done to prevent future events:
+
+1. Extend the scheduled busy periods, i.e. times with an increased base number of tasks, to start one hour earlier.
+
+2. Double the memory in each task.
+
+3. Stricter measures of task health for faster and more reliable automated scaling responses.
+
+4. Stricter and more accurate memory thresholds to trigger automated scaling.
+
+5. New monitoring of slowdowns in event loops, for better detection of issues.
+
+6. Common highest-memory-usage DB queries were optimised to reduce typical memory loads on the tasks.
+
+N~2, effect = 3, duration = ~0.5. Severity ~3 (SIGNIFICANT)
+
+
 ## 2025 December 4th: Brief DB outage (Severity: LOW):
 
 ### Timeline (GMT/UTC)