Let's Encrypt: Delay updating OCSP responses

Let’s Encrypt reported a preliminary incident in which its OCSP updater system fell behind its target schedule for updating OCSP responses. Let’s Encrypt said internal monitoring detected the updater was about 2 hours behind the 3-day update target on 2021-09-05, but a warning alert was configured as working-hours-only and was not received by the oncall. On 2021-09-06 and 2021-09-07, Let’s Encrypt began serving OCSP responses whose `thisUpdate` field was more than 3.5 days and then more than 4 days in the past, which it stated violated the Microsoft Trusted Root Program Requirements and the Baseline Requirements. Let’s Encrypt began incident response after becoming aware during an oncall shift change on 2021-09-07 14:00 UTC and deployed technical mitigations, but said they were not sufficient to clear the backlog within program requirements at first. On 2021-09-08, Let’s Encrypt stated OCSP responses were back in compliance with both the Baseline Requirements and the Microsoft Trusted Root Program Requirements, and that the root cause was identified and fixed, with additional remediation items identified. Mozilla closed the bug as fixed on 2021-10-15, stating there were no open action items remaining and no further questions.

1. 2021-09-05 Internal monitoring detected the OCSP updater was behind the target OCSP update schedule.
2. 2021-09-06 Let’s Encrypt began serving OCSP responses with `thisUpdate` older than 3.5 days.
3. 2021-09-07 Let’s Encrypt began serving OCSP responses with `thisUpdate` older than 4 days and initiated incident response after oncall shift change awareness.
4. 2021-09-08 OCSP responses returned to compliance with the Baseline Requirements and Microsoft Trusted Root Program Requirements.
5. 2021-10-15 Mozilla closed the bug as fixed.

1. 2021-09-07 Internet Security Research Group — Opened a preliminary incident report describing delayed OCSP updates, the alerting issue, and stated requirement violations.
2. 2021-09-08 Internet Security Research Group — Reported that OCSP responses were back in compliance and that the root cause was identified and fixed, with remediation items planned.
3. 2021-09-08 Mozilla representative — Said complete certificate data for all affected certificates was not necessary for an OCSP-related incident.
4. 2021-09-09 Community commenter — Quoted Mozilla incident-reporting language requesting aggregates and summaries for each type of problem identified.
5. 2021-09-09 Internet Security Research Group — Provided a detailed incident report including timeline and described OCSP update behavior and mitigations.
6. 2021-09-17 Internet Security Research Group — Provided an update after inactivity, describing ongoing work on stricter config loading and alert review tasks.
7. 2021-09-30 Internet Security Research Group — Reported completion of alert review and deployment of strict config loading change, and said no further updates were intended absent questions.
8. 2021-10-15 Mozilla representative — Closed the bug as fixed because there were no open action items and no questions.