Let's Encrypt: 302 total OCSP responses served beyond acceptable timelines

This case reports that, for a set of certificate serial numbers, Let's Encrypt served OCSP responses older than acceptable timelines. The issue was discovered during an on-call shift rotation when SREs triaged a non-paging alert about elevated error-level logs and began investigating on 2020-09-08. From 2020-09-07 to 2020-09-08, OCSP responses older than 3.5 days were served for 268 certificate serial numbers, and from 2020-09-12 to 2020-09-13 this occurred for an additional 34 serial numbers; none were served beyond their validity period (nextUpdate), and the maximum OCSP age reached was 5 days. Let's Encrypt executed remediation queries (first on 2020-09-08 17:47 UTC for 268 entries, and again on 2020-09-13 17:22 UTC for the remaining problematic entries) and verified that all potentially affected Certificate Status entries had been remediated. A fix for the root cause was deployed on 2020-09-10 17:37 UTC and concluded at 17:59 UTC, with remediation completed as of 2020-09-13 17:22:17. The bug is marked RESOLVED with resolution FIXED, and Mozilla indicated an intent to close it as completed on or about 2020-10-09 if no further questions were raised.

1. 2020-09-08 Let's Encrypt began investigating after an on-call alert about elevated error-level logs led to discovery of OCSP responses served beyond acceptable timelines.
2. 2020-09-10 Let's Encrypt deployed a production fix for the root cause of the OCSP timing issue.
3. 2020-09-13 Let's Encrypt executed a final remediation query and verified all potentially affected certificate status entries were updated.

1. 2020-09-18 Internet Security Research Group — Created the incident report describing OCSP responses served older than acceptable timelines for 302 certificate serial numbers and the remediation actions taken.
2. 2020-09-20 Community commenter — Asked about whether the proto2-to-proto3 migration introduced latent bugs and what additional OCSP remediations might be useful.
3. 2020-09-21 Internet Security Research Group — Explained three changes that could have prevented/caught the issue, identified the immediate fix deployed on 2020-09-10, noted other in-flight fixes, and described next steps after proto3 migration.
4. 2020-10-05 Mozilla representative — Stated an intent to close the bug as completed on or about 2020-10-09 if there were no additional questions.