Certainly: Serving Expired OCSP Responses
Certainly reported that it became aware of an OCSP problem where its OCSP service was serving expired responses. It stated that the OCSP updater process that periodically generates new OCSP responses for existing certificates had been failing since 17-May 2022, and that during annual audit evidence gathering it found some returned OCSP responses were expired. Certainly investigated and declared an incident on 24-May 2022, determining the issue was likely caused by a configuration change related to the latest Boulder release; it also reported that reverting the configuration did not immediately resolve the problem. On 25-May 2022, Certainly rolled back Boulder to a prior release in production and deployed a fix so that OCSP response generation resumed, and it reported the service was fully restored once new OCSP responses were generated for all non-expired certificates. The thread also documents remediation actions, including upgrades and fixes to startup/monitoring behavior, pre-release OCSP testing, and implementing stale OCSP response alerts and external OCSP monitors. The bug was resolved as FIXED, and Mozilla indicated it would close the case on or about 5-Aug-2022 after remediation completion.
- OCSP updater service began panicking and became unable to sign fresh OCSP responses.
- Incident was declared after expired OCSP responses were found during investigation.
- Production was rolled back and OCSP response generation was restored for non-expired certificates.
- All planned remediation tasks were reported as completed.
- Fastly representative — Reported that Certainly discovered it was serving expired OCSP responses, provided a timeline, and stated production was rolled back and service restored after new OCSP responses were generated.
- Fastly representative — Explained how the CA became aware (annual audit evidence), described the actions taken, and provided root-cause details and affected certificate count (~10,647).
- Internet Security Research Group — Asked for more detail on the root-cause explanation and suggested adding safety checks and a Boulder PR.
- Fastly representative — Confirmed the analysis matched Certainly’s understanding and stated it would submit a Boulder PR, referencing issue #6150.
- Fastly representative — Requested a next update (23-June) and described alerting improvements, including catch-all alerts for panics and Boulder errors.
- Fastly representative — Provided a remediation plan status table, noting some items were done and others delayed (e.g., stale OCSP response alerts).
- Fastly representative — Updated remediation progress, stating remaining tasks were on track and requesting a next update for 30-July.
- Fastly representative — Reported that all planned remediation tasks for the incident were completed.
- Mozilla representative — Stated Mozilla would close the bug on or about 5-Aug-2022.