Certainly: Serving Bad OCSP Responses
On 24 October 2022, Certainly became aware that its OCSP service was returning an “unauthorized” response for some valid certificates. The issue was discovered while testing a new monitoring tool, when a Certainly engineer observed OCSP query errors for certificates on www.certainly.com. Certainly investigated and declared an incident, applied fixes to an active data center, identified a bug in the OCSP monitor, and applied additional fixes to an inactive data center; it also deployed a fix for the monitoring bug and updated its testing checklist. The problem was not related to the certificates themselves; it affected OCSP responses for approximately 50,000 certificates issued between 18 September and 17 October 2022. Certainly attributed the cause to a change in leaf certificate serial number prefixes where the OCSP responder required both old and new prefixes to remain configured, and it also found that an external check bug prevented an alert from firing in this scenario. The thread states that remediation was completed, correct OCSP responses were restored for all certificates on 25 October 2022 at 21:05 UTC, and Certainly continued monitoring the bug for questions until it was closed.
- Certainly’s OCSP service began returning “unauthorized” responses for some valid certificates, triggering an incident response.
- Correct OCSP responses were restored for all affected certificates.
- Remediation was reported complete and no further questions were received; community closure was planned.
- Fastly representative — Wayne Thayer reported that Certainly discovered the OCSP “unauthorized” responses during monitoring-tool testing, provided a detailed incident timeline, described the approximate scope (~50,000 certificates), and explained the serial-prefix configuration and monitoring-check issues.
- Fastly representative — Wayne Thayer stated that remediation was completed and that Certainly would continue monitoring the bug for comments.
- Fastly representative — Wayne Thayer said monitoring would continue for questions or feedback.
- Fastly representative — Wayne Thayer reported remediation was complete, no questions had been received, and monitoring would continue until closure.
- Mozilla representative — Ben Wilson asked whether the community had additional questions or issues and said he planned to close the bug around 23-Nov-2022.
- Fastly representative — Wayne Thayer said Certainly representatives continued to monitor the bug for questions or feedback.