IdenTrust: OCSP Outage
The case concerns an OCSP outage experienced by IdenTrust around April 25, 2020, affecting the OCSP responder URI http://isrg.trustid.ocsp.identrust.com. An external report (referenced in the bug) alleged a multi-day OCSP outage, and IdenTrust acknowledged receipt and began investigating. IdenTrust reported that on 2020-04-23 15:01 (GMT) there was a significant spike in requests to the OCSP responder, peaking at 650k requests per second versus a typical 10k rps, and that the exact cause of the spike was unknown. IdenTrust stated that the spike led the ISRG OCSP responder to start serving errors and that the firewall at the primary data center became unresponsive; after monitoring alerted them to potential impact, they troubleshot and later performed a hard boot on the firewall, after which the request rate returned to normal and the OCSP responder stopped serving errors. IdenTrust also stated that the outage was lengthened by the hard boot requiring air travel to the datacenter, during which users of the OCSP responder were receiving errors. In response to questions about DR testing, IdenTrust stated that OCSP validation servers are part of annual DR testing and that the monitoring method used during the last DR exercise did not detect issues related to traffic switch between CDN and their sites. IdenTrust later reported completion of enhanced end-to-end monitoring for the affected OCSP responder to detect user-facing availability issues, and Mozilla indicated an intent to close the bug on or after 5-Aug-2020 unless new issues arose.
- IdenTrust experienced a significant spike in OCSP responder requests to http://isrg.trustid.ocsp.identrust.com.
- The OCSP responder began serving errors and IdenTrust performed troubleshooting and a hard boot of the primary data center firewall.
- IdenTrust became aware via the community that relying parties may have experienced service disruption and provided an incident summary.
- IdenTrust completed enhanced end-to-end monitoring implementation for the affected OCSP responder.
- Fastly representative — Requested that IdenTrust explain or provide an incident report after a report alleged a multi-day OCSP outage around April 25, 2020.
- IdenTrust Services, LLC — Acknowledged receipt and said they would start investigating.
- IdenTrust Services, LLC — Reported progress investigating and said a formal response would be coordinated by May 22, 2020.
- IdenTrust Services, LLC — Provided an incident summary including the request spike, resulting errors, firewall unresponsiveness, troubleshooting, and the hard boot that ended the errors.
- Fastly representative — Asked whether OCSP failover is part of IdenTrust’s DR procedures and why the issue was not detected during DR review/testing.
- IdenTrust Services, LLC — Answered that OCSP validation servers are part of annual DR testing and that monitoring used in the last DR exercise did not detect issues related to CDN-to-origin traffic switching; said monitoring would be updated.
- IdenTrust Services, LLC — Reported completion of enhanced end-to-end monitoring for http://isrg.trustid.ocsp.identrust.com to detect user-facing availability issues.
- Mozilla representative — Stated an intent to close the bug on or after 5-Aug-2020 unless additional issues or questions were raised.