DigiCert: OCSP responder returning “good” for revoked certificates (on-demand OCSP)

DigiCert reported that, while making system changes on 8/24/2020, it discovered a bug introduced on 7/22/2020 that caused its on-demand OCSP responder (ocspx.digicert.com) to return “good” for revoked certificates. DigiCert stated that the correct response was returned at ocsp.digicert.com and included in CRLs when the certificate included a CRL URI, but the AIA for affected certificates pointed to ocspx.digicert.com, so certificates had the wrong OCSP information. DigiCert investigated and determined that two issuing CAs were impacted, and it deployed a code fix on 8/27/2020 to send correct responses to both the on-demand and pre-signed services. DigiCert then moved remaining CAs to pre-signing only and shut down on-demand signing on 8/31/2020, stating that it had fixed the issue and finished shutting down on-demand signing. In the thread, Mozilla asked about the adequacy of pre-/post-deployment testing and systemic controls; DigiCert responded that it added acceptance automated tests for end-entity certificate status on the on-demand endpoint path and discussed expanding testing and PR process controls. DigiCert also described implementing PR templates and checklists to ensure major workflows are manually tested and/or automated on each PR, and indicated the process was implemented earlier than 10/15. The bug was resolved as FIXED.

1. 2020-07-22 A code change intended to migrate on-demand OCSP to pre-signed responses inadvertently included end-entity certificates.
2. 2020-07-23 The CA began signing all responses as “good” for ocspx.digicert.com after deploying the change.
3. 2020-08-24 DigiCert discovered the on-demand OCSP bug while planning shutdown steps for on-demand OCSP.
4. 2020-08-27 DigiCert deployed a code fix to send correct responses to both on-demand and pre-signed OCSP services.
5. 2020-08-31 DigiCert moved remaining CAs to pre-signing only and shut down on-demand signing.

1. 2020-09-01 Community commenter — Opened the bug with an incident timeline describing how on-demand OCSP returned “good” for revoked certificates and how DigiCert fixed the issue and shut down on-demand signing.
2. 2020-09-09 DigiCert — Asked what additional information Mozilla wanted and stated DigiCert had turned off its on-demand signing service for TLS.
3. 2020-09-14 Mozilla representative — Questioned whether more thorough pre- or post-deployment testing should be included to prevent similar incidents.
4. 2020-09-20 Community commenter — Expressed concern that the response focused on incident-specific plans rather than systemic root causes and remediation.
5. 2020-09-22 DigiCert — Explained the issue was noticed during routine sprint planning for shutting down the system and described added acceptance tests and testing expectations for workflow changes.
6. 2020-09-22 Community commenter — Pressed for robust controls beyond testing, including how issues with test development would be detected.
7. 2020-09-22 DigiCert — Said independent developer review missed the impact on end-entity certificates and described additional process improvements, including PR templates and checklists.
8. 2020-09-25 DigiCert — Outlined a plan to implement PR templates/checklists and improve visibility and workflow testing across CA services, targeting completion by Oct 15.
9. 2020-10-07 DigiCert — Reported that PR templates were implemented earlier than Oct 15 and were being used in CA-related code repositories.
10. 2020-10-09 Mozilla representative — Scheduled the bug for closure on or about 16-October-2020 unless additional questions remained.