DigiCert: CRL and OCSP availability issues due to CDN misconfiguration (resolved)
This case describes a DigiCert incident where CRL responses were unavailable or returned 404 errors for four CRL URLs. DigiCert said it first became aware of the problem by monitoring SSLMate’s CRL-Watch website and noticing four CRLs listed with 404 errors. DigiCert reported that the issue was triggered by a misconfigured path in its CDN after a CDN migration, specifically that the origin path for kr.crl.digicert.com was misconfigured when the old CDN was shut down. DigiCert stated that Engineering resolved the CDN misconfiguration and pushed the change to the CDN, after which Engineering started receiving correct responses from the affected domain. DigiCert also reported that Apple notified it of stale OCSP responses, leading to an investigation and manual OCSP pushes to keep data fresh until the issue was isolated and confirmed fixed. DigiCert stated that the CRL misconfiguration and stale OCSP responses had no impact on certificates issued, and that remediation was completed; the bug was resolved with resolution FIXED. DigiCert further stated it implemented monitoring for CRL availability/age across all CRL distribution endpoints listed in CCADB and completed OCSP monitoring enhancements.
- DigiCert migrated its CDN for CRL/OCSP, during which some configurations were missed.
- DigiCert personnel discovered CRL problems (four CRLs returning 404) and Engineering resolved the CDN misconfiguration.
- Apple notified DigiCert of stale OCSP responses, prompting investigation and manual OCSP pushes.
- DigiCert confirmed the OCSP/related issue was fixed.
- DigiCert implemented CRL monitoring and reported OCSP monitoring enhancements were still in progress.
- DigiCert reported OCSP monitoring was completed and live.
- DigiCert reported remediation was complete and requested closure; Mozilla indicated intent to close on 2023-06-14.
- Community commenter — Martin Sullivan reported that DigiCert noticed four CRLs with 404 errors on SSLMate’s CRL-Watch, described the CDN misconfiguration cause, and outlined remediation and monitoring changes.
- Community commenter — Martin Sullivan said that during investigation DigiCert found other irregularities linked to the change and planned a revised bug report.
- Google representative — Ryan Dickson asked Martin Sullivan to add specificity about the meaning of the proposed change-management auditing and to provide more detail on anticipated timing.
- Community commenter — Martin Sullivan provided an updated timeline including Apple’s notification of stale OCSP responses, described the CDN migration/configuration issues, and reiterated monitoring and review-process changes.
- Community commenter — Martin Sullivan stated that if there were no further questions, DigiCert would provide the next update on April 28 and would continue constant monitoring.
- Community commenter — Martin Sullivan reported that CRL monitoring was implemented to monitor all CRLs per the CCADB list, and that OCSP monitoring enhancements were still in progress with an update planned end of month.
- Community commenter — Martin Sullivan reported OCSP monitoring was completed and live, and that CRL testing alerts were working in a real-world environment.
- Community commenter — Martin Sullivan stated remediation was complete and asked to close the bug.
- Mozilla representative — Ben Wilson said he intended to close the bug on 2023-06-14-2023.