IdenTrust: Bad OCSP Responses
IdenTrust reported an OCSP availability/response correctness issue discovered during a planned 24-hour test of OCSP traffic being diverted to AWS cloud infrastructure for scalability. During the test on December 15, 2022, IdenTrust received customer reports of timeout errors for TLS CRLs and later determined that a small number of certificates were mistakenly shown as "revoked" when checked via OCSP. IdenTrust’s investigation found problems replicating OCSP data to AWS: the replication missed status updates and incorrectly mapped target certificate status to "revoked" with an empty "revocationTime" field. The incorrect OCSP response was served 4,257 times over a 20-hour period, affecting 326 certificates, while 76,298,215 OCSP responses were served correctly. IdenTrust stopped the diversion of OCSP traffic and provided a full incident report describing the cause and timeline, then implemented corrective measures including changing the replication approach away from the "Continuous Query Notification" feature and updating testing to verify revocationTime and revocationReason matches. IdenTrust later confirmed that all implemented changes were working as expected and considered the issue fully resolved; Mozilla indicated it would close the bug on or about May 5, 2023. The bug is resolved as FIXED.
- IdenTrust conducted a planned 24-hour OCSP traffic diversion test to AWS and received reports that some certificates appeared as revoked via OCSP.
- IdenTrust stopped the diversion of OCSP traffic after identifying a data issue.
- IdenTrust targeted implementation of corrective measures for the OCSP replication/status issue.
- IdenTrust confirmed implemented changes were working as expected and considered the issue fully resolved.
- IdenTrust Services, LLC — IdenTrust stated it was investigating after receiving customer reports during a scheduled 24+ hour OCSP traffic diversion test and said a full incident report would follow by December 30, 2022.
- IdenTrust Services, LLC — IdenTrust provided a full incident report describing the OCSP replication problem, the number of affected certificates, and the timeline of actions taken.
- IdenTrust Services, LLC — IdenTrust confirmed it was on track to implement corrective measures by March 31, 2023 and planned an update by February 28, 2023.
- IdenTrust Services, LLC — IdenTrust reiterated it was on track for March 31, 2023 and listed corrective measures to change replication logic and improve testing.
- IdenTrust Services, LLC — IdenTrust reported a pre-production code change and said additional issues were found, expecting at least another month before production validation.
- IdenTrust Services, LLC — IdenTrust confirmed all implemented changes were working as expected and considered the issue fully resolved.
- Mozilla representative — Mozilla stated it would close the bug on or about May 5, 2023.