IdenTrust: Service Degradation

IdenTrust reported a service degradation incident affecting its OCSP responder connectivity due to a DNS lookup failure. The CA said it first became aware of the problem on November 6, 2020 via an OCSP responder monitoring system alert, and later on November 7, 2020 after a customer reported an intermittent OCSP connection error. In its incident response, IdenTrust stated that one of its DNS servers became unstable due to a hardware controller error, and that the DNS server continued to respond with an empty value, causing some validation attempts to fail to reach the OCSP responder. IdenTrust said it temporarily removed the problematic DNS server from service and from the global registrar on November 7, 2020, and testing confirmed services were available afterward. The CA also reported post-incident actions including working with external vendors on virtual environment configuration changes and adjusting monitoring rules for greater granularity. In response to Mozilla’s question about detection and prevention, IdenTrust stated there was no issue with detection because internal and external monitoring performed as expected, and that the issue was an extreme edge case isolated to a specific load balancer; it also said the vendor provided changes to affect a more optimal failure mode. The bug was marked RESOLVED with resolution FIXED, and Mozilla indicated it could be closed around March 5, 2021.

1. 2020-11-06 IdenTrust’s OCSP responder monitoring system alerted to abnormal behavior that later related to a DNS lookup failure affecting OCSP connectivity.
2. 2020-11-07 IdenTrust identified a DNS service degradation cause and temporarily removed the problematic DNS server from service and the global registrar to restore availability.
3. 2021-03-05 Mozilla planned to close the bug around this date after review.

1. 2020-11-14 IdenTrust Services, LLC — IdenTrust provided an incident report describing a November 6–7, 2020 service degradation caused by unstable DNS behavior impacting OCSP validation traffic and outlined remediation steps and post-incident actions.
2. 2021-01-06 Community commenter — Mozilla’s Ryan Sleevi asked for clearer explanation of how the issue avoided detection and what systemic mitigations were being put in place to improve detection and prevention.
3. 2021-01-07 IdenTrust Services, LLC — IdenTrust replied that detection was not the issue, stating monitoring systems performed as expected, and described the edge-case scenario and subsequent monitoring rule granularity changes and vendor-provided failure-mode improvements.
4. 2021-02-26 Mozilla representative — Mozilla stated the bug could be closed and planned to do so around March 5, 2021.